o
h?? @ s d Z ddlZddlZddlZddlZddlZddlmZ ddlm Z ddl
m Z m Z ddl
mZ ddlmZ dd lmZ dd
lmZmZmZ d d lmZ d d
lmZmZm Z m Z d dl m Z d dl m!Z! d dl"m#Z# e$e%Z&eg dZ'eee'Z(de)fddZ*G dd dZ+dS )z AutoProcessor class. N) OrderedDict )PretrainedConfig) get_class_from_dynamic_moduleresolve_trust_remote_code)FeatureExtractionMixin)ImageProcessingMixin)TOKENIZER_CONFIG_FILE)FEATURE_EXTRACTOR_NAMEget_file_from_repologging )_LazyAutoMapping)CONFIG_MAPPING_NAMES
AutoConfigmodel_type_to_module_name!replace_list_option_in_docstrings)AutoFeatureExtractor)AutoImageProcessor)
AutoTokenizer)1)alignAlignProcessor)altclipAltCLIPProcessor)bark
BarkProcessor)blip
BlipProcessor)zblip-2Blip2Processor) bridgetowerBridgeTowerProcessor) chinese_clipChineseCLIPProcessor)clap
ClapProcessor)clip
CLIPProcessor)clipsegCLIPSegProcessor)clvp
ClvpProcessor)flavaFlavaProcessor)fuyu
FuyuProcessor)git GitProcessor)groupvitr& )hubertWav2Vec2Processor)ideficsIdeficsProcessor) instructblipInstructBlipProcessor)zkosmos-2Kosmos2Processor)
layoutlmv2LayoutLMv2Processor)
layoutlmv3LayoutLMv3Processor)llavaLlavaProcessor)markuplmMarkupLMProcessor)mctctMCTCTProcessor)zmgp-strMgpstrProcessor) oneformerOneFormerProcessor)owlv2Owlv2Processor)owlvitOwlViTProcessor)
pix2structPix2StructProcessor) pop2pianoPop2PianoProcessor)sam SamProcessor) seamless_m4tSeamlessM4TProcessor)sewr3 )zsew-dr3 )speech_to_textSpeech2TextProcessor)speech_to_text_2Speech2Text2Processor)speecht5SpeechT5Processor)trocrTrOCRProcessor)tvlt
TvltProcessor)tvp TvpProcessor) unispeechr3 )z
unispeech-satr3 )vilt
ViltProcessor)zvision-text-dual-encoder VisionTextDualEncoderProcessor)wav2vec2r3 )zwav2vec2-conformerr3 )wavlmr3 )whisperWhisperProcessor)xclipXCLIPProcessor
class_namec C s t D ]'\}}| |v r+t|}td| d}zt|| W S ty* Y qw qtj D ]}t|dd | kr?| S q1td}t
|| rOt|| S d S )N.ztransformers.models__name__ transformers) PROCESSOR_MAPPING_NAMESitemsr importlib
import_modulegetattrAttributeErrorPROCESSOR_MAPPING_extra_contentvalueshasattr)ri module_name
processorsmodule processor main_module r| ^/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/auto/processing_auto.pyprocessor_class_from_namee s$
r~ c @ s: e Zd ZdZdd Zeeedd Ze d
ddZ
d S )
AutoProcessora
This is a generic processor class that will be instantiated as one of the processor classes of the library when
created with the [`AutoProcessor.from_pretrained`] class method.
This class cannot be instantiated directly using `__init__()` (throws an error).
c C s t d)Nz}AutoProcessor is designed to be instantiated using the `AutoProcessor.from_pretrained(pretrained_model_name_or_path)` method.)EnvironmentError)selfr| r| r} __init__ s zAutoProcessor.__init__c s| dd}|dur tdt dddur td| d< dd} dd}d d < d}d} fd
d ttj
D }t|t fi |} | durot j
|fi \}
} |
d d}d
|
di v ro|
d d
}| dur|du rtj|fi \}
} |
d d}d
|
di v r|
d d
}|du rt|tfi |} | durt| dd
}
t|
}
W d n1 sw Y |
d d}d
|
di v r|
d d
}|du rt|tstj|fd|i }t|d d}t|drd
|jv r|jd
}|durt|}|du}|dupt |t v }t ||||}|rI|rIt ||fi } dd} t j!"|r>|# |j|fd|i S |durY|j|fd|i S t |t v rmt t | j|fi S z t$j|fd|i W S t%y zt&j|fd|i W Y S t%y Y nw zt'j|fd|i W Y S t%y Y nw Y nw td| d)a
Instantiate one of the processor classes of the library from a pretrained model vocabulary.
The processor class to instantiate is selected based on the `model_type` property of the config object (either
passed as an argument or loaded from `pretrained_model_name_or_path` if possible):
List options
Params:
pretrained_model_name_or_path (`str` or `os.PathLike`):
This can be either:
- a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on
huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or
namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`.
- a path to a *directory* containing a processor files saved using the `save_pretrained()` method,
e.g., `./my_model_directory/`.
cache_dir (`str` or `os.PathLike`, *optional*):
Path to a directory in which a downloaded pretrained model feature extractor should be cached if the
standard cache should not be used.
force_download (`bool`, *optional*, defaults to `False`):
Whether or not to force to (re-)download the feature extractor files and override the cached versions
if they exist.
resume_download (`bool`, *optional*, defaults to `False`):
Whether or not to delete incompletely received file. Attempts to resume the download if such a file
exists.
proxies (`Dict[str, str]`, *optional*):
A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request.
token (`str` or *bool*, *optional*):
The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
when running `huggingface-cli login` (stored in `~/.huggingface`).
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
identifier allowed by git.
return_unused_kwargs (`bool`, *optional*, defaults to `False`):
If `False`, then this function returns just the final feature extractor object. If `True`, then this
functions returns a `Tuple(feature_extractor, unused_kwargs)` where *unused_kwargs* is a dictionary
consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of
`kwargs` which has not been used to update `feature_extractor` and is otherwise ignored.
trust_remote_code (`bool`, *optional*, defaults to `False`):
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to `True` for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.
kwargs (`Dict[str, Any]`, *optional*):
The values in kwargs of any keys which are feature extractor attributes will be used to override the
loaded values. Behavior concerning key/value pairs whose keys are *not* feature extractor attributes is
controlled by the `return_unused_kwargs` keyword parameter.
Passing `token=True` is required when you want to use a private model.
Examples:
```python
>>> from transformers import AutoProcessor
>>> # Download processor from huggingface.co and cache.
>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base-960h")
>>> # If processor files are in a directory (e.g. processor was saved using *save_pretrained('./test/saved_model/')*)
>>> # processor = AutoProcessor.from_pretrained("./test/saved_model/")
```use_auth_tokenNzrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.tokenzV`token` and `use_auth_token` are both specified. Please set only the argument `token`.configtrust_remote_codeT
_from_autoc s i | ] }| v r| | qS r| r| ).0keykwargsr| r}
s z1AutoProcessor.from_pretrained..processor_classr auto_mapzutf-8)encoding
code_revisionz!Unrecognized processing class in z. Can't instantiate a processor, a tokenizer, an image processor or a feature extractor for this model. Make sure the repository contains the files of at least one of those processing classes.)(popwarningswarn
FutureWarningget
ValueErrorinspect signaturer
parameterskeysr
r get_image_processor_dictr get_feature_extractor_dictr openjsonload
isinstancer r from_pretrainedrq rv r r~ typers r r ospathisdirregister_for_auto_classr Exceptionr r )cls pretrained_model_name_or_pathr r r r r processor_auto_mapget_file_from_repo_kwargspreprocessor_config_file config_dict_tokenizer_config_filereaderhas_remote_codehas_local_coder| r r} r s F
z AutoProcessor.from_pretrainedFc C s t j| ||d dS )a
Register a new processor for this class.
Args:
config_class ([`PretrainedConfig`]):
The configuration corresponding to the model to register.
processor_class ([`FeatureExtractorMixin`]): The processor to register.
)exist_okN)rs register) config_classr r r| r| r} r I s
zAutoProcessor.registerN)F) rk
__module__ __qualname____doc__r classmethodr rm r staticmethodr r| r| r| r} r } s =r ),r ro r r r r collectionsr configuration_utilsr dynamic_module_utilsr r feature_extraction_utilsr image_processing_utilsr tokenization_utilsr utilsr
r r auto_factoryr configuration_autor r r r feature_extraction_autor image_processing_autor tokenization_autor
get_loggerrk loggerrm rs strr~ r r| r| r| r} s2
6