o h0@sdZddlZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZeeZeeejZd d d d ZGd dde Zeeje_ejjdurdejjjddddej_dSdS)z9 Processing saving/loading class for common processors. N)Path)OptionalUnion)custom_object_save)PreTrainedTokenizerBase)PushToHubMixin copy_funcdirect_transformers_importloggingrFeatureExtractionMixinImageProcessingMixin) AutoTokenizerAutoFeatureExtractorAutoImageProcessorc@seZdZdZddgZdZdZdZddZddZ dd e fd d Z e   dde eejfdee eejfde de dee ee fdef ddZe dddZe ddZeddZdS) ProcessorMixinza This is a mixin used to provide saving/loading functionality for all processor classes. feature_extractor tokenizerNc Os,|D]}||jvrtd|dqt||jD]\}}||vr(td|d|||<qt|t|jkrMtdt|jdd|jdt|d|D]B\}}t||d }t ||}t |t rrt d d |D}ntt |}t ||std t |jd |d|dt|||qQdS)NzUnexpected keyword argument .z!Got multiple values for argument zThis processor requires z arguments: z, z. Got z arguments instead._classcss"|] }|durtt|VqdSNgetattrtransformers_module.0nrS/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/processing_utils.py Ms z*ProcessorMixin.__init__..z Received a z for argument z, but a z was expected.) attributes TypeErrorziplen ValueErrorjoinitemsrAUTO_TO_BASE_CLASS_MAPPINGget isinstancetuplertype__name__setattr)selfargskwargskeyargattribute_name class_name proper_classrrr__init__6s4      zProcessorMixin.__init__cs0fddjD}d|}jjd|S)Nc s&g|]}d|dtt|qS)z- z: )reprr)rnamer.rr Ys&z+ProcessorMixin.__repr__.. z: )r r% __class__r,)r.attributes_reprrr9r__repr__Xs zProcessorMixin.__repr__F push_to_hubc  s^|dd}|dur tdt|dddurtd||d<tj|dd|rJ|dd}|d |tj j d }j |fi|} |}j durgfd d jD}d d |D} t|| djD]} t| } t| dr}| jj| |qjj durjD]} t| } t| tr| jd=q|rj|||||dddSdS)a Saves the attributes of this processor (feature extractor, tokenizer...) in the specified directory so that it can be reloaded using the [`~ProcessorMixin.from_pretrained`] method. This class method is simply calling [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`]. Please refer to the docstrings of the methods above for more information. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.tokenV`token` and `use_auth_token` are both specified. Please set only the argument `token`.T)exist_okcommit_messagerepo_idcsg|]}t|qSr)r)rr3r9rrr:sz2ProcessorMixin.save_pretrained..cSs g|] }t|tr |jn|qSr)r)r init_kwargs)rarrrr:s )config_set_processor_classauto_map)rErB)popwarningswarn FutureWarningr(r$osmakedirssplitpathsep _create_repo_get_files_timestamps _auto_classr rrhasattrrKr<r,save_pretrainedr)rrH_upload_modified_files) r.save_directoryr?r0r@rErFfiles_timestampsattrsconfigsr3 attributerr9rrZ]sR             zProcessorMixin.save_pretrainedmainpretrained_model_name_or_path cache_dirforce_downloadlocal_files_onlyrBrevisionc Ks~||d<||d<||d<||d<|dd}|dur*tdt|dur(td|}|dur2||d <|j|fi|} || S) a Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`], image processor [`~image_processing_utils.ImageProcessingMixin`] and the tokenizer [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] methods. Please refer to the docstrings of the methods above for more information. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - a path to a *directory* containing a feature extractor file saved using the [`~SequenceFeatureExtractor.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved feature extractor JSON *file*, e.g., `./my_model_directory/preprocessor_config.json`. **kwargs Additional keyword arguments passed along to both [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`]. rcrdrerfr@NrArCrB)rMrNrOrPr$_get_arguments_from_pretrained) clsrbrcrdrerBrfr0r@r/rrrfrom_pretraineds&( zProcessorMixin.from_pretrained AutoProcessorcCsDt|ts|j}ddlmm}t||st|d||_dS)a Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with `AutoProcessor`. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoProcessor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) r)strr,transformers.models.automodelsautorYr$rX)rh auto_class auto_modulerrrregister_for_auto_classs   z&ProcessorMixin.register_for_auto_classc Ksg}|jD]A}t||d}t|tr5tdd|D}|dd}|r0|ddur0|d}n |d}ntt|}||j|fi|q|S)Nrcss&|]}|dur tt|ndVqdSrrrrrrrs$z@ProcessorMixin._get_arguments_from_pretrained..use_fastTrr)r rr)r*r(rappendri) rhrbr0r/r3r4classesrrattribute_classrrrrgs      z-ProcessorMixin._get_arguments_from_pretrainedcCst||jd}t|ddS)Nrmodel_input_names)rr )r.first_attributerrrrvs z ProcessorMixin.model_input_names)F)NFFNra)rj)r, __module__ __qualname____doc__r feature_extractor_classtokenizer_classrXr6r>boolrZ classmethodrrkrQPathLikerrirqrgpropertyrvrrrrr*sD"K  >  r processorrjzprocessor files)object object_class object_files)rzrQrNpathlibrtypingrrdynamic_module_utilsrtokenization_utils_baserutilsrr r r get_loggerr,logger__file__parentrr'rr?formatrrrrs,     p