o òÜÓhaã@s`dZddlmZmZmZddlmZddlmZm Z m Z m Z m Z ddl mZGdd„deƒZd S) z Processor class for Bros. é)ÚListÚOptionalÚUnioné)ÚProcessorMixin)Ú BatchEncodingÚPaddingStrategyÚPreTokenizedInputÚ TextInputÚTruncationStrategy)Ú TensorTypec"söeZdZdZdgZdZd"‡fdd„ Z              d#d eee e ee e fd e d ee e e fd ee e efdeededeedee dee de de de de de deee efdef dd„Zdd„Zdd„Zed d!„ƒZ‡ZS)$Ú BrosProcessora‘ Constructs a Bros processor which wraps a BERT tokenizer. [`BrosProcessor`] offers all the functionalities of [`BertTokenizerFast`]. See the docstring of [`~BrosProcessor.__call__`] and [`~BrosProcessor.decode`] for more information. Args: tokenizer (`BertTokenizerFast`, *optional*): An instance of ['BertTokenizerFast`]. The tokenizer is a required input. Ú tokenizer)Ú BertTokenizerÚBertTokenizerFastNc s |durtdƒ‚tƒ |¡dS)Nz"You need to specify a `tokenizer`.)Ú ValueErrorÚsuperÚ__init__)ÚselfrÚkwargs©Ú __class__©ú^/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/bros/processing_bros.pyr)szBrosProcessor.__init__TFrÚtextÚadd_special_tokensÚpaddingÚ truncationÚ max_lengthÚstrideÚpad_to_multiple_ofÚreturn_token_type_idsÚreturn_attention_maskÚreturn_overflowing_tokensÚreturn_special_tokens_maskÚreturn_offsets_mappingÚ return_lengthÚverboseÚreturn_tensorsÚreturncKs4|jd||||||||| | | | | ||dœ|¤Ž}|S)z¶ This method uses [`BertTokenizerFast.__call__`] to prepare text for the model. Please refer to the docstring of the above two methods for more information. )rrrrrrr r!r"r#r$r%r&r'r(Nr)r)rrrrrrrr r!r"r#r$r%r&r'r(rÚencodingrrrÚ__call__/s(ñðzBrosProcessor.__call__cOó|jj|i|¤ŽS)zÁ This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )rÚ batch_decode©rÚargsrrrrr-\ózBrosProcessor.batch_decodecOr,)z» This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rÚdecoder.rrrr1cr0zBrosProcessor.decodecCs|jj}tt |¡ƒS©N)rÚmodel_input_namesÚlistÚdictÚfromkeys)rÚtokenizer_input_namesrrrr3jszBrosProcessor.model_input_namesr2)NTFNNrNNNFFFFTN)Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú attributesÚtokenizer_classrrr r rÚboolÚstrrr rÚintr rr+r-r1Úpropertyr3Ú __classcell__rrrrr sr ðþý ü ûúùø ÷ ö õ ô óòñð î-r N)r;ÚtypingrrrÚprocessing_utilsrÚtokenization_utils_baserrr r r Úutilsr r rrrrÚs