o h@s@dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZddlmZmZmZmZd d lmZd d lmZd d l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*d d l+m,Z,m-Z-m.Z.e./e0Z1dZ2dZ3dZ4dZ5e!d7Z!eeeedZ6de2iZ7e-e!Gddde&Z8dS)z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)AnyDictListOptionalTupleUnion)Encoding) Tokenizer)Decoder) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer)convert_slow_tokenizer)PreTrainedTokenizer) INIT_TOKENIZER_DOCSTRING AddedToken BatchEncodingPreTokenizedInputPreTokenizedInputPairPreTrainedTokenizerBaseSpecialTokensMixin TextInput TextInputPairTruncationStrategy)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )BPEUnigram WordLevel WordPiecetokenizer_filec%s>eZdZUdZeZdZeed<fddZ e de fddZ e de fd d Z e defd d Zdeeeffd dZe deeeffddZe deeeffddZe deeeffddZdeeeffddZdefddZe defddZe defddZ       d^ded ee d!ee d"e d#e d$e d%e d&e deeee fe!effd'd(Z"d)e#ee!efde#ee!effd*d+Z$d,edefd-d.Z%d/edeefd0d1Z&d_d2e!e#eefdefd3d4Z'd_d5e defd6d7Z( d_d8e#ee!efd9e de#ee!effd:d;Z)d`dd?Z*d@e+dAe,dBedCedDeef dEdFZ-de+j.e,j/ddGddddddddddfdHe#e!e0e!e1e!e2e!e3fd=e d@e+dAe,dBeedCedIe dDeedJeed ee d!ee d"e d#e d$e d%e d&e de4f"dKdLZ5dde+j.e,j/ddGddddddddddfds  z4PreTrainedTokenizerFast.__init__..cS|dSNrr7)xr7r7r=z2PreTrainedTokenizerFast.__init__..keycSsg|]}t|qSr7)strr8r:r7r7r=r>scs g|] }|vr|vr|qSr7r7rG)encoder tokens_to_addr7r=r>sr)special_tokensr7)"popr& ValueErrorcopydeepcopy TokenizerFast from_filer _tokenizerupdate init_kwargs_decode_use_source_tokenizer truncationenable_truncation setdefault no_truncationpaddingenable_paddingsuper__init__sorteditemslistadded_tokens_encoderkeysall_special_tokens_extendedlenall_special_tokens isinstancerspecialrFappend _add_tokens)r<argskwargsr'slow_tokenizerfast_tokenizer_filer)r*fast_tokenizer _truncation_paddingis_last_specialtokensrJr: is_special __class__)rHr<rIr=r\^s                z PreTrainedTokenizerFast.__init__returncCdS)NTr7r;r7r7r=is_fastszPreTrainedTokenizerFast.is_fastcCrv)z `bool`: Whether or not the slow tokenizer can be saved. Usually for sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. Tr7r;r7r7r=can_save_slow_tokenizersz/PreTrainedTokenizerFast.can_save_slow_tokenizercC|jjddS)zP `int`: Size of the base vocabulary (without the added tokens). Fwith_added_tokensrQget_vocab_sizer;r7r7r= vocab_sizesz"PreTrainedTokenizerFast.vocab_sizecCry)NTrz)rQ get_vocabr;r7r7r=rz!PreTrainedTokenizerFast.get_vocabcCs|SN)rr;r7r7r=vocabszPreTrainedTokenizerFast.vocabcC ddt|jdddDS)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. cSi|]\}}|j|qSr7contentr8vkr7r7r= z@PreTrainedTokenizerFast.added_tokens_encoder..cSr?r@r7itemr7r7r=rBrCz>PreTrainedTokenizerFast.added_tokens_encoder..rDr]r*r^r;r7r7r=r`s z,PreTrainedTokenizerFast.added_tokens_encodercCs |jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `Dict[str, int]`: The added tokens. )rQget_added_tokens_decoderr;r7r7r=r*s z,PreTrainedTokenizerFast.added_tokens_decodercCr)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. cSrr7rrr7r7r=rrz;PreTrainedTokenizerFast.get_added_vocab..cSr?r@r7rr7r7r=rBrCz9PreTrainedTokenizerFast.get_added_vocab..rDrr;r7r7r=get_added_vocabs z'PreTrainedTokenizerFast.get_added_vocabcCry)zD Size of the full vocabulary with the added tokens. Trzr|r;r7r7r=__len__szPreTrainedTokenizerFast.__len__cCs|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rQr;r7r7r=backend_tokenizersz)PreTrainedTokenizerFast.backend_tokenizercCs|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rQdecoderr;r7r7r=rszPreTrainedTokenizerFast.decoderFTencodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec Cs|dur d|jv}|durd|jv}|r |jdur |g|j} n|g} tt} | D]>} | d| j|r=| d| j|rG| d| j|rQ| d| j|r[| d| j |rg| dt | jq)| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) Ntoken_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingr5) model_input_names overflowingrr_rgidstype_idsrroffsetsrc) r<rrrrrrrr encodings encoding_dicter7r7r=_convert_encoding s,  z)PreTrainedTokenizerFast._convert_encodingrqcs2|durdSt|tr|Sfdd|DS)aT Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. Ncg|]}|qSr7)#_convert_token_to_id_with_added_vocrGr;r7r=r>KzAPreTrainedTokenizerFast.convert_tokens_to_ids..)rerFrr<rqr7r;r=convert_tokens_to_ids:s    z-PreTrainedTokenizerFast.convert_tokens_to_idsr:cCs|j|}|dur |jS|Sr)rQ token_to_id unk_token_id)r<r:r9r7r7r=rMs z;PreTrainedTokenizerFast._convert_token_to_id_with_added_vocr9cCs|jt|Sr)rQ id_to_tokenint)r<r9r7r7r=_convert_id_to_tokenSsz,PreTrainedTokenizerFast._convert_id_to_token new_tokenscCs|r|j|S|j|Sr)rQadd_special_tokens add_tokens)r<rrJr7r7r=rhVs  z#PreTrainedTokenizerFast._add_tokenspaircCs |j|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rQnum_special_tokens_to_add)r<rr7r7r=r\s z1PreTrainedTokenizerFast.num_special_tokens_to_addrskip_special_tokenscCsRt|tr |j|Sg}|D]}t|}|r||jvrq||j|q|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). )rerrQrall_special_idsrg)r<rrrqr9r7r7r=convert_ids_to_tokensqs  z-PreTrainedTokenizerFast.convert_ids_to_tokenstextrcKs|jd|||d|S)N)r text_pairrr7) encode_plusrq)r<rrrrjr7r7r=tokenizesz PreTrainedTokenizerFast.tokenizepadding_strategyr/r+r.r6c s|jj|jj}|tjkrdur|jn&|||j|jd}dur'd}n fdd|D}||kr=|jjdi||t j krO|durM|j dSdS|t j krV|nd} | |j |j|j|j|d}||krt|jjdi|dSdS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). N)r+r.r0r-csi|] }||dqSrget)r8rrnr7r=rszFPreTrainedTokenizerFast.set_truncation_and_padding..)r5r-pad_idr1r3r6r7)rQrUrYrDO_NOT_TRUNCATErXvaluer,rVr DO_NOT_PAD no_padding MAX_LENGTHr4 pad_token_idr1r2rZ) r<rr/r+r.r6rotargetcurrentr5r7rr=set_truncation_and_paddings>    z2PreTrainedTokenizerFast.set_truncation_and_paddingrbatch_text_or_text_pairsis_split_into_wordsreturn_tensorscst|ttfstdt|dj|||||djj|||d}fdd|D}i}|ddD]fdd|D}||<q>d d|D}rsg}t |D]\}\}}||gt |d 7}q]||d <|d D] } ||qwt ||| d S) Nz:batch_text_or_text_pairs has to be a list or a tuple (got ))rr/r+r.r6)ris_pretokenizedc s&g|]}j|dqS))rrrrrrrr)r)r8r)rrrrrrr<rr7r=r>s z>PreTrainedTokenizerFast._batch_encode_plus..rcs"g|] \}}|D]}|q qSr7r7)r8r_rrDr7r=r>s"cSsg|] \}}|D]}|qqSr7r7)r8rrrr7r7r=r>sroverflow_to_sample_mapping) tensor_type) retupler_ TypeErrortyperrQ encode_batchra enumeraterc&_eventual_warn_about_too_long_sequencer)r<rrrr/r+r.rr6rrrrrrrrrtokens_and_encodingssanitized_tokensstacksanitized_encodingsritoksrrr7) rErrrrrrr<rr=_batch_encode_pluss@    z*PreTrainedTokenizerFast._batch_encode_plusrcKs|r||fgn|g}|j|f||||||| | | | | ||||d|}| dur6| s6tdd|D|j}||d|||S)N)rrrr/r+r.r6rrrrrrrrcSs8i|]\}}|t|dkrt|dtr|dn|qS)r)rcrer_)r8rErr7r7r=rXs&z8PreTrainedTokenizerFast._encode_plus..r)rrr^rr)r<rrrrr/r+r.rr6rrrrrrrrrj batched_inputbatched_outputr7r7r= _encode_plus*s> z$PreTrainedTokenizerFast._encode_pluscCs|jj|Sr)rrdecoderr7r7r=convert_tokens_to_stringcrz0PreTrainedTokenizerFast.convert_tokens_to_string token_idsclean_up_tokenization_spacescKsV|dd|_t|tr|g}|jj||d}|dur|n|j}|r)||}|S|S)Nuse_source_tokenizerF)r)rKrTrerrQrrclean_up_tokenization)r<rrrrjr clean_textr7r7r=_decodefs  zPreTrainedTokenizerFast._decodesave_directory file_names legacy_formatfilename_prefixc s6t|}jdur|durtd|dus|duo jduo j}|dup(|du}|r}tj||r5|dndt}fddj D}|rot |d d d } t j |d ddd d} | | Wdn1sjwYj||d} || |f}|rtj||r|dndt} j| || f}|S)z Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well as in a unique JSON file containing {config + vocab + added-tokens}. NTzYour tokenizer does not have a legacy version defined and therefore cannot register this version. You might consider leaving the legacy_format at `None` or setting it to `False`.F-cs i|] \}}|jkr||qSr7)r~)r8tokr9r;r7r=rs z.wzutf-8)r)indent sort_keys ensure_ascii )r)rFr&rLrxospathjoinADDED_TOKENS_FILEr`r^openjsondumpswritesave_vocabularyTOKENIZER_FILErsave) r<rrrr save_slow save_fastadded_tokens_file added_vocabfout_str vocab_filesr$r7r;r=_save_pretrained~s<    z(PreTrainedTokenizerFast._save_pretrainedc st|j}|d}|d} d} |dddkr)i|dd<g|dd<nW|ddd kre|dd durd|dd } |dd| d } durU| vrU| } d |dd <| d gg|dd<n|ddd vrti|dd<n td|ddddurd|dvr|ddvr|dd|dd<tt|g} |D]5} | dd}| dd}|ddd kr|sqdur| dvrՈ| d| d<| t d'i| q|dur| ||dddkrd|vr|dddur|dd|d<|dddkr'd|vr'|dddur'|dd|d<|ddd kr9| dur9| |d<|ddurP|dddkrPt j |d<t|dd}|d'|| d|}j|||d| durt}d| vr| dD]0}| d|d}durfdd|D}|| d|d<fdd|D| d|d <qd!D]'}|| vr| |\}}dur|vr͈|}|}||g| |<q| |d<tt||j}tj}|d"|D]E}t|d#|dur;t||}dur|vr|}t|d#|}t|t r7t ||j|j|j|jd$d%||<q|||<q|j}|durJ| |t|d krU||d"<|jd'd&i|S)(uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `List[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`Dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelrr rmergesr!unk_idrg)r"r#z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenrfidrcontinuing_subword_prefixend_of_word_suffix pre_tokenizer ByteLevelinitial_alphabet)r~rJ)r5trainerrJrqcsg|]}||qSr7rrG)special_tokens_mapr7r=r> rzCPreTrainedTokenizerFast.train_new_from_iterator..crr7)rrG) tokenizerr7r=r>"rr)clssepadditional_special_tokensrT) single_wordlstriprstrip normalizedrfr'r7) rloadsrQto_strrKrLrOfrom_strrrgrextendpre_tokenizers_fastralphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrrSrMrSPECIAL_TOKENS_ATTRIBUTESremovegetattrrer r!r"r#rrcrt)r< text_iteratorr~r5new_special_tokensrrjtokenizer_jsonrrrrrJ added_tokenrfr trainer_classrtrained_tokenizer_jsonrErq special_tokenr:token_idspecial_tokens_listspecial_token_fullrr7)rrr=train_new_from_iterators"                "            z/PreTrainedTokenizerFast.train_new_from_iterator)NNFFFFT)F)NF)FN)NN)NNN)>__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesr&r__annotations__r\propertyboolrwrxrr~rrFrrr`rr*rrrOr DecoderFastr EncodingFastrrrrrrrrrrhrrrrrrrrrrrrrrrrrrPathLiker r9 __classcell__r7r7rsr=r%Ms   d     */  $ L      X       9   5r%)9r=rMrr collectionsrtypingrrrrrrtokenizers.pre_tokenizerspre_tokenizersr( tokenizersr rDr rOtokenizers.decodersr rCtokenizers.trainersr r rrrtokenization_utilsrtokenization_utils_baserrrrrrrrrrutilsrrr get_loggerr:loggerrSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILErr*r>r%r7r7r7r=s:        0