o h9@sddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZeeZdZdd iZdd d d iZd d d ZgdZGdddeZdS)N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)loggingu▁ vocab_filezsentencepiece.bpe.modelzVhttps://huggingface.co/facebook/mbart-large-en-ro/resolve/main/sentencepiece.bpe.modelzUhttps://huggingface.co/facebook/mbart-large-cc25/resolve/main/sentencepiece.bpe.model)zfacebook/mbart-large-en-rozfacebook/mbart-large-cc25i)ar_ARcs_CZde_DEen_XXes_XXet_EEfi_FIfr_XXgu_INhi_INit_ITja_XXkk_KZko_KRlt_LTlv_LVmy_MMne_NPnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CNc s$eZdZUdZeZeZeZ ddgZ gZ e e ed<gZe e ed<      dGd eeeefffd d ZddZddZeddZedefddZejdedd fddZ dHde e dee e dede e ffdd Z dIde e dee e de e fd d!Z dIde e dee e de e fd"d#Zd$ed%eed&eefd'd(Zd)d*Z d+ede efd,d-Z!d.d/Z"d0d1Z#d2d3Z$dId4ed5eede%efd6d7Z& 8 9dJd:e ed%ed;ee ed&ede'f fdd?Z)d@dAZ*dKdBdCZ+dDedd fdEdFZ,Z-S)LMBartTokenizeruT Construct an MBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizer >>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO") >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ``` input_idsattention_mask prefix_tokens suffix_tokensNsp_model_kwargsc s~t|tr t|dddn|}| durin| _tjdij_jt||_ddddd_ d_ t j_ fd d t tD_d d jD_t jt jj j d <j jd d j D_tj| durfdd| Dtjd|||||||d| | jd || dur| nd_jj_| _jdS)NTF)lstrip normalizedrr)r,r/r-r.cs"i|] \}}|j|jqS) sp_model_sizefairseq_offset).0icodeselfr6b/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/mbart/tokenization_mbart.py ysz+MBartTokenizer.__init__..cSi|]\}}||qSr6r6r9kvr6r6r>r?|r0cSr@r6r6rAr6r6r>r?rDcsg|]}|vr|qSr6r6)r9t)_additional_special_tokensr6r> z+MBartTokenizer.__init__..) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensr1rr6) isinstancestrr r1spmSentencePieceProcessorsp_modelLoadr fairseq_tokens_to_idsr8lenr7 enumerateFAIRSEQ_LANGUAGE_CODESlang_code_to_iditemsid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextendsuper__init__ _src_langcur_lang_code_idrRset_src_lang_special_tokens)r=r rIrJrLrMrKrNrOrPrQrRr1rSkwargs __class__)rFr=r>rgPsR     zMBartTokenizer.__init__cCs$|j}d|d<|j|d<|S)NrXsp_model_proto)__dict__copyrXserialized_model_proto)r=stater6r6r> __getstate__s zMBartTokenizer.__getstate__cCs<||_t|ds i|_tjdi|j|_|j|jdS)Nr1r6)rohasattrr1rVrWrXLoadFromSerializedProtorn)r=dr6r6r> __setstate__s  zMBartTokenizer.__setstate__cCst|jt|j|jdS)Nr4)r[rXr^r8r<r6r6r> vocab_sizeszMBartTokenizer.vocab_sizereturncCs|jSN)rhr<r6r6r>rQszMBartTokenizer.src_lang new_src_langcCs||_||jdSrz)rhrj)r=r{r6r6r>rQsF token_ids_0 token_ids_1already_has_special_tokenscsx|r tj||ddSdgt|j}dgt|j}|dur*|dgt||S|dgt|dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r|r}r~r4Nr)rfget_special_tokens_maskr[r*r+)r=r|r}r~ prefix_ones suffix_onesrlr6r>rs$z&MBartTokenizer.get_special_tokens_maskcCs,|dur |j||jS|j|||jS)ab Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)r*r+)r=r|r}r6r6r> build_inputs_with_special_tokenssz/MBartTokenizer.build_inputs_with_special_tokenscCsP|jg}|jg}|durt|||dgSt||||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr) sep_token_id cls_token_idr[)r=r|r}sepclsr6r6r>$create_token_type_ids_from_sequencess "z3MBartTokenizer.create_token_type_ids_from_sequencesreturn_tensorsrQrRcKsJ|dus|dur td||_||fd|d|}||}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensrforced_bos_token_id) ValueErrorrQconvert_tokens_to_ids)r= raw_inputsrrQrR extra_kwargsinputs tgt_lang_idr6r6r>_build_translation_inputs s z(MBartTokenizer._build_translation_inputscs(fddtjD}|j|S)Ncsi|]}||qSr6)convert_ids_to_tokens)r9r:r<r6r>r?rHz,MBartTokenizer.get_vocab..)rangerxraadded_tokens_encoder)r=vocabr6r<r> get_vocabs zMBartTokenizer.get_vocabtextcCs|jj|tdS)N)out_type)rXencoderU)r=rr6r6r> _tokenizeszMBartTokenizer._tokenizecCs4||jvr |j|S|j|}|r||jS|jS)z0Converts a token (str) in an id using the vocab.)rZrX PieceToIdr8 unk_token_id)r=tokenspm_idr6r6r>_convert_token_to_id!s   z#MBartTokenizer._convert_token_to_idcCs&||jvr |j|S|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rbrX IdToPiecer8)r=indexr6r6r>_convert_id_to_token*s  z#MBartTokenizer._convert_id_to_tokencCsd|td}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r=tokens out_stringr6r6r>convert_tokens_to_string0sz'MBartTokenizer.convert_tokens_to_stringsave_directoryfilename_prefixcCstj|std|ddStj||r|dndtd}tj|jtj|kr?tj |jr?t |j||fStj |jsgt |d}|j }||Wd|fS1sbwY|fS)NzVocabulary path (z) should be a directory-rr wb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathr isfileropenrXrqwrite)r=rrout_vocab_fileficontent_spiece_modelr6r6r>save_vocabulary5s" (     zMBartTokenizer.save_vocabularyrr! src_texts tgt_textsc s"||_||_tj||fi|Srz)rQrRrfprepare_seq2seq_batch)r=rrQrrRrkrlr6r>rFsz$MBartTokenizer.prepare_seq2seq_batchcC ||jSrz)rjrQr<r6r6r>_switch_to_input_modeR z$MBartTokenizer._switch_to_input_modecCrrz)set_tgt_lang_special_tokensrRr<r6r6r>_switch_to_target_modeUrz%MBartTokenizer._switch_to_target_modecC$|j||_g|_|j|jg|_dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].Nr^ cur_lang_coder* eos_token_idr+)r=rQr6r6r>rjX z*MBartTokenizer.set_src_lang_special_tokenslangcCr)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nr)r=rr6r6r>r^rz*MBartTokenizer.set_tgt_lang_special_tokens) r,r-r-r,r.r/r0NNNNN)NFrz)rNr!)ryN).__name__ __module__ __qualname____doc__rvocab_files_names&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESmax_model_input_sizesPRETRAINED_VOCAB_FILES_MAPpretrained_vocab_files_mapmodel_input_namesr*rint__annotations__r+rrrUrrgrsrwpropertyrxrQsetterboolrrrrrrrrrrrr rrrrjr __classcell__r6r6rlr>r'3s  N             r')rshutilrtypingrrrrr sentencepiecerVtokenization_utilsr r r utilsr get_loggerrrrrrrr]r'r6r6r6r>s&