o h\E@s ddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZeeZdd d d d d ZddiddiddiddidZddiZiZdZGdddeZdede ee fdejfddZdeddfddZ dedee e ffdd Z!dS)!N)Path)copyfile)AnyDictListOptionalTupleUnion)PreTrainedTokenizer)loggingz source.spmz target.spmz vocab.jsonztarget_vocab.jsonztokenizer_config.json) source_spm target_spmvocabtarget_vocab_filetokenizer_config_filezHelsinki-NLP/opus-mt-en-dezIhttps://huggingface.co/Helsinki-NLP/opus-mt-en-de/resolve/main/source.spmzIhttps://huggingface.co/Helsinki-NLP/opus-mt-en-de/resolve/main/target.spmzIhttps://huggingface.co/Helsinki-NLP/opus-mt-en-de/resolve/main/vocab.jsonzThttps://huggingface.co/Helsinki-NLP/opus-mt-en-de/resolve/main/tokenizer_config.json)r rrru▁c seZdZdZeZeZeZ e Z ddgZ e dZ        dEd eeeefd dffd d ZddZded efddZddZdefddZded eefddZded efddZfddZfd d!Zd"eed efd#d$ZdFd eefd%d&Z d'd(Z!d)d*Z"e#d efd+d,Z$dFd-ed.eed e%efd/d0Z&d efd1d2Z'd3d4Z(d5d6Z)d efd7d8Z*d9ed dfd:d;Z+dd?Z- dGd@edAeedBe.d eefdCdDZ/Z0S)HMarianTokenizeraB Construct a Marian tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: source_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the source language. target_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the target language. source_lang (`str`, *optional*): A string representing the source language. target_lang (`str`, *optional*): A string representing the target language. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. model_max_length (`int`, *optional*, defaults to 512): The maximum sentence length the model accepts. additional_special_tokens (`List[str]`, *optional*, defaults to `["", ""]`): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import MarianForCausalLM, MarianTokenizer >>> model = MarianForCausalLM.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> src_texts = ["I am a small frog.", "Tom asked his teacher for advice."] >>> tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."] # optional >>> inputs = tokenizer(src_texts, text_target=tgt_texts, return_tensors="pt", padding=True) >>> outputs = model(**inputs) # should work ``` input_idsattention_maskz>>.+<rFsp_model_kwargsreturnc  s.| durin| |_t|sJd|| |_t||_t||jvr)tdt| |jvs2J| rHt||_dd|j D|_ g|_ ndd|j D|_ dd|jD|_ ||_ ||_ ||g|_t||j|_t||j|_|j|_|j|_|tjd ||||| | |j|| d | dS) Nzcannot find spm source z token must be in the vocabcSi|]\}}||qSr.0kvrrd/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/marian/tokenization_marian.py z,MarianTokenizer.__init__..cSrrrrrrr!r"r#cSs$g|]}|dr|dr|qS)z>>z<<) startswithendswithrrrrr! s$z,MarianTokenizer.__init__..) source_lang target_lang unk_token eos_token pad_tokenmodel_max_lengthrrseparate_vocabsr)rrexistsr. load_jsonencoderstrKeyErrortarget_encoderitemsdecodersupported_language_codesr(r) spm_filesload_spm spm_source spm_target current_spmcurrent_encoder_setup_normalizersuper__init__)selfr rrrr(r)r*r+r,r-rr.kwargs __class__rr!r@sD    zMarianTokenizer.__init__c CsNzddlm}||jj|_WdSttfy&tddd|_YdSw)Nr)MosesPunctNormalizerz$Recommended: pip install sacremoses.cSs|SNr)xrrr!sz3MarianTokenizer._setup_normalizer..) sacremosesrEr( normalizepunc_normalizer ImportErrorFileNotFoundErrorwarningswarn)rArErrr!r>s  z!MarianTokenizer._setup_normalizerrGcCs|r||SdS)zHCover moses empty string edge case. They return empty list for '' input!)rK)rArGrrr!rJszMarianTokenizer.normalizecCs|j||j|jSrF)r=getr*)rAtokenrrr!_convert_token_to_idsz$MarianTokenizer._convert_token_to_idtextcCs2|j|}|r|dgng}||jd|fS)z6Remove language codes like >>fr<< before sentencepiecerrP)language_code_rematchgroupsub)rArTrVcoderrr!remove_language_codes z$MarianTokenizer.remove_language_codecCs&||\}}|jj|td}||S)N)out_type)rZr<encoder2)rArTrYpiecesrrr! _tokenizeszMarianTokenizer._tokenizeindexcCs|j||jS)z?Converts an index (integer) in a token (str) using the decoder.)r6rQr*)rAr_rrr!_convert_id_to_tokensz$MarianTokenizer._convert_id_to_tokenc tj|fi|S)ao Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]`: The list of decoded sentences. )r? batch_decode)rA sequencesrBrCrr!rbszMarianTokenizer.batch_decodec ra)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. )r?decode)rA token_idsrBrCrr!rdszMarianTokenizer.decodetokenscCsv|jr|jn|j}g}d}|D]}||jvr$||||d7}g}q||q|||7}|td}|S)zQUses source spm if _decode_use_source_tokenizer is True, and target spm otherwiserP ) _decode_use_source_tokenizerr:r;all_special_tokens decode_piecesappendreplaceSPIECE_UNDERLINEstrip)rArfsp_modelcurrent_sub_tokens out_stringrRrrr!convert_tokens_to_strings   z(MarianTokenizer.convert_tokens_to_stringcCs$|dur ||jgS|||jgS)z=Build model inputs from a sequence by appending eos_token_id.N) eos_token_id)rA token_ids_0 token_ids_1rrr! build_inputs_with_special_tokenss z0MarianTokenizer.build_inputs_with_special_tokenscCs|j|_|j|_dSrF)r:r<r1r=rArrr!_switch_to_input_mode&s z%MarianTokenizer._switch_to_input_modecCs|j|_|jr |j|_dSdSrF)r;r<r.r4r=rwrrr!_switch_to_target_mode*s z&MarianTokenizer._switch_to_target_modecCs t|jSrF)lenr1rwrrr! vocab_size/s zMarianTokenizer.vocab_sizesave_directoryfilename_prefixc Cstj|std|ddSg}|jrOtj||r |dndtd}tj||r1|dndtd}t|j |t|j || || |ntj||rY|dndtd}t|j || |t tdtdg|j |j|jgD]Z\}}} tj||r|dnd|} tj|tj| krtj|rt|| | | q|tj|st| d } | } | | Wdn1swY| | q|t|S) NzVocabulary path (z) should be a directory-rPrrr rwb)ospathisdirloggererrorr.joinVOCAB_FILES_NAMES save_jsonr1r4rkzipr8r:r;abspathisfileropenserialized_model_protowritetuple) rAr|r} saved_filesout_src_vocab_fileout_tgt_vocab_fileout_vocab_filespm_save_filename spm_orig_path spm_model spm_save_pathficontent_spiece_modelrrr!save_vocabulary3sR        $      zMarianTokenizer.save_vocabularycCs|SrF) get_src_vocabrwrrr! get_vocab`szMarianTokenizer.get_vocabcCt|jfi|jSrF)dictr1added_tokens_encoderrwrrr!rczMarianTokenizer.get_src_vocabcCrrF)rr4added_tokens_decoderrwrrr! get_tgt_vocabfrzMarianTokenizer.get_tgt_vocabcCs"|j}|dddD|S)NcSsi|]}|dqSrFrr&rrr!r"lsz0MarianTokenizer.__getstate__..)r:r;r<rKr)__dict__copyupdate)rAstaterrr! __getstate__is  zMarianTokenizer.__getstate__dcsF|_tds i_fddjD\__j_dS)Nrc3s|] }t|jVqdSrF)r9r)rfrwrr! wsz/MarianTokenizer.__setstate__..)rhasattrrr8r:r;r<r>)rArrrwr! __setstate__ps   zMarianTokenizer.__setstate__cOsdS)zJust EOSr)rAargsrBrrr!num_special_tokens_to_add{sz)MarianTokenizer.num_special_tokens_to_addcs(t|j|jfdd|DS)Ncsg|] }|vr dndqS)rrr)rrGall_special_idsrr!r'sz7MarianTokenizer._special_token_mask..)setrremove unk_token_id)rAseqrrr!_special_token_masks  z#MarianTokenizer._special_token_maskrtrualready_has_special_tokenscCs:|r||S|dur||dgS|||dgS)zCGet list where entries are [1] if a token is [eos] or [pad] else 0.Nr)r)rArtrurrrr!get_special_tokens_masks  z'MarianTokenizer.get_special_tokens_mask) NNNrrrrNFrF)NF)1__name__ __module__ __qualname____doc__rvocab_files_namesPRETRAINED_VOCAB_FILES_MAPpretrained_vocab_files_mapPRETRAINED_INIT_CONFIGURATIONpretrained_init_configuration&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESmax_model_input_sizesmodel_input_namesrecompilerUrrr2rr@r>rJrSrZrr^intr`rbrdrrrvrxrypropertyr{rrrrrrrrrboolr __classcell__rrrCr!r?sl:  >    - rrrrcCstjdi|}|||S)Nr) sentencepieceSentencePieceProcessorLoad)rrspmrrr!r9s r9cCs@t|d}tj||ddWddS1swYdS)Nw)indent)rjsondump)datarrrrr!rs "rcCs8t|d }t|WdS1swYdS)Nr)rrload)rrrrr!r0s $r0)"rrrrNpathlibrshutilrtypingrrrrrr rtokenization_utilsr utilsr get_loggerrrrrrrrmrr2rr9rr0rrrr!sH         S