o hN@sdZddlZddlZddlZddlZddlmZmZmZm Z ddl m Z ddl m Z e eZddd d Zd d id d id did Zd diZd ddgddddiZddZddZddZGddde ZdS)zTokenization classes for FSMT.N)DictListOptionalTuple)PreTrainedTokenizer)loggingzvocab-src.jsonzvocab-tgt.jsonz merges.txt)src_vocab_filetgt_vocab_file merges_filezstas/tiny-wmt19-en-dezHhttps://huggingface.co/stas/tiny-wmt19-en-de/resolve/main/vocab-src.jsonzHhttps://huggingface.co/stas/tiny-wmt19-en-de/resolve/main/vocab-tgt.jsonzDhttps://huggingface.co/stas/tiny-wmt19-en-de/resolve/main/merges.txtiende)langsmodel_max_lengthspecial_tokens_map_filefull_tokenizer_filecCs6t}|d}|ddD] }|||f|}q |S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charcharr`/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/fsmt/tokenization_fsmt.py get_pairs9s rcCs|dd}tdd|}|dd}|dd}|dd}|d d }|d d }|d d }|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|d d!}|d"d#}|d$d%}|d&d'}|d(d)}|d*d+}|d,d-}td.d|}|d/d0}|d1d2}|d3d4}|d5d6}|d7d8}|d9d:}|d;d<}|d=d>}|d?d@}|S)Azz Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/replace-unicode-punctuation.perl u，,u。\s*z. u、u”"u“u∶:u：u？?u《u》u）)u！!u（(u；;u１1u」u「u０0u３3u２2u５5u６6u９9u７7u８8u４4u．\s*u～~u’'u…z...u━-u〈u【[u】]u％%)replaceresub)textrrrreplace_unicode_punctFsJ                                  r:cCs8g}|D]}t|}|drq||qd|S)zw Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/remove-non-printing-char.perl C) unicodedatacategory startswithappendjoin)r9outputrcatrrrremove_non_printing_charqs    rDc seZdZdZeZeZeZ e Z ddgZ         d=fd d Z d eeeffd dZed efddZddZddZddZddZeddZeddZddZdd Zd!d"Zd>d$d%Zd&d'Zd(d)Zd*d+Z d?d,e!ed-e"e!ed e!efd.d/Z# d@d,e!ed-e"e!ed0e$d e!effd1d2 Z% d?d,e!ed-e"e!ed e!efd3d4Z&d?d5ed6e"ed e'efd7d8Z(d9d:Z)d;d<Z*Z+S)A FSMTTokenizera Construct an FAIRSEQ Transformer tokenizer. Based on Byte-Pair Encoding. The tokenization process is the following: - Moses preprocessing and tokenization. - Normalizing all inputs text. - The arguments `special_tokens` and the function `set_special_tokens`, can be used to add additional symbols (like "__classify__") to a vocabulary. - The argument `langs` defines a pair of languages. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: langs (`List[str]`, *optional*): A list of two languages to translate from and to, for instance `["en", "ru"]`. src_vocab_file (`str`, *optional*): File containing the vocabulary for the source language. tgt_vocab_file (`st`, *optional*): File containing the vocabulary for the target language. merges_file (`str`, *optional*): File containing the merges. do_lower_case (`bool`, *optional*, defaults to `False`): Whether or not to lowercase the input when tokenizing. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskNFc  szddl} Wn tytdw| |_||_||_||_||_i|_i|_i|_ |r9t |dkr9|\|_ |_ nt d|dt|dd} t| |_Wdn1sXwYt|dd} t| }dd |D|_Wdn1s}wYt|dd}|d dd }Wdn1swYd d |D}tt|tt ||_i|_tjd||||||||| d | dS)NrnYou need to install sacremoses to use XLMTokenizer. See https://pypi.org/project/sacremoses/ for installation.zFarg `langs` needs to be a list of 2 langs, e.g. ['en', 'ru'], but got z. Usually that means that tokenizer can't find a mapping for the given model path in PRETRAINED_VOCAB_FILES_MAP, and other maps of this tokenizer.utf-8encodingcSi|]\}}||qSrr.0kvrrr z*FSMTTokenizer.__init__.. cSs g|] }t|ddqS)NrM)tuplesplit)rSmergerrr  z*FSMTTokenizer.__init__..) rr r r do_lower_case unk_token bos_token sep_token pad_tokenr) sacremoses ImportErrorsmr r r r_cache_moses_punct_normalizercache_moses_tokenizercache_moses_detokenizerlensrc_langtgt_lang ValueErroropenjsonloadencoderitemsdecoderreadr[dictziprange bpe_rankscachesuper__init__)selfrr r r r_r`rarbrckwargsrdsrc_vocab_handletgt_vocab_handle tgt_vocab merges_handlemerges __class__rrr{s\     zFSMTTokenizer.__init__returncCs|SN) get_src_vocabr|rrr get_vocabszFSMTTokenizer.get_vocabcCs|jSr)src_vocab_sizerrrr vocab_sizeszFSMTTokenizer.vocab_sizecC2||jvr|jj|d}||j|<|j||SNlang)rgrfMosesPunctNormalizer normalize)r|r9rpunct_normalizerrrrmoses_punct_norm  zFSMTTokenizer.moses_punct_normcCs:||jvr|jj|d}||j|<|j|j|ddddS)NrTF)aggressive_dash_splits return_strescape)rhrfMosesTokenizertokenize)r|r9rmoses_tokenizerrrrmoses_tokenize s   zFSMTTokenizer.moses_tokenizecCrr)rirfMosesDetokenizer detokenize)r|tokensrmoses_detokenizerrrrmoses_detokenizerzFSMTTokenizer.moses_detokenizecCs t|}|||}t|}|Sr)r:rrD)r|r9rrrrmoses_pipelines zFSMTTokenizer.moses_pipelinecC t|jSr)rjrqrrrrr  zFSMTTokenizer.src_vocab_sizecCrr)rjrsrrrrtgt_vocab_size$rzFSMTTokenizer.tgt_vocab_sizecCt|jfi|jSr)rurqadded_tokens_encoderrrrrr(zFSMTTokenizer.get_src_vocabcCrr)rursadded_tokens_decoderrrrr get_tgt_vocab+rzFSMTTokenizer.get_tgt_vocabc s~t|dd|ddf}|jvrj|St|}|s#|dS t|fddd}|jvr4ny|\}}g}d}|t|krz|||} Wnty\|||dYn?w|||| | }|||kr|t|dkr||d|kr| |||d 7}n | |||d7}|t|ksBt|}|}t|dkrnt|}q$d |}|d krd }|j|<|S) NrYTcsj|tdS)Ninf)rxgetfloat)pairrrr8sz#FSMTTokenizer.bpe..keyrrrM z z ) rZryrminrxrjindexrmextendr@rA) r|tokenrrbigramfirstsecondnew_wordijrrrbpe.sN     ,     zFSMTTokenizer.bper cCsn|j}|jr |}|r|}n|j||d}|j||d}g}|D]}|r4|t||dq#|S)av Tokenize a string given language code using Moses. Details of tokenization: - [sacremoses](https://github.com/alvations/sacremoses): port of Moses - Install with `pip install sacremoses` Args: - lang: ISO language code (default = 'en') (string). Languages should belong of the model supported languages. However, we don't enforce it. - bypass_tokenizer: Allow users to preprocess and tokenize the sentences externally (default = False) (bool). If True, we only apply BPE. Returns: List of tokens. rr) rkr_lowerr[rrrlistr)r|r9rbypass_tokenizer split_tokensrrrr _tokenizeZs zFSMTTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)rqrr`)r|rrrr_convert_token_to_idsz"FSMTTokenizer._convert_token_to_idcCs|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rsrr`)r|rrrr_convert_id_to_tokensz"FSMTTokenizer._convert_id_to_tokencCs.dd|D}d|}|||j}|S)z:Converts a sequence of tokens (string) in a single string.cSs g|] }|ddddqS)rr<r)r6)rStrrrr]r^z:FSMTTokenizer.convert_tokens_to_string..r<)rAr[rrl)r|rr9rrrconvert_tokens_to_stringsz&FSMTTokenizer.convert_tokens_to_string token_ids_0 token_ids_1cCs(|jg}|dur ||S||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A FAIRSEQ Transformer sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N) sep_token_idr|rrseprrr build_inputs_with_special_tokenssz.FSMTTokenizer.build_inputs_with_special_tokensalready_has_special_tokenscsZ|r tj||ddS|dur#dgt|dgdgt|dgSdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNrr)rzget_special_tokens_maskrj)r|rrrrrrrs(z%FSMTTokenizer.get_special_tokens_maskcCsF|jg}|durt||dgSt||dgt||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. A FAIRSEQ Transformer sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An FAIRSEQ_TRANSFORMER sequence pair mask has the following format: Nrr)rrjrrrr$create_token_type_ids_from_sequencess$z2FSMTTokenizer.create_token_type_ids_from_sequencessave_directoryfilename_prefixc Cstj|std|ddStj||r|dndtd}tj||r,|dndtd}tj||r=|dndtd}t|dd d }|t j |j d d d ddWdn1sewYt|dd d  }dd|j D}|t j |d d d ddWdn1swYd}t|dd d 5} t|j dddD]!\} } || krtd|d| }| d| d|d7}qWdn1swY|||fS)NzVocabulary path (z) should be a directoryr0r<r r r wrNrOrMTF)indent sort_keys ensure_asciirXcSrQrrrRrrrrVrWz1FSMTTokenizer.save_vocabulary..rcSs|dS)Nrr)kvrrrrsz/FSMTTokenizer.save_vocabulary..rzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rr)ospathisdirloggererrorrAVOCAB_FILES_NAMESrnwriterodumpsrqrsrrsortedrxwarning) r|rrr r r frrwriter bpe_tokens token_indexrrrsave_vocabularys@     zFSMTTokenizer.save_vocabularycCs|j}d|d<|S)Nrf)__dict__copy)r|staterrr __getstate__s zFSMTTokenizer.__getstate__cCs4||_zddl}Wn tytdw||_dS)NrrL)rrdrerf)r|drdrrr __setstate__s   zFSMTTokenizer.__setstate__) NNNNFrHrIrJrK)r Fr)NF),__name__ __module__ __qualname____doc__rvocab_files_namesPRETRAINED_VOCAB_FILES_MAPpretrained_vocab_files_mapPRETRAINED_INIT_CONFIGURATIONpretrained_init_configuration&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESmax_model_input_sizesmodel_input_namesr{rstrintrpropertyrrrrrrrrrrrrrrrrrboolrrrrrr __classcell__rrrrrEs-C   ,'      !$rE)rrorr7r=typingrrrrtokenization_utilsrutilsr get_loggerrrrrrrrr:rDrErrrrs>      +