o h<@sdZddlZddlZddlZddlZddlmZmZddlm Z m Z m Z m Z ddl mZeeZddd Zd d id d id Zd d iZddZGdddeZddZddZGddde ZdS)z$Tokenization classes for OpenAI GPT.N)OptionalTuple)PreTrainedTokenizer _is_control_is_punctuation_is_whitespace)loggingz vocab.jsonz merges.txt) vocab_file merges_filez openai-gptz9https://huggingface.co/openai-gpt/resolve/main/vocab.jsonz9https://huggingface.co/openai-gpt/resolve/main/merges.txticCs|}|sgS|}|S)z@Runs basic whitespace cleaning and splitting on a piece of text.)stripsplit)texttokensrd/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/openai/tokenization_openai.pywhitespace_tokenize.s rc@sXeZdZdZ     dddZdddZdd Zdd d Zd d ZddZ ddZ dS)BasicTokenizera Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). do_split_on_punc (`bool`, *optional*, defaults to `True`): In some instances we want to skip the basic punctuation splitting so that later tokenization can capture the full context of the words, such as contractions. TNcCs2|durg}||_t||_||_||_||_dSN) do_lower_caseset never_splittokenize_chinese_chars strip_accentsdo_split_on_punc)selfrrrrrrrr__init__Os  zBasicTokenizer.__init__cCs|r |jt|n|j}||}|jr||}td|}t|}g}|D])}||vrH|j r@| }|j dur?| |}n|j rH| |}| |||q(td|}|S)aj Basic Tokenization of a piece of text. For sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. NFCF )runionr _clean_textr_tokenize_chinese_chars unicodedata normalizerrlowerr_run_strip_accentsextend_run_split_on_puncjoin)rrrunicode_normalized_text orig_tokens split_tokenstoken output_tokensrrrtokenize_s&      zBasicTokenizer.tokenizecCsBtd|}g}|D]}t|}|dkrq ||q d|S)z$Strips accents from a piece of text.NFDMn)r"r#categoryappendr()rroutputcharcatrrrr%s    z!BasicTokenizer._run_strip_accentscCs|jr |dur||vr|gSt|}d}d}g}|t|krI||}t|r/||gd}n|r6|gd}|d||d7}|t|ksdd|DS) z&Splits punctuation on a piece of text.NrTFcSsg|]}d|qS)r1)r().0xrrr z5BasicTokenizer._run_split_on_punc..)rlistlenrr3)rrrcharsistart_new_wordr4r5rrrr's$     z!BasicTokenizer._run_split_on_punccCsTg}|D] }t|}||r|d|||dq||qd|S)z)Adds whitespace around any CJK character.rr1)ord_is_chinese_charr3r(rrr4r5cprrrr!s      z&BasicTokenizer._tokenize_chinese_charscCs|dkr|dks@|dkr|dks@|dkr|dks@|dkr |dks@|d kr(|d ks@|d kr0|d ks@|d kr8|dks@|dkrB|dkrBdSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr)rrErrrrCs zBasicTokenizer._is_chinese_charcCsXg}|D]"}t|}|dks|dkst|rqt|r!|dq||qd|S)zBPerforms invalid character removal and whitespace cleanup on text.rirr1)rBrrr3r(rDrrrr s   zBasicTokenizer._clean_text)TNTNTr) __name__ __module__ __qualname____doc__rr.r%r'r!rCr rrrrr8s &  rcCs6t}|d}|ddD] }|||f|}q |S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rr8N)radd)wordpairs prev_charr5rrr get_pairss rNcCsn|dd}|dd}|dd}|dd}|dd}td d |}td d |}td d|}|S)zm fixes some issues the spacy tokenizer had on books corpus also does some whitespace standardization u—-u–u―u…z...´'zD(-+|~+|!+|"+|;+|\?+|\++|,+|\)+|\(+|\\+|\/+|\*+|\[+|\]+|}+|{+|\|+|_+)z \1 z\s*\n\s*z z[^\S\n]+r)replaceresubr )rrrrtext_standardizes     rUcseZdZdZeZeZeZ ddgZ dfdd Z e ddZ e d d Zd d Zd dZddZddZddZddZddedeedeefddZZS)OpenAIGPTTokenizera( Construct a GPT Tokenizer. Based on Byte-Pair-Encoding with the following peculiarities: - lowercases all inputs, - uses `SpaCy` tokenizer and `ftfy` for pre-BPE tokenization if they are installed, fallback to BERT's `BasicTokenizer` if not. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. input_idsattention_maskc  s4zddl}ddlm}|}|j|_|j|_Wnty.tdt dd|_d|_Ynwt |dd}t ||_ Wdn1sFwYdd |j D|_t |dd} | d d d } Wdn1srwYd d| D} tt| tt| |_i|_tjdd|i|dS)Nr)EnglishzQftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.T)rutf-8encodingcSsi|]\}}||qSrr)r9kvrrr r<z/OpenAIGPTTokenizer.__init__.. r8r7cSsg|]}t|qSr)tupler )r9mergerrrr;!sz/OpenAIGPTTokenizer.__init__.. unk_tokenr)ftfy spacy.lang.enrZ tokenizernlpfix_text ImportErrorloggerwarningropenjsonloadencoderitemsdecoderreadr dictzipranger> bpe_rankscachesuperr) rr r rdkwargsrerZ_nlp vocab_handle merges_handlemerges __class__rrrs,      zOpenAIGPTTokenizer.__init__cCsdS)NTrrrrrr'sz OpenAIGPTTokenizer.do_lower_casecCs t|jSr)r>rprrrr vocab_size+s zOpenAIGPTTokenizer.vocab_sizecCst|jfi|jSr)rtrpadded_tokens_encoderrrrr get_vocab/szOpenAIGPTTokenizer.get_vocabc s~t|dd|ddf}|jvrj|St|}|s#|dS t|fddd}|jvr4ny|\}}g}d}|t|krz|||} Wnty\|||dYn?w|||| | }|||kr|t|dkr||d|kr| |||d 7}n | |||d7}|t|ksBt|}|}t|dkrnt|}q$d |}|d krd }|j|<|S) Nr7Tcsj|tdS)Ninf)rwgetfloat)pairrrr<sz(OpenAIGPTTokenizer.bpe..keyrr8rz z ) rbrxrNminrwr>index ValueErrorr&r3r() rr,rKrLbigramfirstsecondnew_wordr@jrrrbpe2sN     ,     zOpenAIGPTTokenizer.bpecCsg}|jdur!|j|}|D]}|t||dq|S|t||}|D]}|t||j dq-|S)zTokenize a string.Nr) rirhr.r&r=rr rUrr$)rrr+r,rrr _tokenize^s  "zOpenAIGPTTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)rprrd)rr,rrr_convert_token_to_idmsz'OpenAIGPTTokenizer._convert_token_to_idcCs|j||jS)z0Converts an id in a token (BPE) using the vocab.)rrrrd)rrrrr_convert_id_to_tokenqsz'OpenAIGPTTokenizer._convert_id_to_tokencCsd|dd}|S)z:Converts a sequence of tokens (string) in a single string.r1rr)r(rRr )rr out_stringrrrconvert_tokens_to_stringusz+OpenAIGPTTokenizer.convert_tokens_to_stringNsave_directoryfilename_prefixreturnc CsVtj|std|ddStj||r|dndtd}tj||r,|dndtd}t|ddd }|t j |j d d d d dWdn1sTwYd}t|ddd =}|dt |j dddD]!\}} || krtd|d| }|d|d|d7}qsWd||fS1swY||fS)NzVocabulary path (z) should be a directoryrOr1r r wr[r\rTF)indent sort_keys ensure_asciirarz#version: 0.2 cSs|dS)Nr8r)kvrrrrsz4OpenAIGPTTokenizer.save_vocabulary..rzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rr8)ospathisdirrkerrorr(VOCAB_FILES_NAMESrmwriterndumpsrpsortedrwrqrl) rrrr merge_filefrwriter bpe_tokens token_indexrrrsave_vocabularyzs8        z"OpenAIGPTTokenizer.save_vocabulary)rYr)rFrGrHrIrvocab_files_namesPRETRAINED_VOCAB_FILES_MAPpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESmax_model_input_sizesmodel_input_namesrpropertyrrrrrrrrstrrrr __classcell__rrrrrVs$  ,(rV)rIrnrrSr"typingrrtokenization_utilsrrrrutilsr get_loggerrFrkrrrrobjectrrNrUrVrrrrs.   "