o
h5 @ s d Z ddlZddlZddlmZ ddlmZmZmZ ddl m
Z
ddl m Z e
eZdd d
Zd d d
ddd
d
Zddd
Zdd ZG dd de
ZdS )z! Tokenization classes for PhoBERT N)copyfile)ListOptionalTuple )PreTrainedTokenizer)loggingz vocab.txtz bpe.codes)
vocab_file merges_filez@https://huggingface.co/vinai/phobert-base/resolve/main/vocab.txtzAhttps://huggingface.co/vinai/phobert-large/resolve/main/vocab.txt)zvinai/phobert-basezvinai/phobert-largez@https://huggingface.co/vinai/phobert-base/resolve/main/bpe.codeszAhttps://huggingface.co/vinai/phobert-large/resolve/main/bpe.codes c C s> t }| d }| dd D ] }|||f |}q
t |}|S )z
Return set of symbol pairs in a word.
Word is represented as tuple of symbols (symbols being variable-length strings).
r N)setadd)wordpairs prev_charchar r f/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/phobert/tokenization_phobert.py get_pairs4 s r c
s& e Zd ZdZeZeZeZ d) fdd Z
d*d
e e d e
e e d e e fd
dZ d+d
e e d e
e e ded e e f fdd
Z d*d
e e d e
e e d e e fddZedd Zdd Zdd Zdd Zd d Zd d Zd!d" Zd*d#ed$e
e d ee fd%d&Z d'd( Z Z S ),PhobertTokenizeraO
Construct a PhoBERT tokenizer. Based on Byte-Pair-Encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
bos_token (`st`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N) cls_token_id sep_token_id)r@ rF rG clssepr r r build_inputs_with_special_tokens s
z1PhobertTokenizer.build_inputs_with_special_tokensFalready_has_special_tokensc sh |r t j||ddS |du r dgdgt| dg S dgdgt| ddg dgt| dg S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rF rG rN Nr r )r> get_special_tokens_maskr; )r@ rF rG rN rD r r rO s 0z(PhobertTokenizer.get_special_tokens_maskc C sP | j g}| jg}|du rt|| | dg S t|| | | | | dg S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. PhoBERT does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )rJ rI r; )r@ rF rG rL rK r r r $create_token_type_ids_from_sequences s
"z5PhobertTokenizer.create_token_type_ids_from_sequencesc C s
t | jS N)r; r1 r@ r r r
vocab_size s
zPhobertTokenizer.vocab_sizec C s t | jfi | jS rQ )r8 r1 added_tokens_encoderrR r r r get_vocab s zPhobertTokenizer.get_vocabc
s | j v r
j | S t|}tt|d d |d d g }t|}|s'|S t| fddd}| jvr8ny|\}}g }d}|t|k rz|||} W n ty` | ||d Y n?w | ||| | }|| |kr|t|d k r||d |kr|
|| |d 7 }n |
|| |d7 }|t|k sFt|}|}t|dkrnt|}q(d
|}|d d }| j |< |S ) Nr% zTc s j | tdS )Ninf)r< getfloat)pairrR r r s z&PhobertTokenizer.bpe..)keyr r r @@ ) r= r& listr minr< r; index
ValueErrorextendappendjoin)
r@ tokenr r bigramfirstsecondnew_wordijr rR r bpe sN
"
,
zPhobertTokenizer.bpec C s8 g }t d|}|D ]}|t| |d q
|S )zTokenize a string.z\S+\n? )refindallrb r^ rl r' )r@ text split_tokenswordsre r r r _tokenize& s
zPhobertTokenizer._tokenizec C s | j || j | jS )z0Converts a token (str) in an id using the vocab.)r1 rW r, )r@ re r r r _convert_token_to_id0 s z%PhobertTokenizer._convert_token_to_idc C s | j || jS )z=Converts an index (integer) in a token (str) using the vocab.)r5 rW r, )r@ r` r r r _convert_id_to_token4 s z%PhobertTokenizer._convert_id_to_tokenc C s d |dd }|S )z:Converts a sequence of tokens (string) in a single string.rm r\ )rd replacestrip)r@ tokens
out_stringr r r convert_tokens_to_string8 s z)PhobertTokenizer.convert_tokens_to_stringsave_directoryfilename_prefixc C s t j|std| d d S t j||r|d ndtd }t j||r,|d ndtd }t j| jt j|krNt j | jrNt
| j| n&t j | jstt |d}| j
}|| W d n1 sow Y t j| jt j|krt
| j| ||fS )NzVocabulary path (z) should be a directory-rv r r
wb)ospathisdirloggererrorrd VOCAB_FILES_NAMESabspathr isfiler r6 sp_modelserialized_model_protowriter
)r@ r| r} out_vocab_fileout_merge_fileficontent_spiece_modelr r r save_vocabulary= s& (
z PhobertTokenizer.save_vocabularyc
C s t |trCz!t|ddd}| | W d W dS 1 s w Y W dS ty4 } z|d}~w tyB td| dw | }|D ]!}| }| d}|dkr\t
d |d| }t | j | j |< qIdS )
zi
Loads a pre-existing dictionary from a text file and adds its symbols to this instance.
rr! r" Nz Incorrect encoding detected in z , please rebuild the datasetrm r% z5Incorrect dictionary format, expected ' ')
isinstancer2 r6 r3 FileNotFoundError UnicodeError Exception readlinesrx rfindra r; r1 ) r@ ffdfnfelineslineTmplineidxr r r r r3 Z s.
z PhobertTokenizer.add_from_file)r r r r r r r rQ )NF) __name__
__module__ __qualname____doc__r vocab_files_namesPRETRAINED_VOCAB_FILES_MAPpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESmax_model_input_sizesr? r intr rM boolrO rP propertyrS rU rl rs rt ru r{ r2 r r r3
__classcell__r r rD r r D sb 0-
,
r )r r rn shutilr typingr r r tokenization_utilsr utilsr
get_loggerr r r r r r r r r r r s.