o
h;2 @ s d Z ddlZddlmZ ddlmZmZmZmZm Z ddl
Z ddl m
Z
ddlmZ eeZdZd d
iZd d d iiZd d
iZG dd de
ZdS )zTokenization classes for . N)copyfile)AnyDictListOptionalTuple )PreTrainedTokenizer)loggingu ▁
vocab_filezsentencepiece.bpe.modelzfacebook/xglm-564MzNhttps://huggingface.co/facebook/xglm-564M/resolve/main/sentencepiece.bpe.modeli c
sR e Zd ZdZeZeZeZ ddgZ
d,d e e e
ef d
df fd d
Zd
d Zdd Z d-dee de ee d
ee fddZ d.dee de ee ded
ee f fdd
Z d-dee de ee d
ee fddZedd Zd d Zd e
d
ee
fd d!Zd"d# Z d$d% Z d&d' Z d-d(e
d)e e
d
e e
fd*d+Z Z!S )/
XGLMTokenizera
Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on
[SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_mask X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N) sep_token_idr% rL rM sepr r r build_inputs_with_special_tokens s z.XGLMTokenizer.build_inputs_with_special_tokensFalready_has_special_tokensc s\ |r t j||ddS |du rdgdgt| S dgdgt| ddg dgt| S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rL rM rR Nr" r )r= get_special_tokens_maskr9 )r% rL rM rR r@ r r rS s *z%XGLMTokenizer.get_special_tokens_maskc C s@ | j g}|du rt|| dg S t|| | | | dg S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. XLM-RoBERTa does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )rN r9 rO r r r $create_token_type_ids_from_sequences s z2XGLMTokenizer.create_token_type_ids_from_sequencesc C s t | j| j | j S N)r9 r5 r$ r0 r% r r r
vocab_size s zXGLMTokenizer.vocab_sizec s( fddt jD }| j |S )Nc s i | ]} ||qS r )convert_ids_to_tokensr rV r r r'
r z+XGLMTokenizer.get_vocab..)r1 rW r: added_tokens_encoder)r% vocabr rV r get_vocab s zXGLMTokenizer.get_vocabtextc C s | j j|tdS )N)out_type)r5 encoder7 )r% r\ r r r _tokenize s zXGLMTokenizer._tokenizec C s4 || j v r
| j | S | j|}|r|| j S | jS )z0Converts a token (str) in an id using the vocab.)r8 r5 PieceToIdr$ unk_token_id)r% tokenspm_idr r r _convert_token_to_id s
z"XGLMTokenizer._convert_token_to_idc C s& || j v r
| j | S | j|| j S )z=Converts an index (integer) in a token (str) using the vocab.)r< r5 IdToPiecer$ )r% indexr r r _convert_id_to_token s
z"XGLMTokenizer._convert_id_to_tokenc C s d |td }|S )zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r% tokens
out_stringr r r convert_tokens_to_string s z&XGLMTokenizer.convert_tokens_to_stringsave_directoryfilename_prefixc C s t j|std| d d S t j||r|d ndtd }t j| jt j|kr?t j | jr?t
| j| |fS t j | jsgt |d}| j
}|| W d |fS 1 sbw Y |fS )NzVocabulary path (z) should be a directory-rh r wb)ospathisdirloggererrorrj VOCAB_FILES_NAMESabspathr isfiler openr5 rE write)r% rq rr out_vocab_fileficontent_spiece_modelr r r save_vocabulary% s" (
z XGLMTokenizer.save_vocabulary)r r r r r r NrU )NF)"__name__
__module__ __qualname____doc__rz vocab_files_namesPRETRAINED_VOCAB_FILES_MAPpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESmax_model_input_sizesmodel_input_namesr r r7 r r> rG rK r intrQ boolrS rT propertyrW r[ r_ rd rg rp r r
__classcell__r r r@ r r + sl ? 8
(r )r ru shutilr typingr r r r r
sentencepiecer3 tokenization_utilsr utilsr
get_loggerr rx rl rz r r r r r r r s"