o
h+ @ sL d Z ddlZddlmZ ddlmZ eeZdddZ G d d
d
eZ
dS ) z XLNet configuration N )PretrainedConfig)loggingz@https://huggingface.co/xlnet-base-cased/resolve/main/config.jsonzAhttps://huggingface.co/xlnet-large-cased/resolve/main/config.json)zxlnet-base-casedzxlnet-large-casedc s e Zd ZdZdZdgZdddddZ
d" fd d Zed d Z e j
d!d Z Z S )# XLNetConfiga
This is the configuration class to store the configuration of a [`XLNetModel`] or a [`TFXLNetModel`]. It is used to
instantiate a XLNet model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the
[xlnet-large-cased](https://huggingface.co/xlnet-large-cased) architecture.
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
documentation from [`PretrainedConfig`] for more information.
Args:
vocab_size (`int`, *optional*, defaults to 32000):
Vocabulary size of the XLNet model. Defines the number of different tokens that can be represented by the
`inputs_ids` passed when calling [`XLNetModel`] or [`TFXLNetModel`].
d_model (`int`, *optional*, defaults to 1024):
Dimensionality of the encoder layers and the pooler layer.
n_layer (`int`, *optional*, defaults to 24):
Number of hidden layers in the Transformer encoder.
n_head (`int`, *optional*, defaults to 16):
Number of attention heads for each attention layer in the Transformer encoder.
d_inner (`int`, *optional*, defaults to 4096):
Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
ff_activation (`str` or `Callable`, *optional*, defaults to `"gelu"`):
The non-linear activation function (function or string) in the If string, `"gelu"`, `"relu"`, `"silu"` and
`"gelu_new"` are supported.
untie_r (`bool`, *optional*, defaults to `True`):
Whether or not to untie relative position biases
attn_type (`str`, *optional*, defaults to `"bi"`):
The attention type used by the model. Set `"bi"` for XLNet, `"uni"` for Transformer-XL.
initializer_range (`float`, *optional*, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
layer_norm_eps (`float`, *optional*, defaults to 1e-12):
The epsilon used by the layer normalization layers.
dropout (`float`, *optional*, defaults to 0.1):
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
mem_len (`int` or `None`, *optional*):
The number of tokens to cache. The key/value pairs that have already been pre-computed in a previous
forward pass won't be re-computed. See the
[quickstart](https://huggingface.co/transformers/quickstart.html#using-the-past) for more information.
reuse_len (`int`, *optional*):
The number of tokens in the current batch to be cached and reused in the future.
bi_data (`bool`, *optional*, defaults to `False`):
Whether or not to use bidirectional input pipeline. Usually set to `True` during pretraining and `False`
during finetuning.
clamp_len (`int`, *optional*, defaults to -1):
Clamp all relative distances larger than clamp_len. Setting this attribute to -1 means no clamping.
same_length (`bool`, *optional*, defaults to `False`):
Whether or not to use the same attention length for each token.
summary_type (`str`, *optional*, defaults to "last"):
Argument used when doing sequence summary. Used in the sequence classification and multiple choice models.
Has to be one of the following options:
- `"last"`: Take the last token hidden state (like XLNet).
- `"first"`: Take the first token hidden state (like BERT).
- `"mean"`: Take the mean of all tokens hidden states.
- `"cls_index"`: Supply a Tensor of classification token position (like GPT/GPT-2).
- `"attn"`: Not implemented now, use multi-head attention.
summary_use_proj (`bool`, *optional*, defaults to `True`):
Argument used when doing sequence summary. Used in the sequence classification and multiple choice models.
Whether or not to add a projection after the vector extraction.
summary_activation (`str`, *optional*):
Argument used when doing sequence summary. Used in the sequence classification and multiple choice models.
Pass `"tanh"` for a tanh activation to the output, any other value will result in no activation.
summary_proj_to_labels (`boo`, *optional*, defaults to `True`):
Used in the sequence classification and multiple choice models.
Whether the projection outputs should have `config.num_labels` or `config.hidden_size` classes.
summary_last_dropout (`float`, *optional*, defaults to 0.1):
Used in the sequence classification and multiple choice models.
The dropout ratio to be used after the projection and activation.
start_n_top (`int`, *optional*, defaults to 5):
Used in the SQuAD evaluation script.
end_n_top (`int`, *optional*, defaults to 5):
Used in the SQuAD evaluation script.
use_mems_eval (`bool`, *optional*, defaults to `True`):
Whether or not the model should make use of the recurrent memory mechanism in evaluation mode.
use_mems_train (`bool`, *optional*, defaults to `False`):
Whether or not the model should make use of the recurrent memory mechanism in train mode.
For pretraining, it is recommended to set `use_mems_train` to `True`. For fine-tuning, it is recommended to
set `use_mems_train` to `False` as discussed
[here](https://github.com/zihangdai/xlnet/issues/41#issuecomment-505102587). If `use_mems_train` is set to
`True`, one has to make sure that the train batches are correctly pre-processed, *e.g.* `batch_1 = [[This
line is], [This is the]]` and `batch_2 = [[ the first line], [ second line]]` and that all batches are of
equal size.
Examples:
```python
>>> from transformers import XLNetConfig, XLNetModel
>>> # Initializing a XLNet configuration
>>> configuration = XLNetConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = XLNetModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
```xlnetmems
vocab_sized_modeln_headn_layer)n_token hidden_sizenum_attention_headsnum_hidden_layers } geluTbi{Gz?-q=皙? NFlasttanh c s< || _ || _|| _|| _|| dkr td|| dd| v r7| d || kr7td| d d|| d|| | _|| _|| _|| _|| _ | | _
|
| _ | | _ | | _
|
| _|| _|| _|| _|| _|| _|| _|| _|| _|| _|| _|| _|| _d| v rt d t | d }|| _ || _ t j!d |||d
| d S )
zConstructs XLNetConfig.r z'd_model % n_head' (z) should be equal to 0d_headz
`d_head` (z*) should be equal to `d_model // n_head` () use_cachezlThe `use_cache` argument is deprecated and will be removed in a future version, use `use_mems_eval` instead.) pad_token_id bos_token_id eos_token_idN )"r r r r
ValueErrorr!
ff_activationd_inneruntie_r attn_typeinitializer_rangelayer_norm_epsdropoutmem_len reuse_lenbi_data clamp_len same_length summary_typesummary_use_projsummary_activationsummary_last_dropout start_n_top end_n_topr% r$ r& warningswarn
FutureWarning
use_mems_evaluse_mems_trainsuper__init__) selfr r r r
r* r) r+ r, r- r. r/ r0 r1 r> r? r2 r3 r4 r5 r6 r7 r8 r9 r: r$ r% r& kwargs __class__r' c/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/xlnet/configuration_xlnet.pyrA sT
zXLNetConfig.__init__c C s t d| j d dS )N
The model < is one of the few models that has no sequence length limit.r )loggerinfo
model_type)rB r' r' rF max_position_embeddings s z#XLNetConfig.max_position_embeddingsc C s t d| j d)NrG rH )NotImplementedErrorrK )rB valuer' r' rF rL s )r r r r r r Tr r r r r NTFFr Fr Tr r r r r r r ) __name__
__module__ __qualname____doc__rK keys_to_ignore_at_inference
attribute_maprA propertyrL setter
__classcell__r' r' rD rF r sR l R
r ) rR r; configuration_utilsr utilsr
get_loggerrO rI #XLNET_PRETRAINED_CONFIG_ARCHIVE_MAPr r' r' r' rF s