o h@sdZddlmZddlmZmZmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZdd lmZmZmZmZmZdd lmZdd lmZdd lmZmZmZmZd dlm Z e!e"Z#dZ$dZ%dZ&dZ'dZ(dZ)dZ*dZ+dZ,dZ-gdZ.Gddde j/Z0Gdddej1j2Z3Gddde4Z5d d!Z6Gd"d#d#ej1j2Z7Gd$d%d%e j/Z8Gd&d'd'e j/Z9Gd(d)d)e j/Z:Gd*d+d+e j/Z;Gd,d-d-e j/ZGd2d3d3e j/Z?d4d5Z@ejAjBd6d7ZCejAjBd8d9ZDejAjBd:d;ZEGdd?d?e j/ZGGd@dAdAeZHdBZIdCZJedDeIGdEdFdFeHZKedGeIGdHdIdIeHZLGdJdKdKe j/ZMGdLdMdMe j/ZNGdNdOdOe j/ZOedPeIGdQdRdReHZPedSeIGdTdUdUeHZQedVeIGdWdXdXeHZRdS)Yz PyTorch DeBERTa model.)Sequence)OptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)softmax_backward_data)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) DebertaConfigrmicrosoft/deberta-basez!lsanochkin/deberta-large-feedbackz' Paris'z0.54z#Palak/microsoft_deberta-large_squadz' a nice puppet'gQ? )rzmicrosoft/deberta-largezmicrosoft/deberta-xlargezmicrosoft/deberta-base-mnlizmicrosoft/deberta-large-mnlizmicrosoft/deberta-xlarge-mnlics0eZdZfddZddZeddZZS) ContextPoolercs2tt|j|j|_t|j|_||_ dSN) super__init__rLinearpooler_hidden_sizedense StableDropoutpooler_dropoutdropoutconfigselfr& __class__b/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.pyrCs   zContextPooler.__init__cCs8|dddf}||}||}t|jj|}|SNr)r%r"r r&pooler_hidden_act)r( hidden_states context_token pooled_outputr+r+r,forwardIs   zContextPooler.forwardcC|jjSr)r& hidden_sizer(r+r+r, output_dimSszContextPooler.output_dim)__name__ __module__ __qualname__rr2propertyr6 __classcell__r+r+r)r,rBs   rc@s4eZdZdZeddZeddZeddZdS) XSoftmaxa Masked Softmax which is optimized for saving memory Args: input (`torch.tensor`): The input tensor that will apply softmax. mask (`torch.IntTensor`): The mask matrix where 0 indicate that element will be ignored in the softmax calculation. dim (int): The dimension that will apply softmax Example: ```python >>> import torch >>> from transformers.models.deberta.modeling_deberta import XSoftmax >>> # Make a tensor >>> x = torch.randn([4, 20, 100]) >>> # Create a mask >>> mask = (x > 0).int() >>> # Specify the dimension to apply softmax >>> dim = -1 >>> y = XSoftmax.apply(x, mask, dim) ```cCsX||_|tj}||tt|jj}t ||j}| |d| ||Sr-) dimtotorchbool masked_filltensorfinfodtypeminsoftmax masked_fill_save_for_backward)r(inputmaskr=rmaskoutputr+r+r,r2ts  zXSoftmax.forwardcCs$|j\}t||||j|}|ddfSr) saved_tensorsrr=)r( grad_outputrL inputGradr+r+r,backwards zXSoftmax.backwardc Csddlmm}ddlm}m}|jd||jdd}|jd|d|jdtj dtj d d ||jd d}|||||jdt t | jd } ||| |} ||| ||jdtj dtjd d S) Nr)rArFCastLong)to_iSubConstantrrD)value_tBool)torch.onnx.symbolic_helperonnxsymbolic_helpertorch.onnx.symbolic_opset9rArFopcast_pytorch_to_onnxr?rBint64rCtyperDrEr@) gr(rJr=sym_helprArFmask_cast_valuer_maskrLr+r+r,symbolics"( $zXSoftmax.symbolicN)r7r8r9__doc__ staticmethodr2rPrer+r+r+r,r<Xs  r<c@seZdZddZdS)DropoutContextcCsd|_d|_d|_d|_dS)NrrT)r%rJscale reuse_maskr5r+r+r,rs zDropoutContext.__init__N)r7r8r9rr+r+r+r,rhs rhcCst|ts |}d}n|j}||j9}|jr|jnd}|dkr2|dur2dt|d| tj }t|tr?|jdur?||_||fS)Nrr) isinstancerhr%rirjrJr? empty_like bernoulli_r>r@)rI local_contextr%rJr+r+r,get_masks     roc @sZeZdZdZeddZeddZedejj dejj de e e fd ejj fd d Zd S) XDropoutzlOptimized dropout function to save computation and memory by using mask operation instead of multiplication.cCsDt||\}}dd||_|dkr ||||d|jS|S)Ng?rr)rorirHrA)ctxrI local_ctxrJr%r+r+r,r2s  zXDropout.forwardcCs0|jdkr|j\}||d|jdfS|dfS)Nrr)rirMrA)rqrNrJr+r+r,rPs zXDropout.backwardrarIrrreturncCs4ddlm}|}t|tr|j}d}|||||S)Nr)symbolic_opset12T) torch.onnxrtrkrhr%)rarIrrrt dropout_ptrainr+r+r,res  zXDropout.symbolicN)r7r8r9rfrgr2rPr?_CGraphValuerfloatrhrer+r+r+r,rps  4rpcsBeZdZdZfddZddZddZdd d Zd d ZZ S)r#z Optimized dropout module for stabilizing the training Args: drop_prob (float): the dropout probabilities cs t||_d|_d|_dSr-)rr drop_probcount context_stack)r(r|r)r+r,rs  zStableDropout.__init__cCs$|jr|jdkrt||S|S)zr Call the module Args: x (`torch.tensor`): The input tensor to apply dropout r)trainingr|rpapply get_context)r(xr+r+r,r2szStableDropout.forwardcCsd|_d|_dSr-)r}r~r5r+r+r, clear_contexts zStableDropout.clear_contextTrcCs2|jdurg|_d|_|jD]}||_||_qdSr-)r~r}rjri)r(rjricr+r+r, init_contexts  zStableDropout.init_contextcCsT|jdur'|jt|jkr|jt|j|j}|j|_|jd7_|S|jS)Nr)r~r}lenappendrhr|r%)r(rqr+r+r,rs  zStableDropout.get_context)Tr) r7r8r9rfrr2rrrr;r+r+r)r,r#s  r#cs*eZdZdZdfdd ZddZZS)DebertaLayerNormzBLayerNorm module in the TF style (epsilon inside the square root).-q=cs8ttt||_tt||_||_ dSr) rrr Parameterr?onesweightzerosbiasvariance_epsilon)r(sizeepsr)r+r,r s  zDebertaLayerNorm.__init__cCsj|j}|}|jddd}||djddd}||t||j}||}|j||j }|S)NT)keepdim) rDr{meanpowr?sqrtrr>rr)r(r/ input_typervarianceyr+r+r,r2s zDebertaLayerNorm.forward)rr7r8r9rfrr2r;r+r+r)r,r src$eZdZfddZddZZS)DebertaSelfOutputcs<tt|j|j|_t|j|j|_t |j |_ dSr) rrrr r4r"rlayer_norm_eps LayerNormr#hidden_dropout_probr%r'r)r+r,rs zDebertaSelfOutput.__init__cC&||}||}|||}|Srr"r%rr(r/ input_tensorr+r+r,r2%  zDebertaSelfOutput.forwardr7r8r9rr2r;r+r+r)r,rs rcs.eZdZfddZ    dddZZS)DebertaAttentioncs(tt||_t||_||_dSr)rrDisentangledSelfAttentionr(rrLr&r'r)r+r,r-s    zDebertaAttention.__init__FNc CsJ|j||||||d}|r|\}}|dur|}|||} |r#| |fS| S)N) query_states relative_posrel_embeddings)r(rL) r(r/attention_maskoutput_attentionsrrr self_output att_matrixattention_outputr+r+r,r23s   zDebertaAttention.forwardFNNNrr+r+r)r,r,  rcs2eZdZfddZdejdejfddZZS)DebertaIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rrrr r4intermediate_sizer"rk hidden_actstrr intermediate_act_fnr'r)r+r,rRs   zDebertaIntermediate.__init__r/rscC||}||}|Sr)r"rr(r/r+r+r,r2Z  zDebertaIntermediate.forward)r7r8r9rr?Tensorr2r;r+r+r)r,rQs rcr) DebertaOutputcsBtt|j|j|_t|j|j|_ t |j |_ ||_ dSr)rrrr rr4r"rrrr#rr%r&r'r)r+r,ras   zDebertaOutput.__init__cCrrrrr+r+r,r2hrzDebertaOutput.forwardrr+r+r)r,r`s rcs.eZdZfddZ    dddZZS) DebertaLayercs,tt||_t||_t||_dSr)rrr attentionr intermediaterrLr'r)r+r,rps   zDebertaLayer.__init__NFc CsH|j||||||d}|r|\}}||} || |} |r"| |fS| S)Nrrrr)rrrL) r(r/rrrrrrrintermediate_output layer_outputr+r+r,r2vs   zDebertaLayer.forward)NNNFrr+r+r)r,rorrcsNeZdZdZfddZddZddZdd d Z   dd dZZ S)DebertaEncoderz8Modified BertEncoder with relative position bias supportcs~ttfddtjD|_tdd|_|jr:tdd|_ |j dkr/j |_ t |j dj |_ d|_dS) Ncsg|]}tqSr+)r.0_r&r+r, sz+DebertaEncoder.__init__..relative_attentionFmax_relative_positionsrrr)rrr ModuleListrangenum_hidden_layerslayergetattrrrmax_position_embeddings Embeddingr4rgradient_checkpointingr'r)rr,rs    zDebertaEncoder.__init__cCs|jr |jj}|Sd}|Sr)rrr)r(rr+r+r,get_rel_embeddingsz DebertaEncoder.get_rel_embeddingcCsN|dkr|dd}||dd}|S|dkr%|d}|S)Nrrrr )r= unsqueezesqueeze)r(rextended_attention_maskr+r+r,get_attention_masks  z!DebertaEncoder.get_attention_maskNcCsB|jr|dur|dur|dn|d}t||d|j}|S)Nr)rrbuild_relative_positiondevice)r(r/rrqr+r+r, get_rel_posszDebertaEncoder.get_rel_posTFc Cs<||}||||}|rdnd}|rdnd} t|tr"|d} n|} |} t|jD]T\} } |r8||f}|jrK|jrK| | j | |||| |}n | | |||| |d}|r[|\}}|durx|}t|trw| dt |jkru|| dnd} n|} |r| |f} q-|r||f}|st dd||| fDSt ||| dS)Nr+r)rrrrrcss|] }|dur|VqdSrr+)rvr+r+r, sz)DebertaEncoder.forward..last_hidden_stater/ attentions)rrrkrr enumeraterrr_gradient_checkpointing_func__call__rtupler )r(r/routput_hidden_statesrrr return_dictall_hidden_statesall_attentionsnext_kvri layer_moduleatt_mr+r+r,r2s^          "  zDebertaEncoder.forward)NN)TFNNT) r7r8r9rfrrrrr2r;r+r+r)r,rs    rcCsjtj|tj|d}tj|tj|d}|dddf|dd|d}|d|ddf}|d}|S)a Build relative position according to the query and key We assume the absolute position of query \(P_q\) is range from (0, query_size) and the absolute position of key \(P_k\) is range from (0, key_size), The relative positions from query to key is \(R_{q \rightarrow k} = P_q - P_k\) Args: query_size (int): the length of query key_size (int): the length of key Return: `torch.LongTensor`: A tensor with shape [1, query_size, key_size] rDrNrrr)r?arangelongviewrepeatr) query_sizekey_sizerq_idsk_ids rel_pos_idsr+r+r,rs $ rcCs*||d|d|d|dgS)Nrrrrexpandr)c2p_pos query_layerrr+r+r,c2p_dynamic_expand*rcCs*||d|d|d|dgS)Nrrrr)rr key_layerr+r+r,p2c_dynamic_expandrrcCs*||dd|d|dfS)Nrrr) pos_indexp2c_attrr+r+r,pos_dynamic_expandrrcsBeZdZdZfddZddZ    d dd Zd d ZZS) ra Disentangled self-attention module Parameters: config (`str`): A model config class instance with the configuration to build a new model. The schema is similar to *BertConfig*, for more details, please refer [`DebertaConfig`] cst|j|jdkrtd|jd|jd|j|_t|j|j|_|j|j|_tj |j|jddd|_ t t j |jt jd|_t t j |jt jd|_|jdur]|jng|_t|d d|_t|d d|_|jrtj |j|jdd|_tj |j|jdd|_|jrt|d d |_|jd kr|j|_t|j|_d|jvrtj |j|jdd|_d|jvrt |j|j|_t|j|_dS)NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r FrrVr talking_headrrrc2pp2c) rrr4num_attention_heads ValueErrorintattention_head_size all_head_sizerr in_projrr?rr{q_biasv_bias pos_att_typerrrhead_logits_projhead_weights_projrrr#r pos_dropoutpos_proj pos_q_projattention_probs_dropout_probr%r'r)r+r,r)s:      z"DisentangledSelfAttention.__init__cCs4|dd|jdf}||}|ddddS)Nrrrrr )rrrpermute)r(r new_x_shaper+r+r,transpose_for_scoresLs z.DisentangledSelfAttention.transpose_for_scoresFNcsR|dur}|jddd\}} } nRddjjjjdddfdd tdDdgddd|jdjd } fd d td dD\} } fd d | | | fD\}} } |jddddf}| j ddddf} d}d t j }t t j|dt jd |}||j|jd }t || dd}jrĈ|}|| |||}|dur||}jr|dddd ddd d}t||d}|}jr|dddd ddd d}t || }|ddd d}|ddd}||}|r'||fS|S)a Call the module Args: hidden_states (`torch.FloatTensor`): Input states to the module usually the output from previous layer, it will be the Q,K and V in *Attention(Q,K,V)* attention_mask (`torch.BoolTensor`): An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j* th token. output_attentions (`bool`, optional): Whether return the attention matrix. query_states (`torch.FloatTensor`, optional): The *Q* state in *Attention(Q,K,V)*. relative_pos (`torch.LongTensor`): The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with values ranging in [*-max_relative_positions*, *max_relative_positions*]. rel_embeddings (`torch.FloatTensor`): The embedding of relative distances. It's a tensor of shape [\(2 \times \text{max_relative_positions}\), *hidden_size*]. Nr rr=cSs0|durt|||St||Sr)r?matmult)wbrr+r+r,linear|sz1DisentangledSelfAttention.forward..linearrcs0g|]tjfddtjDddqS)csg|] }|dqS)r r+rr)kwsr+r,rsz@DisentangledSelfAttention.forward...rr)r?catrr)r)r(r")r!r,rs0z5DisentangledSelfAttention.forward..rVcs.g|]}||j|jdqS)rV)r>rDr )r/rqkvbqkvwr+r,rs.rcsg|]}|qSr+)r)rrr5r+r,rsrrr)r rchunkrrrr>rDrrrrr?rrBrr{r transposerrdisentangled_att_biasrrrr<rr%r contiguousr)r(r/rrrrrqprr value_layerrr!rrel_att scale_factorriattention_scoresattention_probs context_layernew_context_layer_shaper+)r/rr$r%r(r"r,r2QsF&  ""  " "  z!DisentangledSelfAttention.forwardcCsl|dur|d}t||d|j}|dkr"|dd}n|dkr.|d}n|dkr=td|tt|d|d|j}| |j}||j||j|ddfd}d}d|j vr| |} | | } t|| d d} t||d|dd} tj| d t| ||d } || 7}d |j vr4||} | | } | ttj| d tjd |} |d|dkrt|d|d|j} n|} t| |d|dd}t|| d dj |jd }tj|d t|||d d d}|d|dkr0|dddddddfd }tj|dt|||d }||7}|S) Nrrrr rz2Relative position ids must be of dim 2 or 3 or 4. rr)r=indexrrV)rrrr=rr rEmaxrrr>rrrr?rr(clampgatherrrrrBr{rDrr)r(rrrrr.ratt_spanscore pos_key_layerc2p_attrpos_query_layerr_posp2c_posrrr+r+r,r)sV           $"z/DisentangledSelfAttention.disentangled_att_biasr) r7r8r9rfrrr2r)r;r+r+r)r,rs #  [rcs*eZdZdZfddZdddZZS)DebertaEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cstt|dd}t|d|j|_tj|j|j|d|_t|dd|_ |j s,d|_ n t|j |j|_ |j dkrCt|j |j|_ |j|jkrTtj|j|jdd|_t|j|j|_t|j|_||_|jd t|j d dd dS) N pad_token_idrembedding_size) padding_idxposition_biased_inputTFr position_ids)rr) persistent)rrrr4rArr vocab_sizeword_embeddingsrCposition_embeddingsrtype_vocab_sizetoken_type_embeddingsr embed_projrrrr#rr%r&register_bufferr?rr)r(r&r@r)r+r,rs$      zDebertaEmbeddings.__init__Nc CsN|dur |}n|dd}|d}|dur$|jddd|f}|dur3tj|tj|jjd}|dur<||}|jdurI||}nt|}|} |j rW| |7} |j j dkrf| |} | | 7} |j |j jkrr|| } || } |dur|| kr|dkr|dd}|d}|| j}| |} || } | S)Nrrrrr3r)rrDr?rrrrGrH zeros_likerCr&rIrJrAr4rKrr=rrr>rDr%) r( input_idstoken_type_idsrDrJ inputs_embeds input_shape seq_lengthrH embeddingsrJr+r+r,r2s>            zDebertaEmbeddings.forward)NNNNNrr+r+r)r,r?s r?c@s*eZdZdZeZdZdgZdZddZ dS)DebertaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. debertarHTcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rA|jjjd|jjd|j durC|jj|j  dSdSdS)zInitialize the weights.g)rstdN) rkrr rdatanormal_r&initializer_rangerzero_rrB)r(moduler+r+r, _init_weights7s    z$DebertaPreTrainedModel._init_weightsN) r7r8r9rfr config_classbase_model_prefix"_keys_to_ignore_on_load_unexpectedsupports_gradient_checkpointingr\r+r+r+r,rT,s rTa The DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen. It's build on top of BERT/RoBERTa with two improvements, i.e. disentangled attention and enhanced mask decoder. With those two improvements, it out perform BERT/RoBERTa on a majority of tasks with 80GB pretraining data. This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`DebertaConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zaThe bare DeBERTa Model transformer outputting raw hidden-states without any specific head on top.cseZdZfddZddZddZddZee d e e e e d  dd eejd eejdeejdeejdeejdeedeedeedeee ffddZZS) DebertaModelcs8t|t||_t||_d|_||_|dSr-) rrr?rSrencoderz_stepsr& post_initr'r)r+r,rs    zDebertaModel.__init__cCr3rrSrGr5r+r+r,get_input_embeddingssz!DebertaModel.get_input_embeddingscCs ||j_dSrrer(new_embeddingsr+r+r,set_input_embeddingss z!DebertaModel.set_input_embeddingscCstd)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel z7The prune function is not implemented in DeBERTa model.)NotImplementedError)r(heads_to_pruner+r+r, _prune_headsszDebertaModel._prune_headsbatch_size, sequence_length checkpoint output_typer]NrNrrOrDrPrrrrsc  s|dur|njj}|dur|njj}|dur|njj}|dur*|dur*td|dur9|||} n|durF|dd} ntd|durQ|jn|j} |dur_tj | | d}|durltj | tj | d}j |||||d} j | |d||d} | d } jd kr| d }fd d tjD}| d}j }j |}j | }|d dD]}|||d |||d}| |q| d}|s|f| |rd nddSt||r| jnd| jdS)NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embeds)rr)rNrOrDrJrPT)rrrrrcsg|]}jjdqSr&)rbrrr5r+r,rsz(DebertaModel.forward..Frrr)r&rruse_return_dictr %warn_if_padding_and_no_attention_maskrrr?rrrrSrbrcrrrrrr r/r)r(rNrrOrDrPrrrrQrembedding_outputencoder_outputsencoded_layersr/layersrrrel_posrsequence_outputr+r5r,r2sr        zDebertaModel.forward)NNNNNNNN)r7r8r9rrfrirlrDEBERTA_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrr?rr@rrr2r;r+r+r)r,rasL     raz5DeBERTa Model with a `language modeling` head on top.cseZdZddgZfddZddZddZee d e e e e d eed  dd eejdeejdeejdeejdeejdeejdeedeedeedeee ffddZZS)DebertaForMaskedLMzcls.predictions.decoder.weightzcls.predictions.decoder.biascs,t|t||_t||_|dSr)rrrarUDebertaOnlyMLMHeadclsrdr'r)r+r,rs    zDebertaForMaskedLM.__init__cCs |jjjSrr predictionsdecoderr5r+r+r,get_output_embeddings z(DebertaForMaskedLM.get_output_embeddingscCs||jj_dSrrrgr+r+r,set_output_embeddingssz(DebertaForMaskedLM.set_output_embeddingsrmz[MASK])rorpr]rJexpected_output expected_lossNrNrrOrDrPlabelsrrrrsc  Cs| dur| n|jj} |j|||||||| d} | d} || } d} |dur7t}|| d|jj|d} | sM| f| dd}| durK| f|S|St| | | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` NrrOrDrPrrrrrrlosslogitsr/r) r&rqrUrrrrFr r/r)r(rNrrOrDrPrrrroutputsrxprediction_scoresmasked_lm_lossloss_fctrLr+r+r,r2s4 zDebertaForMaskedLM.forward NNNNNNNNN)r7r8r9_tied_weights_keysrrrrryrzr_CHECKPOINT_FOR_MASKED_LMr r|_MASKED_LM_EXPECTED_OUTPUT_MASKED_LM_EXPECTED_LOSSrr?rr@rrr2r;r+r+r)r,r}sX       r}cr)DebertaPredictionHeadTransformcsftt|d|j|_t|j|j|_t|j t r#t |j |_ n|j |_ tj |j|jd|_ dS)NrA)r)rrrr4rArr r"rkrrr transform_act_fnrrr'r)r+r,rFs  z'DebertaPredictionHeadTransform.__init__cCs"||}||}||}|Sr)r"rrrr+r+r,r2Qs   z&DebertaPredictionHeadTransform.forwardrr+r+r)r,rEs  rcr)DebertaLMPredictionHeadcs\tt||_t|d|j|_tj|j|j dd|_ t t |j |_|j|j _dS)NrAFr)rrr transformrr4rArr rFrrr?rrr'r)r+r,rYs  z DebertaLMPredictionHead.__init__cCrr)rrrr+r+r,r2grzDebertaLMPredictionHead.forwardrr+r+r)r,rXs rcr)r~cstt||_dSr)rrrrr'r)r+r,ros zDebertaOnlyMLMHead.__init__cCs||}|Sr)r)r(rxrr+r+r,r2ss zDebertaOnlyMLMHead.forwardrr+r+r)r,r~ns r~z DeBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZfddZddZddZeede e e e d dd e ejd e ejd e ejd e ejde ejde ejde ede ede edeee ffddZZS) DebertaForSequenceClassificationcs~t|t|dd}||_t||_t||_|jj}t |||_ t|dd}|dur2|j j n|}t||_|dS)N num_labelsr cls_dropout)rrrrrarUrpoolerr6rr classifierr&rr#r%rd)r(r&rr6drop_outr)r+r,rs       z)DebertaForSequenceClassification.__init__cCs |jSr)rUrfr5r+r+r,rfrz5DebertaForSequenceClassification.get_input_embeddingscCs|j|dSr)rUrirgr+r+r,risz5DebertaForSequenceClassification.set_input_embeddingsrmrnNrNrrOrDrPrrrrrsc  Cs:| dur| n|jj} |j|||||||| d} | d} || } || } || } d}|dur|jjdur|jdkrQt }| d |j } || | d}n| dks^|ddkr|dk}|}|ddkrt| d||d| d}t|d| d}t}|| d|j| d}n^td | }nUtd}|| |d }nC|jjdkrt }|jdkr|| |}n+|| |}n%|jjdkrt}|| d|j| d}n|jjdkrt}|| |}| s| f| dd}|dur|f|S|St|| | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rOrrDrPrrrrrr regressionsingle_label_classificationmulti_label_classificationr)r&rqrUrr%r problem_typerrr rr>rDr=rnonzerorr?r7rrr{rB LogSoftmaxsumrrrrr/r)r(rNrrOrDrPrrrrr encoder_layerr1rrloss_fn label_indexlabeled_logitsr log_softmaxrLr+r+r,r2sh               z(DebertaForSequenceClassification.forwardr)r7r8r9rrfrirryrzrr{rr|rr?rr@rrr2r;r+r+r)r,rxsP     rz DeBERTa Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeedeee e d         dde e j de e j de e j d e e j d e e j d e e j d e ed e ede edeee ffddZZS)DebertaForTokenClassificationcsJt||j|_t||_t|j|_t |j |j|_ | dSr) rrrrarUrDropoutrr%r r4rrdr'r)r+r,rs   z&DebertaForTokenClassification.__init__rmrnNrNrrOrDrPrrrrrsc  Cs| dur| n|jj} |j|||||||| d} | d} || } || } d} |dur;t}|| d|j|d} | sQ| f| dd}| durO| f|S|St| | | j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrrr) r&rqrUr%rrrrrr/r)r(rNrrOrDrPrrrrrrxrrrrLr+r+r,r2s0   z%DebertaForTokenClassification.forwardr)r7r8r9rrryrzrr{rr|rr?rr@rrr2r;r+r+r)r,rsL     rz DeBERTa Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZfddZeedeee e e e e ed          ddeejdeejdeejd eejd eejd eejd eejd eedeedeedeee ffddZZS)DebertaForQuestionAnsweringcs<t||j|_t||_t|j|j|_| dSr) rrrrarUrr r4 qa_outputsrdr'r)r+r,r?s   z$DebertaForQuestionAnswering.__init__rm)rorpr]rrqa_target_start_indexqa_target_end_indexNrNrrOrDrPstart_positions end_positionsrrrrsc  CsF| dur| n|jj} |j||||||| | d} | d} || } | jddd\}}|d}|d}d}|dur|durt|dkrN|d}t|dkr[|d}|d}| d|}| d|}t |d}|||}|||}||d}| s||f| dd}|dur|f|S|St |||| j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrrr) ignore_indexr)r start_logits end_logitsr/r)r&rqrUrsplitrr*rrr6rrr/r)r(rNrrOrDrPrrrrrrrxrrr total_loss ignored_indexr start_lossend_lossrLr+r+r,r2IsN!          z#DebertaForQuestionAnswering.forward) NNNNNNNNNN)r7r8r9rrryrzr_CHECKPOINT_FOR_QArr|_QA_EXPECTED_OUTPUT_QA_EXPECTED_LOSS_QA_TARGET_START_INDEX_QA_TARGET_END_INDEXrr?rr@rrr2r;r+r+r)r,r7sZ       r)Srfcollections.abcrtypingrrrr?torch.utils.checkpointrtorch.nnrrr activationsr modeling_outputsr r rrrmodeling_utilsr pytorch_utilsrutilsrrrrconfiguration_debertar get_loggerr7loggerr|r{rrrrrrrr%DEBERTA_PRETRAINED_MODEL_ARCHIVE_LISTModulerautogradFunctionr<objectrhrorpr#rrrrrrrrjitscriptrrrrr?rTDEBERTA_START_DOCSTRINGryrar}rrr~rrrr+r+r+r,s        ?'1%"e   CL,oO oB