o hi @svdZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z m Z mZddlmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(e&)e*Z+dZ,dZ-dZ.dgZ/ddZ0da1e%re$re+2dze0Wne3yZ4ze+5de4WYdZ4[4ndZ4[4ww ddZ6d\ddZ7d\ddZ8d\ddZ9d d!Z:Gd"d#d#ej;j<Z=Gd$d%d%ej;j<Z>Gd&d'd'Z?d]d(d)Z@d*d+ZA   d^d,d-ZBGd.d/d/e jCZDGd0d1d1e jCZEGd2d3d3e jCZFGd4d5d5e jCZGGd6d7d7e jCZHGd8d9d9e jCZIGd:d;d;e jCZJGdd?d?e jCZLGd@dAdAe jCZMGdBdCdCe jCZNGdDdEdEeZOdFZPdGZQe"dHePGdIdJdJeOZRe"dKePGdLdMdMeOZSGdNdOdOe jCZTe"dPePGdQdRdReOZUe"dSePGdTdUdUeOZVe"dVePGdWdXdXeOZWe"dYePGdZd[d[eOZXdS)_z PyTorch MRA model.N)Path)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)load)ACT2FN)"BaseModelOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_ninja_availableis_torch_cuda_availablelogging) MraConfigzuw-madison/mra-base-512-4r AutoTokenizercsLttjjjddfdd}|gd}td|ddad dladS) Nkernelsmracsfdd|DS)Ncsg|]}|qSr").0file src_folderr"Z/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/mra/modeling_mra.py Cz:load_cuda_kernels..append_root..r")filesr%r"r' append_rootBsz&load_cuda_kernels..append_root)zcuda_kernel.cuzcuda_launch.cuztorch_extension.cpp cuda_kernelT)verboser)r__file__resolveparentr r,)r+ src_filesr"r%r'load_cuda_kernels>s    r2zLoading custom CUDA kernels...zFailed to load CUDA kernels. Mra requires custom CUDA kernels. Please verify that compatible versions of PyTorch and CUDA Toolkit are installed: cCst|dkr tdt|dkrtd|ddkr#td|ddkr.td|jd d jd d }|}|}|}t ||||\}}|d d d d d d d d d f}||fS) z8 Computes maximum values for softmax stability. z.sparse_qk_prod must be a 4-dimensional tensor.'indices must be a 2-dimensional tensor. z>The size of the second dimension of sparse_qk_prod must be 32.r z=The size of the third dimension of sparse_qk_prod must be 32.dimN) lensize ValueErrormaxvalues transpose contiguousintr, index_max)sparse_qk_prodindicesquery_num_block key_num_block index_valsmax_valsmax_vals_scatterr"r"r' sparse_max]s$rKr6cCst|dkr tdt|dkrtd|jd|jdkr&td|j\}}||}tj|dtj|jd}||||}||dddf||ddf}|S)zN Converts attention mask to a sparse mask for high resolution logits. r4z$mask must be a 2-dimensional tensor.r5rzBmask and indices must have the same size in the zero-th dimension.dtypedeviceN) r;r<r=shapetorcharangelongrNreshape)maskrE block_size batch_sizeseq_len num_block batch_idxr"r"r' sparse_maskys &rZc Cs"|\}}}|\}}}||dkrtd||dkr"td||||||dd}||||||dd}t|dkrJtdt|dkrVtdt|d krbtd |d d krmtd |d d krxtd|}|}|}|}t|||S)z7 Performs Sampled Dense Matrix Multiplication. rzTquery_size (size of first dimension of dense_query) must be divisible by block_size.Pkey_size (size of first dimension of dense_key) must be divisible by block_size.r:r7r3z+dense_query must be a 4-dimensional tensor.)dense_key must be a 4-dimensional tensor.r4r5r r6z.The third dimension of dense_query must be 32.z,The third dimension of dense_key must be 32.) r<r=rSr@r;rArBr, mm_to_sparse) dense_query dense_keyrErUrV query_sizer9_key_sizer"r"r'r]s.  r]c Cs|\}}}||dkrtd|d|krtd|d|kr'td||||||dd}t|d krAtd t|d krMtd t|dkrYtd |dd krdtd|}|}|}|}t||||}|dd||||}|S)zP Performs matrix multiplication of a sparse matrix with a dense matrix. rr[r4zQThe size of the second dimension of sparse_query must be equal to the block_size.r zPThe size of the third dimension of sparse_query must be equal to the block_size.r:r7r3,sparse_query must be a 4-dimensional tensor.r\r5r6z8The size of the third dimension of dense_key must be 32.) r<r=rSr@r;rArBr,sparse_dense_mm) sparse_queryrEr_rFrUrVrbr9 dense_qk_prodr"r"r'rds. rdcCs |||tj||ddS)Nfloor rounding_mode)rPdivrR)rE dim_1_block dim_2_blockr"r"r'transpose_indicess rmc@s2eZdZeddZeddZed ddZdS) MraSampledDenseMatMulcC&t||||}||||||_|SN)r]save_for_backwardrU)ctxr^r_rErUrDr"r"r'forwardzMraSampledDenseMatMul.forwardc Csj|j\}}}|j}|d|}|d|}t|||}t|dd|||} t||||} | | ddfSNrr:r7) saved_tensorsrUr<rmrdr@) rrgradr^r_rErUrFrG indices_Tgrad_key grad_queryr"r"r'backwards   zMraSampledDenseMatMul.backwardr6cCt||||Srp)rnapply)r^r_rErUr"r"r' operator_callz#MraSampledDenseMatMul.operator_callNr6__name__ __module__ __qualname__ staticmethodrsr{r~r"r"r"r'rns   rnc@s0eZdZeddZeddZeddZdS)MraSparseDenseMatMulcCrorp)rdrqrF)rrrerEr_rFrDr"r"r'rsrtzMraSparseDenseMatMul.forwardc Cs`|j\}}}|j}|d|d}t|||}t|dd|||}t|||} | d|dfSru)rvrFr<rmrdr@r]) rrrwrerEr_rFrGrxryrzr"r"r'r{s    zMraSparseDenseMatMul.backwardcCr|rp)rr})rerEr_rFr"r"r'r~rz"MraSparseDenseMatMul.operator_callNrr"r"r"r'rs   rc@seZdZeddZdS) MraReduceSumc Cs|\}}}}t|dkrtdt|dkr td|\}}}}|\}}|jdd|||}tj|dtj|jd}tj ||dd |dddf|||} tj |||f|j |jd} | d| ||||} | |||} | S) Nr3rcr4r5r8rrLrgrh) r<r;r=sumrSrPrQrRrNrjzerosrM index_add) rerErFrGrVrXrUrarY global_idxestempoutputr"r"r'r~s$ & zMraReduceSum.operator_callN)rrrrr~r"r"r"r'rsrcCs|\}}}||}d} |durl||||jdd} |||||jdd| dddddfd} |||||jdd| dddddfd} |durk|||||jdd| dddddfd} n5|tj||tj|jd} |||||jdd} |||||jdd} |dur|||||jdd} t| | ddt |} | j dddj }|dur| d | dddddf| dddddfd k} | | || fS) z/ Compute low resolution approximation. Nr:r8r7ư>rLT)r9keepdims@g?)r<rSrrPonesfloatrNmeanmatmulr@mathsqrtr>r?)querykeyrUrTvaluerVrWhead_dimnum_block_per_row value_hat token_count query_hatkey_hatlow_resolution_logitlow_resolution_logit_row_maxr"r"r'get_low_resolution_logit0s6: rcCsT|j\}}}|dkr3|d}tj|||jd} tjtj| | d|d} || dddddfd}|dkrk|ddd|ddfd|ddd|ddf<|ddddd|fd|ddddd|f<tj||d|ddd d } | j} |d kr| j j dd j } || ddddfk }| |fS|d krd}| |fSt |d)zZ Compute the indices of the subset of components to be used in the approximation. rr4rN)diagonalNg@r:TF)r9largestsortedfullr8sparsez# is not a valid approx_model value.) rOrPrrNtriltriutopkrSrEr?minrr=)r num_blocks approx_modeinitial_prior_first_n_blocksinitial_prior_diagonal_n_blocksrVtotal_blocks_per_rowraoffset temp_mask diagonal_mask top_k_valsrE thresholdhigh_resolution_maskr"r"r'get_block_idxesXs. rc $Cstdur t|S|\} } } } | | } | |dkr!td| |}|| | | }|| | | }|| | | }|dure||dddddf}||dddddf}||dddddf}|dkrvt|||||\}}}}n(|dkrtt||||\}}}}Wdn1swYnt dt||}t |||||\}}Wdn1swYt j ||||dt | }t||||\}}||}|dur|dd t||dddddddf}t|}t ||||}t ||||}|dkrt||d||dddddf}t||dddddddfd d |d | | | }|jd d dddddfd d || | }|d d || | |} |durq| |} t| | dk}!||!dddddf}||!}t| | dk}"||"dddddf}||"}|||dddddf|dddddfd }#n|dkr||dddddfd }#nt d |dur|#|dddddf}#|#| | | | }#|#S)z0 Use Mra to approximate self-attention. Nrz4sequence length must be divisible by the block_size.rrz&approx_mode must be "full" or "sparse")rUrrr:r8rz-config.approx_mode must be "full" or "sparse")r,rP zeros_likerequires_grad_r<r=rSrno_grad Exceptionrrnr~rrrKrZexprrrrepeatrr)$rrrrTrrrUrrrVnum_headrWr meta_batchrrrrrralow_resolution_logit_normalizedrErhigh_resolution_logitrIrJhigh_resolution_attnhigh_resolution_attn_outhigh_resolution_normalizerlow_resolution_attnlow_resolution_attn_outlow_resolution_normalizerlog_correctionlow_resolution_corrhigh_resolution_corr context_layerr"r"r'mra2_attention~s      .  "  . .   rcs*eZdZdZfddZdddZZS) MraEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|jd|j|_ t|j |j|_ tj |j|j d|_ t|j|_|dt|jddt|dd|_|jdtj|jtj|jjd d d dS) N) padding_idxr4eps position_ids)rr:position_embedding_typeabsolutetoken_type_idsrLF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferrPrQexpandgetattrrrrr<rRrNselfconfig __class__r"r'rs  zMraEmbeddings.__init__Nc Cs|dur |}n|dd}|d}|dur$|jddd|f}|durNt|drC|jddd|f}||d|}|}n tj|tj|jjd}|durW| |}| |} || } |j dkrn| |} | | 7} | | } || } | S)Nr:rrrrLr)r<rhasattrrrrPrrRrNrrrrrr) r input_idsrr inputs_embeds input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr embeddingsrr"r"r'rss,        zMraEmbeddings.forward)NNNNrrr__doc__rrs __classcell__r"r"rr'rs rc0eZdZdfdd ZddZdddZZS) MraSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|dur[|n|j|_|jd|j|_t|jt|jdd|_|j|_|j|_|j|_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r6r4)rrrnum_attention_headsrr=rBattention_head_size all_head_sizerLinearrrrrattention_probs_dropout_probrrr block_per_rowrXrrrrrrrrr"r'r,s*   zMraSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nr:rr4rr )r<rrviewpermute)rlayernew_layer_shaper"r"r'transpose_for_scoresHs z%MraSelfAttention.transpose_for_scoresc Cs||}|||}|||}||}|\}}} } d|d}|d|d||| }d} | | krt||| | | f} t j |t j | |j dgdd}t j |t j | |j dgdd}t j |t j | |j dgdd}t |||||j|j|j|jd} | | kr| ddddddd| f} | ||| | } | d d dd } | dd |jf}| j|} | f}|S) N?rrr6rr:r8)rrrrr4r r7)rr rrr<squeezerrSrBrPcatrrNrrrXrrrrrArr)r hidden_statesattention_maskmixed_query_layer key_layer value_layer query_layerrV num_headsrWr gpu_warp_sizepad_sizernew_context_layer_shapeoutputsr"r"r'rsMs@      zMraSelfAttention.forwardrp)rrrrr rsrr"r"rr'r+src8eZdZfddZdejdejdejfddZZS) MraSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr) rrrrrdenserrrrrrrr"r'r zMraSelfOutput.__init__r  input_tensorreturncC&||}||}|||}|Srprrrrr rr"r"r'rs  zMraSelfOutput.forwardrrrrrPTensorrsrr"r"rr'r $rcr) MraAttentionNcs.tt||d|_t||_t|_dS)N)r)rrrrrrset pruned_headsrrr"r'rs   zMraAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr8)r;rrrrr(rrrrrrrunion)rheadsindexr"r"r' prune_headss zMraAttention.prune_headscCs2|||}||d|}|f|dd}|SNrr)rr)rr r self_outputsattention_outputrr"r"r'rss zMraAttention.forwardrp)rrrrr,rsrr"r"rr'r&sr&c2eZdZfddZdejdejfddZZS)MraIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrp) rrrrrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrrr"r'rs   zMraIntermediate.__init__r rcC||}||}|Srp)rr6rr r"r"r'rs  zMraIntermediate.forwardr#r"r"rr'r1s r1cr) MraOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rrrrr2rrrrrrrrrr"r'rrzMraOutput.__init__r rrcCrrpr r!r"r"r'rsr"zMraOutput.forwardr#r"r"rr'r:r%r:cs.eZdZfddZdddZddZZS) MraLayercsBt|j|_d|_t||_|j|_t||_t ||_ dSNr) rrchunk_size_feed_forward seq_len_dimr& attentionadd_cross_attentionr1 intermediater:rrrr"r'rs   zMraLayer.__init__NcCsB|||}|d}|dd}t|j|j|j|}|f|}|Sr-)r?rfeed_forward_chunkr=r>)rr rself_attention_outputsr/r layer_outputr"r"r'rss   zMraLayer.forwardcCs||}|||}|Srp)rAr)rr/intermediate_outputrDr"r"r'rBs  zMraLayer.feed_forward_chunkrp)rrrrrsrBrr"r"rr'r;s   r;cs.eZdZfddZ    dddZZS) MraEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr")r;)r#rarr"r'r(r)z'MraEncoder.__init__..F) rrrr ModuleListrangenum_hidden_layersrgradient_checkpointingrrrGr'rs   zMraEncoder.__init__NFTc Cs|rdnd}t|jD]#\}}|r||f}|jr%|jr%||j||} n|||} | d}q |r6||f}|sCtdd||fDSt||dS)Nr"rcss|] }|dur|VqdSrpr")r#vr"r"r' sz%MraEncoder.forward..)last_hidden_stater ) enumeraterrKtraining_gradient_checkpointing_func__call__tupler ) rr r head_maskoutput_hidden_states return_dictall_hidden_statesi layer_module layer_outputsr"r"r'rss(      zMraEncoder.forward)NNFTrrrrrsrr"r"rr'rFs  rFcr0)MraPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSr) rrrrrrr3r4r5r transform_act_fnrrrrr"r'rs  z#MraPredictionHeadTransform.__init__r rcCs"||}||}||}|Srp)rr]rr8r"r"r'rs!s   z"MraPredictionHeadTransform.forwardr#r"r"rr'r\s  r\cs$eZdZfddZddZZS)MraLMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dS)NF)bias) rrr\ transformrrrrdecoder ParameterrPrr_rrr"r'r*s  zMraLMPredictionHead.__init__cCr7rp)r`rar8r"r"r'rs7r9zMraLMPredictionHead.forwardr[r"r"rr'r^)s  r^cr0)MraOnlyMLMHeadcstt||_dSrp)rrr^ predictionsrrr"r'r?s zMraOnlyMLMHead.__init__sequence_outputrcCs||}|Srp)rd)rreprediction_scoresr"r"r'rsCs zMraOnlyMLMHead.forwardr#r"r"rr'rc>s rcc@s$eZdZdZeZdZdZddZdS)MraPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r!TcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsg)rstdNr )r3rrweightdatanormal_rinitializer_ranger_zero_rrrfill_)rmoduler"r"r' _init_weightsSs      z MraPreTrainedModel._init_weightsN) rrrrr config_classbase_model_prefixsupports_gradient_checkpointingrpr"r"r"r'rgIs  rgaF This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MraConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. ak Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z]The bare MRA Model transformer outputting raw hidden-states without any specific head on top.cseZdZfddZddZddZddZee d e e e e d  dd eejd eejdeejdeejdeejdeejdeedeedeee ffddZZS)MraModelcs2t|||_t||_t||_|dSrp)rrrrrrFencoder post_initrrr"r'rs    zMraModel.__init__cCs|jjSrprrrr"r"r'get_input_embeddingsszMraModel.get_input_embeddingscCs ||j_dSrprw)rrr"r"r'set_input_embeddingss zMraModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrurr?r,)rheads_to_prunerr*r"r"r' _prune_headsszMraModel._prune_headsbatch_size, sequence_length checkpoint output_typerqNrrrrrTrrUrVrc Cs||dur|n|jj}|dur|n|jj}|dur |dur td|dur/||||} n|dur<|dd} ntd| \} } |durK|jn|j} |dur[tj| | f| d}|durt |j drz|j j ddd| f} | | | }|}n tj | tj| d}||| }|||jj}|j ||||d}|j|||||d}|d }|s|f|d dSt||j|j|jd S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer:z5You have to specify either input_ids or inputs_embedsrrrL)rrrr)rrTrUrVrr)rNr  attentionscross_attentions)rrUuse_return_dictr=%warn_if_padding_and_no_attention_maskr<rNrPrrrrrrrRget_extended_attention_mask get_head_maskrJrur r rr)rrrrrrTrrUrVrrVrrNrrextended_attention_maskembedding_outputencoder_outputsrer"r"r'rssZ     zMraModel.forward)NNNNNNNN)rrrrryrzr}rMRA_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrrPr$boolrrrsrr"r"rr'rtsL     rtz1MRA Model with a `language modeling` head on top.cseZdZddgZfddZddZddZee d e e e e d  dd eejd eejdeejdeejdeejdeejdeejdeedeedeee ffddZZS)MraForMaskedLMzcls.predictions.decoder.weightzcls.predictions.decoder.biascs,t|t||_t||_|dSrp)rrrtr!rcclsrvrrr"r'rs    zMraForMaskedLM.__init__cCs |jjjSrprrdrarxr"r"r'get_output_embeddingss z$MraForMaskedLM.get_output_embeddingscCs||jj_dSrpr)rnew_embeddingsr"r"r'set_output_embeddingssz$MraForMaskedLM.set_output_embeddingsr~rNrrrrrTrlabelsrUrVrc  Cs| dur| n|jj} |j|||||||| d} | d} || } d} |dur7t}|| d|jj|d} | sM| f| dd}| durK| f|S|St| | | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. NrrrrTrrUrVrr:rlosslogitsr r) rrr!rrrrrr r)rrrrrrTrrrUrVrrerfmasked_lm_lossloss_fctrr"r"r'rs!s4 zMraForMaskedLM.forward NNNNNNNNN)rrr_tied_weights_keysrrrrrrrrrrrrPr$rrrrsrr"r"rr'rsR      rcs(eZdZdZfddZddZZS)MraClassificationHeadz-Head for sentence-level classification tasks.csFtt|j|j|_t|j|_t|j|j |_ ||_ dSrp) rrrrrrrrr num_labelsout_projrrrr"r'r^s  zMraClassificationHead.__init__cKsR|dddddf}||}||}t|jj|}||}||}|S)Nr)rrr rr4r)rfeatureskwargsxr"r"r'rsfs    zMraClassificationHead.forwardrr"r"rr'r[s rzMRA Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.ceZdZfddZeedeee e d         dde e j de e j de e j d e e j d e e j d e e j d e e j d e ede edeee ffddZZS)MraForSequenceClassificationcs4t||j|_t||_t||_|dSrp)rrrrtr!r classifierrvrrr"r'rvs    z%MraForSequenceClassification.__init__r~rNrrrrrTrrrUrVrc  Csf| dur| n|jj} |j|||||||| d} | d} || } d} |dur|jjdurP|jdkr6d|j_n|jdkrL|jtjksG|jtj krLd|j_nd|j_|jjdkrnt }|jdkrh|| | } n+|| |} n%|jjdkrt }|| d|j| d} n|jjdkrt}|| |} | s| f| dd}| dur| f|S|St| | | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr regressionsingle_label_classificationmulti_label_classificationr:r)rrr!r problem_typerrMrPrRrBr r rrrrr r)rrrrrrTrrrUrVrrerrrrr"r"r'rssR    "       z$MraForSequenceClassification.forwardr)rrrrrrrrrrrrrPr$rrrrsrr"r"rr'rpsL     rzMRA Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.cr)MraForMultipleChoicecsDt|t||_t|j|j|_t|jd|_| dSr<) rrrtr!rrrpre_classifierrrvrrr"r'rs   zMraForMultipleChoice.__init__z(batch_size, num_choices, sequence_lengthrNrrrrrTrrrUrVrc  Cs| dur| n|jj} |dur|jdn|jd} |dur%|d|dnd}|dur4|d|dnd}|durC|d|dnd}|durR|d|dnd}|dure|d|d|dnd}|j|||||||| d} | d} | dddf} || } t| } | | }|d| }d}|durt }|||}| s|f| dd}|dur|f|S|St ||| j | j dS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr:r7rrr)rrrOrr<r!rrReLUrrrr r)rrrrrrTrrrUrV num_choicesr hidden_state pooled_outputrreshaped_logitsrrrr"r"r'rssN      zMraForMultipleChoice.forwardr)rrrrrrrrrrrrrPr$rrrrsrr"r"rr'rsL     rzMRA Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.cr)MraForTokenClassificationcsJt||j|_t||_t|j|_t |j |j|_ | dSrp) rrrrtr!rrrrrrrrvrrr"r'r(s   z"MraForTokenClassification.__init__r~rNrrrrrTrrrUrVrc  Cs| dur| n|jj} |j|||||||| d} | d} || } || } d} |durdt}|durW|ddk}| d|j}t ||dt |j  |}|||} n || d|j|d} | sz| f| dd}| durx| f|S|St | | | j| jdS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr:rr)rrr!rrrrrrPwheretensor ignore_indextype_asrr r)rrrrrrTrrrUrVrrerrr active_loss active_logits active_labelsrr"r"r'rs3sD   z!MraForTokenClassification.forwardr)rrrrrrrrrrrrrPr$rrrrsrr"r"rr'r"sL     rzMRA Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`).cseZdZfddZeedeee e d          dde e j de e j de e j d e e j d e e j d e e j d e e j d e e j de ede edeee ffddZZS)MraForQuestionAnsweringcsBt|d|_|j|_t||_t|j|j|_| dS)Nr4) rrrrtr!rrr qa_outputsrvrrr"r'r{s   z MraForQuestionAnswering.__init__r~rNrrrrrTrstart_positions end_positionsrUrVrc  Cs>| dur| n|jj} |j||||||| | d} | d} || } | jddd\}}|d}|d}d}|dur}|dur}t|dkrJ|d}t|dkrW|d}|d}|d|}|d|}t |d}|||}|||}||d}| s||f| dd}|dur|f|S|St |||| j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrr:r8)rr4)r start_logits end_logitsr r) rrr!rsplitr r;r<clamprrr r)rrrrrrTrrrrUrVrrerrr total_loss ignored_indexr start_lossend_lossrr"r"r'rssN            zMraForQuestionAnswering.forward) NNNNNNNNNN)rrrrrrrrrrrrrPr$rrrrsrr"r"rr'rusR      rr)NN)r6rr)YrrpathlibrtypingrrrrPtorch.utils.checkpointrtorch.nnrrr torch.utils.cpp_extensionr activationsr modeling_outputsr rrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrrrrconfiguration_mrar get_loggerrloggerrr_TOKENIZER_FOR_DOC!MRA_PRETRAINED_MODEL_ARCHIVE_LISTr2r,inforewarningrKrZr]rdrmautogradFunctionrnrrrrrModulerrrr&r1r:r;rFr\r^rcrgMRA_START_DOCSTRINGrrtrrrrrrr"r"r"r's               (( (- s:V!,  /lLTTN