o h@sdZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZmZdd lmZmZmZmZd dlm Z e!e"Z#dZ$dZ%gdZ&Gddde j'Z(Gddde j'Z)Gddde j'Z*Gddde j'Z+Gddde j'Z,Gddde j'Z-Gddde j'Z.Gd d!d!e j'Z/Gd"d#d#eZ0d$Z1d%Z2ed&e1Gd'd(d(e0Z3Gd)d*d*e j'Z4Gd+d,d,e j'Z5ed-e1Gd.d/d/e0Z6eGd0d1d1eZ7ed2e1Gd3d4d4e0Z8dS)5z PyTorch Splinter model.N) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN))BaseModelOutputWithPastAndCrossAttentions ModelOutputQuestionAnsweringModelOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)SplinterConfigtau/splinter-baser)rztau/splinter-base-qassztau/splinter-largeztau/splinter-large-qassc sjeZdZdZfddZ     ddeejdeejdeejd eejd ee d e f d d Z Z S)SplinterEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jdddt|dd|_dS) N) padding_idxeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandgetattrr selfconfig __class__d/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/splinter/modeling_splinter.pyr#4s zSplinterEmbeddings.__init__Nr input_idstoken_type_idsr inputs_embedspast_key_values_lengthreturnc Cs|dur |}n|dd}|d}|dur&|jdd|||f}|dur5tj|tj|jjd}|dur>||}||}||} |jdkrU| |} | | 7} | | } | | } | S)Nrrdtypedevicer!) sizerr3zeroslongrEr(r,r r*r-r1) r8r>r?rr@rA input_shape seq_lengthr, embeddingsr*r<r<r=forwardEs$       zSplinterEmbeddings.forward)NNNNr) __name__ __module__ __qualname____doc__r#rr3 LongTensor FloatTensorintrrL __classcell__r<r<r:r=r1s* rcseZdZdfdd ZdejdejfddZ      dd ejd eejd eejd eejd eejdee e ejdee de ejfddZ Z S)SplinterSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|p\t|dd|_|jdksh|jd kry|j|_t d |jd |j|_|j|_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r r! relative_keyrelative_key_queryr)r"r#r&num_attention_headshasattr ValueErrorrSattention_head_size all_head_sizerLinearquerykeyvaluer/attention_probs_dropout_probr1r6r r)r$distance_embedding is_decoderr8r9r r:r<r=r#is*   zSplinterSelfAttention.__init__xrBcCs6|dd|j|jf}||}|ddddS)NrrrZrr )rFr[r^viewpermute)r8rh new_x_shaper<r<r=transpose_for_scoress z*SplinterSelfAttention.transpose_for_scoresF hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionscCs||}|du} | r|dur|d} |d} |}nP| r/|||} |||} |}n;|durZ|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} |du} |jrz| | f}t| | dd}|j dks|j dkr | j d| j d}}| rtj |dtj |j d dd}ntj|tj |j d dd}tj|tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}||}n|j dkr td | |}td | |}|||}|t|j}|dur||}tjj|dd}||}|dur0||}t|| }|dddd }|dd|jf}||}|rX||fn|f}|jrd||f}|S)NrrrZdimrrXrYrC)rDzbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rarlrbrcr3catrfmatmul transposer shapetensorrHrErir4rer)torDeinsummathsqrtr^r functionalsoftmaxr1rj contiguousrFr_)r8rmrnrorprqrrrsmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputsr<r<r=rLsn             zSplinterSelfAttention.forwardNNNNNNF) rMrNrOr#r3TensorrlrrRrboolrLrTr<r<r:r=rUhs4 rUc8eZdZfddZdejdejdejfddZZS)SplinterSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr) r"r#rr`r&denser-r.r/r0r1r7r:r<r=r# zSplinterSelfOutput.__init__rm input_tensorrBcC&||}||}|||}|Srrr1r-r8rmrr<r<r=rL  zSplinterSelfOutput.forwardrMrNrOr#r3rrLrTr<r<r:r=r $rcseZdZdfdd ZddZ      ddejdeejd eejd eejd eejd ee e ejd ee de ejfddZ Z S)SplinterAttentionNcs.tt||d|_t||_t|_dS)Nr )r"r#rUr8routputset pruned_headsrgr:r<r=r#s   zSplinterAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrt)lenrr8r[r^rrrarbrcrrr_union)r8headsindexr<r<r= prune_headss zSplinterAttention.prune_headsFrmrnrorprqrrrsrBc Cs<||||||||}||d|} | f|dd} | S)Nrr)r8r) r8rmrnrorprqrrrs self_outputsattention_outputrr<r<r=rLs  zSplinterAttention.forwardrr) rMrNrOr#rr3rrrRrrrLrTr<r<r:r=rs4 rcs2eZdZfddZdejdejfddZZS)SplinterIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r"r#rr`r&intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr7r:r<r=r#1s   zSplinterIntermediate.__init__rmrBcCs||}||}|Sr)rr)r8rmr<r<r=rL9s  zSplinterIntermediate.forwardrr<r<r:r=r0s rcr)SplinterOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r"r#rr`rr&rr-r.r/r0r1r7r:r<r=r#ArzSplinterOutput.__init__rmrrBcCrrrrr<r<r=rLGrzSplinterOutput.forwardrr<r<r:r=r@rrcseZdZfddZ      ddejdeejdeejdeejd eejd eeeejd ee d eejfd dZ ddZ Z S) SplinterLayercsrt|j|_d|_t||_|j|_|j|_|jr-|js&t|dt|dd|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedr!r)r"r#chunk_size_feed_forward seq_len_dimr attentionrfadd_cross_attentionr]crossattentionr intermediaterrr7r:r<r=r#Ps   zSplinterLayer.__init__NFrmrnrorprqrrrsrBc Cs|dur |ddnd}|j|||||d} | d} |jr(| dd} | d} n| dd} d} |jro|durot|dsDtd|d|durN|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) NrZ)rsrrrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`rv) rrfr\r]rrfeed_forward_chunkrr)r8rmrnrorprqrrrsself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputr<r<r=rL^sP         zSplinterLayer.forwardcCs||}|||}|Sr)rr)r8rintermediate_outputrr<r<r=rs  z SplinterLayer.feed_forward_chunkr) rMrNrOr#r3rrrRrrrLrrTr<r<r:r=rOs4  ArcseZdZfddZ         ddejdeejdeejd eejd eejd eeeejd ee d ee dee dee de eeje ffddZ Z S)SplinterEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr<)r).0_r9r<r= sz,SplinterEncoder.__init__..F) r"r#r9r ModuleListrangenum_hidden_layerslayergradient_checkpointingr7r:rr=r#s   zSplinterEncoder.__init__NFTrmrnrorprqpast_key_valuesrrsoutput_hidden_states return_dictrBc  Cs^| rdnd} |r dnd} |r|jjrdnd} |jr%|jr%|r%tdd}|r)dnd}t|jD]^\}}| r;| |f} |durC||nd}|durM||nd}|jrc|jrc||j |||||||}n ||||||||}|d}|rz||df7}|r| |df} |jjr| |df} q0| r| |f} | st dd ||| | | fDSt ||| | | d S) Nr<zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FrrrrZcss|] }|dur|VqdSrr<)rvr<r<r= sz*SplinterEncoder.forward..last_hidden_staterrm attentionscross_attentions) r9rrtraininglogger warning_once enumerater_gradient_checkpointing_func__call__tupler )r8rmrnrorprqrrrsrrall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskrr layer_outputsr<r<r=rLsz         zSplinterEncoder.forward) NNNNNNFFT)rMrNrOr#r3rrrRrrrr rLrTr<r<r:r=rsD      rc@s$eZdZdZeZdZdZddZdS)SplinterPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. splinterTcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsg)meanstdNg?)rrr`weightdatanormal_r9initializer_rangebiaszero_r$rr-fill_)r8moduler<r<r= _init_weightss      z%SplinterPreTrainedModel._init_weightsN) rMrNrOrPr config_classbase_model_prefixsupports_gradient_checkpointingrr<r<r<r=rs  raK This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SplinterConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a/ Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `{0}`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `{0}`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `{0}`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare Splinter Model transformer outputting raw hidden-states without any specific head on top.c!seZdZdZfddZddZddZdd Zee d e e e ed  dd eejdeejdeejdeejdeejdeejdeejdeejdeeejdeedeedeedeedeee ffddZZS) SplinterModela* The model is an encoder (with only self-attention) following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. cs2t|||_t||_t||_|dSr)r"r#r9rrKrencoder post_initr7r:r<r=r#gs    zSplinterModel.__init__cCs|jjSrrKr()r8r<r<r=get_input_embeddingsqsz"SplinterModel.get_input_embeddingscCs ||j_dSrr)r8rcr<r<r=set_input_embeddingsts z"SplinterModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r8heads_to_prunerrr<r<r= _prune_headswszSplinterModel._prune_headsbatch_size, sequence_length checkpoint output_typerNr>rnr?rror@rprqrrrsrrrBc Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jjr-| dur(| n|jj} nd} |dur;|dur;td|durJ||||}n|durW|dd}ntd|\}}|durf|j n|j }| durv| ddj dnd}|durt j |||f|d}|durt j |t j|d }|||}|jjr|dur|\}}}||f}|durt j ||d}||}nd}|||jj}|j|||||d }|j|||||| | | | | d }|d}| s|f|d dSt||j|j|j|jd S)a  encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrrZ)rErC)r>rr?r@rA) rnrorprqrrrsrrrr)r9rsruse_return_dictrfrr]%warn_if_padding_and_no_attention_maskrFrErzr3onesrGrHget_extended_attention_maskinvert_attention_mask get_head_maskrrKrr rrmrr)r8r>rnr?rror@rprqrrrsrrrI batch_sizerJrErAextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputssequence_outputr<r<r=rLsx)     zSplinterModel.forward) NNNNNNNNNNNNN)rMrNrOrPr#rrrrSPLINTER_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrr3rrrRrrrrLrTr<r<r:r=r\sl         rcs4eZdZdfdd ZdejdejfddZZS) SplinterFullyConnectedLayergelucsDt||_||_t|j|j|_t||_t |j|_ dSr) r"r# input_dim output_dimrr`rr act_fnr-)r8rrrr:r<r=r#s  z$SplinterFullyConnectedLayer.__init__inputsrBcCs"||}||}||}|Sr)rr r-)r8r!rmr<r<r=rL s   z#SplinterFullyConnectedLayer.forward)rrr<r<r:r=rs rcs(eZdZdZfddZddZZS)QuestionAwareSpanSelectionHeadzf Implementation of Question-Aware Span Selection (QASS) head, described in Splinter's paper: csztt|j|j|_t|j|j|_t|j|j|_t|j|j|_tj |j|jdd|_ tj |j|jdd|_ dS)NF)r) r"r#rr&query_start_transformquery_end_transformstart_transform end_transformrr`start_classifierend_classifierr7r:r<r=r#s z'QuestionAwareSpanSelectionHead.__init__cCs|\}}}|ddd|}tj|d|d}||}||}||} ||} | |} | ddd} t | | } | |} | ddd} t | | } | | fS)Nrr)rurrrZ) rF unsqueezerepeatr3gatherr#r$r%r&r'rjrxr()r8r! positionsrrur gathered_repsquery_start_repsquery_end_reps start_repsend_repsrm start_logits end_logitsr<r<r=rL"s        z&QuestionAwareSpanSelectionHead.forward)rMrNrOrPr#rLrTr<r<r:r=r"s  r"z Splinter Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZfddZeedeee e d            dde e j de e j de e j d e e j d e e j d e e j d e e jd e e jde ede ede ede e jdeee ffddZZS)SplinterForQuestionAnsweringc4t|t||_t||_|j|_|dSrr"r#rrr" splinter_qassquestion_token_idrr7r:r<r=r#?    z%SplinterForQuestionAnswering.__init__rrNr>rnr?rror@start_positions end_positionsrsrrquestion_positionsrBc  Cs| dur| n|jj} d} | dur9|dur#tjt||jdd}ntj|dtj |j |j d}| d} d} |j ||||||| | | d }|d}||| \}}| r`|d |d }}|dur~|d |t|jj}|d |t|jj}d}|dur|durt|d kr|d}t|d kr|d}|d }|d||d|t|d }|||}|||}||d }| s||f|d d}|dur|f|S|St||||j|jd S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. NFrrtr)rDlayoutrETrnr?rror@rsrrr ignore_indexrZlossr2r3rmr)r9r r3argmaxeqr8rSrGrFrHr=rEr)rr7squeezefinforDminrclamp_rr rmr)r8r>rnr?rror@r:r;rsrrr<question_positions_were_none"question_position_for_each_examplerrr2r3 total_loss ignored_indexloss_fct start_lossend_lossrr<r<r=rLIsj$           z$SplinterForQuestionAnswering.forward NNNNNNNNNNNN)rMrNrOr#rrrrrr rrr3rrQrrrrLrTr<r<r:r=r47s^        r4c@sleZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)SplinterForPreTrainingOutputa Class for outputs of Splinter as a span selection model. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when start and end positions are provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-end scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. NrBr2r3rmr)rMrNrOrPrBrr3rR__annotations__r2r3rmrrr<r<r<r=rQs rQz Splinter Model for the recurring span selection task as done during the pretraining. The difference to the QA task is that we do not have a question, but multiple question tokens that replace the occurrences of recurring spans instead. cseZdZfddZeed            ddeej deej deej deej d eej d eej d eej d eej d ee dee dee deej de e effddZdej dej fddZZS)SplinterForPreTrainingcr5rr6r7r:r<r=r#r9zSplinterForPreTraining.__init__z*batch_size, num_questions, sequence_lengthNr>rnr?rror@r:r;rsrrr<rBc  Cs| dur| n|jj} | dur|dur|durtd| dur&|dur&td| dur/||} |j||||||| | | d } | d}|\}}}||| \}}| d}|dur}|d|||}|d|t |j j }|d|t |j j }d}|dur|dur| dtd|d| dtd|dt|jjd}||||||||}||||||||}||d}| s||f| dd}|dur|f|S|St|||| j| jd S) a start_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. NzCquestion_positions must be specified in order to calculate the lossz>question_positions must be specified when input_embeds is usedr>rrr?rZrA)r9r  TypeError_prepare_question_positionsrrFr7r)r5r3rFrDrGrHmaxrr'rirQrmr)r8r>rnr?rror@r:r;rsrrr<rrrsequence_lengthrur2r3 num_questions attention_mask_for_each_questionrKrMrNrOrr<r<r=rLsh!       zSplinterForPreTraining.forwardcCslt||jjk\}}t|}tj|d|f|jjtj |j d}t dd|D}||||f<|S)NrrCcSsg|]}t|qSr<)r3r4)rnr<r<r=rSszFSplinterForPreTraining._prepare_question_positions..) r3wherer9r8bincountfullrFrVr'rHrErw)r8r>rowsflat_positionsrXr,colsr<r<r=rUJs  z2SplinterForPreTraining._prepare_question_positionsrP)rMrNrOr#rrrrr3rrQrrrrQrLrUrTr<r<r:r=rSsZ         drS)9rPr~ dataclassesrtypingrrrrr3torch.utils.checkpointrtorch.nnr activationsr modeling_outputsr r r modeling_utilsr pytorch_utilsrrrutilsrrrrconfiguration_splinterr get_loggerrMrrr&SPLINTER_PRETRAINED_MODEL_ARCHIVE_LISTModulerrUrrrrrrrSPLINTER_START_DOCSTRINGrrrr"r4rQrSr<r<r<r=sd        72W] 2 &r