o h@sHdZddlZddlZddlZddlmZmZmZmZddl Z ddl m m Z ddlZ ddl m Z ddlmZmZmZddlmZddlmZmZdd lmZdd lmZmZmZdd lmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&d dl'm(Z(e#rddl)m*Z*m+Z+ddl,m-Z-m.Z.m/Z/de0e1e*j2vZ3e%4e5Z6dZ7ddZ8Gddde j9Z:Gddde j9Z;ddZde j?d e@d!e j?fd"d#ZAGd$d%d%e j9ZBGd&d'd'eBZCeBeCd(ZDGd)d*d*e j9ZEd+ZFe!d,eFGd-d.d.eZGd/ZHe!d,eFGd0d1d1eGZIGd2d3d3eGZJe!d4eFGd5d6d6eGZKdS)8z PyTorch Mistral model.N)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache)!_prepare_4d_causal_attention_mask)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)PreTrainedModel)add_start_docstrings%add_start_docstrings_to_model_forwardis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10loggingreplace_return_docstrings) MistralConfig)flash_attn_funcflash_attn_varlen_func)index_first_axis pad_input unpad_input window_sizercCsX|jdtjd}tj|dd}|}ttj |dtjjdd}|||fS)NdimdtypeF)as_tupler)rr) sumtorchint32nonzeroflattenmaxitemFpadcumsum)attention_maskseqlens_in_batchindicesmax_seqlen_in_batch cu_seqlensr5b/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py_get_unpad_data=s r7cs&eZdZdfdd ZddZZS)MistralRMSNormư>cs&ttt||_||_dS)z= MistralRMSNorm is equivalent to T5LayerNorm N)super__init__r Parameterr'onesweightvariance_epsilon)self hidden_sizeeps __class__r5r6r;Ks  zMistralRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)Nr!T)keepdim) r$tor'float32powmeanrsqrtr?r>)r@ hidden_states input_dtypevariancer5r5r6forwardSs  zMistralRMSNorm.forward)r9__name__ __module__ __qualname__r;rO __classcell__r5r5rCr6r8Jsr8cs0eZdZd fdd ZddZd dd ZZS) MistralRotaryEmbedding'Ncsrt||_||_||_d|jtd|jd||j}|j d|dd|j ||j j t ddS)Ng?rrEinv_freqF persistentseq_lendevicer$)r:r;r#max_position_embeddingsbaser'arangefloatrGregister_buffer_set_cos_sin_cacherXr]get_default_dtype)r@r#r^r_r]rXrCr5r6r;]s * zMistralRotaryEmbedding.__init__cCst||_tj|j||jjd}t||j}tj||fdd}|jd| |dd|jd| |dddS)N)r]r$r!r# cos_cachedFrY sin_cached) max_seq_len_cachedr'r`rXr$outercatrbcosrGsin)r@r\r]r$tfreqsembr5r5r6rcks z)MistralRotaryEmbedding._set_cos_sin_cachecCsN||jkr|j||j|jd|jd|j|jd|jd|j|jdfS)Nr[)r$)rhrcr]r$rfrGrg)r@xr\r5r5r6rOus zMistralRotaryEmbedding.forward)rVrWNN)rQrRrSr;rcrOrTr5r5rCr6rU\s rUcCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr!rEre)shaper'rj)rpx1x2r5r5r6 rotate_halfsrucCsL|||}|||}||t||}||t||}||fS)anApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`): The position indices of the tokens corresponding to the query and key tensors. For example, this can be used to pass offsetted position ids when working with a KV-cache. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezeru)qkrkrl position_ids unsqueeze_dimq_embedk_embedr5r5r6apply_rotary_pos_embs r}cs$eZdZfddZddZZS) MistralMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFbias) r:r;configrAintermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnr@rrCr5r6r;s zMistralMLP.__init__cCs ||||||Srq)rrrr)r@rpr5r5r6rO zMistralMLP.forwardrPr5r5rCr6r~s  r~rLn_repreturncCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rrexpandreshape)rLrbatchnum_key_value_headsslenhead_dimr5r5r6 repeat_kvs 0rcseZdZdZddedeeffdd Zdej ded efd d Z    dd ej deej deej dee de de deej eej eeej ffddZZS)MistralAttentionz Multi-headed attention from 'Attention Is All You Need' paper. Modified to use sliding window attention: Longformer and "Generating Long Sequences with Sparse Transformers". Nr layer_idxcs<t||_||_|durtd|jjd|j|_|j |_ |j|j |_ |j |_ |j |j |_ |j|_|j|_d|_|j|_|j |j |jkrYtd|jd|j dtj|j|j |j dd|_tj|j|j |j dd|_tj|j|j |j dd|_tj|j |j |jdd|_t|j |j|jd |_dS) NzInstantiating z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).Fr)r^r_)r:r;rrlogger warning_oncerDrQrAnum_attention_heads num_headsrrnum_key_value_groupsr^ rope_theta is_causalattention_dropout ValueErrorrrq_projk_projv_projo_projrU rotary_embr@rrrCr5r6r;s>   zMistralAttention.__init__tensorr\bszcCs ||||j|jddS)NrrE)viewrr transpose contiguous)r@rr\rr5r5r6_shaperzMistralAttention._shapeFrLr0rypast_key_valueoutput_attentions use_cachercKsd|vr td|\}} } ||} ||} ||} | || |j|j dd} | || |j |j dd} | || |j |j dd} | j d}|durj|j durat d|jjd||||j 7}|j| |d\}}t| | |||\} } |dur||d }|| | |j |\} } t| |j} t| |j} t| | dd t|j}|||j| |fkrt d ||j| |fd ||dur||d| |fkrt d |d| |fd |||}tjj|dtjd| j}tjj ||j!|j"d}t|| }|||j| |jfkr)t d||j| |jfd || dd#}|$|| |j%}|&|}|sCd}|||fS)N padding_maskrPassing `padding_mask` is deprecated and will be removed in v4.37. Please make sure use `attention_mask` instead.`rrEzFThe cache structure has changed since version v4.36. If you are using zv for auto-regressive decoding with k/v caching, please make sure to initialize the attention class with a layer index.r\rlrkr z$Attention weights should be of size z , but is z!Attention mask should be of size r!r")ptrainingz `attn_output` should be of size )'warningswarnsizerrrrrrrrrrrrrDrQget_usable_lengthrr}updaterrr'matmulmathsqrtr functionalsoftmaxrHrGr$dropoutrrrrrAr)r@rLr0ryrrrkwargsrq_len_ query_states key_states value_states kv_seq_lenrkrl cache_kwargs attn_weights attn_outputr5r5r6rOsj             zMistralAttention.forwardrqNNNFF)rQrRrS__doc__rrintr;r'Tensorr LongTensorr boolrrOrTr5r5rCr6rs0% rc sxeZdZdZfddZ     ddejdeejdeejd ee d e d e f d d Z   dddZ ddZ ZS)MistralFlashAttention2aJ Mistral flash attention module. This module inherits from `MistralAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cs tj|i|t |_dSrq)r:r;r_flash_attn_uses_top_left_mask)r@argsrrCr5r6r;IszMistralFlashAttention2.__init__NFrLr0ryrrrc KsXd|vrtd|d}|\}} } ||} ||} ||} | || |j|j  dd} | || |j |j  dd} | || |j |j  dd} | j d}|dur`|| ||j7}t||dddfd}|j| |d\}}t| | |||\} } tot|jddduo||jjk}tstd |dur.t|jdddur||jjkrd|jj}|d }|d}|dddd|dddf}|dddd|dddf}|j d|jjdkrtd |j ||f}|dur|dd|df}tj|t|ddddfgdd }||d }|| | |j|\} } t| |j } t| |j } |j!s@dn|j"}| j#}|tj$krut%|jdrX|jj&}n|jj'j#}td|d| (|} | (|} | (|} | dd} | dd} | dd} |j)| | | || ||d}|*|| |j+}|,|}|sd}|||fS)NrrrrErr!rsliding_windowzThe current flash attention version does not support sliding window attention, for a more memory efficient implementation make sure to upgrade flash-attn library.rzepast key must have a shape of (`batch_size, num_heads, self.config.sliding_window-1, head_dim`), got rer_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)ruse_sliding_windows)-rrpoprrrrrrrrrrrrrr+r,rr}_flash_supports_window_sizegetattrrrrrrrr'rj ones_likerrrrrr$rHhasattrrr>rG_flash_attention_forwardrrAr)r@rLr0ryrrrrrrrrrrrrotary_seq_lenrkrlrslicing_tokenspast_key past_valuer dropout_raterM target_dtyperrr5r5r6rOQs      "  " $$ (              zMistralFlashAttention2.forwardrc  Cs|js|j} n|jo |dk} |dur]|jd} ||||||\}}}} } } | \}}| \}}|s?t|||||||||| d }nt|||||||||| |jj|jjfd }t|| | |}|S|skt|||||| d}|St|||||| |jj|jjfd}|S)a Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then computes the attention scores and pad the final attention scores. Args: query_states (`torch.Tensor`): Input query states to be passed to Flash Attention API key_states (`torch.Tensor`): Input key states to be passed to Flash Attention API value_states (`torch.Tensor`): Input value states to be passed to Flash Attention API attention_mask (`torch.Tensor`): The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the position of padding tokens and 1 for the position of non-padding tokens. dropout (`int`, *optional*): Attention dropout softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim) use_sliding_windows (`bool`, *optional*): Whether to activate sliding window attention. rNr) cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_k dropout_p softmax_scalecausal)rrrrrrrr )rr)rrr ) rrrr _upad_inputrrrrr)r@rrrr0 query_lengthrrrr batch_size indices_q cu_seq_lens max_seq_lensrrmax_seqlen_in_batch_qmax_seqlen_in_batch_kattn_output_unpadrr5r5r6rst     z/MistralFlashAttention2._flash_attention_forwardcCs&|j\}}}} ||jdkr|jd} |dd| |df}t|\} } } t||||| | }t||||| | }||krUt||||| | }| }| }| }n2|dkrsd}tj|dtj|jd}|dd}|d}n|dd| df}t ||\}}}}|||||| f|| ffS)Nr!rr$r]) rrr7rrr'r`r(r]squeezer)r@ query_layer key_layer value_layerr0rrrrrattention_mask_num_tokens indices_krrrrrr5r5r6r1s<   z"MistralFlashAttention2._upad_inputr)rNF)rQrRrSrr;r'rrrr rrOrrrTr5r5rCr6rAs6    dr)eagerflash_attention_2cseZdZdedeffdd Z     ddejdeejd eej d ee ejd ee d ee d e ej ee ej ej fffddZ ZS)MistralDecoderLayerrrcsVt|j|_t|j|||_t||_t|j|j d|_ t|j|j d|_ dS)NrB) r:r;rAMISTRAL_ATTENTION_CLASSES_attn_implementation self_attnr~mlpr8 rms_norm_epsinput_layernormpost_attention_layernormrrCr5r6r;cs  zMistralDecoderLayer.__init__NFrLr0ryrrrrc Ksd|vr td |}||}|j||||||d\}} } ||}|}||}||}||}|f} |r=| | f7} |rD| | f7} | S)Nrr)rLr0ryrrr)rrrrrr) r@rLr0ryrrrrresidualself_attn_weightspresent_key_valueoutputsr5r5r6rOms4       zMistralDecoderLayer.forwardr)rQrRrSrrr;r'rrrrr FloatTensorrOrTr5r5rCr6rbs,   raL This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MistralConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zUThe bare Mistral Model outputting raw hidden-states without any specific head on top.c@s2eZdZeZdZdZdgZdZdZ dZ ddZ dS)MistralPreTrainedModelmodelTrpast_key_valuescCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rA|jjjd|d|j durC|jj|j  dSdSdS)Nr)rJstd) rinitializer_range isinstancerrr>datanormal_rzero_ Embedding padding_idx)r@modulerr5r5r6 _init_weightss    z$MistralPreTrainedModel._init_weightsN) rQrRrSr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_cache_classrr5r5r5r6r s r aL Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*): Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Two formats are allowed: - a [`~cache_utils.Cache`] instance; - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy cache format. The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned. If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cseZdZdZdeffdd ZddZddZee  dd e j d e e j d e e j d e ee jde e jde ede ede ede edeeeffddZZS) MistralModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MistralDecoderLayer`] Args: config: MistralConfig rcstj|_j|_tjj|j|_t fddt j D|_ j dk|_tjjd|_d|_|dS)Ncsg|]}t|qSr5)r).0rrr5r6 ,sz)MistralModel.__init__..rrF)r:r; pad_token_idr vocab_sizerrrA embed_tokens ModuleListrangenum_hidden_layerslayersr_use_flash_attention_2r8rnormgradient_checkpointing post_initrrCr"r6r;%s   zMistralModel.__init__cC|jSrqr&r@r5r5r6get_input_embeddings5z!MistralModel.get_input_embeddingscC ||_dSrqr0r@valuer5r5r6set_input_embeddings8 z!MistralModel.set_input_embeddingsN input_idsr0ryr inputs_embedsrroutput_hidden_states return_dictrc  Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |dur4|dur4td|dur>|j\} } n|durI|j\} } } ntdd} |rct|t }|r^t |}| | } |dur|durn|j n|j }t j| | | t j|d}|dd| }n|d| }|dur||}|dur|jr|r|dddf| k}|rtd|jr|durd|vr|nd}n t|| | f|| |jjd}|}|jr|jr|rtdd }|rd nd}|rd nd}d}|jD]B}|r||f7}|jr|jr||j||||||}n |||||||d }|d}|r,||r)d nd }|r6||d f7}q||}|rD||f7}d}|rR|rP| n|}| sbt!dd||||fDSt"||||dS)NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timezEYou have to specify either decoder_input_ids or decoder_inputs_embedsrrr!zYou are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input. )rzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr5)r0ryrrrrErcss|] }|dur|VqdSrqr5)r!vr5r5r6 sz'MistralModel.forward..)last_hidden_staterrL attentions)#rrr;ruse_return_dictrrrrr r from_legacy_cacherr]r'r`longrvrr&r+r&r,rrr-rrrr*_gradient_checkpointing_func__call__r,to_legacy_cachetupler)r@r9r0ryrr:rrr;r<r seq_lengthrpast_key_values_lengthuse_legacy_cacher]is_padding_rightrLall_hidden_statesall_self_attnsnext_decoder_cache decoder_layer layer_outputs next_cacher5r5r6rO;s               zMistralModel.forward) NNNNNNNNN)rQrRrSrrr;r2r7rMISTRAL_INPUTS_DOCSTRINGr'rrrrr rrrrrOrTr5r5rCr6r sH    r cseZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e e e eed          d#dejdeejdeejdeeejdeejdeejdeedeedeedeedeeeffddZ d$dd Zed!d"ZZS)%MistralForCausalLMzlm_head.weightcs@t|t||_|j|_tj|j|jdd|_| dSr) r:r;r r r%rrrAlm_headr.rrCr5r6r;s   zMistralForCausalLM.__init__cC|jjSrqr r&r1r5r5r6r2z'MistralForCausalLM.get_input_embeddingscC ||j_dSrqrVr5r5r5r6r7 z'MistralForCausalLM.set_input_embeddingscCr/rqrTr1r5r5r6get_output_embeddingsr3z(MistralForCausalLM.get_output_embeddingscCr4rqrZ)r@new_embeddingsr5r5r6set_output_embeddingsr8z(MistralForCausalLM.set_output_embeddingscCr4rqr )r@decoderr5r5r6 set_decoderr8zMistralForCausalLM.set_decodercCr/rqr^r1r5r5r6 get_decoderr3zMistralForCausalLM.get_decoder) output_typerNr9r0ryrr:labelsrrr;r<rc  Cs&|dur|n|jj}| dur| n|jj} | dur| n|jj} |j|||||||| | d } | d} || } | } d}|durq| dddddf}|dddf}t}| d|jj }| d}| |j }|||}| s| f| dd}|dur|f|S|St || | j| j| jdS)a Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Returns: Example: ```python >>> from transformers import AutoTokenizer, MistralForCausalLM >>> model = MistralForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS) >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER) >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) r9r0ryrr:rrr;r<r.r!rlosslogitsrrLr@)rrr;rAr rTrarrrr%rGr]rrrLr@)r@r9r0ryrr:rcrrr;r<r rLrfre shift_logits shift_labelsloss_fctoutputr5r5r6rOsJ)    zMistralForCausalLM.forwardc Ksz|durnt|tr|}|j}|}n |ddjd}}d}|durA|jd|jdkrA|dd|jd| df}n||jdkrR|dd|df}|durn|durn||jd|krn|dd| df}|dd} |dur| dur|dd} | |dkd|r| dd|jd df} |dur|durd|i} nd|i} | | ||d|d | S) NrrErryr!r:r9r)ryrrr0) rr get_seq_length seen_tokensget_max_lengthrrgetrCr/ masked_fill_r) r@r9rr0r:r cache_length past_lengthmax_cache_lengthry model_inputsr5r5r6prepare_inputs_for_generation=s@  "  z0MistralForCausalLM.prepare_inputs_for_generationcs.d}|D]}|tfdd|Df7}q|S)Nr5c3s$|] }|d|jVqdS)rN) index_selectrGr])r! past_statebeam_idxr5r6r>{s"z4MistralForCausalLM._reorder_cache..)rG)rrxreordered_past layer_pastr5rwr6_reorder_cachevs z!MistralForCausalLM._reorder_cache NNNNNNNNNN)NNN)rQrRrS_tied_weights_keysr;r2r7r[r]r`rarrRrr_CONFIG_FOR_DOCr'rrrrr rrrrOrt staticmethodr{rTr5r5rCr6rSs`        X 9rSa The Mistral Model transformer with a sequence classification head on top (linear layer). [`MistralForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZfddZddZddZee          ddej d e ej d e ej d e e ej d e ej d e ej de ede ede ede edeeeffddZZS) MistralForSequenceClassificationcs@t||j|_t||_tj|j|jdd|_| dSr) r:r; num_labelsr r rrrAscorer.rrCr5r6r;s   z)MistralForSequenceClassification.__init__cCrUrqrVr1r5r5r6r2rWz5MistralForSequenceClassification.get_input_embeddingscCrXrqrVr5r5r5r6r7rYz5MistralForSequenceClassification.set_input_embeddingsNr9r0ryrr:rcrrr;r<rc  Cs| dur| n|jj} |j|||||||| | d } | d} || } |dur+|jd}n|jd}|jjdur>|dkr>td|jjdurGd}n|dur_t||jj dd | j }nd}| tj || j d|f}d}|dur| | j }|jjdur|jdkrd|j_n|jdkr|jtjks|jtj krd |j_nd |j_|jjdkrt}|jdkr|||}n+|||}n%|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)r0ryrr:rrr;r<rrz=Cannot handle batch sizes > 1 if no padding token is defined.r!)r] regressionsingle_label_classificationmulti_label_classificationrd)rrAr rrrr$rr'eqrargmaxrGr]r` problem_typerr$rCr rrrrrrrLr@)r@r9r0ryrr:rcrrr;r<transformer_outputsrLrfrsequence_lengths pooled_logitsrerirjr5r5r6rOsr        "       z(MistralForSequenceClassification.forwardr|)rQrRrSr;r2r7rrRr'rrrrr rrrrrOrTr5r5rCr6rsL       r)r)Lrinspectrrtypingrrrrr'torch.nn.functionalrrr-torch.utils.checkpointtorch.nnrrr activationsr cache_utilsr r modeling_attn_mask_utilsrmodeling_outputsrrrmodeling_utilsrutilsrrrrrrconfiguration_mistralr flash_attnrrflash_attn.bert_paddingrrrlist signature parametersr get_loggerrQrr~r7Moduler8rUrur}r~rrrrrrrMISTRAL_START_DOCSTRINGr rRr rSrr5r5r5r6sv        %  GF*;