o h@sdZddlZddlZddlZddlmZmZmZmZddl Z ddl m m Z ddlZ ddl m Z ddlmZmZmZddlmZddlmZmZdd lmZdd lmZmZmZdd lmZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,e%rddl-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3de4e5e.j6vZ7e*re!sddl8Z e j9:eZe';e<Z=dZ>d@de j?de j?de@fddZAddZBGddde jCZDGdd d e jCZEd!d"ZFdAd#d$ZGd%e j?d&eHde j?fd'd(ZIGd)d*d*e jCZJGd+d,d,eJZKGd-d.d.e jCZLeJeKd/ZMGd0d1d1e jCZNGd2d3d3e jCZOd4ZPe#d5ePGd6d7d7eZQd8ZRe#d5ePGd9d:d:eQZSGd;d<dd?d?eQZUdS)Bz PyTorch Mixtral model.N)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache)!_prepare_4d_causal_attention_mask)MoeCausalLMOutputWithPastMoeModelOutputWithPast SequenceClassifierOutputWithPast)PreTrainedModel)#is_torch_greater_or_equal_than_1_13)add_start_docstrings%add_start_docstrings_to_model_forwardis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10loggingreplace_return_docstrings)is_torch_fx_available) MixtralConfig)flash_attn_funcflash_attn_varlen_func)index_first_axis pad_input unpad_input window_sizer gate_logits num_expertsreturncCs|durdSt|trtj|dd}tj||dd\}}|jdd}|jtjkr.|tj}t |j dkr:| d}tj j ||}tj|ddj}|tj}tj|dd}tj|dd}t|| d|dS)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://arxiv.org/abs/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits (Union[`torch.Tensor`, Tuple[torch.Tensor]): Logits from the `gate`, should be a tuple of tensors. Shape: [batch_size, seqeunce_length, num_experts]. num_experts (`int`, *optional*): Number of experts Returns: The auxiliary loss. Nrdimr#)axis) isinstancetupletorchcattopksoftmaxdtypeint64tolenshape unsqueezer functionalone_hotmaxvaluesfloat32mean)r$r%top_krouting_weightsselected_experts expert_masktokens_per_group_and_expert router_prob_per_group_and_expertrDb/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/mixtral/modeling_mixtral.pyload_balancing_loss_funcLs       rFcCsX|jdtjd}tj|dd}|}ttj |dtjjdd}|||fS)Nr)r(r2F)as_tupler)rr) sumr.int32nonzeroflattenr:itemFpadcumsum)attention_maskseqlens_in_batchindicesmax_seqlen_in_batch cu_seqlensrDrDrE_get_unpad_data|s rVcs&eZdZdfdd ZddZZS)MixtralRMSNormư>cs&ttt||_||_dS)z= MixtralRMSNorm is equivalent to T5LayerNorm N)super__init__r Parameterr.onesweightvariance_epsilon)self hidden_sizeeps __class__rDrErZs  zMixtralRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)Nr#r)T)keepdim) r2r4r.r<powr=rsqrtr^r])r_ hidden_states input_dtypevariancerDrDrEforwards  zMixtralRMSNorm.forward)rX)__name__ __module__ __qualname__rZrj __classcell__rDrDrbrErWsrWcs0eZdZd fdd ZddZd dd ZZS) MixtralRotaryEmbedding'Ncsrt||_||_||_d|jtd|jd||j}|j d|dd|j ||j j t ddS)Ng?rr#inv_freqF persistentseq_lendevicer2)rYrZr(max_position_embeddingsbaser.arangefloatr4register_buffer_set_cos_sin_cacherrrwget_default_dtype)r_r(rxryrwrrrbrDrErZs * zMixtralRotaryEmbedding.__init__cCst||_tj|j||jjd}t||j}tj||fdd}|jd| |dd|jd| |dddS)N)rwr2r)r' cos_cachedFrs sin_cached) max_seq_len_cachedr.rzrrr2outerr/r|cosr4sin)r_rvrwr2tfreqsembrDrDrEr}s z)MixtralRotaryEmbedding._set_cos_sin_cachecCsN||jkr|j||j|jd|jd|j|jd|jd|j|jdfS)Nru)r2)rr}rwr2rr4r)r_xrvrDrDrErjs zMixtralRotaryEmbedding.forward)rprqNN)rkrlrmrZr}rjrnrDrDrbrEros rocCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..Nr)r#r')r6r.r/)rx1x2rDrDrE rotate_halfsrcCsL|||}|||}||t||}||t||}||fS)anApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`): The position indices of the tokens corresponding to the query and key tensors. For example, this can be used to pass offsetted position ids when working with a KV-cache. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )r7r)qkrr position_ids unsqueeze_dimq_embedk_embedrDrDrEapply_rotary_pos_embs rrgn_repcCs^|j\}}}}|dkr |S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r6expandreshape)rgrbatchnum_key_value_headsslenhead_dimrDrDrE repeat_kvs 0rcseZdZdZddedeeffdd Zdej ded efd d Z    dd ej deej deej dee de de deej eej eeej ffddZZS)MixtralAttentionz Multi-headed attention from 'Attention Is All You Need' paper. Modified to use sliding window attention: Longformer and "Generating Long Sequences with Sparse Transformers". Nconfig layer_idxcs<t||_||_|durtd|jjd|j|_|j |_ |j|j |_ |j |_ |j |j |_ |j|_|j|_d|_|j|_|j |j |jkrYtd|jd|j dtj|j|j |j dd|_tj|j|j |j dd|_tj|j|j |j dd|_tj|j |j |jdd|_t|j |j|jd |_dS) NzInstantiating z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).Fbias)rxry)rYrZrrlogger warning_oncercrkr`num_attention_heads num_headsrrnum_key_value_groupsrx rope_theta is_causalattention_dropout ValueErrorrLinearq_projk_projv_projo_projro rotary_embr_rrrbrDrErZs>   zMixtralAttention.__init__tensorrvbszcCs ||||j|jddS)Nrr#)viewrr transpose contiguous)r_rrvrrDrDrE_shapes zMixtralAttention._shapeFrgrQrpast_key_valueoutput_attentions use_cacher&cKsd|vr td|\}} } ||} ||} ||} | || |j|j dd} | || |j |j dd} | || |j |j dd} | j d}|durj|j durat d|jjd||||j 7}|j| |d\}}t| | |||\} } |dur||d }|| | |j |\} } t| |j} t| |j} t| | dd t|j}|||j| |fkrt d ||j| |fd ||dur||d| |fkrt d |d| |fd |||}tjj|dtjd| j}tjj ||j!|j"d}t|| }|||j| |jfkr)t d||j| |jfd || dd#}|$|| |j%}|&|}|sCd}|||fS)N padding_maskrPassing `padding_mask` is deprecated and will be removed in v4.37. Please make sure use `attention_mask` instead.`rr#r*zFThe cache structure has changed since version v4.36. If you are using zv for auto-regressive decoding with k/v caching, please make sure to initialize the attention class with a layer index.rvrrr z$Attention weights should be of size z , but is z!Attention mask should be of size r)rG)ptrainingz `attn_output` should be of size )'warningswarnsizerrrrrrrrr6rrrcrkget_usable_lengthrrupdaterrr.matmulmathsqrtrr8r1r<r4r2dropoutrrrrr`r)r_rgrQrrrrkwargsrq_len_ query_states key_states value_states kv_seq_lenrr cache_kwargs attn_weights attn_outputrDrDrErj sj             zMixtralAttention.forwardrNNNFF)rkrlrm__doc__rrintrZr.Tensorr LongTensorr boolrrjrnrDrDrbrErs0% rc sxeZdZdZfddZ     ddejdeejdeejd ee d e d e f d d Z   dddZ ddZ ZS)MixtralFlashAttention2aJ Mixtral flash attention module. This module inherits from `MixtralAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cs tj|i|t |_dSr)rYrZr_flash_attn_uses_top_left_mask)r_argsrrbrDrErZ{szMixtralFlashAttention2.__init__NFrgrQrrrrc KsXd|vrtd|d}|\}} } ||} ||} ||} | || |j|j  dd} | || |j |j  dd} | || |j |j  dd} | j d}|dur`|| ||j7}t||dddfd}|j| |d\}}t| | |||\} } tot|jddduo||jjk}tstd |dur.t|jdddur||jjkrd|jj}|d }|d}|dddd|dddf}|dddd|dddf}|j d|jjdkrtd |j ||f}|dur|dd|df}tj|t|ddddfgdd }||d }|| | |j|\} } t| |j } t| |j } |j!s@dn|j"}| j#}|tj$krut%|jdrX|jj&}n|jj'j#}td|d| (|} | (|} | (|} | dd} | dd} | dd} |j)| | | || ||d}|*|| |j+}|,|}|sd}|||fS)Nrrrr#r*r)rsliding_windowzThe current flash attention version does not support sliding window attention, for a more memory efficient implementation make sure to upgrade flash-attn library.rzepast key must have a shape of (`batch_size, num_heads, self.config.sliding_window-1, head_dim`), got r'r_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)ruse_sliding_windows)-rrpoprrrrrrrrrr6rrr:rMrr_flash_supports_window_sizegetattrrrrrrrr.r/ ones_likerrrrrr2r<hasattrrr]r4_flash_attention_forwardrr`r)r_rgrQrrrrrrrrrrrrrotary_seq_lenrrrslicing_tokenspast_key past_valuer dropout_raterh target_dtyperrrDrDrErjs      "  " $$ (              zMixtralFlashAttention2.forwardrc  Cs|js|j} n|jo |dk} |dur]|jd} ||||||\}}}} } } | \}}| \}}|s?t|||||||||| d }nt|||||||||| |jj|jjfd }t|| | |}|S|skt|||||| d}|St|||||| |jj|jjfd}|S)a Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then computes the attention scores and pad the final attention scores. Args: query_states (`torch.Tensor`): Input query states to be passed to Flash Attention API key_states (`torch.Tensor`): Input key states to be passed to Flash Attention API value_states (`torch.Tensor`): Input value states to be passed to Flash Attention API attention_mask (`torch.Tensor`): The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the position of padding tokens and 1 for the position of non-padding tokens. dropout (`int`, *optional*): Attention dropout softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim) use_sliding_windows (`bool`, *optional*): Whether to activate sliding window attention. rNr) cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_k dropout_p softmax_scalecausal)rrrrrrrr")rr)rrr") rrr6 _upad_inputrrrr r)r_rrrrQ query_lengthrrrr batch_size indices_q cu_seq_lens max_seq_lensrrmax_seqlen_in_batch_qmax_seqlen_in_batch_kattn_output_unpadrrDrDrErst     z/MixtralFlashAttention2._flash_attention_forwardcCs&|j\}}}} ||jdkr|jd} |dd| |df}t|\} } } t||||| | }t||||| | }||krUt||||| | }| }| }| }n2|dkrsd}tj|dtj|jd}|dd}|d}n|dd| df}t ||\}}}}|||||| f|| ffS)Nr)rr2rw) r6rVrrr.rzrJrwsqueezer!)r_ query_layer key_layer value_layerrQrrrrrattention_mask_num_tokens indices_krrrrrrDrDrErcs<   z"MixtralFlashAttention2._upad_inputr)rNF)rkrlrmrrZr.rrrr rrjrrrnrDrDrbrErss6    drcs*eZdZdeffdd ZddZZS)MixtralBLockSparseTop2MLPrcslt|j|_|j|_tj|j|jdd|_tj|j|jdd|_ tj|j|jdd|_ t |j |_ dSNFr)rYrZintermediate_sizeffn_dimr` hidden_dimrrw1w2w3r hidden_actact_fnr_rrbrDrErZs z"MixtralBLockSparseTop2MLP.__init__cCs,|||||}||}||Sr)r r r r )r_rgr?current_hidden_statesrDrDrErjs z!MixtralBLockSparseTop2MLP.forward)rkrlrmrrZrjrnrDrDrbrErs r)eagerflash_attention_2cs6eZdZdZfddZdejdejfddZZS)MixtralSparseMoeBlocka This implementation is strictly equivalent to standard MoE with full capacity (no dropped tokens). It's faster since it formulates MoE operations in terms of block-sparse operations to accomodate imbalanced assignments of tokens to experts, whereas standard MoE either (1) drop tokens at the cost of reduced performance or (2) set capacity factor to number of experts and thus waste computation and memory on padding. csdtj|_j|_j|_j|_ t j |j|jdd|_ t fddt|jD|_dS)NFrcsg|]}tqSrD)r).0rrrDrE sz2MixtralSparseMoeBlock.__init__..)rYrZr`rrrnum_local_expertsr%num_experts_per_tokr>rrgate ModuleListrangeexpertsrrbrrErZs $zMixtralSparseMoeBlock.__init__rgr&cCsB|j\}}}|d|}||}tj|dtjd}tj||jdd\}}||j ddd}| |j }tj |||f|j |j d}tjjj||jdd dd } t|jD]?} |j| } t| | \} } | jd d krnqV| }| }|d |fd|}| ||||d f}|d | | |j qV||||}||fS)  r)rrGr'T)r(rdr) num_classesr#rN)r6rrrNr1r.r{r0r>rIr4r2zerosrwrr8r9r%permuterrwheretolistr index_add_)r_rgrsequence_lengthr router_logitsr?r@final_hidden_statesrA expert_idx expert_layeridxtop_x top_x_listidx_list current_staterrDrDrErjs.     zMixtralSparseMoeBlock.forward) rkrlrmrrZr.rrjrnrDrDrbrErs  rcseZdZdedeffdd Z      ddejdeejd eej d ee ejd ee d ee d ee de ej ee ej ej fffddZ ZS)MixtralDecoderLayerrrcsVt|j|_t|j|||_t||_t|j|j d|_ t|j|j d|_ dS)Nra) rYrZr`MISTRAL_ATTENTION_CLASSES_attn_implementation self_attnrblock_sparse_moerW rms_norm_epsinput_layernormpost_attention_layernormrrbrDrErZs  zMixtralDecoderLayer.__init__NFrgrQrrroutput_router_logitsrr&cKsd|vr td |} ||}|j||||||d\}} } | |}|} ||}||\}} | |}|f} |r?| | f7} |rF| | f7} |rM| | f7} | S)Nrr)rgrQrrrr)rrr4r1r5r2)r_rgrQrrrr6rrresidualself_attn_weightspresent_key_valuer$outputsrDrDrErjs8       zMixtralDecoderLayer.forward)NNNFFF)rkrlrmrrrZr.rrrrr FloatTensorrjrnrDrDrbrEr-s2   r-aL This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MixtralConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zUThe bare Mixtral Model outputting raw hidden-states without any specific head on top.c@s2eZdZeZdZdZdgZdZdZ dZ ddZ dS)MixtralPreTrainedModelmodelTr-past_key_valuescCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rA|jjjd|d|j durC|jj|j  dSdSdS)Nr)r=std) rinitializer_ranger,rrr]datanormal_rzero_ Embedding padding_idx)r_moduler?rDrDrE _init_weightsYs    z$MixtralPreTrainedModel._init_weightsN) rkrlrmr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_cache_classrGrDrDrDrEr<Ks r<a5 Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cseZdZdZdeffdd ZddZddZee  dd e j d e e j d e e j d e ee jde e jde ede ede ede ede edeeeffddZZS) MixtralModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MixtralDecoderLayer`] Args: config: MixtralConfig rcstj|_j|_tjj|j|_t fddt j D|_ j dk|_tjjd|_d|_|dS)Ncsg|]}t|qSrD)r-)rrrrDrErsz)MixtralModel.__init__..rr.F)rYrZ pad_token_idrE vocab_sizerrDr` embed_tokensrrnum_hidden_layerslayersr0_use_flash_attention_2rWr3normgradient_checkpointing post_initrrbrrErZs   zMixtralModel.__init__cC|jSrrRr_rDrDrEget_input_embeddingsz!MixtralModel.get_input_embeddingscC ||_dSrrZr_valuerDrDrEset_input_embeddings z!MixtralModel.set_input_embeddingsN input_idsrQrr> inputs_embedsrroutput_hidden_statesr6 return_dictr&c  Cs|dur|n|jj}| dur| n|jj} |dur|n|jj}|dur$|n|jj}| dur.| n|jj} |dur>|dur>td|durH|j\} } n|durS|j\} } } ntdd}|rmt|t  }|rht |}| | }|dur|durx|j n|j }tj|| |tj|d}|dd| }n|d| }|dur||}|dur|jr|r|dddf| k}|rtd|jr|durd|vr|nd}n t|| | f|||jjd}|}|jr|jr|rtdd }|rd nd}|rd nd}| rd nd}d}|jD]P}|r||f7}|jr$|jr$||j|||||| |}n ||||||| |d }|d}|r?||rXsz'MixtralModel.forward..)last_hidden_stater>rg attentionsr$)$rrr6reruse_return_dictrr6r,r r from_legacy_cacherrwr.rzlongr7rrRrUrIrMrrrWrrrrT_gradient_checkpointing_func__call__rVto_legacy_cacher-r)r_rcrQrr>rdrrrer6rfr seq_lengthrpast_key_values_lengthuse_legacy_cacherwis_padding_rightrgall_hidden_statesall_self_attnsall_router_logitsnext_decoder_cache decoder_layer layer_outputs next_cacherDrDrErjs                zMixtralModel.forward NNNNNNNNNN)rkrlrmrrrZr\rarMIXTRAL_INPUTS_DOCSTRINGr.rrrrr;rrrrrjrnrDrDrbrErOsN     rOcseZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e e e eed           d$dejdeejdeejdeeejdeejdeejdeedeedeedeedeedeeeffddZ d%d d!Zed"d#ZZS)&MixtralForCausalLMzlm_head.weightcsXt|t||_|j|_tj|j|jdd|_|j |_ |j |_ |j |_ | dSr)rYrZrOr=rQrrr`lm_headrouter_aux_loss_coefrr%rrXrrbrDrErZis   zMixtralForCausalLM.__init__cC|jjSrr=rRr[rDrDrEr\tz'MixtralForCausalLM.get_input_embeddingscC ||j_dSrrr_rDrDrEraw z'MixtralForCausalLM.set_input_embeddingscCrYrrr[rDrDrEget_output_embeddingszr]z(MixtralForCausalLM.get_output_embeddingscCr^rr)r_new_embeddingsrDrDrEset_output_embeddings}rbz(MixtralForCausalLM.set_output_embeddingscCr^rr=)r_decoderrDrDrE set_decoderrbzMixtralForCausalLM.set_decodercCrYrrr[rDrDrE get_decoderr]zMixtralForCausalLM.get_decoder) output_typerHNrcrQrr>rdlabelsrrrer6rfr&c  Cs|dur|n|jj}| dur| n|jj} | dur| n|jj} | dur$| n|jj} |j|||||||| | | d } | d} || }|}d}|dur||dddddf}|dddf}t }| d|jj }| d}| |j }|||}d}| rt| r| jn| d|j|j}|dur||j|7}| s|f| dd}| r|f|}|dur|f|S|St|||| j| j| j| jdS)a Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Returns: Example: ```python >>> from transformers import AutoTokenizer, MixtralForCausalLM >>> model = MixtralForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS) >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER) >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rcrQrr>rdrrrer6rfr.r)r)lossaux_losslogitsr>rgrjr$)rrr6rerkr=rr{rrrrQr4rwrFr$r%rrrr>rgrj)r_rcrQrr>rdrrrrer6rfr:rgrr shift_logits shift_labelsloss_fctroutputrDrDrErjsf+     zMixtralForCausalLM.forwardc Ksz|durnt|tr|}|j}|}n |ddjd}}d}|durA|jd|jdkrA|dd|jd| df}n||jdkrR|dd|df}|durn|durn||jd|krn|dd| df}|dd} |dur| dur|dd} | |dkd|r| dd|jd df} |dur|durd|i} nd|i} | | ||d|d | S) Nrr#rrr)rdrcr)rr>rrQ) r,r get_seq_length seen_tokensget_max_lengthr6getrmrP masked_fill_r) r_rcr>rQrdr cache_length past_lengthmax_cache_lengthr model_inputsrDrDrEprepare_inputs_for_generations@  "  z0MixtralForCausalLM.prepare_inputs_for_generationcs.d}|D]}|tfdd|Df7}q|S)NrDc3s$|] }|d|jVqdS)rN) index_selectr4rw)r past_statebeam_idxrDrErh0s"z4MixtralForCausalLM._reorder_cache..)r-)r>rreordered_past layer_pastrDrrE_reorder_cache+s z!MixtralForCausalLM._reorder_cache) NNNNNNNNNNN)NNN)rkrlrm_tied_weights_keysrZr\rarrrrrr}rr_CONFIG_FOR_DOCr.rrrrr;rrrrjr staticmethodrrnrDrDrbrEr~fsf         j 9r~a The Mixtral Model transformer with a sequence classification head on top (linear layer). [`MixtralForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZfddZddZddZee          ddej d e ej d e ej d e e ej d e ej d e ej de ede ede ede edeeeffddZZS) MixtralForSequenceClassificationcs@t||j|_t||_tj|j|jdd|_| dSr) rYrZ num_labelsrOr=rrr`scorerXrrbrDrErZFs   z)MixtralForSequenceClassification.__init__cCrrrr[rDrDrEr\Orz5MixtralForSequenceClassification.get_input_embeddingscCrrrr_rDrDrEraRrz5MixtralForSequenceClassification.set_input_embeddingsNrcrQrr>rdrrrrerfr&c  Cs| dur| n|jj} |j|||||||| | d } | d} || } |dur+|jd}n|jd}|jjdur>|dkr>td|jjdurGd}n|dur_t||jj dd | j }nd}| tj || j d|f}d}|dur| | j }|jjdur|jdkrd|j_n|jdkr|jtjks|jtj krd |j_nd |j_|jjdkrt}|jdkr|||}n+|||}n%|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rQrr>rdrrrerfrrz=Cannot handle batch sizes > 1 if no padding token is defined.r))rw regressionsingle_label_classificationmulti_label_classification)rrr>rgrj)rrkr=rr6rPrr.eqrargmaxr4rwrz problem_typerr2rmr rrrrrr>rgrj)r_rcrQrr>rdrrrrerftransformer_outputsrgrrsequence_lengths pooled_logitsrrrrDrDrErjUsr        "       z(MixtralForSequenceClassification.forwardr|)rkrlrmrZr\rarr}r.rrrrr;rrrrrjrnrDrDrbrEr5sL       r)Nr#)r)Vrinspectrrtypingrrrrr.torch.nn.functionalrr8rNtorch.utils.checkpointtorch.nnrrr activationsr cache_utilsr r modeling_attn_mask_utilsrmodeling_outputsrrrmodeling_utilsr pytorch_utilsrutilsrrrrrrutils.import_utilsrconfiguration_mixtralr flash_attnrrflash_attn.bert_paddingrr r!list signature parametersrtorch.fxfxwrap get_loggerrkrrrr{rFrVModulerWrorrrrrrrr/rr-MIXTRAL_START_DOCSTRINGr<r}rOr~rrDrDrDrEs          0 %  FNC:P