o h@sdZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZdd lmZmZmZmZdd lmZdd lmZmZmZmZd d lmZee Z!dZ"dZ#ddgZ$Gdddej%Z&Gdddej'Z(Gdddej'Z)GdddeZ*dZ+dZ,ede+Gddde*Z-ede+Gdd d e*Z.ed!e+Gd"d#d#e*Z/ed$e+Gd%d&d&e*Z0dS)'z PyTorch BioGPT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)!_prepare_4d_causal_attention_mask))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) BioGptConfigzmicrosoft/biogptrzmicrosoft/BioGPT-LargecsDeZdZdZdedeffdd Zd dejdeffd d ZZ S) BioGptLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcsd|_t||j|dS)N)offsetsuper__init__)selfrr __class__`/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/biogpt/modeling_biogpt.pyr?sz)BioGptLearnedPositionalEmbedding.__init__rattention_maskpast_key_values_lengthcsN|}tj|dd||d}|dd|df}t||jS)z3`input_ids_shape` is expected to be [bsz x seqlen].rdimN)longtorchcumsumtype_asrforwardr)rr#r$ positionsrr!r"r+Es z(BioGptLearnedPositionalEmbedding.forward)r) __name__ __module__ __qualname____doc__intrr( LongTensorr+ __classcell__r!r!rr"r:s"rcseZdZdZ     ddededed ed ed ed eeffd d Z de j dedefddZ     dde j dee j dee e j dee j dee j dede e j ee j ee e j ffddZZS)BioGptAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias is_causalconfigcst||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩r:)rrr6r7r8head_dimr< ValueErrorscalingr9r;rLineark_projv_projq_projout_proj)rr6r7r8r9r:r;r<rr!r"rVs&    zBioGptAttention.__init__tensorseq_lenbszcCs ||||j|jddS)Nrr)viewr7r> transpose contiguous)rrFrGrHr!r!r"_shapeus zBioGptAttention._shape hidden_stateskey_value_statespast_key_valuer#layer_head_maskoutput_attentionsreturncCsr|du}|\}} } |||j} |r.|dur.|djd|jdkr.|d} |d} nZ|rE|||d|} |||d|} nC|durt|||d|} |||d|} tj|d| gdd} tj|d| gdd} n|||d|} |||d|} |j r| | f}||j d|j f}|| | |j |} | j |} | j |} | d}t| | dd}|||j | |fkrtd||j | |fd||dur ||d| |fkrtd |d| |fd|| ||j | ||}| ||j | |}tjj|dd}|durL||j fkr1td |j fd|| dddd| ||j | |}| ||j | |}|rc| ||j | |}| ||j | |}nd}tjj||j|jd }t|| }|||j | |j fkrtd ||j | |j fd|| ||j | |j }|dd}| || |j}||}|||fS) z#Input shape: Batch x Time x ChannelNrrrr%z$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )sizerDr@shaperLrBrCr(catr9r7r>rIreshapebmmrJr?r functionalsoftmaxr8rVr6rE)rrMrNrOr#rPrQis_cross_attentionrHtgt_len_ query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputr!r!r"r+xs        "    zBioGptAttention.forward)r5FTFN)NNNNF)r-r.r/r0r1floatboolrrrr(TensorrLrr+r3r!r!rr"r4SsV r4cseZdZdeffdd Z     ddejdeejd eejd eeejd ee d ee d eej eeej ej fffddZ Z S)BioGptDecoderLayerr<cst|j|_t|j|j|jdd|_|j|_ t |j |_ |j |_ t|j|_t|j|j|_t|j|j|_t|j|_dS)NT)r6r7r8r9)rr hidden_sizer6r4num_attention_headsattention_probs_dropout_prob self_attnhidden_dropout_probr8r hidden_act activation_fnactivation_dropoutr LayerNormself_attn_layer_normrAintermediate_sizefc1fc2final_layer_normrr<rr!r"rs  zBioGptDecoderLayer.__init__NFTrMr#rPrOrQ use_cacherRc Cs|}||}|dur|ddnd}|j|||||d\}} } tjj||j|jd}||}|}||}||}||}tjj||j |jd}| |}tjj||j|jd}||}|f} |ri| | f7} |rp| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Nr)rMrOr#rPrQrT) rwrqrr\r8rVr{ryrtrurz) rrMr#rPrOrQr}residualself_attn_past_key_valueself_attn_weightspresent_key_valueoutputsr!r!r"r+s4        zBioGptDecoderLayer.forward)NNNFT) r-r.r/rrr(rlrrrk FloatTensorr+r3r!r!rr"rms, rmc@s$eZdZdZeZdZdZddZdS)BioGptPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. biogptTcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsr5)meanstdN?) isinstancerrAweightdatanormal_r<initializer_ranger:zero_ Embedding padding_idxrvfill_)rmoduler!r!r" _init_weightsOs      z#BioGptPreTrainedModel._init_weightsN) r-r.r/r0r config_classbase_model_prefixsupports_gradient_checkpointingrr!r!r!r"rEs  raJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`~BioGptConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare BioGPT Model transformer outputting raw hidden-states without any specific head on top.cseZdZdeffdd ZddZddZee de e e e d  dd eejd eejd eejdeejdeeeejdeedeedeedeedeee ffddZZS) BioGptModelr<cst|_j|_j|_j|_j|_ j r"t jnd|_ tj|j|j |_tj|j|_tfddtjD|_t|j|_d|_|dS)Nrcsg|]}tqSr!)rm).0r`r<r!r" sz(BioGptModel.__init__..F)rrr< layerdroprrr8rnr6 pad_token_idrscale_embeddingmathsqrt embed_scalerr vocab_size embed_tokensrmax_position_embeddingsembed_positions ModuleListrangenum_hidden_layerslayersrv layer_normgradient_checkpointing post_initr|rrr"rs   zBioGptModel.__init__cC|jSNrrr!r!r"get_input_embeddingsz BioGptModel.get_input_embeddingscC ||_dSrrrvaluer!r!r"set_input_embeddings z BioGptModel.set_input_embeddingsbatch_size, sequence_length checkpoint output_typerN input_idsr# head_mask inputs_embedspast_key_valuesr}rQoutput_hidden_states return_dictrRc  Cs&|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |dur4|dur4td|dur?|} | } n|durW|dd} |dddddf} ntd|durh|ddjdnd} |durv|| |j }|durt j |jd|jd| ft j |j d}n|jd| | dkrtd|jdd | | dd ||| } t|| || }|| }tjj||j|jd }|jr|jr|rtd d }|rdnd}|rdnd}d}|rdnd}t|jD]t\}}|r||f7}|jr t g}||jkr q|dur||nd}|jr4|jr4||j|||dur-||ndd||}n||||dur@||nd|||d}|d}|rZ|||rUdndf7}|rd||df7}q|rm||f7}||}|rw|nd}| stdd|||||fDSt|||||dS)NzDYou cannot specify both input_ids and inputs_embeds at the same timerSz5You have to specify either input_ids or inputs_embedsrrr)dtypedevicez'The provided attention mask has length z, but its length should be z0 (sum of the lengths of current and past inputs)rTzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr!)r#rPrOrQr}css|] }|dur|VqdSrr!)rvr!r!r" =sz&BioGptModel.forward..)last_hidden_staterrM attentionscross_attentions) r<rQrr}use_return_dictr?rWrXrrr(onesrkrrr rr\r8rVrlogger warning_once enumeraterrandr_gradient_checkpointing_func__call__rtupler )rrr#rrrr}rQrrinput input_shaper$r,rMall_hidden_statesall_self_attnsall_cross_attentionsnext_decoder_cacheidx decoder_layerdropout_probabilityrO layer_outputs next_cacher!r!r"r+s               zBioGptModel.forward) NNNNNNNNN)r-r.r/rrrrrBIOGPT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrr(r2rrrlrkrr+r3r!r!rr"rsP    rzHBioGPT Model with a `language modeling` head on top for CLM fine-tuning.cseZdZdgZfddZddZddZee de e e e d  dd eejd eejd eejdeejdeeeejdeejdeedeedeedeedeee ffddZ dddZeddZZS)BioGptForCausalLMzoutput_projection.weightcs8t|t||_tj|j|jdd|_| dSNFr=) rrrrrrArnroutput_projectionrr|rr!r"rQs   zBioGptForCausalLM.__init__cCrrrrr!r!r"get_output_embeddingsZrz'BioGptForCausalLM.get_output_embeddingscCrrr)rnew_embeddingsr!r!r"set_output_embeddings]rz'BioGptForCausalLM.set_output_embeddingsrrNrr#rrrlabelsr}rQrrrRc  Cs| dur| n|jj} |j|||||||| | d } | d} || } d}|durS| ddddddf}|ddddf}t}||d|jj|d}| si| f| dd}|durg|f|S|St|| | j | j | j | j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)r#rrrr}rQrrrrSr)losslogitsrrMrr) r<rrrrKrrIrr rrMrr)rrr#rrrrr}rQrrrsequence_outputprediction_scoreslm_lossshifted_prediction_scoresloss_fctoutputr!r!r"r+`s> zBioGptForCausalLM.forwardc Ks|dur(|ddjd}|jd|kr|}n|jdd}|dd|df}|dur5|dur5d|i}nd|i}||||dd|S)Nrrrrrr})r#rr})rXupdateget) rrr#rrkwargs past_lengthremove_prefix_length model_inputsr!r!r"prepare_inputs_for_generations  z/BioGptForCausalLM.prepare_inputs_for_generationcs.d}|D]}|tfdd|Df7}q|S)Nr!c3s$|] }|d|jVqdS)rN) index_selecttor)r past_statebeam_idxr!r"rs"z3BioGptForCausalLM._reorder_cache..)r)rrreordered_past layer_pastr!rr"_reorder_caches z BioGptForCausalLM._reorder_cache NNNNNNNNNN)NN)r-r.r/_tied_weights_keysrrrrrrrrr rrr(r2rrrlrkrr+r staticmethodrr3r!r!rr"rKs`       : rz BioGPT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeeeeee d           dde e j de e j de e j de e j d e eee jd e e j d e e j d e ed e ede ede edeeeffddZZS)BioGptForTokenClassificationcsjt||j|_t||_t|dr|jdur|j}n|j}t ||_ t |j |j|_ |dS)Nclassifier_dropout)rr num_labelsrrhasattrrrrrDropoutr8rArn classifierr)rr<rrr!r"rs    z%BioGptForTokenClassification.__init__rNrtoken_type_idsr#rrrrr}rQrrrRc  Cs | dur| n|jj} |j||||||| | | d } | d} || } || }d}|duret}|durX|ddk}|d|j}t ||dt |j  |}|||}n ||d|j|d}| s{|f| dd}|dury|f|S|St ||| j| jdS) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr#rrr}rQrrrrSrr)rrrMr)r<rrr8rrrIrr(whererF ignore_indexr*rrMr)rrrr#rrrrr}rQrrtransformer_outputsrMrrr active_loss active_logits active_labelsrr!r!r"r+sF   z$BioGptForTokenClassification.forward) NNNNNNNNNNN)r-r.r/rrrrrrrrr(r2rrrlrkrr+r3r!r!rr"rsX      ra The BioGpt Model transformer with a sequence classification head on top (linear layer). [`BioGptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it is required to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZdeffdd Zeeeee e d          dde e j de e jde e jd e eee jd e e jd e e j d e ed e ede ede edeee ffddZddZddZZS)BioGptForSequenceClassificationr<cs@t||j|_t||_tj|j|jdd|_| dSr) rrrrrrrArnscorerr|rr!r"r4s   z(BioGptForSequenceClassification.__init__rNrr#rrrrr}rQrrrRc  Cs| dur| n|jj} |j|||||||| | d } | d} || } |dur/|jdd\}}n |jdd\}}|jjdurAd}n"|durWt||jjdd | j }n d}t |j jd| tj|| j d|f}d}|dur|jjdur|jdkrd |j_n|jdkr|jtjks|jtjkrd |j_nd |j_|jjd krt}|jdkr|||}n+|||}n%|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S) r Nr rrrSrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)rrrrMr)r<rrrrXrr(nesumrrrwarningr r-arange problem_typerrr'r1rsqueezerrIrrrrMr)rrr#rrrrr}rQrrr rMr batch_sizesequence_length pooled_logitsrrrr!r!r"r+=sn  $    "       z'BioGptForSequenceClassification.forwardcCs|jjSrrrrr!r!r"rsz4BioGptForSequenceClassification.get_input_embeddingscCs ||j_dSrrrr!r!r"rs z4BioGptForSequenceClassification.set_input_embeddingsr)r-r.r/rrrrrrrrrr(r2rrrlrkrr+rrr3r!r!rr"r$sV     Xr)1r0rtypingrrrr(torch.utils.checkpointrtorch.nnrrr activationsr modeling_attn_mask_utilsr modeling_outputsr r rrmodeling_utilsrutilsrrrrconfiguration_biogptr get_loggerr-rrr$BIOGPT_PRETRAINED_MODEL_ARCHIVE_LISTrrModuler4rmrBIOGPT_START_DOCSTRINGrrrrrr!r!r!r"s\      T 6'zU