o h @sfdZddlZddlZddlmZmZmZddlZddl Z ddl Z ddl m Z ddl m Z ddlmZddlmZdd lmZmZmZdd lmZdd lmZmZmZmZd d lmZee Z!d Z"dZ#dZ$gdZ%dZ&dZ'dZ(dZ)dZ*gdZ+  dDdee,e,fde-de,dee j.de,dej/f ddZ0Gdd d e j1Z2Gd!d"d"e j1Z3Gd#d$d$e j1Z4Gd%d&d&e j1Z5Gd'd(d(e j1Z6Gd)d*d*e j1Z7Gd+d,d,e j1Z8Gd-d.d.e8Z9Gd/d0d0e j1Z:Gd1d2d2e j1Z;Gd3d4d4e j1Zd9Z?d:Z@ed;e?GdZAed>e?Gd?d@d@e>ZBedAe?GdBdCdCe>ZCdS)Ez PyTorch SEW model.N)OptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)BaseModelOutputCausalLMOutputSequenceClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) SEWConfigrzasapp/sew-tiny-100k-ft-ls100h)ri$iz_'MISTER QUILTER IS THE APPOSTILE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPOLLE'gzG?z(anton-l/sew-mid-100k-ft-keyword-spottingz '_unknown_'g ףp= #@)zasapp/sew-tiny-100kzasapp/sew-small-100kzasapp/sew-mid-100kshape mask_prob mask_lengthattention_mask min_masksreturncs|\}dkr tdkrtdddtjdfdd}|dur:|d n fd d t|D}tj |ft d }g} |} | d krZ|S|D];} || } tjj t | d| dd} t | d kr}d}n| d }t| tj| | tjd |g} | | q\t| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkrd| | dk<t|| dd |S)af Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://arxiv.org/abs/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr}|d|kr*t|dd}|S)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonrrrsequence_lengthZ/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/sew/modeling_sew.pycompute_num_masked_spanis  z6_compute_mask_indices..compute_num_masked_spanNcsg|]}qSr"r".0_)r!r"r# |sz)_compute_mask_indices..dtyperF)replace) ValueErrornprandomranditemsumdetachtolistrangezerosboolchoicearangelen concatenateonesint32appendarray broadcast_toreshaperput_along_axis)rrrrr batch_sizer$ input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrrspec_aug_mask_idxdummy_mask_idxoffsetsr"rr#_compute_mask_indicesCs\    rKc&eZdZdfdd ZddZZS)SEWNoLayerNormConvLayerrcsjt|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvrfeat_extract_activation activationselfconfiglayer_id __class__r"r#rSs  z SEWNoLayerNormConvLayer.__init__cCs||}||}|SN)r[r]r_ hidden_statesr"r"r#forwards  zSEWNoLayerNormConvLayer.forwardr__name__ __module__ __qualname__rSrg __classcell__r"r"rbr#rMsrMcrL)SEWLayerNormConvLayerrcs|t|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)NrrrNT)elementwise_affine)rRrSrTrUrVrrWrXrYrZr[ LayerNorm layer_normrr\r]r^rbr"r#rSs  zSEWLayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)Nr%)r[ transposerqr]rer"r"r#rgs     zSEWLayerNormConvLayer.forwardrhrir"r"rbr#rnsrncrL)SEWGroupNormConvLayerrcst|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)NrrrNT) num_groups num_channelsaffine)rRrSrTrUrVrrWrXrYrZr[rr\r] GroupNormrqr^rbr"r#rSs   zSEWGroupNormConvLayer.__init__cC"||}||}||}|Srd)r[rqr]rer"r"r#rgs   zSEWGroupNormConvLayer.forwardrhrir"r"rbr#rtsrtc$eZdZfddZddZZS)SEWPositionalConvEmbeddingcsttj|j|j|j|jd|j|jd|_t rXddl }|j j |jj ddtjj|jddd|_Wdn1s@wY|j ||jj|j ||jjn tjj|jddd|_t|j|_t|j|_dS)N)rOpaddinggroupsrPr modifier_rankweight)namedim)rRrSrrW hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupssqueeze_factorr[r deepspeedzeroGatheredParametersrutils weight_normregister_external_parameterweight_vweight_gSEWSamePadLayerr}rr\r])r_r`rrbr"r#rSs&   z#SEWPositionalConvEmbedding.__init__cCryrd)r[r}r]rer"r"r#rgs   z"SEWPositionalConvEmbedding.forwardrir"r"rbr#r{s r{crz)rcs*t|ddkrd|_dSd|_dS)Nr|rr)rRrSnum_pad_remove)r_rrbr"r#rS(s  zSEWSamePadLayer.__init__cCs,|jdkr|ddddd|j f}|SNr)rrer"r"r#rg,s zSEWSamePadLayer.forwardrir"r"rbr#r's rcrz) SEWUpsamplingcs:tt|j|j|j|_t|j|_ |j|_dSrd) rRrSrLinearrr projectionrr\r]r_r`rbr"r#rS3s   zSEWUpsampling.__init__cCsd||}||}|jdkr0|\}}}||j}||j}||||j|}||||}|S)Nr)rr]rsizerA)r_rfbszsrc_len src_embed_dimtgt_len tgt_embed_dimr"r"r#rg9s     zSEWUpsampling.forwardrir"r"rbr#r2s rcs0eZdZdZfddZddZddZZS)SEWFeatureEncoderz.Construct the features from raw audio waveformcstjdkr tddgfddtjdD}njdkr2fddtjD}n td jd t||_ d |_ d |_ dS) Ngrouprracsg|] }t|ddqS)rr)rMr'ir`r"r#r)Psz.SEWFeatureEncoder.__init__..rlayercsg|]}t|dqS)r)rnrrr"r#r)Tsz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rRrSfeat_extract_normrtr5num_feat_extract_layersr-r ModuleList conv_layersgradient_checkpointing_requires_grad)r_r`rrbrr#rSLs       zSEWFeatureEncoder.__init__cCs|D]}d|_qd|_dSNF) parameters requires_gradrr_paramr"r"r#_freeze_parameters]s  z$SEWFeatureEncoder._freeze_parameterscCs\|dddf}|jr|jrd|_|jD]}|jr'|jr'|jr'||j|}q||}q|S)NT)rtrainingrrr_gradient_checkpointing_func__call__)r_ input_valuesrf conv_layerr"r"r#rgbs   zSEWFeatureEncoder.forward)rjrkrl__doc__rSrrgrmr"r"rbr#rIs  rcseZdZfddZZS)SEWFeatureExtractorcs8t|td|jjd|jjdjdtdS)Nz The class `zD` has been depreciated and will be removed in Transformers v5. Use `rz ` instead.)rRrSwarningswarnrcrj __bases__ FutureWarningrrbr"r#rSvs   zSEWFeatureExtractor.__init__)rjrkrlrSrmr"r"rbr#rusrcseZdZdZ     ddededed ed ed ed eeffd d Z de j dedefddZ     dde j dee j dee e j dee j dee j dede e j ee j ee e j ffddZZS) SEWAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderrQ is_causalr`cst||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g)rQ)rRrSrrrhead_dimr`r-scalingrrrrk_projv_projq_projout_proj)r_rrrrrQrr`rbr"r#rSs&    zSEWAttention.__init__tensorseq_lenrcCs ||||j|jddS)Nrr|)viewrrrs contiguous)r_rrrr"r"r#_shapes zSEWAttention._shaperfkey_value_statespast_key_valuerlayer_head_maskoutput_attentionsrcCsr|du}|\}} } |||j} |r.|dur.|djd|jdkr.|d} |d} nZ|rE|||d|} |||d|} nC|durt|||d|} |||d|} tj|d| gdd} tj|d| gdd} n|||d|} |||d|} |j r| | f}||j d|j f}|| | |j |} | j |} | j |} | d}t| | dd}|||j | |fkrtd||j | |fd||dur ||d| |fkrtd |d| |fd|| ||j | ||}| ||j | |}tjj|dd}|durL||j fkr1td |j fd|| dddd| ||j | |}| ||j | |}|rc| ||j | |}| ||j | |}nd}tjj||j|jd }t|| }|||j | |j fkrtd ||j | |j fd|| ||j | |j }|dd}| || |j}||}|||fS) z#Input shape: Batch x Time x ChannelNrr|rr%rz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size )prz `attn_output` should be of size )rrrrrrrtorchcatrrrrrAbmmrsr-r functionalsoftmaxrrrr)r_rfrrrrris_cross_attentionrrr( query_states key_states value_states proj_shaper attn_weightsattn_weights_reshaped attn_probs attn_outputr"r"r#rgs        "    zSEWAttention.forward)rFTFN)NNNNF)rjrkrlrrfloatr7rrrSrTensorrrrgrmr"r"rbr#rsV rcrz)SEWFeedForwardcsptt|j|_t|j|j|_ t |j t r"t |j |_n|j |_t|j|j|_t|j|_dSrd)rRrSrDropoutactivation_dropoutintermediate_dropoutrrintermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrrbr"r#rS!s  zSEWFeedForward.__init__cCs6||}||}||}||}||}|Srd)rrrrrrer"r"r#rg.s     zSEWFeedForward.forwardrir"r"rbr#r s  rcs&eZdZfddZdddZZS)SEWEncoderLayercsftt|j|j|jdd|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_dS)NF)rrrreps)rRrSrrnum_attention_headsattention_dropout attentionrrrrrplayer_norm_epsrqr feed_forwardfinal_layer_normrrbr"r#rS:s  zSEWEncoderLayer.__init__NFcCsf|}|j|||d\}}}||}||}||}|||}||}|f}|r1||f7}|S)Nrr)rrrqrr)r_rfrr attn_residualrr(outputsr"r"r#rgGs    zSEWEncoderLayer.forwardrrir"r"rbr#r9s  rcs.eZdZfddZ    dddZZS) SEWEncodercst|_t|_tjj|_tj j j d|_ t j|_tfddtjD|_t|_d|_dS)Nrcsg|]}tqSr")rr&rr"r#r)csz'SEWEncoder.__init__..F)rRrSr`r{pos_conv_embedr AvgPool1drpoolrprrrqrrrrr5num_hidden_layerslayersrupsamplerrrbrr#rS\s     zSEWEncoder.__init__NFTc Cs|rdnd}|r dnd}|durvd||<|d}||jj} |jd|jj} tjd| | jddd | jdd} | | ddk}d|ddddddfj |j d}|t |j j }| |jdd|jd|jd}|jd} |dd }||} ||}t | d|d}|d d|f| d d|f}|dd }||}||}t}|jD]H}|r||f}tg}|jr||jjkrd nd }|r|r|jr|jr||j|||}n||||d }|d}|rd}|r||df}q|r ||f}||}|jd| kr*tj|ddd| |jdf}|s9t dd|||fDSt!|||dS)Nr"rr%rrdevice?r*r|.TFrNNcss|] }|dur|VqdSrdr")r'vr"r"r# sz%SEWEncoder.forward..last_hidden_staterf attentions)"longr2r`rrrr9rrexpandtor+finfominrsrrrrqrr rr0r layerdroprrrrrrpadtupler )r_rfrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsrDoutput_lengthsmax_encoder_length attention_idsn_input_timestepsposition_embeddingspooled_hidden_states min_lengthdeepspeed_zero3_is_enabledrdropout_probabilityskip_the_layer layer_outputsr"r"r#rggsz    &               zSEWEncoder.forward)NFFTrir"r"rbr#r[s rc@sTeZdZdZeZdZdZdZddZ de e j e ffdd Zd e d e j fd d ZdS)SEWPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. sewrTc Cst|tr)tjj|jjddtd|jj d|jj dtj |jj dnt|tj r;|jjjd|jjdn}t|tjtjfrR|j j|jjdnft|tjrtrddl}t|drt|d r|jj|j|jgdd tj|jjWdn1swYn*|jj|jdd tj|jjWdn1swYntj|jjt|tj tjfr|j dur|j jdSdSdS) zInitialize the weightsrr|r)meanstdrrNrrr)rr{rinitnormal_r[rmathsqrtrO in_channels constant_rQrdatar`initializer_rangerprxzero_fill_rWr rhasattrrrrrkaiming_normal_)r_modulerr"r"r# _init_weightss8     z SEWPreTrainedModel._init_weightsrDcCs4dd}t|jj|jjD] \}}||||}q |S)zH Computes the output length of the convolutional layers cSstj|||dddS)Nfloor) rounding_moder)rdiv)rrOrPr"r"r#_conv_out_lengthszMSEWPreTrainedModel._get_feat_extract_output_lengths.._conv_out_length)zipr`rXrY)r_rDr8rOrPr"r"r# _get_feat_extract_output_lengthssz3SEWPreTrainedModel._get_feat_extract_output_lengthsfeature_vector_lengthrcCs~||dtj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dg d dg }|S)Nr%r)r+rrr) r:r2rrr rr6r+rr9flipcumsumr7)r_r;rrrCr"r"r#"_get_feature_vector_attention_masks "z5SEWPreTrainedModel._get_feature_vector_attention_maskN)rjrkrlrr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingr4rr LongTensorrr:r>r"r"r"r#r#s r#a SEW was proposed in [Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition](https://arxiv.org/abs/2109.06870) by Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving etc.). This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SEWConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Wav2Vec2Processor.__call__`] for details. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z]The bare SEW Model transformer outputting raw hidden-states without any specific head on top.cseZdZdeffdd Z  ddejdeejdeejfdd Z e e e e eed ed      dd eejdeejdeejd eedeedeedeeeffddZZS)SEWModelr`cst|||_t||_tj|jd|jd|_ |jd|j k|_ |j r1t |jd|j |_ t|j|_|jdksB|jdkrNtt|j |_t||_|dS)Nr%rr)rRrSr`rfeature_extractorrrprTrrqrproject_featuresrfeature_projectionrfeat_proj_dropoutfeature_dropoutmask_time_probmask_feature_prob Parameterr FloatTensoruniform_masked_spec_embedrencoder post_initrrbr"r#rS=s    zSEWModel.__init__Nrfmask_time_indicesrcCst|jdds |S|\}}}|dur|j|j||<n-|jjdkrK|jrKt||f|jj|jj ||jj d}t j ||j t jd}|j|j||<|jjdkr|jrt||f|jj|jj|jjd}t j ||j t jd}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). apply_spec_augmentTNr)rrrr)rr+)rrrr%)getattrr`rrOrr+rJrrKmask_time_lengthmask_time_min_masksrrrr7rKmask_feature_lengthmask_feature_min_masksr )r_rfrRrrCr!rmask_feature_indicesr"r"r#_mask_hidden_statesQs4 zSEWModel._mask_hidden_statesaudio) checkpoint output_typer?modalityexpected_outputrrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}||}|dd}||}|jr6||}| |}|durH| |j d|}|j ||d}|j |||||d} | d}|sh|f| ddSt|| j| jdS)Nrr|)rRrrrrrr )r`rruse_return_dictrErsrqrFrGrIr>rrZrPr rfr ) r_rrrRrrrextract_featuresrfencoder_outputsr"r"r#rgs8     zSEWModel.forwardrNNNNN)rjrkrlrrSrrMrrCrZrSEW_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr7rrrgrmr"r"rbr#rD8sN .  rDzaSEW Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).cseZdZddeeffdd ZddZddZd d Zd d Z e e e e eeeed      ddeejdeejdeedeedeedeejdeeeffddZZS) SEWForCTCN target_langcs~t|t||_t|j|_||_|j dur#t d|j dt |dr.|j r.|jn|j}t||j |_|dS)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SEWForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)rRrSrDr$rr final_dropoutrrj vocab_sizer-rcr1rkoutput_hidden_sizerrlm_headrQ)r_r`rjrnrbr"r#rSs     zSEWForCTC.__init__cCsv|j}|durt|jdddurtd|d|dur,t|jdddur,tddS|dur9|j|dddSdS)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. Nadapter_attn_dimzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load)rjrTr`r-loggerinfo load_adapter)r_rjr"r"r# tie_weightss zSEWForCTC.tie_weightscCtdt|dS) Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nrrrfreeze_feature_encoderr_r"r"r#freeze_feature_extractor  z"SEWForCTC.freeze_feature_extractorcC|jjdSrwNr$rErr{r"r"r#rzz SEWForCTC.freeze_feature_encodercC|jD]}d|_qdSz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNr$rrrr"r"r#freeze_base_modelzSEWForCTC.freeze_base_model)r\r]r?r_ expected_lossrrrrrlabelsrc Cst|dur|n|jj}|j|||||d}|d}||}||} d} |dur||jjkr9td|jj|dur?|ntj |tj d}| | d tj } |dk} | d} || }tjj| dtjddd}tjjjd d tjj||| | |jj|jj|jjd } Wdn1swY|s| f|td}| dur| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nr`rz$Label values must be <= vocab_size: r*r%)rr+rF)enabled)blank reduction zero_infinitylosslogitsrfr )r`rar$rrorrmr-r ones_liker r:r2r masked_selectrr log_softmaxfloat32rsbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONr rfr )r_rrrrrrrrfrrrD labels_masktarget_lengthsflattened_targets log_probsoutputr"r"r#rgsN      zSEWForCTC.forwardrdrd)rjrkrlrrrSrur|rzrrrerrfr rg_CTC_EXPECTED_OUTPUT_CTC_EXPECTED_LOSSrrr7rrrgrmr"r"rbr#risD   riz SEW Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. cseZdZfddZddZddZddZeee e e e d e ed  dd eejd eejdeedeedeedeejdeee ffddZZS)SEWForSequenceClassificationcst|t|dr|jrtdt||_|jd}|jr*t t |||_ t |j|j|_t |j|j|_|dS)NrkzZSequence classification does not support the use of SEW adapters (config.add_adapter=True)r)rRrSr1rkr-rDr$ruse_weighted_layer_sumrrLrr< layer_weightsrrclassifier_proj_size projector num_labels classifierrQ)r_r` num_layersrbr"r#rS^s    z%SEWForSequenceClassification.__init__cCrv)z Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. rxNryr{r"r"r#r|or}z5SEWForSequenceClassification.freeze_feature_extractorcCr~rrr{r"r"r#rz{rz3SEWForSequenceClassification.freeze_feature_encodercCrrrrr"r"r#rrz.SEWForSequenceClassification.freeze_base_modelr[)r\r]r?r^r_rNrrrrrrrcCs`|dur|n|jj}|jjrdn|}|j|||||d}|jjrB|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|durV|jdd} n||jd|} d|| <|j dd| j dd dd} || } d} |durt}|| d|jj| d} |s| f|td}| dur| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTr`rrr%rrr)r`rarr$rrstackrrrrrr2rr%r>rrrrr rfr )r_rrrrrrrrf norm_weights pooled_output padding_maskrrloss_fctrr"r"r#rgsF    z$SEWForSequenceClassification.forwardrd)rjrkrlrSr|rzrrrer_SEQ_CLASS_CHECKPOINTr rg_SEQ_CLASS_EXPECTED_OUTPUT_SEQ_CLASS_EXPECTED_LOSSrrrr7rrrgrmr"r"rbr#rUsD    rr)Drr)rtypingrrrnumpyr.rtorch.utils.checkpointrtorch.nnr activationsrintegrations.deepspeedr modeling_outputsr r r modeling_utilsr rrrrrconfiguration_sewr get_loggerrjrrrrgrfrhrrrrr!SEW_PRETRAINED_MODEL_ARCHIVE_LISTrrrCndarrayrKModulerMrnrtr{rrrrrrrrr#SEW_START_DOCSTRINGrerDrirr"r"r"r#s          x" ,  "hG|