o h @sdZddlZddlmZmZmZddlZddlZddl Zddlm Z ddl m Z ddl mZddlmZdd lmZmZmZdd lmZdd lmZmZmZmZmZd d lmZee Z!d Z"dZ#dZ$gdZ%dZ&dZ'dZ(dZ)dZ*dgZ+  dJdee,e,fde-de,deej.de,dej/f ddZ0Gdd d e j1Z2Gd!d"d"e j1Z3Gd#d$d$e j1Z4Gd%d&d&e j1Z5Gd'd(d(e j1Z6Gd)d*d*e j1Z7Gd+d,d,e7Z8Gd-d.d.e j1Z9Gd/d0d0e j1Z:Gd1d2d2e j1Z;Gd3d4d4e j1ZGd9d:d:e j1Z?Gd;d<dd>eZAd?ZBd@ZCedAeBGdBdCdCeAZDedDeBGdEdFdFeAZEedGeBGdHdIdIeAZFdS)Kz PyTorch Hubert model.N)OptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)BaseModelOutputCausalLMOutputSequenceClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) HubertConfigrzfacebook/hubert-large-ls960-ft)ri$iz['MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL'gGz6@zsuperb/hubert-base-superb-ksz '_unknown_'g(\!@zfacebook/hubert-base-ls960shape mask_prob mask_lengthattention_mask min_masksreturncs|\}dkr tdkrtdddtjdfdd}|dur:|d n fd d t|D}tj |ft d }g} |} | d krZ|S|D];} || } tjj t | d| dd} t | d kr}d}n| d }t| tj| | tjd |g} | | q\t| } t| dddddf|| f} | || } t ddddf}t||| f|| }| |} | dkrd| | dk<t|| dd |S)af Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://arxiv.org/abs/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `csXt|}t|}|kr}|d|kr*t|dd}|S)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonrrrsequence_length`/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/hubert/modeling_hubert.pycompute_num_masked_spanjs  z6_compute_mask_indices..compute_num_masked_spanNcsg|]}qSr#r#.0_)r"r#r$ }sz)_compute_mask_indices..dtyperF)replace) ValueErrornprandomranditemsumdetachtolistrangezerosboolchoicearangelen concatenateonesint32appendarray broadcast_toreshaperput_along_axis)rrrrr batch_sizer% input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrrspec_aug_mask_idxdummy_mask_idxoffsetsr#r r$_compute_mask_indicesDs\    rLc&eZdZdfdd ZddZZS)HubertNoLayerNormConvLayerrcsjt|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ dS)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvrfeat_extract_activation activationselfconfiglayer_id __class__r#r$rTs  z#HubertNoLayerNormConvLayer.__init__cCs||}||}|SN)r\r^r` hidden_statesr#r#r$forwards  z"HubertNoLayerNormConvLayer.forwardr__name__ __module__ __qualname__rTrh __classcell__r#r#rcr$rNsrNcrM)HubertLayerNormConvLayerrcs|t|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ tj |jdd|_ t |j|_dS)NrrrOT)elementwise_affine)rSrTrUrVrWrrXrYrZr[r\ LayerNorm layer_normrr]r^r_rcr#r$rTs  z!HubertLayerNormConvLayer.__init__cCs:||}|dd}||}|dd}||}|S)Nr&)r\ transposerrr^rfr#r#r$rhs     z HubertLayerNormConvLayer.forwardrirjr#r#rcr$rosrocrM)HubertGroupNormConvLayerrcst|dkr|j|dnd|_|j||_tj|j|j|j||j||j d|_ t |j |_ tj|j|jdd|_dS)NrrrOT) num_groups num_channelsaffine)rSrTrUrVrWrrXrYrZr[r\rr]r^ GroupNormrrr_rcr#r$rTs   z!HubertGroupNormConvLayer.__init__cCs"||}||}||}|Sre)r\rrr^rfr#r#r$rhs   z HubertGroupNormConvLayer.forwardrirjr#r#rcr$rusruc$eZdZfddZddZZS)HubertPositionalConvEmbeddingcsttj|j|j|j|jd|jd|_tjj }t tjj dr'tjj j }t rdddl }|jj|jjdd||jddd|_Wdn1sLwY|j||jj|j||jjn ||jddd|_t|j|_t|j|_dS)N)rPpaddinggroups weight_normr modifier_rankweight)namedim)rSrTrrX hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupsr\utilsrhasattrparametrizationsr deepspeedzeroGatheredParametersrregister_external_parameterweight_vweight_gHubertSamePadLayerr}rr]r^)r`rarrrcr#r$rTs*   z&HubertPositionalConvEmbedding.__init__cCs:|dd}||}||}||}|dd}|SNrr|)rtr\r}r^rfr#r#r$rh#s     z%HubertPositionalConvEmbedding.forwardrjr#r#rcr$r{s r{crz)rcs*t|ddkrd|_dSd|_dS)Nr|rr)rSrTnum_pad_remove)r`rrcr#r$rT0s  zHubertSamePadLayer.__init__cCs,|jdkr|ddddd|j f}|SNr)rrfr#r#r$rh4s zHubertSamePadLayer.forwardrjr#r#rcr$r/s rcs0eZdZdZfddZddZddZZS)HubertFeatureEncoderz.Construct the features from raw audio waveformcstjdkr tddgfddtjdD}njdkr2fddtjD}n td jd t||_ d |_ d |_ dS) Ngrouprrbcsg|] }t|ddqS)rr)rNr(irar#r$r*Bsz1HubertFeatureEncoder.__init__..rlayercsg|]}t|dqS)r)rorrr#r$r*Fsz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rSrTfeat_extract_normrur6num_feat_extract_layersr.r ModuleList conv_layersgradient_checkpointing_requires_grad)r`rarrcrr$rT>s       zHubertFeatureEncoder.__init__cCs|D]}d|_qd|_dSNF) parameters requires_gradrr`paramr#r#r$_freeze_parametersOs  z'HubertFeatureEncoder._freeze_parameterscCs\|dddf}|jr|jrd|_|jD]}|jr'|jr'|jr'||j|}q||}q|S)NT)rtrainingrrr_gradient_checkpointing_func__call__)r` input_valuesrg conv_layerr#r#r$rhTs   zHubertFeatureEncoder.forward)rkrlrm__doc__rTrrhrnr#r#rcr$r;s  rcseZdZfddZZS)HubertFeatureExtractorcs8t|td|jjd|jjdjdtdS)Nz The class `zD` has been depreciated and will be removed in Transformers v5. Use `rz ` instead.)rSrTwarningswarnrdrk __bases__ FutureWarningr`rarcr#r$rThs   zHubertFeatureExtractor.__init__)rkrlrmrTrnr#r#rcr$rgsrcrz)HubertFeatureProjectioncsXt|j|_|jrtj|jd|jd|_t|jd|j |_ t |j |_ dS)Nr&eps)rSrTfeat_proj_layer_normrrqrUlayer_norm_epsrrLinearr projectionDropoutfeat_proj_dropoutdropoutrrcr#r$rTss z HubertFeatureProjection.__init__cCs(|jr||}||}||}|Sre)rrrrrrfr#r#r$rh{s    zHubertFeatureProjection.forwardrjr#r#rcr$rrs rcseZdZdZ     ddededed ed ed ed eeffd d Z de j dedefddZ     dde j dee j dee e j dee j dee j dede e j ee j ee e j ffddZZS)HubertAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsr is_decoderrR is_causalracst||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ t j |||d|_ t j |||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g)rR)rSrTrrrhead_dimrar.scalingrrrrk_projv_projq_projout_proj)r`rrrrrRrrarcr#r$rTs&    zHubertAttention.__init__tensorseq_lenbszcCs ||||j|jddSr)viewrrrt contiguous)r`rrrr#r#r$_shapes zHubertAttention._shapergkey_value_statespast_key_valuerlayer_head_maskoutput_attentionsrcCsr|du}|\}} } |||j} |r.|dur.|djd|jdkr.|d} |d} nZ|rE|||d|} |||d|} nC|durt|||d|} |||d|} tj|d| gdd} tj|d| gdd} n|||d|} |||d|} |j r| | f}||j d|j f}|| | |j |} | j |} | j |} | d}t| | dd}|||j | |fkrtd||j | |fd||dur ||d| |fkrtd |d| |fd|| ||j | ||}| ||j | |}tjj|dd}|durL||j fkr1td |j fd|| dddd| ||j | |}| ||j | |}|rc| ||j | |}| ||j | |}nd}tjj||j|jd }t|| }|||j | |j fkrtd ||j | |j fd|| ||j | |j }|dd}| || |j}||}|||fS) z#Input shape: Batch x Time x ChannelNrr|rr&rz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size )prz `attn_output` should be of size )sizerrrrrrtorchcatrrrrrBbmmrtr.r functionalsoftmaxrrrr)r`rgrrrrris_cross_attentionrtgt_lenr) query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputr#r#r$rhs        "    zHubertAttention.forward)rFTFN)NNNNF)rkrlrmrrfloatr8rrrTrTensorrrrhrnr#r#rcr$rsV rcrz)HubertFeedForwardcsptt|j|_t|j|j|_ t |j t r"t |j |_n|j |_t|j|j|_t|j|_dSre)rSrTrractivation_dropoutintermediate_dropoutrrintermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrrcr#r$rT%s  zHubertFeedForward.__init__cCs6||}||}||}||}||}|Sre)rrrrrrfr#r#r$rh2s     zHubertFeedForward.forwardrjr#r#rcr$r$s  rcs&eZdZfddZdddZZS)HubertEncoderLayercsftt|j|j|jdd|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_dS)NFrrrrr)rSrTrrnum_attention_headsattention_dropout attentionrrrrrqrrrr feed_forwardfinal_layer_normrrcr#r$rT>s  zHubertEncoderLayer.__init__NFcCsf|}|j|||d\}}}||}||}||}|||}||}|f}|r1||f7}|SNrr)rrrrrrr`rgrr attn_residualrr)outputsr#r#r$rhKs    zHubertEncoderLayer.forwardrrjr#r#rcr$r=s  rcs,eZdZfddZdejfddZZS)HubertAttnAdapterLayercsZt|j|_|j|_t|j|_t |j|j|_ t |_ t |j|j|_ dS)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)rSrTadapter_attn_dim input_dimr hidden_dimrrqnormrlinear_1ReLUact_fnlinear_2rrcr#r$rTas  zHubertAttnAdapterLayer.__init__rgcCs,||}||}||}||}|Sre)rrrr rfr#r#r$rhos    zHubertAttnAdapterLayer.forward)rkrlrmrTr FloatTensorrhrnr#r#rcr$r`s rcs@eZdZfddZ  d dejdeejdefdd ZZ S) !HubertEncoderLayerStableLayerNormcstt|j|j|jdd|_t|j |_ tj |j|j d|_ t||_tj |j|j d|_t|dddur@t||_dSd|_dS)NFrrr)rSrTrrrrrrrrrrqrrrrrrgetattrr adapter_layerrrcr#r$rT{s   z*HubertEncoderLayerStableLayerNorm.__init__NFrgrrcCsz|}||}|j|||d\}}}||}||}||||}|jdur1|||}|f}|r;||f7}|Sr)rrrrrrr rr#r#r$rhs    z)HubertEncoderLayerStableLayerNorm.forwardr) rkrlrmrTrrrr8rhrnr#r#rcr$r zs r c sLeZdZfddZ    d dejdeejded ed ef d d Z Z S) HubertEncodercft|_t|_tjjjd|_ t j |_ t fddtjD|_d|_dS)Nrcg|]}tqSr#)rr'rr#r$r*z*HubertEncoder.__init__..FrSrTrar{pos_conv_embedrrqrrrrrrrrr6num_hidden_layerslayersrrrcrr$rTs    zHubertEncoder.__init__NFTrgrroutput_hidden_states return_dictcCs|rdnd}|r dnd}|durO|ddd|jd}d||<d|ddddddfj|jd}|t|jj}||jdd|jd|jd}| |} || }| |}| |}t } |j D]G} |rq||f}tg} |jr| |jjkrdnd } | r| r|jr|jr|| j|||}n| |||d }|d}| rd }|r||df}qh|r||f}|std d |||fDSt|||dS)Nr#r&rr|r?r+TFrNNcs|] }|dur|VqdSrer#r(vr#r#r$ z(HubertEncoder.forward..last_hidden_staterg attentions) unsqueezerepeatrtor,rfinfominexpandrrrrr rr1rra layerdroprrrtupler r`rgrrrrall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddingsdeepspeed_zero3_is_enabledrdropout_probabilityskip_the_layer layer_outputsr#r#r$rhs\   &        zHubertEncoder.forwardNFFT) rkrlrmrTrrrrr8rhrnr#r#rcr$rs"  rcs.eZdZfddZ    dddZZS) HubertEncoderStableLayerNormcr)Nrcrr#)r r'rr#r$r*rz9HubertEncoderStableLayerNorm.__init__..Frrrcrr$rTs   z%HubertEncoderStableLayerNorm.__init__NFTcCs|rdnd}|r dnd}|durO|ddd|jd}d||<d|ddddddfj|jd}|t|jj}||jdd|jd|jd}| |} || }| |}t } |j D]G} |rl||f}t g} |jr|| |jjkr|dnd } | r| r|jr|jr|| j|||}n| |||d }|d}| rd }|r||df}qc||}|r||f}|std d |||fDSt|||dS)Nr#r&rr|rrr+TFrrcsrrer#rr#r#r$rGrz7HubertEncoderStableLayerNorm.forward..r)r"r#rr$r,rr%r&r'rrr rr1rrar(rrrrrr)r r*r#r#r$rhs\   &        z$HubertEncoderStableLayerNorm.forwardr3rjr#r#rcr$r4s r4c@sTeZdZdZeZdZdZdZddZ de e j e ffdd Zd e d e j fd d ZdS)HubertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. hubertrTcCsRt|tjr|jjjd|jjdn}t|tjtj fr)|j j |jj dnft|tj rtrddl}t|dret|dre|jj|j|jgddtj|jjWdn1s_wYn*|jj|jddtj|jjWdn1swYntj|jjt|tjtj fr|j dur|j j dSdSdS) zInitialize the weightsr)meanstdrrNrrr)rrrrdatanormal_rainitializer_rangerqryrRzero_fill_rXr rrrrrrinitkaiming_normal_)r`modulerr#r#r$ _init_weightsZs*   z#HubertPreTrainedModel._init_weightsrEcCs4dd}t|jj|jjD] \}}||||}q |S)zH Computes the output length of the convolutional layers cSstj|||dddS)Nfloor) rounding_moder)rdiv)rrPrQr#r#r$_conv_out_lengthxszPHubertPreTrainedModel._get_feat_extract_output_lengths.._conv_out_length)ziprarYrZ)r`rErErPrQr#r#r$ _get_feat_extract_output_lengthsssz6HubertPreTrainedModel._get_feat_extract_output_lengthsfeature_vector_lengthrcCs~||dtj}|jd}tj||f|j|jd}d|tj |jd|jd|df<| dg d dg }|S)Nr&r)r,devicer)rI) rGr3r$rlongrr7r,rIr:flipcumsumr8)r`rHroutput_lengthsrDr#r#r$"_get_feature_vector_attention_masks "z8HubertPreTrainedModel._get_feature_vector_attention_maskN)rkrlrmrr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrArr LongTensorrrGrNr#r#r#r$r5Osr5a! Hubert was proposed in [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://arxiv.org/abs/2106.07447) by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving etc.). This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`HubertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Wav2Vec2Processor.__call__`] for details. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) `attention_mask` should only be passed if the corresponding processor has `config.return_attention_mask == True`. For all models whose processor has `config.return_attention_mask == False`, such as [hubert-base](https://huggingface.co/facebook/hubert-base-ls960), `attention_mask` should **not** be passed to avoid degraded performance when doing batched inference. For such models `input_values` should simply be padded with 0 and passed without `attention_mask`. Be aware that these models also yield slightly different results depending on whether `input_values` is padded or not. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare Hubert Model transformer outputting raw hidden-states without any specific head on top.cseZdZdeffdd Z  ddejdeejdeejfdd Z e e e e ed      dd eejdeejdeejd eed eedeedeee ffddZZS) HubertModelracszt|||_t||_t||_|jdks|jdkr)t t |j |_|jr2t||_nt||_|dS)Nr)rSrTrarfeature_extractorrfeature_projectionmask_time_probmask_feature_probr Parameterrr runiform_masked_spec_embeddo_stable_layer_normr4encoderr post_initrrcr#r$rTs      zHubertModel.__init__Nrgmask_time_indicesrcCst|jdds |S|\}}}|dur|j|j||<n-|jjdkrK|jrKt||f|jj|jj ||jj d}t j ||j t jd}|j|j||<|jjdkr|jrt||f|jj|jj|jjd}t j ||j t jd}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). apply_spec_augmentTNr)rrrr)rIr,)rrrr&)r rarr[r$r,rWrrLmask_time_lengthmask_time_min_masksrrrIr8rXmask_feature_lengthmask_feature_min_masksr')r`rgr_rrDr"rmask_feature_indicesr#r#r$_mask_hidden_statess4 zHubertModel._mask_hidden_states) output_typerOrrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}||}|dd}|dur6||jd|}||}|j ||d}|j |||||d} | d}|s[|f| ddSt || j | j dS)aZ Returns: Example: ```python >>> from transformers import AutoProcessor, HubertModel >>> from datasets import load_dataset >>> import soundfile as sf >>> processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft") >>> model = HubertModel.from_pretrained("facebook/hubert-large-ls960-ft") >>> def map_to_array(batch): ... speech, _ = sf.read(batch["file"]) ... batch["speech"] = speech ... return batch >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.map(map_to_array) >>> input_values = processor(ds["speech"][0], return_tensors="pt").input_values # Batch size 1 >>> hidden_states = model(input_values).last_hidden_state ```Nrr|)r_rrrrrr)rarruse_return_dictrUrtrNrrVrfr]r rgr!) r`rrr_rrrextract_featuresrgencoder_outputsr#r#r$rh s2&   zHubertModel.forwardrNNNNN)rkrlrmrrTrr rrSrfrHUBERT_INPUTS_DOCSTRINGrr _CONFIG_FOR_DOCrr8rrrhrnr#r#rcr$rTsB .  rTzdHubert Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).cseZdZddeeffdd ZddZddZd d Zd d Z e e e e eeeed      ddeejdeejdeedeedeedeejdeeeffddZZS) HubertForCTCN target_langcs~t|t||_t|j|_||_|j dur#t d|j dt |dr.|j r.|jn|j}t||j |_|dS)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `HubertForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)rSrTrTr6rr final_dropoutrrp vocab_sizer.rdrrqoutput_hidden_sizerrlm_headr^)r`rarprtrcr#r$rT]s     zHubertForCTC.__init__cCsv|j}|durt|jdddurtd|d|dur,t|jdddur,tddS|dur9|j|dddSdS)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. NrzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load)rpr rar.loggerinfo load_adapter)r`rpr#r#r$ tie_weightsts zHubertForCTC.tie_weightscCtdt|dS) Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nrrrfreeze_feature_encoderr`r#r#r$freeze_feature_extractor  z%HubertForCTC.freeze_feature_extractorcC|jjdSr|Nr6rUrrr#r#r$rz#HubertForCTC.freeze_feature_encodercC|jD]}d|_qdSz Calling this function will disable the gradient computation for the base model so that its parameters will not be updated during training. Only the classification head will be updated. FNr6rrrr#r#r$freeze_base_modelzHubertForCTC.freeze_base_model) checkpointrgrOexpected_output expected_lossrrrrrlabelsrc Cst|dur|n|jj}|j|||||d}|d}||}||} d} |dur||jjkr9td|jj|dur?|ntj |tj d}| | d tj } |dk} | d} || }tjj| dtjddd}tjjjd d tjj||| | |jj|jj|jjd } Wdn1swY|s| f|td}| dur| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nrhrz$Label values must be <= vocab_size: r+r&)rr,rF)enabled)blank reduction zero_infinitylosslogitsrgr!)rarir6rrurrsr.r ones_likerJrGr3r$ masked_selectrr log_softmaxfloat32rtbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONr rgr!)r`rrrrrrrrgrrrE labels_masktarget_lengthsflattened_targets log_probsoutputr#r#r$rhsN      zHubertForCTC.forwardrerl)rkrlrmrrrTrzrrrrrmr_CHECKPOINT_FOR_DOCr rn_CTC_EXPECTED_OUTPUT_CTC_EXPECTED_LOSSrrr8rrrhrnr#r#rcr$roWsD   roz Hubert Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. cseZdZfddZddZddZddZeee e e e d e ed  dd eejd eejdeedeedeedeejdeee ffddZZS)HubertForSequenceClassificationcst|t|dr|jrtdt||_|jd}|jr*t t |||_ t |j|j|_t |j|j|_|dS)Nrqz]Sequence classification does not support the use of Hubert adapters (config.add_adapter=True)r)rSrTrrqr.rTr6ruse_weighted_layer_sumrrYrr= layer_weightsrrclassifier_proj_size projector num_labels classifierr^)r`ra num_layersrcr#r$rTs    z(HubertForSequenceClassification.__init__cCr{)z Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. r}Nr~rr#r#r$rrz8HubertForSequenceClassification.freeze_feature_extractorcCrrrrr#r#r$rrz6HubertForSequenceClassification.freeze_feature_encodercCrrrrr#r#r$r!rz1HubertForSequenceClassification.freeze_base_modelaudio)rrgrOmodalityrrNrrrrrrrcCs`|dur|n|jj}|jjrdn|}|j|||||d}|jjrB|t}tj|dd}tjj |j dd} || dddj dd}n|d}| |}|durV|jdd} n||jd|} d|| <|j dd| j dd dd} || } d} |durt}|| d|jj| d} |s| f|td}| dur| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrhrrr&rrr)rarirr6rrstackrrrrrr3rr7rNrrrrr rgr!)r`rrrrrrrrg norm_weights pooled_output padding_maskrrloss_fctrr#r#r$rh)sF    z'HubertForSequenceClassification.forwardrl)rkrlrmrTrrrrrmr_SEQ_CLASS_CHECKPOINTr rn_SEQ_CLASS_EXPECTED_OUTPUT_SEQ_CLASS_EXPECTED_LOSSrrrr8rrrhrnr#r#rcr$rsD    rr)Grrtypingrrrnumpyr/rtorch.utils.checkpointrtorch.nnr activationsrintegrations.deepspeedr modeling_outputsr r r modeling_utilsr rrrrrrconfiguration_hubertr get_loggerrkrwrrnr_EXPECTED_OUTPUT_SHAPErrrrr$HUBERT_PRETRAINED_MODEL_ARCHIVE_LISTrrrSndarrayrLModulerNrorur{rrrrrrrrr rr4r5HUBERT_START_DOCSTRINGrmrTrorr#r#r#r$s          x( ,  #.RU@&