o hn @sdZddlZddlZddlmZmZmZmZddlZddl m m Z ddl Zddlm Z ddlmZmZmZddlmZddlmZmZdd lmZdd lmZmZdd lmZmZmZm Z d d l!m"Z"e #e$Z%dZ&dZ'gdZ(dZ)dZ*dgZ+d4dej,de-de.dej,fddZ/Gddde j0Z1Gddde j0Z2Gddde j0Z3Gd d!d!e j0Z4Gd"d#d#e j0Z5Gd$d%d%e j0Z6Gd&d'd'e j0Z7Gd(d)d)e j0Z8Gd*d+d+eZ9d,Z:d-Z;ed.e:Gd/d0d0e9Zr(r(r0r)r?^s(( r?cs<eZdZdedeffdd ZdejdejfddZZ S) PvtSelfOutputr@rEcs*tt|||_t|j|_dSr,)r-r.rLineardenserYrZr[)r/r@rEr0r(r)r.s zPvtSelfOutput.__init__r2rcCs||}||}|Sr,)rzr[r3r(r(r)r4s  zPvtSelfOutput.forward) r7r8r9rrvr.r!r<r4r>r(r(r0r)rxsrxc speZdZdZdedededeffdd Zded ej fd d Z ddej d edede d e ej f ddZ ZS)PvtEfficientSelfAttentionzpEfficient self-attention mechanism with reduction of the sequence [PvT paper](https://arxiv.org/abs/2102.12122).r@rEnum_attention_headssequences_reduction_ratiocst||_||_|j|jdkr td|jd|jdt|j|j|_|j|j|_tj |j|j|j d|_ tj |j|j|j d|_ tj |j|j|j d|_ t|j|_||_|dkrwtj||||d|_tj||jd|_dSdS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ())biasrrGrI)r-r.rEr|rmrvattention_head_size all_head_sizerryqkv_biasquerykeyvaluerYattention_probs_dropout_probr[r}rTsequence_reductionrVrWrXr/r@rEr|r}r0r(r)r.s,  z"PvtEfficientSelfAttention.__init__r2rcCs6|dd|j|jf}||}|ddddS)Nr_rr`rr )rbr|rviewre)r/r2 new_shaper(r(r)transpose_for_scoress z.PvtEfficientSelfAttention.transpose_for_scoresFr]r^output_attentionscCs&|||}|jdkr6|j\}}}|ddd||||}||}|||dddd}||}|||} || |} t || dd} | t |j} tjj| dd} || } t | | } | dddd} | dd|jf}| |} |r| | f}|S| f}|S)Nrrr`r_rkr )rrr}rrerdrrXrrr!matmulromathsqrtrr functionalsoftmaxr[ contiguousrbrr)r/r2r]r^r query_layerrrseq_lenrD key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr(r(r)r4s*        z!PvtEfficientSelfAttention.forwardru)r7r8r9r:rrvr;r.r!r<rrwrr4r>r(r(r0r)r{s0 r{c s`eZdZdedededeffdd ZddZ dd ej d ed ed e de ej f ddZ Z S) PvtAttentionr@rEr|r}cs6tt||||d|_t||d|_t|_dS)N)rEr|r})rE)r-r.r{r/rxr'set pruned_headsrr0r(r)r.s  zPvtAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrk)lenrr/r|rrrrrrr'rzrunion)r/headsindexr(r(r) prune_heads s zPvtAttention.prune_headsFr2r]r^rrcCs4|||||}||d}|f|dd}|S)Nrr)r/r')r/r2r]r^r self_outputsattention_outputrr(r(r)r4szPvtAttention.forwardru)r7r8r9rrvr;r.rr!r<rwrr4r>r(r(r0r)rs. rc sReZdZ  d dededeedeeffdd Zdejd ejfd d Z Z S) PvtFFNNr@ in_featureshidden_features out_featurescsjt|dur |n|}t|||_t|jtr!t|j|_ n|j|_ t|||_ t |j |_ dSr,)r-r.rrydense1rL hidden_actr=r intermediate_act_fndense2rYrZr[)r/r@rrrr0r(r)r.)s  zPvtFFN.__init__r2rcCs6||}||}||}||}||}|Sr,)rrr[rr3r(r(r)r4:s     zPvtFFN.forward)NN) r7r8r9rrvrr.r!r<r4r>r(r(r0r)r(src sTeZdZdedededededef fdd Zdd ejd ed ed e fddZ Z S)PvtLayerr@rEr|r*r} mlp_ratiocszttj||jd|_t||||d|_|dkrt|nt |_ tj||jd|_ t ||}t |||d|_dS)NrI)r@rEr|r}r)r@rr)r-r.rrVrW layer_norm_1r attentionr+Identityr* layer_norm_2rvrmlp)r/r@rEr|r*r}rmlp_hidden_sizer0r(r)r.Ds  zPvtLayer.__init__Fr2r]r^rc Csn|j|||||d}|d}|dd}||}||}|||}||}||} | f|}|S)N)r2r]r^rrr)rrr*rr) r/r2r]r^rself_attention_outputsrr mlp_output layer_outputr(r(r)r4Zs    zPvtLayer.forwardru) r7r8r9rrvr;r.r!r<rwr4r>r(r(r0r)rCs&rc s^eZdZdeffdd Z   d dejdeedeed eed e e e ff d d Z Z S) PvtEncoderr@c stt||_td|jt|j}g}t |j D]9}| t ||dkr)|j n |jj d|d|j||j||dkrA|jn|j|d|j|||j dkdqt||_g}d}t |j D]@}g}|dkrv||j|d7}t |j|D]}| t||j||j|||||j||j|dq}| t|qet||_tj|jd|jd|_dS)Nrr`r)r@rArBrCrDrErF)r@rEr|r*r}rr_rI)r-r.r@r!linspacedrop_path_ratesumdepthstolistrangenum_encoder_blocksappendr?rA patch_sizesstridesrD hidden_sizesr ModuleListpatch_embeddingsrr|sequence_reduction_ratios mlp_ratiosblockrVrWrX) r/r@drop_path_decaysr\iblockscurlayersjr0r(r)r.rsJ      zPvtEncoder.__init__FTrjroutput_hidden_states return_dictrcCs |rdnd}|r dnd}|jd}t|j}|} tt|j|jD]C\} \} } | | \} } }| D]}|| | ||}|d} |rF||df}|rM|| f}q0| |dkrd| || |ddddd} q!| | } |rq|| f}|st dd| ||fDSt | ||d S) Nr(rrr_r r`css|] }|dur|VqdSr,r().0vr(r(r) sz%PvtEncoder.forward..last_hidden_stater2 attentions) rrr enumerateziprrdrerrXtupler )r/rjrrrall_hidden_statesall_self_attentionsrr num_blocksr2idxembedding_layer block_layerr]r^r layer_outputsr(r(r)r4s8         zPvtEncoder.forward)FFT)r7r8r9rr.r! FloatTensorrrwrrr r4r>r(r(r0r)rqs 5 rc@s>eZdZdZeZdZdZdee j e j e j fddfddZ dS) PvtPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. pvtrjmodulerNcCst|tjr$tjj|jjd|jjd|j_|j dur"|j j dSdSt|tj r9|j j |jj ddSt|t rctjj|jjd|jjd|j_|jduretjj|jjd|jjd|j_dSdSdS)zInitialize the weightsr)meanstdNg?)rLrryinit trunc_normal_weightdatar@initializer_rangerzero_rVfill_r?rRrF)r/rr(r(r) _init_weightss,      z PvtPreTrainedModel._init_weights)r7r8r9r:r config_classbase_model_prefixmain_input_namerrryrTrVrr(r(r(r)rs &raG This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`~PvtConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a  Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`PvtImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zSThe bare Pvt encoder outputting raw hidden-states without any specific head on top.cseZdZdeffdd ZddZeede e e e de d dd ejd eed eed eedeee ff ddZZS)PvtModelr@cs(t|||_t||_|dSr,)r-r.r@rencoder post_initr/r@r0r(r)r. s   zPvtModel.__init__cCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrlayerrr)r/heads_to_prunerrr(r(r) _prune_headsszPvtModel._prune_heads%(batch_size, channels, height, width)vision) checkpoint output_typermodalityexpected_outputNrjrrrrcCs~|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||d}|d}|s6|f|ddSt||j|jdS)Nrjrrrrrr)r@rruse_return_dictrr r2r)r/rjrrrencoder_outputssequence_outputr(r(r)r4s$zPvtModel.forward)NNN)r7r8r9rr.rrPVT_INPUTS_DOCSTRINGr5r_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPEr!rrrwrrr4r>r(r(r0r)rs2   rz Pvt Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. cseZdZdeddffdd Zeedee e e e d    dde ejd e ejd e ed e ed e edeee ff d dZZS)PvtForImageClassificationr@rNcsRt||j|_t||_|jdkrt|jd|jnt|_ | dS)Nrr_) r-r. num_labelsrrrryrr classifierrrr0r(r)r.Ns  $ z"PvtForImageClassification.__init__r)rrrrrjlabelsrrrc Csp|dur|n|jj}|j||||d}|d}||dddddf}d} |dur|jjdurU|jdkr;d|j_n|jdkrQ|jtjksL|jtj krQd|j_nd|j_|jjdkrst } |jdkrm| | | } n+| ||} n%|jjdkrt } | | d|j| d} n|jjdkrt} | ||} |s|f|dd} | dur| f| S| St| ||j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr regressionsingle_label_classificationmulti_label_classificationr_)losslogitsr2r)r@rrr  problem_typer rr!longrvr squeezerrrr r2r) r/rjr rrrrrrrloss_fctr'r(r(r)r4\sJ   "       z!PvtForImageClassification.forward)NNNN)r7r8r9rr.rrr5r_IMAGE_CLASS_CHECKPOINTr r_IMAGE_CLASS_EXPECTED_OUTPUTrr!r<rwrrr4r>r(r(r0r)r Fs4   r )rF)>r:rMrtypingrrrrr!torch.nn.functionalrrrftorch.utils.checkpointtorch.nnrrr activationsr modeling_outputsr r modeling_utilsr pytorch_utilsrrutilsrrrrconfiguration_pvtr get_loggerr7loggerrrr rr!PVT_PRETRAINED_MODEL_ARCHIVE_LISTr<r;rwr*Moduler+r?rxr{rrrrrPVT_START_DOCSTRINGrrr r(r(r(r)sZ      B R*.Y# :