o hޕ @sRdZddlmZmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZmZdd lmZdd lmZmZmZmZmZd d lmZeeZd Z dZ!gdZ"dZ#dZ$dgZ%dCde&de&dee&de&fddZ'e(de(dfde(de(de(de(fddZ)Gdddej*Z+Gd d!d!ej*Z,Gd"d#d#ej*Z-Gd$d%d%ej*Z.Gd&d'd'ej*Z/Gd(d)d)ej*Z0Gd*d+d+ej*Z1Gd,d-d-ej*Z2Gd.d/d/ej*Z3Gd0d1d1eZ4d2Z5d3Z6ed4e5Gd5d6d6e4Z7ed7e5Gd8d9d9e4Z8Gd:d;d;ej*Z9Gdd?d?ej*Z;ed@e5GdAdBdBe4Zg?T) num_featuresepsmomentumaffinetrack_running_stats)super__init__r ValueErrorrConv2d convolution BatchNorm2d normalization isinstancestrr activation hidden_act) selfr,r-r.r/r0r1r2r3r4r5r7 __class__r!r"r?XsB         zMobileViTV2ConvLayer.__init__featurescCs6||}|jdur||}|jdur||}|Sr')rBrDrG)rIrLr!r!r"forwards     zMobileViTV2ConvLayer.forward)rrFrTT) __name__ __module__ __qualname__rrboolrrFr?torchTensorrM __classcell__r!r!rJr"r+Ws>    6r+c sTeZdZdZ ddedededededd f fd d Zd ejdejfd dZ Z S)MobileViTV2InvertedResidualzQ Inverted residual block (MobileNetv2): https://arxiv.org/abs/1801.04381 rr,r-r.r0r3rNc stttt||jd}|dvrtd|d|dko$||k|_t|||dd|_ t|||d|||d|_ t|||dd d |_ dS) Nr)rrzInvalid stride .r)r-r.r/r )r-r.r/r0r1r3Fr-r.r/r5) r>r?r#rround expand_ratior@ use_residualr+ expand_1x1conv_3x3 reduce_1x1)rIr,r-r.r0r3expanded_channelsrJr!r"r?s0   z$MobileViTV2InvertedResidual.__init__rLcCs4|}||}||}||}|jr||S|Sr')r[r\r]rZ)rIrLresidualr!r!r"rMs    z#MobileViTV2InvertedResidual.forward)r rNrOrP__doc__rrr?rRrSrMrTr!r!rJr"rUs"!rUc sPeZdZ ddedededededdf fd d Zd ejdejfd d ZZ S)MobileViTV2MobileNetLayerrr,r-r.r0 num_stagesrNcsRtt|_t|D]}t||||dkr|ndd}|j||}qdS)Nrr)r-r.r0)r>r?r ModuleListlayerrangerUappend)rIr,r-r.r0rcirerJr!r"r?s    z"MobileViTV2MobileNetLayer.__init__rLcC|jD]}||}q|Sr're)rIrL layer_moduler!r!r"rM  z!MobileViTV2MobileNetLayer.forward)rr rNrOrPrrr?rRrSrMrTr!r!rJr"rbs rbcsDeZdZdZdededdffdd Zdejdejfd d Z Z S) MobileViTV2LinearSelfAttentionaq This layer applies a self-attention with linear complexity, as described in MobileViTV2 paper: https://arxiv.org/abs/2206.02680 Args: config (`MobileVitv2Config`): Model configuration object embed_dim (`int`): `input_channels` from an expected input of size :math:`(batch_size, input_channels, height, width)` r, embed_dimrNc s\tt||dd|ddddd|_tj|jd|_t|||ddddd|_||_dS)NrrTF)r,r-r.r2r/r4r5p) r>r?r+qkv_projrDropout attn_dropoutout_projro)rIr,rorJr!r"r?s*    z'MobileViTV2LinearSelfAttention.__init__ hidden_statesc Cs||}tj|d|j|jgdd\}}}tjjj|dd}||}||}tj|ddd}tjj || |}| |}|S)Nr)split_size_or_sectionsdimrxTrxkeepdim) rrrRsplitror functionalsoftmaxrtsumrelu expand_asru) rIrvqkvquerykeyrcontext_scorescontext_vectoroutr!r!r"rMs    z&MobileViTV2LinearSelfAttention.forwardr`r!r!rJr"rns rnc LeZdZ d dededededdf fdd Zd ejdejfd d Z Z S)MobileViTV2FFNr,roffn_latent_dim ffn_dropoutrNc sZtt|||dddddd|_t||_t|||dddddd|_t||_dS)NrTF)r,r-r.r/r0r2r4r5) r>r?r+conv1rrsdropout1conv2dropout2)rIr,rorrrJr!r"r? s.   zMobileViTV2FFN.__init__rvcCs,||}||}||}||}|Sr')rrrr)rIrvr!r!r"rM@s    zMobileViTV2FFN.forwardr rNrOrPrrfloatr?rRrSrMrTr!r!rJr"rs rc r)MobileViTV2TransformerLayerrr,rordropoutrNcsbttjd||jd|_t|||_tj|d|_ tjd||jd|_ t ||||j |_ dS)Nr num_groups num_channelsr:rp)r>r?r GroupNormlayer_norm_epslayernorm_beforern attentionrsrlayernorm_afterrrffn)rIr,rorrrJr!r"r?Is  z$MobileViTV2TransformerLayer.__init__rvcCs<||}||}||}||}||}||}|Sr')rrrr)rIrvlayernorm_1_outattention_output layer_outputr!r!r"rMWs    z#MobileViTV2TransformerLayer.forwardrrr!r!rJr"rHsrcDeZdZdedededdffdd Zdejdejfd d ZZ S) MobileViTV2Transformerr,n_layersd_modelrNcsft|j}||g|}dd|D}t|_t|D]}t||||d}|j|qdS)NcSsg|] }t|ddqS))r).0dr!r!r" lsz3MobileViTV2Transformer.__init__..)ror) r>r?ffn_multiplierrrdrerfrrg)rIr,rrrffn_dims block_idxtransformer_layerrJr!r"r?ds    zMobileViTV2Transformer.__init__rvcCrir'rj)rIrvrkr!r!r"rMurlzMobileViTV2Transformer.forwardrmr!r!rJr"rcsrcseZdZdZ   ddededededed ed ed d ffd d Zdejd e eje eefffddZ dejde eefd ejfddZ dejd ejfddZ Z S)MobileViTV2Layerz= MobileViTV2 layer: https://arxiv.org/abs/2206.02680 rrr,r-r. attn_unit_dim n_attn_blocksr3r0rNc st|j|_|j|_|}|dkr.t||||dkr|nd|dkr&|dndd|_|}nd|_t||||j|d|_ t|||dddd|_ t |||d|_ t jd||jd|_t|||dd dd|_dS) Nrr)r-r.r0r3)r-r.r/r1F)r-r.r/r4r5)rrrT)r>r? patch_size patch_width patch_heightrUdownsampling_layerr+conv_kernel_sizeconv_kxkconv_1x1r transformerrrr layernormconv_projection) rIr,r-r.rrr3r0 cnn_out_dimrJr!r"r?sN   zMobileViTV2Layer.__init__ feature_mapcCsT|j\}}}}tjj||j|jf|j|jfd}||||j|jd}|||ffS)N)r/r0ry)shaperr~unfoldrrreshape)rIr batch_sizer- img_height img_widthpatchesr!r!r" unfoldings   zMobileViTV2Layer.unfoldingr output_sizecCsH|j\}}}}|||||}tjj|||j|jf|j|jfd}|S)N)rr/r0)rrrr~foldrr)rIrrrin_dimr n_patchesrr!r!r"foldings  zMobileViTV2Layer.foldingrLcCs`|jr||}||}||}||\}}||}||}|||}||}|Sr')rrrrrrrr)rIrLrrr!r!r"rMs       zMobileViTV2Layer.forward)rrr)rNrOrPrarrr?rRrSrrrrMrTr!r!rJr"r{s2  &="  rc sPeZdZdeddffdd Z  d dejd ed edee e ffd d Z Z S)MobileViTV2Encoderr,rNcst||_t|_d|_d}}|jdkrd}d}n|jdkr%d}d}tt d|j dddddd }td|j dd }td |j dd }td |j dd }td |j dd } td|j dd } t |||ddd} |j | t |||ddd} |j | t |||t|jd|j dd |jdd} |j | |r|d9}t ||| t|jd|j dd |jd|d}|j ||r|d9}t || | t|jd|j dd |jd|d}|j |dS)NFrTrr @r)rrrir)r-r.r0rcrr)r-r.rr)r-r.rrr3)r>r?r,rrdregradient_checkpointing output_strider#r*width_multiplierrbrgrbase_attn_unit_dimsr)rIr,dilate_layer_4dilate_layer_5r3 layer_0_dim layer_1_dim layer_2_dim layer_3_dim layer_4_dim layer_5_dimlayer_1layer_2layer_3layer_4layer_5rJr!r"r?s        zMobileViTV2Encoder.__init__FTrvoutput_hidden_states return_dictcCsx|rdnd}t|jD]\}}|jr|jr||j|}n||}|r(||f}q |s6tdd||fDSt||dS)Nr!css|] }|dur|VqdSr'r!)rvr!r!r" Tsz-MobileViTV2Encoder.forward..)last_hidden_staterv) enumeraterertraining_gradient_checkpointing_func__call__tupler )rIrvrrall_hidden_statesrhrkr!r!r"rM?s    zMobileViTV2Encoder.forward)FT) rNrOrPrr?rRrSrQrrr rMrTr!r!rJr"rsT rc@sBeZdZdZeZdZdZdZde e j e j e j fddfdd ZdS) MobileViTV2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. mobilevitv2 pixel_valuesTmodulerNcCstt|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj r8|j j |jj ddSdS)zInitialize the weightsr)meanstdNg?) rErLinearrAweightdatanormal_r,initializer_ranger2zero_ LayerNormfill_)rIrr!r!r" _init_weightses   z(MobileViTV2PreTrainedModel._init_weights)rNrOrPrar config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrrrrArrr!r!r!r"rZs&raM This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MobileViTV2Config`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aF Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`MobileViTImageProcessor.__call__`] for details. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zYThe bare MobileViTV2 model outputting raw hidden-states without any specific head on top.c seZdZddedeffdd ZddZeee e e e de d  dd eejd eed eedeee ffddZZS)MobileViTV2ModelTr, expand_outputc sft|||_||_ttd|jdddddd}t||j|ddd d d |_ t ||_ | dS) Nrrrr)rrr rTr-r.r/r0r4r5) r>r?r,rr#r*rr+r conv_stemrencoder post_init)rIr,rrrJr!r"r?s"  zMobileViTV2Model.__init__cCsF|D]\}}|jj|}t|tr |jjD]}|j|qqdS)zPrunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrerErrr prune_heads)rIheads_to_prune layer_indexheadsmobilevitv2_layerrr!r!r" _prune_headss   zMobileViTV2Model._prune_headsvision) checkpoint output_typermodalityexpected_outputNrrrrc Cs|dur|n|jj}|dur|n|jj}|durtd||}|j|||d}|jr;|d}tj|ddgdd}n|d}d}|sV|durK||fn|f}||ddSt |||j d S) Nz You have to specify pixel_valuesrrrryFr{r)r pooler_outputrv) r,ruse_return_dictr@rrrrRrr rv) rIrrrembedding_outputencoder_outputsr pooled_outputoutputr!r!r"rMs0 zMobileViTV2Model.forward)T)NNN)rNrOrPrrQr?r rMOBILEVITV2_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErrRrSrrrMrTr!r!rJr"rs.   rz MobileViTV2 model with an image classification head on top (a linear layer on top of the pooled features), e.g. for ImageNet. cseZdZdeddffdd Zeeeee e e d    d de e jde ed e e jd e edeee ff d d ZZS)!MobileViTV2ForImageClassificationr,rNcs`t||j|_t||_td|jdd}|jdkr%tj||jdnt |_ | dS)Nrrrr) in_features out_features) r>r? num_labelsrrr#rrrIdentity classifierr)rIr,r.rJr!r"r?s    z*MobileViTV2ForImageClassification.__init__)rrrrrrlabelsrc Csb|dur|n|jj}|j|||d}|r|jn|d}||}d}|dur|jjdurP|jdkr6d|j_n|jdkrL|jtj ksG|jtj krLd|j_nd|j_|jjdkrnt } |jdkrh| | | }n+| ||}n%|jjdkrt } | |d|j|d}n|jjdkrt} | ||}|s|f|dd} |dur|f| S| St|||jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss). If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr regressionsingle_label_classificationmulti_label_classificationryr)losslogitsrv)r,rrrr$ problem_typer"dtyperRlongrrsqueezerviewrr rv) rIrrr%routputsrr*r)loss_fctrr!r!r"rMs>    "       z)MobileViTV2ForImageClassification.forwardNNNN)rNrOrPrr?rrr_IMAGE_CLASS_CHECKPOINTr r_IMAGE_CLASS_EXPECTED_OUTPUTrrRrSrQrrrMrTr!r!rJr"rs0 rcr) MobileViTV2ASPPPoolingr,r-r.rNc s4ttjdd|_t|||ddddd|_dS)Nr)rTrr)r>r?rAdaptiveAvgPool2d global_poolr+r)rIr,r-r.rJr!r"r?;s  zMobileViTV2ASPPPooling.__init__rLcCs:|jdd}||}||}tjj||ddd}|S)NrbilinearFsizemode align_corners)rr7rrr~ interpolate)rIrL spatial_sizer!r!r"rMJs   zMobileViTV2ASPPPooling.forwardrmr!r!rJr"r5:sr5c@eZdZdZdeddffdd Zdejdejfdd ZZ S) MobileViTV2ASPPzs ASPP module defined in DeepLab papers: https://arxiv.org/abs/1606.00915, https://arxiv.org/abs/1706.05587 r,rNcsttdjdd}|jtjdkrtdt |_ t ddd}|j ||j fd d jDt}|j |t d ddd|_tjjd |_dS) Nrrrr z"Expected 3 values for atrous_ratesrrrWc s g|] }td|ddqS)r r)r-r.r/r3r5)r+)rrater,r-r.r!r"rms z,MobileViTV2ASPP.__init__..rp)r>r?r#raspp_out_channelslen atrous_ratesr@rrdconvsr+rgextendr5projectrsaspp_dropout_probr)rIr,encoder_out_channels in_projection pool_layerrJrBr"r?Ws4      zMobileViTV2ASPP.__init__rLcCsDg}|jD] }|||qtj|dd}||}||}|S)Nrrz)rGrgrRcatrIr)rIrLpyramidconvpooled_featuresr!r!r"rMs   zMobileViTV2ASPP.forward rNrOrPrarr?rRrSrMrTr!r!rJr"r@Rs,r@cr?) MobileViTV2DeepLabV3zB DeepLabv3 architecture: https://arxiv.org/abs/1706.05587 r,rNc sBtt||_t|j|_t||j |j ddddd|_ dS)NrFT)r-r.r/r4r5r2) r>r?r@asppr Dropout2dclassifier_dropout_probrr+rDr"r$rIr,rJr!r"r?s   zMobileViTV2DeepLabV3.__init__rvcCs&||d}||}||}|S)Nry)rTrr$)rIrvrLr!r!r"rMs  zMobileViTV2DeepLabV3.forwardrRr!r!rJr"rSsrSzZ MobileViTV2 model with a semantic segmentation head on top, e.g. for Pascal VOC. cs~eZdZdeddffdd Zeeeee d    d de e j de e j d e e d e e deeeff d d ZZS)"MobileViTV2ForSemanticSegmentationr,rNcs8t||j|_t|dd|_t||_|dS)NF)r)r>r?r"rrrSsegmentation_headrrWrJr!r"r?s   z+MobileViTV2ForSemanticSegmentation.__init__)rrrr%rrc Cs|dur|n|jj}|dur|n|jj}|j|d|d}|r!|jn|d}||}d}|durU|jjdkr:tdtj j ||j ddddd } t |jj d } | | |}|sw|rc|f|dd} n |f|d d} |duru|f| S| St|||r|jdd Sddd S) a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> import requests >>> import torch >>> from PIL import Image >>> from transformers import AutoImageProcessor, MobileViTV2ForSemanticSegmentation >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("apple/mobilevitv2-1.0-imagenet1k-256") >>> model = MobileViTV2ForSemanticSegmentation.from_pretrained("apple/mobilevitv2-1.0-imagenet1k-256") >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```NTrrz/The number of labels should be greater than onerr8Fr9) ignore_indexr)r)r*rv attentions)r,rrrrvrYr"r@rr~r=rrsemantic_loss_ignore_indexr) rIrr%rrr0encoder_hidden_statesr*r)upsampled_logitsr1rr!r!r"rMsB'   z*MobileViTV2ForSemanticSegmentation.forwardr2)rNrOrPrr?rrrrrrrRrSrQrrrMrTr!r!rJr"rXs&  rX)rN)=ratypingrrrrRtorch.utils.checkpointrtorch.nnrrr activationsr modeling_outputsr r r rmodeling_utilsrutilsrrrrrconfiguration_mobilevitv2r get_loggerrNloggerrrrr3r4)MOBILEVITV2_PRETRAINED_MODEL_ARCHIVE_LISTrr#rr*Moduler+rUrbrnrrrrrrMOBILEVITV2_START_DOCSTRINGrrrr5r@rSrXr!r!r!r"sf      (A1?)rm  TQ=