o h=@sdZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZddlmZmZmZmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"m#Z#d d l$m%Z%e"&e'Z(dZ)dZ*gdZ+dZ,dZ-dgZ.Gddde j/Z0Gddde j/Z1Gddde j/Z2Gddde j/Z3Gddde j/Z4Gddde j/Z5Gdd d e j/Z6Gd!d"d"e j/Z7Gd#d$d$e j/Z8Gd%d&d&eZ9d'Z:d(Z;e d)e:Gd*d+d+e9Ze d1e:Gd2d3d3e9Z?dS)4z PyTorch ViT model.N)DictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedImageModelingOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) ViTConfigrz!google/vit-base-patch16-224-in21k)rizgoogle/vit-base-patch16-224z Egyptian catc sxeZdZdZddededdffdd Zd ejd e d e dejfd d Z  ddejde ej dedejfddZ ZS) ViTEmbeddingszb Construct the CLS token, position and patch embeddings. Optionally, also the mask token. Fconfiguse_mask_tokenreturnNcstttdd|j|_|rttdd|jnd|_ t ||_ |j j }ttd|d|j|_ t|j|_||_dS)Nr)super__init__r Parametertorchrandn hidden_size cls_tokenzeros mask_tokenViTPatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropoutr)selfrrr, __class__Z/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/vit/modeling_vit.pyr"Gs    zViTEmbeddings.__init__ embeddingsheightwidthc CsJ|jdd}|jjdd}||kr||kr|jS|jdddf}|jddddf}|jd}||jj} ||jj} | d| d} } |dtt|tt||}|dddd}t j j || t|| t|fdd d }t| |jd krt| |jdksJ|dddd dd|}t j|d|fdd S) a# This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. Source: https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174 rNrg?r bicubicF) scale_factormode align_cornersdim)shaper-r patch_sizereshapeintmathsqrtpermuter functional interpolateviewr$cat unsqueeze) r1r6r7r8r, num_positionsclass_pos_embedpatch_pos_embedrAh0w0r4r4r5interpolate_pos_encodingRs*    $(z&ViTEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posrSc Cs|j\}}}}|j||d}|dur1|jd} |j|| d} |d| } |d| | | }|j|dd} tj| |fdd}|rN|| |||}n||j }| |}|S)N)rSrr9?r@) rBr+r)expandrMtype_asr'r$rLrSr-r0) r1rTrUrS batch_size num_channelsr7r8r6 seq_length mask_tokensmask cls_tokensr4r4r5forwardss   zViTEmbeddings.forwardFNF)__name__ __module__ __qualname____doc__rboolr"r$TensorrErSr BoolTensorr_ __classcell__r4r4r2r5rBs $rcs<eZdZdZfddZd dejdedejfdd ZZ S) r*z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizestride)r!r" image_sizerCrZr& isinstance collectionsabcIterabler,rConv2d projection)r1rrlrCrZr&r,r2r4r5r"s  zViTPatchEmbeddings.__init__FrTrSr c Cs|j\}}}}||jkrtd|jd|d|s?||jdks(||jdkr?td|d|d|jdd|jdd ||d dd }|S) NzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rrzInput image size (*z) doesn't match model (z).r:)rBrZ ValueErrorrlrrflatten transpose)r1rTrSrYrZr7r8r6r4r4r5r_s( zViTPatchEmbeddings.forwardr`) rbrcrdrer"r$rgrfr_rir4r4r2r5r*s $r*c sveZdZdeddffdd ZdejdejfddZ dd eejd e de e ejejfe ejffd d Z Z S)ViTSelfAttentionrr Ncst|j|jdkr t|ds td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads rs)bias)r!r"r&num_attention_headshasattrrurEattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer.attention_probs_dropout_probr0r1rr2r4r5r"s  zViTSelfAttention.__init__xcCs6|dd|j|jf}||}|ddddS)Nr9rr:rr )sizer{r}rKrH)r1r new_x_shaper4r4r5transpose_for_scoress z%ViTSelfAttention.transpose_for_scoresF head_maskoutput_attentionsc Cs||}|||}|||}||}t||dd}|t|j }t j j |dd} | | } |durA| |} t| |} | dddd} | dd|jf} | | } |rj| | f} | S| f} | S)Nr9r?r@rr:rr )rrrrr$matmulrwrFrGr}rrIsoftmaxr0rH contiguousrr~rK) r1 hidden_statesrrmixed_query_layer key_layer value_layer query_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr4r4r5r_s$      zViTSelfAttention.forwardra)rbrcrdrr"r$rgrrrfrrr_rir4r4r2r5rxsrxcsFeZdZdZdeddffdd Zdejdejdejfd d ZZ S) ViTSelfOutputz The residual connection is defined in ViTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rr Ncs.tt|j|j|_t|j|_dSN) r!r"rrr&denser.r/r0rr2r4r5r" zViTSelfOutput.__init__r input_tensorcC||}||}|Srrr0r1rrr4r4r5r_  zViTSelfOutput.forward) rbrcrdrerr"r$rgr_rir4r4r2r5rs$rc s~eZdZdeddffdd ZdeeddfddZ  dd ej d e ej d e de e ej ej fe ej ffd dZZS) ViTAttentionrr Ncs*tt||_t||_t|_dSr)r!r"rx attentionroutputset pruned_headsrr2r4r5r"s    zViTAttention.__init__headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr@)lenrrr{r}rrrrrrrr~union)r1rindexr4r4r5 prune_headss zViTAttention.prune_headsFrrrcCs4||||}||d|}|f|dd}|S)Nrr)rr)r1rrr self_outputsattention_outputrr4r4r5r_ szViTAttention.forwardra)rbrcrdrr"rrErr$rgrrfrrr_rir4r4r2r5rsrcs<eZdZdeddffdd ZdejdejfddZZS) ViTIntermediaterr NcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r!r"rrr&intermediate_sizerrm hidden_actstrr intermediate_act_fnrr2r4r5r"/s   zViTIntermediate.__init__rcCrr)rr)r1rr4r4r5r_7rzViTIntermediate.forward rbrcrdrr"r$rgr_rir4r4r2r5r.srcsBeZdZdeddffdd Zdejdejdejfdd ZZS) ViTOutputrr Ncs.tt|j|j|_t|j|_ dSr) r!r"rrrr&rr.r/r0rr2r4r5r"?rzViTOutput.__init__rrcCs ||}||}||}|Srrrr4r4r5r_Ds  zViTOutput.forwardrr4r4r2r5r>s$rc sleZdZdZdeddffdd Z  d dejd eejd e de e ejejfe ejffd d Z Z S)ViTLayerz?This corresponds to the Block class in the timm implementation.rr Ncsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nreps)r!r"chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr&layer_norm_epslayernorm_beforelayernorm_afterrr2r4r5r"Ps    zViTLayer.__init__FrrrcCs`|j||||d}|d}|dd}||}||}||}|||}|f|}|S)N)rrr)rrrrr)r1rrrself_attention_outputsrr layer_outputr4r4r5r_Zs     zViTLayer.forwardra)rbrcrdrerr"r$rgrrfrrr_rir4r4r2r5rMs rcsbeZdZdeddffdd Z    ddejd eejd ed ed ede e e ff d dZ Z S) ViTEncoderrr Ncs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr4)r).0_rr4r5 {sz'ViTEncoder.__init__..F) r!r"rr ModuleListrangenum_hidden_layerslayergradient_checkpointingrr2rr5r"xs   zViTEncoder.__init__FTrrroutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD]8\}} |r||f}|dur$||nd} |jr6|jr6|| j|| |} n| || |} | d}|rI|| df}q|rQ||f}|s_tdd|||fDSt|||dS)Nr4rrcss|] }|dur|VqdSrr4)rvr4r4r5 sz%ViTEncoder.forward..)last_hidden_stater attentions) enumeraterrtraining_gradient_checkpointing_func__call__tupler) r1rrrrrall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputsr4r4r5r_~s6      zViTEncoder.forward)NFFT)rbrcrdrr"r$rgrrfrrrr_rir4r4r2r5rws&  rc@sJeZdZdZeZdZdZdZddgZ de e j e j e jfdd fd d Zd S) ViTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. vitrTTrrmoduler NcCst|tjtjfr0tjj|jjt j d|j j d|jj |j_|jdur.|jjdSdSt|tjrE|jj|jjddSt|tr|tjj|jjt j d|j j d|jj |j_tjj|jjt j d|j j d|jj |j_dSdS)zInitialize the weightsg)meanstdNrV)rmrrrqinit trunc_normal_weightdatator$float32rinitializer_rangedtyperzzero_rfill_rr-r')r1rr4r4r5 _init_weightss8        z ViTPreTrainedModel._init_weights)rbrcrdrer config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrrrrqrrr4r4r4r5rs&raE This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ViTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViTImageProcessor.__call__`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z]The bare ViT Model transformer outputting raw hidden-states without any specific head on top.cseZdZddededeffdd Zdefd d Zd ee e e fdd fd dZ e e eeeeded ddeejdeejdeejdeedeedeedeedeeeffddZZS)ViTModelTFradd_pooling_layerrcs\t|||_t||d|_t||_tj|j |j d|_ |r%t |nd|_ |dS)N)rr)r!r"rrr6rencoderrrr&r layernorm ViTPoolerpooler post_init)r1rrrr2r4r5r"s   zViTModel.__init__r cCs|jjSr)r6r+)r1r4r4r5get_input_embeddingsszViTModel.get_input_embeddingsheads_to_pruneNcCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r1rrrr4r4r5 _prune_heads szViTModel._prune_headsvision) checkpoint output_typermodalityexpected_outputrTrUrrrrSrcCs |dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|||jj}|jjj j j }|j |kr?| |}|j|||d} |j | ||||d} | d} || } |jdurd|| nd} |s{| durp| | fn| f} | | ddSt| | | j| jdS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rUrS)rrrrrr)r pooler_outputrr)rrruse_return_dictru get_head_maskrr6r+rrrrrrrrrrr)r1rTrUrrrrSrexpected_dtypeembedding_outputencoder_outputssequence_output pooled_output head_outputsr4r4r5r_s@   zViTModel.forward)TFNNNNNNN)rbrcrdrrfr"r*rrrErrrVIT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr$rgrhrrr_rir4r4r2r5rsH   rcs*eZdZdeffdd ZddZZS)rrcs*tt|j|j|_t|_dSr)r!r"rrr&rTanh activationrr2r4r5r"Ys zViTPooler.__init__cCs(|dddf}||}||}|S)Nr)rr)r1rfirst_token_tensorrr4r4r5r_^s  zViTPooler.forward)rbrcrdrr"r_rir4r4r2r5rXsraVViT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://arxiv.org/abs/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). cseZdZdeddffdd Zeeeee d       dde e j de e j d e e j d e ed e ed e ed e edeeeffddZZS)ViTForMaskedImageModelingrr NcsXt|t|ddd|_ttj|j|jd|j ddt |j|_ | dS)NFT)rrr:r) in_channels out_channelsrj) r!r"rrr Sequentialrqr&encoder_striderZ PixelShuffledecoderrrr2r4r5r"ts   z"ViTForMaskedImageModeling.__init__)rrrTrUrrrrSrc Cs|dur|n|jj}|dur%|jj|jjkr%td|jjd|jjd|j|||||||d}|d} | ddddf} | j\} } } t| d} }| dd d | | | |} | | }d}|dur|jj |jj}| d ||}| |jjd |jjd d}tjj||d d }|||d |jj}|s|f|dd}|dur|f|S|St|||j|jdS)a= bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, ViTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k") >>> model = ViTForMaskedImageModeling.from_pretrained("google/vit-base-patch16-224-in21k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```NzWhen `bool_masked_pos` is provided, `patch_size` must be equal to `encoder_stride` to ensure that the reconstructed image has the same dimensions as the input. Got `patch_size` = z and `encoder_stride` = rs)rUrrrrSrrrg?r:r9none) reductiongh㈵>)lossreconstructionrr)rrrCrrurrBrFfloorrHrDrrlrepeat_interleaverMrrrIl1_losssumrZrrr)r1rTrUrrrrSrrrrYsequence_lengthrZr7r8reconstructed_pixel_valuesmasked_im_lossrr]reconstruction_lossrr4r4r5r_sX)     z!ViTForMaskedImageModeling.forwardr )rbrcrdrr"rr rrr rr$rgrhrfrrr_rir4r4r2r5rgs8   ra ViT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. cseZdZdeddffdd Zeeeee e e d       dde e jde e jd e e jd e ed e ed e ed e edeee ffddZZS)ViTForImageClassificationrr NcsRt||j|_t|dd|_|jdkrt|j|jnt|_ | dS)NF)rr) r!r" num_labelsrrrrr&Identity classifierrrr2r4r5r"s $ z"ViTForImageClassification.__init__)rrrrrTrlabelsrrrSrcCs|dur|n|jj}|j||||||d}|d} || dddddf} d} |dur|| j}|jjdur]|jdkrCd|j_n|jdkrY|jt j ksT|jt j krYd|j_nd|j_|jjdkr{t } |jdkru| | | } n+| | |} n%|jjdkrt} | | d|j|d} n|jjdkrt} | | |} |s| f|dd} | dur| f| S| St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrrSrrr regressionsingle_label_classificationmulti_label_classificationr9)rlogitsrr)rrrr(rdevice problem_typer&rr$longrEr squeezer rKr rrr)r1rTrr)rrrSrrrr-rloss_fctrr4r4r5r_sP     "       z!ViTForImageClassification.forwardr )rbrcrdrr"rr r_IMAGE_CLASS_CHECKPOINTrr _IMAGE_CLASS_EXPECTED_OUTPUTrr$rgrfrrr_rir4r4r2r5r%sB  r%)@recollections.abcrnrFtypingrrrrrrr$torch.utils.checkpointrtorch.nnr r r activationsr modeling_outputsrrrrmodeling_utilsr pytorch_utilsrrutilsrrrrrconfiguration_vitr get_loggerrbloggerr r rr3r4!VIT_PRETRAINED_MODEL_ARCHIVE_LISTModulerr*rxrrrrrrrVIT_START_DOCSTRINGr rrrr%r4r4r4r5sd      P'<'*3' _  r