o h{ @sVdZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZddlmZmZmZmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"m#Z#dd l$m%Z%d dl&m'Z'e"(e)Z*dZ+dZ,gdZ-dZ.dZ/dgZ0Gddde j1Z2Gddde j1Z3Gddde j1Z4Gddde j1Z5Gddde j1Z6Gddde j1Z7d@d"e j8d#e9d$e:d%e j8fd&d'Z;Gd(d)d)e j1ZGd.d/d/e j1Z?Gd0d1d1e j1Z@Gd2d3d3eZAd4ZBd5ZCd6ZDe d7eBGd8d9d9eAZEe d:eBGd;d<dd?d?eAe%ZGdS)Az PyTorch DINOv2 model.N)DictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) BackboneMixin) Dinov2Configrzfacebook/dinov2-base)riiz(facebook/dinov2-small-imagenet1k-1-layerztabby, tabby catcsjeZdZdZdeddffdd Zdejded edejfd d Z dd ejd e ejdejfddZ Z S)Dinov2EmbeddingszM Construct the CLS token, mask token, position and patch embeddings. configreturnNcs~tttdd|j|_ttd|j|_ t ||_ |j j }ttd|d|j|_ t|j|_||_dS)Nr)super__init__r Parametertorchrandn hidden_size cls_tokenzeros mask_tokenDinov2PatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropoutr)selfrr+ __class__`/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/dinov2/modeling_dinov2.pyr!Hs   zDinov2Embeddings.__init__ embeddingsheightwidthc CsV|jdd}|jjdd}||kr||kr|jS|jdddf}|jddddf}|jd}||jj}||jj}|d|d}}|dtt|tt||}|dddd}t j j |t |t|t |t|fdd d }t||jd kst||jdkrt d |dddddd|}tj|d|fdd S)a# This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. Source: https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174 rNrg?r bicubicF) scale_factormode align_cornerszHWidth or height does not match with the interpolated position embeddingsdim)shaper,r patch_sizereshapeintmathsqrtpermuter functional interpolatefloat ValueErrorviewr#cat unsqueeze) r0r5r6r7r+ num_positionsclass_pos_embedpatch_pos_embedr@r3r3r4interpolate_pos_encodingSs,    $"$z)Dinov2Embeddings.interpolate_pos_encoding pixel_valuesbool_masked_posc Cs|j\}}}}||}|dur"t|d|j|jd|}|j |dd}tj ||fdd}|| |||}| |}|S)Nr8rrr?) rAr*r#whererNr(todtyper&expandrMrRr/) r0rSrT batch_size_r6r7r5 cls_tokensr3r3r4forwardus  zDinov2Embeddings.forwardN) __name__ __module__ __qualname____doc__rr!r#TensorrDrRrr\ __classcell__r3r3r1r4rCs  *"rcs6eZdZdZfddZdejdejfddZZS)r)z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizestride)r r! image_sizerB num_channelsr% isinstance collectionsabcIterabler+rConv2d projection)r0rrfrBrgr%r+r1r3r4r!s  zDinov2PatchEmbeddings.__init__rSrcCsH|jd}||jkrtd|jd|d||ddd}|S)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r9)rArgrKrmflatten transpose)r0rSrgr5r3r3r4r\s  zDinov2PatchEmbeddings.forward) r^r_r`rar!r#rbr\rcr3r3r1r4r)s r)c sveZdZdeddffdd ZdejdejfddZ dd eejd e de e ejejfe ejffd d Z Z S)Dinov2SelfAttentionrrNcst|j|jdkr t|ds td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads rnbias)r r!r%num_attention_headshasattrrKrDattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer-attention_probs_dropout_probr/r0rr1r3r4r!s  zDinov2SelfAttention.__init__xcCs6|dd|j|jf}||}|ddddS)Nr8rr9rr )sizerurwrLrG)r0r new_x_shaper3r3r4transpose_for_scoress z(Dinov2SelfAttention.transpose_for_scoresF head_maskoutput_attentionsc Cs||}|||}|||}||}t||dd}|t|j }t j j |dd} | | } |durA| |} t| |} | dddd} | dd|jf} | | } |rj| | f} | S| f} | S)Nr8r>r?rr9rr )r{rr|r}r#matmulrprErFrwrrHsoftmaxr/rG contiguousrrxrL) r0 hidden_statesrrmixed_query_layer key_layer value_layer query_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr3r3r4r\s$      zDinov2SelfAttention.forwardNF)r^r_r`rr!r#rbrrboolrrr\rcr3r3r1r4rqsrqcsFeZdZdZdeddffdd Zdejdejdejfd d ZZ S) Dinov2SelfOutputz The residual connection is defined in Dinov2Layer instead of here (as is the case with other models), due to the layernorm applied before each block. rrNcs.tt|j|j|_t|j|_dSr]) r r!rryr%denser-r.r/rr1r3r4r!s zDinov2SelfOutput.__init__r input_tensorcCs||}||}|Sr])rr/)r0rrr3r3r4r\s  zDinov2SelfOutput.forward) r^r_r`rarr!r#rbr\rcr3r3r1r4rs$rc s~eZdZdeddffdd ZdeeddfddZ  dd ej d e ej d e de e ej ej fe ej ffd dZZS)Dinov2AttentionrrNcs*tt||_t||_t|_dSr])r r!rq attentionroutputset pruned_headsrr1r3r4r!s    zDinov2Attention.__init__headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr?)lenrrrurwrrr{r|r}rrrxunion)r0rindexr3r3r4 prune_headss zDinov2Attention.prune_headsFrrrcCs4||||}||d|}|f|dd}|S)Nrr)rr)r0rrr self_outputsattention_outputrr3r3r4r\szDinov2Attention.forwardr)r^r_r`rr!rrDrr#rbrrrrr\rcr3r3r1r4rsrc4eZdZdfdd ZdejdejfddZZS) Dinov2LayerScalerNcs(tt|jt|j|_dSr]) r r!rr"layerscale_valuer#onesr%lambda1rr1r3r4r!$s zDinov2LayerScale.__init__ hidden_statecCs ||jSr])rr0rr3r3r4r\(s zDinov2LayerScale.forwardrNr^r_r`r!r#rbr\rcr3r3r1r4r#srFinput drop_probtrainingrcCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)rWdevice)rAndimr#randrWrfloor_div)rrr keep_probrA random_tensorrr3r3r4 drop_path-s rcsTeZdZdZd deeddffdd Zdejdejfdd Z de fd d Z Z S) Dinov2DropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nrrcst||_dSr])r r!r)r0rr1r3r4r!Es  zDinov2DropPath.__init__rcCst||j|jSr])rrr)r0rr3r3r4r\IszDinov2DropPath.forwardcCs d|jS)Nzp={})formatrr0r3r3r4 extra_reprLs zDinov2DropPath.extra_reprr]) r^r_r`rarrJr!r#rbr\strrrcr3r3r1r4rBs rcr) Dinov2MLPrNcsnt|j}}t|j|j}tj||dd|_t|j t r(t |j |_ n|j |_ tj||dd|_ dS)NTrs)r r!r%rD mlp_ratiorryfc1rh hidden_actrr activationfc2r0r in_features out_featureshidden_featuresr1r3r4r!Qs   zDinov2MLP.__init__rcCs"||}||}||}|Sr])rrrrr3r3r4r\\s   zDinov2MLP.forwardrrr3r3r1r4rPs rcr) Dinov2SwiGLUFFNrNcslt|j}}t|j|j}t|ddddd}tj|d|dd|_tj||dd|_dS)Nr9r Trs) r r!r%rDrrry weights_in weights_outrr1r3r4r!ds  zDinov2SwiGLUFFN.__init__rcCs6||}|jddd\}}tj||}||S)Nr9r8r?)rchunkrrHsilur)r0rx1x2hiddenr3r3r4r\ms  zDinov2SwiGLUFFN.forwardrrr3r3r1r4rcs rc sleZdZdZdeddffdd Z  d dejd eejd e de e ejejfe ejffd d Z Z S) Dinov2LayerzCThis corresponds to the Block class in the original implementation.rrNcsttj|j|jd|_t||_t ||_ |j dkr#t |j nt |_tj|j|jd|_|jr;t||_nt||_t ||_|j dkrRt |j |_dSt |_dS)Nepsr)r r!r LayerNormr%layer_norm_epsnorm1rrr layer_scale1drop_path_raterIdentity drop_path1norm2use_swiglu_ffnrmlpr layer_scale2 drop_path2rr1r3r4r!ws      (zDinov2Layer.__init__FrrrcCsp|j||||d}|d}||}|dd}||}||}||}||}||}|f|}|S)N)rrr)rrrrrr)r0rrrself_attention_outputsrr layer_outputr3r3r4r\s      zDinov2Layer.forwardr)r^r_r`rarr!r#rbrrrrr\rcr3r3r1r4rtsrcsbeZdZdeddffdd Z    ddejd eejd ed ed ede e e ff d dZ Z S) Dinov2EncoderrrNcs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr3)r.0rZrr3r4 sz*Dinov2Encoder.__init__..F) r r!rr ModuleListrangenum_hidden_layerslayergradient_checkpointingrr1rr4r!s   zDinov2Encoder.__init__FTrrroutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD]8\}} |r||f}|dur$||nd} |jr6|jr6|| j|| |} n| || |} | d}|rI|| df}q|rQ||f}|s_tdd|||fDSt|||dS)Nr3rrcss|] }|dur|VqdSr]r3)rvr3r3r4 sz(Dinov2Encoder.forward..)last_hidden_stater attentions) enumeraterrr_gradient_checkpointing_func__call__tupler) r0rrrrrall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputsr3r3r4r\s6      zDinov2Encoder.forward)NFFT)r^r_r`rr!r#rbrrrrrr\rcr3r3r1r4rs&  rc@sBeZdZdZeZdZdZdZde e j e j e j fddfdd ZdS) Dinov2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. dinov2rSTmodulerNcCst|tjtjfr0tjj|jjt j d|j j d|jj |j_|jdur.|jjdSdSt|tjrE|jj|jjddSt|tr|tjj|jjt j d|j j d|jj |j_tjj|jjt j d|j j d|jj |j_dSdS)zInitialize the weightsr)meanstdNg?)rhrryrlinit trunc_normal_weightdatarVr#float32rinitializer_rangerWrtzero_rfill_rr,r&)r0rr3r3r4 _init_weightss8        z#Dinov2PreTrainedModel._init_weights)r^r_r`rar config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrrryrlrr r3r3r3r4rs&raH This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Dinov2Config`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a4 Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`BitImageProcessor.preprocess`] for details. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Only relevant for pre-training. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. aM Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`BitImageProcessor.preprocess`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare DINOv2 Model transformer outputting raw hidden-states without any specific head on top.cseZdZdeffdd ZdefddZdeee efddfd d Z e e e eeed ed       dd eejdeejdeejdeedeedeedeeeffddZZS) Dinov2ModelrcsFt|||_t||_t||_tj|j |j d|_ | dS)Nr) r r!rrr5rencoderrrr%r layernorm post_initrr1r3r4r!Cs    zDinov2Model.__init__rcC|jjSr]r5r*rr3r3r4get_input_embeddingsOz Dinov2Model.get_input_embeddingsheads_to_pruneNcCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r0rrrr3r3r4 _prune_headsRszDinov2Model._prune_headsvision) checkpoint output_typer modalityexpected_outputrSrTrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|||jj}|j||d}|j|||||d}|d} | | } | dddddf} |sa| | f} | |ddSt | | |j |j dS)Nz You have to specify pixel_values)rTrrrrrr)r pooler_outputrr) rrruse_return_dictrK get_head_maskrr5rrrrr) r0rSrTrrrrembedding_outputencoder_outputssequence_output pooled_output head_outputsr3r3r4r\Zs6 zDinov2Model.forwardNNNNNN)r^r_r`rr!r)rrrDrrrDINOV2_BASE_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr#rbrrrr\rcr3r3r1r4r>sB   rz Dinov2 Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. cseZdZdeddffdd Zeeeee e e d      dde e jde e jd e e jd e ed e ed e edeee ffd dZZS)Dinov2ForImageClassificationrrNcsRt||j|_t||_|jdkrt|jd|jnt|_ | dS)Nrr9) r r! num_labelsrrrryr%r classifierrrr1r3r4r!s  $ z%Dinov2ForImageClassification.__init__)rrr rrSrlabelsrrrcCs|dur|n|jj}|j|||||d}|d}|dddf} |ddddf} tj| | jddgdd} || } d} |dur|| j}|jj durr|j dkrXd|j_ n|j dkrn|j tj ksi|j tj krnd|j_ nd|j_ |jj dkrt}|j dkr|| |} n+|| |} n%|jj dkrt}|| d |j |d } n|jj dkrt}|| |} |s| f|d d}| dur| f|S|St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrrr? regressionsingle_label_classificationmulti_label_classificationr8r9)losslogitsrr)rr!rr#rMrr/rVr problem_typer.rWlongrDr squeezer rLr rrr)r0rSrr0rrrrr%r& patch_tokens linear_inputr5r4loss_fctrr3r3r4r\sT     "       z$Dinov2ForImageClassification.forwardr()r^r_r`rr!rDINOV2_INPUTS_DOCSTRINGr_IMAGE_CLASS_CHECKPOINTrr+_IMAGE_CLASS_EXPECTED_OUTPUTrr#rbrrrr\rcr3r3r1r4r-s< r-zO Dinov2 backbone, to be used with frameworks like DETR and MaskFormer. c sreZdZfddZdefddZeeee e d   dde j d e ed e ed e ede f d d ZZS)Dinov2BackbonecsjttfddtjdD|_t|_t|_ t j j j d|_|dS)Ncsg|]}jqSr3)r%rrr3r4rsz+Dinov2Backbone.__init__..rr)r r!_init_backbonerr num_featuresrr5rrrrr%rrrrr1rr4r!s     zDinov2Backbone.__init__rcCrr]rrr3r3r4r rz#Dinov2Backbone.get_input_embeddings)rr NrSrrrcCsb|dur|n|jj}|dur|n|jj}|dur|n|jj}||}|j|d||d}|r1|jn|d}d}t|j|D]F\} } | |j vr|jj rO| | } |jj r~| ddddf} |j \} } } }|jj}| | | |||d} | dddd } || f7}q=|s|r|f|dd}|S|f|d d}|St||r|jnd|r|jd Sdd S) a7 Returns: Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("facebook/dinov2-base") >>> model = AutoBackbone.from_pretrained( ... "facebook/dinov2-base", out_features=["stage2", "stage5", "stage8", "stage11"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 16, 16] ```NT)rrrrr3r8rr r9) feature_mapsrr)rr!rrr5rrzip stage_namesrapply_layernormrreshape_hidden_statesrArBrCrGrrr)r0rSrrrr#rrrBstagerrYrZr6r7rBrr3r3r4r\sF#     zDinov2Backbone.forward)NNN)r^r_r`r!r)rrr<rrr+r#rbrrr\rcr3r3r1r4r?s&   r?)rF)Hracollections.abcrirEtypingrrrrrrr#torch.utils.checkpointrtorch.nnr r r activationsr modeling_outputsrrrrmodeling_utilsr pytorch_utilsrrutilsrrrrrutils.backbone_utilsrconfiguration_dinov2r get_loggerr^loggerr+r*r,r=r>$DINOV2_PRETRAINED_MODEL_ARCHIVE_LISTModulerr)rqrrrrbrJrrrrrrrrDINOV2_START_DOCSTRINGr)r<rr-r?r3r3r3r4sl       G"=' 53& R]