o h @sdZddlZddlZddlmZmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZmZddlmZdd lmZmZmZmZdd lmZd d lmZeeZd Z dgZ!Gddde j"Z#ddZ$ddZ%Gddde j"Z&deZdZdZfddZddZdejdejfdd ZZ S) VitDetEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_ ||_||_||_ |j r]|d}t td||j|_nd|_t j||||d|_dS)Nrr) kernel_sizestride)super__init__pretrain_image_size patch_size num_channels hidden_size isinstance collectionsabcIterable image_size num_patches use_absolute_position_embeddingsr Parametertorchzerosposition_embeddingsConv2d projection)selfconfigr!rrrr" num_positions __class__`/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/vitdet/modeling_vitdet.pyr9s  zVitDetEmbeddings.__init__cCs|r |ddddf}|jd}tt|}|||kr"td||ks*||krJtjj|d||d dddd||fdd d }| ddddS|d||dS) a Calculate absolute positional embeddings. If needed, resize embeddings and remove cls_token dimension for the original embeddings. Args: abs_pos_embeddings (`torch.Tensor`): Absolute positional embeddings with (1, num_position, num_channels). has_cls_token (`bool`): If true, has 1 embedding in abs_pos_embeddings for cls token. height (`int`): Height of input image tokens. width (`int`): Width of input image tokens. Returns: Absolute positional embeddings after processing with shape (1, height, width, num_channels) Nrz5Absolute position embeddings must be a square number.rrbicubicF)sizemode align_corners) shapeintmathsqrt ValueErrorr functional interpolatereshapepermute)r*abs_pos_embeddings has_cls_tokenheightwidth num_positionr4new_abs_pos_embeddingsr/r/r0get_absolute_positionsOs  z'VitDetEmbeddings.get_absolute_positions pixel_valuesreturncCs|jd}||jkrtd|jd|d||}|jdurA|dddd}|||jd|jd|jd}|dddd}|S) NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rr2rT)r7rr;r)r'r?rF)r*rGr embeddingsr/r/r0forwardts"    zVitDetEmbeddings.forward) __name__ __module__ __qualname____doc__rrFr%TensorrK __classcell__r/r/r-r0r3s  %rcCstdt||d}|jd|kr3tjj|d|jddddd|dd}|d|dd}n|}t |dddft||d}t |dddft||d}|||dt||d}|| S) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (`int`): Size of query q. k_size (`int`): Size of key k. rel_pos (`torch.Tensor`): Relative position embeddings (num_embeddings, num_channels). Returns: Extracted positional embeddings according to relative positions. r2rrr1linear)r4r5N?) r8maxr7rr<r=r>r?r%arangelong)q_sizek_sizerel_pos max_rel_distrel_pos_resizedq_coordsk_coordsrelative_coordsr/r/r0 get_rel_poss$$ r_cCs|\}}|\}} t|||} t|| |} |j\} } }|| |||}td|| } td|| }|| |||| | dddddddddf|dddddddddf| |||| }|S)a Calculate decomposed Relative Positional Embeddings as introduced in [MViT2](https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py). Args: attn (`torch.Tensor`): Attention map. queries (`torch.Tensor`): Query q in the attention layer with shape (batch_size, queries_height * queries_width, num_channels). rel_pos_h (`torch.Tensor`): Relative position embeddings (Lh, num_channels) for height axis. rel_pos_w (`torch.Tensor`): Relative position embeddings (Lw, num_channels) for width axis. q_size (`Tuple[int]`): Spatial sequence size of query q with (queries_height, queries_width). k_size (`Tuple[int]`]): Spatial sequence size of key k with (keys_height, keys_width). Returns: attn (Tensor): attention map with added relative positional embeddings. zbhwc,hkc->bhwkzbhwc,wkc->bhwkN)r_r7r>r%einsumview)attnqueries rel_pos_h rel_pos_wrWrXqueries_height queries_width keys_height keys_widthrelative_heightrelative_width batch_size_dimr_qrelative_weightr/r/r0!add_decomposed_relative_positionss      rqcs,eZdZdZdfdd Zd ddZZS) VitDetAttentionz=Multi-head Attention block with relative position embeddings.Ncst|j}|j}||_||}|d|_tj||d|jd|_ t|||_ |j |_ |j rSt t d|dd||_t t d|dd||_dSdS)z Args: config (`VitDetConfig`): Model configuration. input_size (`Tuple[int]`, *optional*): Input resolution, only required in case relative position embeddings are added. grbiasr2rrN)rrrnum_attention_heads num_headsscalerLinearqkv_biasqkvproj use_relative_position_embeddingsr$r%r&rdre)r*r+ input_sizernrvhead_dimr-r/r0rs   $zVitDetAttention.__init__FcCs&|j\}}}}|||||d|jdddddd}|d||j||dd\}} } ||j| dd} |jrMt | ||j |j ||f||f} | j dd} | | }| ||j||d}|ddddd}||||d}||}|r| ||j| jd| jd} || f} | S|f} | S) Nrr1r2rr)rn)r7rzr>rvr?unbindrw transposer|rqrdresoftmaxrar{)r* hidden_stateoutput_attentionsrlrBrCrmrzrckeysvaluesattention_scoresattention_probsoutputsr/r/r0rKs,,&  zVitDetAttention.forwardN)FrLrMrNrOrrKrQr/r/r-r0rrsrrFinput drop_probtrainingrHcCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)dtypedevice)r7ndimr%randrrfloor_div)rrr keep_probr7 random_tensoroutputr/r/r0 drop_paths rcsTeZdZdZd deeddffdd Zdejdejfdd Z de fd d Z Z S) VitDetDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).NrrHcst||_dSr)rrr)r*rr-r/r0r.s  zVitDetDropPath.__init__ hidden_statescCst||j|jSr)rrr)r*rr/r/r0rK2szVitDetDropPath.forwardcCs d|jS)Nzp={})formatrr*r/r/r0 extra_repr5s zVitDetDropPath.extra_reprr) rLrMrNrOrfloatrr%rPrKstrrrQr/r/r-r0r+s rcs*eZdZdZdfdd ZddZZS)VitDetLayerNormaL A LayerNorm variant, popularized by Transformers, that performs point-wise mean and variance normalization over the channel dimension for inputs that have shape (batch_size, channels, height, width). https://github.com/facebookresearch/ConvNeXt/blob/d1fa8f6fef0a165b27399986cc2bdacc92777e40/models/convnext.py#L119 ư>cs@ttt||_tt||_||_ |f|_ dSr) rrrr$r%onesweightr&rtepsnormalized_shape)r*rrr-r/r0r@s  zVitDetLayerNorm.__init__cCsn|jddd}||djddd}||t||j}|jddddf||jddddf}|S)NrT)keepdimr2)meanpowr%r:rrrt)r*xusr/r/r0rKGs ,zVitDetLayerNorm.forward)rrr/r/r-r0r9srcs(eZdZdZfddZddZZS)VitDetResBottleneckBlockz The standard bottleneck residual block without the last activation layer. It contains 3 conv layers with kernels 1x1, 3x3, 1x1. csttj||ddd|_t||_t|j|_ tj||dddd|_ t||_ t|j|_ tj||ddd|_ t||_dS)ar Args: config (`VitDetConfig`): Model configuration. in_channels (`int`): Number of input channels. out_channels (`int`): Number of output channels. bottleneck_channels (`int`): Number of output channels for the 3x3 "bottleneck" conv layers. rFrsr)paddingrtN)rrrr(conv1rnorm1r hidden_actact1conv2norm2act2conv3norm3)r*r+ in_channels out_channelsbottleneck_channelsr-r/r0rUs     z!VitDetResBottleneckBlock.__init__cCs&|}|D]}||}q||}|Sr)children)r*routlayerr/r/r0rKms   z VitDetResBottleneckBlock.forwardrr/r/r-r0rOs rcs@eZdZdededdffdd Zdejdejfdd ZZS) VitDetMlp in_featureshidden_featuresrHNcsDtt|||_t|j|_t|||_t |j |_ dSr) rrrrxfc1r ractfc2Dropout dropout_probdrop)r*r+rrr-r/r0rws  zVitDetMlp.__init__rcCs6||}||}||}||}||}|Sr)rrrr)r*rr/r/r0rK~s     zVitDetMlp.forward) rLrMrNr8rr%rPrKrQr/r/r-r0rvsrc Cs|j\}}}}||||}||||}|dks|dkr,tj|ddd|d|f}||||}} |||||| |||}|ddddddd|||} | || ffS)a  Partition into non-overlapping windows with padding if needed. Args: hidden_state (`torch.Tensor`): Input tokens with [batch_size, height, width, num_channels]. window_size (`int`): Window size. Returns: `tuple(torch.FloatTensor)` comprising various elements: - windows: windows after partition with [batch_size * num_windows, window_size, window_size, num_channels]. - (patch_height, patch_width): padded height and width before partition rrrr2rr1)r7rr<padrar? contiguous) r window_sizerlrBrCr pad_height pad_width patch_height patch_widthwindowsr/r/r0window_partitions$ rc Cs|\}}|\}}|jd||||}||||||||d} | dddddd|||d} ||ks=||krO| ddd|d|ddf} | S) a@ Window unpartition into original sequences and removing padding. Args: windows (`torch.Tensor`): Input tokens with [batch_size * num_windows, window_size, window_size, num_channels]. window_size (`int`): Window size. pad_height_width (`Tuple[int]`): Padded height and width (patch_height, patch_width). height_width (`Tuple[int]`): Original height and width before padding. Returns: hidden_state: unpartitioned sequences with [batch_size, height, width, num_channels]. rr1rrr2rrN)r7rar?r) rrpad_height_width height_widthrrrBrCrlrr/r/r0window_unpartitions$$rc s|eZdZdZ ddededededd f fd d Z dd e j d e e j dede e e j e j fe e j ffddZZS) VitDetLayerzCThis corresponds to the Block class in the original implementation.rFr+drop_path_rateruse_residual_blockrHNcst|j}|j|j|j|jf}tj||jd|_t ||dkr%|n||fd|_ |dkr4t |nt |_ tj||jd|_t||t||jd|_||_||_|jret||||dd|_dSdS)N)rr)r}r)r+rrr2)r+rrr)rrrr!rr LayerNormlayer_norm_epsrrr attentionrIdentityrrrr8 mlp_ratiomlprrrresidual)r*r+rrrrnr}r-r/r0rs(  zVitDetLayer.__init__r head_maskrc Cs|dddd}|}||}|jdkr'|jd|jd}}t||j\}}|j||d}|d}|dd} |jdkrGt||j|||f}|||}|||| |}|dddd}|j rk| |}|f| } | S)Nrr2rr)r) r?rrr7rrrrrrrr) r*rrrshortcutrBrCrself_attention_outputsrr/r/r0rKs*      zVitDetLayer.forward)rrF)NF)rLrMrNrOrrr8boolrr%rPrrrrKrQr/r/r-r0rs2 rcsbeZdZdeddffdd Z    ddejd eejd ed ed ede e e ff d dZ Z S) VitDetEncoderr+rHNc st||_|j}ddtd|j|D}g}t|D]}|t |||||j vr/|j nd||j vdqt ||_d|_dS)NcSsg|]}|qSr/)item).0rr/r/r0 sz*VitDetEncoder.__init__..r)rrrF)rrr+num_hidden_layersr%linspacerrangeappendrwindow_block_indicesrresidual_block_indicesr ModuleListrgradient_checkpointing)r*r+depthrlayersir-r/r0rs   zVitDetEncoder.__init__FTrrroutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD]8\}} |r||f}|dur$||nd} |jr6|jr6|| j|| |} n| || |} | d}|rI|| df}q|rQ||f}|s_tdd|||fDSt|||dS)Nr/rrcss|] }|dur|VqdSrr/)rvr/r/r0 Jsz(VitDetEncoder.forward..last_hidden_stater attentions) enumeraterrr_gradient_checkpointing_func__call__tupler ) r*rrrrrall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputsr/r/r0rK&s6      zVitDetEncoder.forward)NFFT)rLrMrNrrr%rPrrrrr rKrQr/r/r-r0rs& rmodulecCs6tjj|jddd|jdurtj|jddSdS)a Initialize `module.weight` using the "MSRAFill" implemented in Caffe2. Also initializes `module.bias` to 0. Source: https://detectron2.readthedocs.io/en/latest/_modules/fvcore/nn/weight_init.html. Args: module (torch.nn.Module): module to initialize. fan_outrelu)r5 nonlinearityNr)rinitkaiming_normal_rrt constant_)rr/r/r0caffe2_msra_fillRs r c@sFeZdZdZeZdZdZdZgZ de e j e j e jfddfdd ZdS) VitDetPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. vitdetrGTrrHNcCst|tjtjfr0tjj|jjt j d|j j d|jj |j_|jdur.|jjdSdSt|tjrE|jj|jjddSt|trdtjj|jjt j d|j j d|jj |j_dSt|tr|j jrtjj|jjt j d|j j d|j_tjj|jjt j d|j j d|j_dSt|tr|j|j|jfD]}t|q|j|jfD]}|jjd|jjq|jjj|jjjdSdS)zInitialize the weightsr)rstdNrS) rrrxr(r trunc_normal_rdatator%float32r+initializer_rangerrtzero_rfill_rr'rrr|rdrerrrrr rrr)r*rrr/r/r0 _init_weightslsP          z#VitDetPreTrainedModel._init_weights)rLrMrNrOr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrrrxr(rrr/r/r/r0r `s&r aH This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`VitDetConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aK Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViTImageProcessor.__call__`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare VitDet Transformer model outputting raw hidden-states without any specific head on top.cseZdZdeffdd ZdefddZdeee efddfd d Z e e e eed      dd eejd eejdeedeedeedeeeff ddZZS) VitDetModelr+cs2t|||_t||_t||_|dSr)rrr+rrJrencoder post_initr*r+r-r/r0rs    zVitDetModel.__init__rHcC|jjSrrJr)rr/r/r0get_input_embeddingsz VitDetModel.get_input_embeddingsheads_to_pruneNcCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrr prune_heads)r*r"rheadsr/r/r0 _prune_headsszVitDetModel._prune_heads output_typerrGrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|||jj}||}|j|||||d}|d}|sL|f|ddSt ||j |j dS)a Returns: Examples: ```python >>> from transformers import VitDetConfig, VitDetModel >>> import torch >>> config = VitDetConfig() >>> model = VitDetModel(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 768, 14, 14] ```Nz You have to specify pixel_values)rrrrrrr) r+rruse_return_dictr; get_head_maskrrJrr rr) r*rGrrrrembedding_outputencoder_outputssequence_outputr/r/r0rKs. zVitDetModel.forward)NNNNN)rLrMrNrrrr rr8rr&rVITDET_INPUTS_DOCSTRINGrr _CONFIG_FOR_DOCrr%rPrrrrKrQr/r/r-r0rs0   rzF ViTDet backbone, to be used with frameworks like Mask R-CNN. c sreZdZfddZdefddZeeee e d   dde j d e ed e ed e ede f d d ZZS)VitDetBackbonecsVttt|_t|_fddtjdD|_ | dS)Ncsg|]}jqSr/)r)rrmr+r/r0r%sz+VitDetBackbone.__init__..r) rr_init_backbonerrJrrrr num_featuresrrr-r1r0rs     zVitDetBackbone.__init__rHcCrrrrr/r/r0r *r!z#VitDetBackbone.get_input_embeddingsr'NrGrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}||}|j|d||d}|r1|jn|d}d}t|j|D]\} } | |j vrK|| f7}q=|sf|r[|f|dd} | S|f|dd} | St ||rm|jnd|j dS)a Returns: Examples: ```python >>> from transformers import VitDetConfig, VitDetBackbone >>> import torch >>> config = VitDetConfig() >>> model = VitDetBackbone(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```NT)rrrrr/r2) feature_mapsrr) r+r)rrrJrrzip stage_names out_featuresr r) r*rGrrrr+rrr4stagerrr/r/r0rK-s8    zVitDetBackbone.forward)NNN)rLrMrNrrr rr.rr r/r%rPrrrKrQr/r/r-r0r0s&   r0)rF)9rOcollections.abcrr9typingrrrrrr%torch.utils.checkpointr activationsr modeling_outputsr r modeling_utilsr utilsr rrrutils.backbone_utilsrconfiguration_vitdetr get_loggerrLloggerr/$VITDET_PRETRAINED_MODEL_ARCHIVE_LISTModulerr_rqrrrPrrrrrrrrrrrr r VITDET_START_DOCSTRINGr.rr0r/r/r/r0sX      W$) ?'KC8 Z