o he  @sdZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddlZ ddl mZddlmZddlmZmZdd lmZmZdd lmZdd lmZmZmZmZmZd d l m!Z!m"Z"m#Z#e$e%Z&dZ'dgZ(de j)de j)fddZ*de j)de j)fddZ+de j)de,fddZ-dOde j)de.de/de,de j)f ddZ0dPdd Z1d!d"Z2Gd#d$d$ej3Z4Gd%d&d&ej3Z5Gd'd(d(ej3Z6eGd)d*d*eZ7Gd+d,d,ej3Z8Gd-d.d.ej3Z9Gd/d0d0ej3Z:Gd1d2d2ej3Z;Gd3d4d4ej3ZGd9d:d:ej3Z?Gd;d<dZBd?ZCd@ZDGdAdBdBej3ZEGdCdDdDej3ZFGdEdFdFej3ZGGdGdHdHe@ZHGdIdJdJej3ZIGdKdLdLe@ZJeeAGdMdNdNe@ZKdS)Qz PyTorch GroupViT model.N) dataclass)AnyOptionalTupleUnion)nn)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)GroupViTConfigGroupViTTextConfigGroupViTVisionConfigznvidia/groupvit-gcc-yfcclogitsreturncCstj|tjt||jdS)Ndevice)r functional cross_entropytorcharangelenr)rr!d/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/groupvit/modeling_groupvit.pycontrastive_loss6sr# similaritycCs t|}t|}||dS)Ng@)r#t)r$ caption_loss image_lossr!r!r" groupvit_loss;s  r(dimcCsJ||}|j|ddd}tj|tjd||d}|||}|S)NTkeepdimr memory_format?)softmaxmaxr zeros_likelegacy_contiguous_formatscatter_detach)rr)y_softindexy_hardretr!r!r" hard_softmaxAs r9Ftauhardc Cstjjtjd|j|jdtjd|j|jd}||j}|||}| |}|rK|j |ddd}tj |tj d ||d}|||} | S|} | S)N)rdtyper.Tr*rr,)r distributionsgumbelGumbeltensorrr>sampleshaper/r0r1r2r3r4) rr;r<r) gumbel_distgumbelsr5r6r7r8r!r!r"gumbel_softmaxKs   rGc Cs|||jdd}||kr tt||}|jd|}ntt||}|jd|}|jd}|jd}|||||}tjj|||fd|d}|S)a Args: attentions (`torch.Tensor`): attention map of shape [batch_size, groups, feat_height*feat_width] height (`int`): height of the output attention map width (`int`): width of the output attention map align_corners (`bool`, *optional*): the `align_corner` argument for `nn.functional.interpolate`. Returns: `torch.Tensor`: resized attention map of shape [batch_size, groups, height, width] g?rrbilinear)sizemode align_corners)rDintnproundreshaperr interpolate) attentionsheightwidthrLscale feat_width feat_height batch_sizegroupsr!r!r"resize_attention_mapas    rZcCsg}t7d}|D]*}|ddd}|dur|}n||}t|dddg|R}||q Wdn1s@wY|d}|S)a1 Args: attentions (`tuple(torch.FloatTensor)`: tuple of attention maps returned by `GroupViTVisionTransformer` hw_shape (`tuple(int)`): height and width of the output attention map Returns: `torch.Tensor`: the attention map of shape [batch_size, groups, height, width] NrrHrr:)rno_gradpermute contiguousrZappend)rRhw_shape attn_mapsprev_attn_masks attn_masks cur_attn_mapfinal_groupingr!r!r"get_grouping_from_attentionss  recs*eZdZdeffdd ZddZZS)GroupViTCrossAttentionLayerconfigcsJtt||_tj|j|jd|_t ||_ tj|j|jd|_ dSNeps) super__init__GroupViTAttentionattnr LayerNorm hidden_sizelayer_norm_epsnorm2 GroupViTMLPmlp norm_postselfrg __class__r!r"rls   z$GroupViTCrossAttentionLayer.__init__cCs<|}||j||dd}||||}||}|S)N)encoder_hidden_statesr)rnrtrrru)rwquerykeyxr!r!r"forwards  z#GroupViTCrossAttentionLayer.forward)__name__ __module__ __qualname__rrlr~ __classcell__r!r!rxr"rfsrfcs4eZdZdeffdd Zd ddZddZZS) GroupViTAssignAttentionrgcsjt|jd|_t|j|j|_t|j|j|_t|j|j|_t|j|j|_ |j |_ dS)N) rkrlrprUrLinearq_projk_projv_projproj assign_epsrvrxr!r"rls   z GroupViTAssignAttention.__init__TcCsD|r|jrt|d|d}|S|rt|dd}|Stjj|dd}|S)N)r)r<r))trainingrGr9rrr/)rwrnr@r<r!r!r"get_attns  z GroupViTAssignAttention.get_attncCs|}||}||}||}||dd|j}||}|j|ddd}||jddd|j}||}||}||fS)Nrr:F)r@r<Tr)r+) rrr transposerUrsumrr)rwr{r|valueraw_attnrn soft_attnoutr!r!r"r~s     zGroupViTAssignAttention.forward)TT)rrrrrlrr~rr!r!rxr"rs  rcs2eZdZdeffdd ZddZddZZS)GroupViTTokenAssignrgcst||_tjjjd|_tj t j j rj nj j f}fdd|D\}}t ||||_tjjjd|_tjjjd|_t|_t|_tjjjd|_tj|j|_dS)Nricsg|] }t|jqSr!)rMrp).0r}rgr!r" sz0GroupViTTokenAssign.__init__..)rkrlnum_output_grouprrorprq norm_tokens isinstanceassign_mlp_ratio collectionsabcIterableGroupViTMixerMLP mlp_internorm_post_tokensnorm_xrfpre_assign_attnrassign norm_new_xrs mlp_channels)rwrgnum_group_tokenrr tokens_dim channels_dimrxrr"rls    zGroupViTTokenAssign.__init__cCs||}||}|S)z Args: group_tokens (torch.Tensor): group tokens, [batch_size, num_group_tokens, channels] Returns: projected_group_tokens (torch.Tensor): [batch_size, num_output_groups, channels] )rr)rw group_tokensprojected_group_tokensr!r!r"project_group_tokens z'GroupViTTokenAssign.project_group_tokencCs^||}||}||}|||}|||\}}||7}||||}||fS)z Args: image_tokens (`torch.Tensor`): image tokens, of shape [batch_size, input_length, channels] group_tokens (`torch.Tensor`): group tokens, [batch_size, num_group_tokens, channels] )rrrrrrr)rw image_tokensrrnew_image_tokens attentionr!r!r"r~s    zGroupViTTokenAssign.forward)rrrrrlrr~rr!r!rxr"rs rc@seZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZeed <dZeed <d eefd d ZdS)GroupViTModelOutputa\ Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. segmentation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels, logits_height, logits_width)`): Classification scores for each pixel. The logits returned do not necessarily have the same size as the `pixel_values` passed as inputs. This is to avoid doing two interpolations and lose some quality when a user needs to resize the logits to the original image size as post-processing. You should always check your logits shape and resize as needed. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`GroupViTTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`GroupViTVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`GroupViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`GroupViTVisionModel`]. Nlosslogits_per_imagelogits_per_textsegmentation_logits text_embeds image_embedstext_model_outputvision_model_outputrcstfddDS)Nc3s.|]}|dvr |nt|VqdS))rrN)getattrto_tuple)rkrwr!r" @s  z/GroupViTModelOutput.to_tuple..)tuplekeysrr!rr"r?s zGroupViTModelOutput.to_tuple)rrr__doc__rrr FloatTensor__annotations__rrrrrrr rrrrr!r!r!r"rs !  rc sheZdZdZ    ddedeeeeeffded effd d Zdd ej de dej fddZ Z S)GroupViTPatchEmbeddingsz# Image to Patch Embedding. r image_size patch_size num_channels embed_dimcstt|tjjr|n||f}t|tjjr|n||f}|d|d|d|d}||_||_||_t j ||||d|_ dS)Nrr) kernel_sizestride) rkrlrrrrrr num_patchesrConv2d projection)rwrrrrrrxr!r"rlKs  z GroupViTPatchEmbeddings.__init__F pixel_valuesinterpolate_pos_encodingrc Csx|j\}}}}|s.||jdks||jdkr.td|d|d|jdd|jdd ||ddd}|S)NrrzInput image size (*z) doesn't match model ().rH)rDr ValueErrorrflattenr)rwrrrXrrSrTr}r!r!r"r~\szGroupViTPatchEmbeddings.forward)rrrrF) rrrrrMrrrlrTensorboolr~rr!r!rxr"rFs $rcs\eZdZdeffdd Zdejdededejfdd Zdd ejd e dejfd dZ Z S)GroupViTVisionEmbeddingsrgcsptt|j|j|j|jd|_|jj}t t d||j|_ t |j|_t j|j|jd|_||_dS)N)rrrrrri)rkrlrrrrrppatch_embeddingsrr Parameterrzerosposition_embeddingsDropoutdropoutrorq layernormrg)rwrgrrxr!r"rlis  z!GroupViTVisionEmbeddings.__init__ embeddingsrSrTrcCs|jd}||jjdkr||kr|jS|j}|jd}|jd}||jj}||jj} |d| d}} t|} } |dt| t| |dddd} || | | f} t j j | | ddd }|dddd dd|}|S) a# This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. Source: https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174 rr:g?rrrHbicubicF) scale_factorrKrL) rDrrgrmathsqrtrPrMr\rrrQview)rwrrSrTnpatchpatch_pos_embednum_original_pos_embedr)rWrVoriginal_heightoriginal_widthreshaped_patch_pos_embedrr!r!r"rxs,     z1GroupViTVisionEmbeddings.interpolate_pos_encodingFrrc Csd|j\}}}}|j||d}||}|\}}} |r&|||||}n||j}||}|S)N)r)rDrrrJrrr) rwrrrXrrSrTrseq_len_r!r!r"r~s   z GroupViTVisionEmbeddings.forwardr) rrrrrlrrrMrrr~rr!r!rxr"rhs$"rc sXeZdZdeffdd Z   d deejdeejdeejdej fd d Z Z S) GroupViTTextEmbeddingsrgcsRt|j}t|j||_t|j||_|j dt |j ddddS)N position_ids)rr:F) persistent) rkrlrpr Embedding vocab_sizetoken_embeddingmax_position_embeddingsposition_embeddingregister_bufferrrexpandrwrgrrxr!r"rls  zGroupViTTextEmbeddings.__init__N input_idsr inputs_embedsrcCsb|dur |jdn|jd}|dur|jddd|f}|dur&||}||}||}|S)Nr:r)rDrrr)rwrrr seq_lengthrrr!r!r"r~s  zGroupViTTextEmbeddings.forwardNNN) rrrrrlrr LongTensorrrr~rr!r!rxr"rsrc seZdZdZdededededef fdd Zed d Zd d Z dde j de e j de j fddZ dde j de e j de edee jfddZZS) GroupViTStagezMThis corresponds to the `GroupingLayer` class in the GroupViT implementation.rgdepthnum_prev_group_tokenrrcst||_||_|dkrttd|j|_ nd|_ t fddt |D|_ |dkr;t ||d|_nd|_|dkr^|dkr^ttjjjdt|jd||_dSd|_dS)Nrrcg|]}tqSr!GroupViTEncoderLayerrrrr!r"rz*GroupViTStage.__init__..)rgrrrirH)rkrlrrrrrrrp group_token ModuleListrangelayersr downsample Sequentialrorqrgroup_projector)rwrgrrrrrxrr"rls(   zGroupViTStage.__init__cCs |jduSN)r rr!r!r"with_group_tokens zGroupViTStage.with_group_tokencCs>|jr|ddd|j f|dd|j dffS|dfSr)rrrwr}r!r!r"split_xs0zGroupViTStage.split_xNr}r rcCs|dur|Stj||gddS)Nrr)rcat)rwr}r r!r!r"concat_xszGroupViTStage.concat_xF hidden_statesprev_group_tokenoutput_attentionsc Cs|jr|j|ddd}|jdur|||}nd}|}|||}|jD] }||ddd}|d}q(||\}}d} |jdurL|||\}} ||f} |rW| | f} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the grouping tensors of Grouping block. rr:N)attention_maskcausal_attention_mask) rr rrJrrr rr ) rwrrrr r}cat_xlayer layer_outroutputsr!r!r"r~s&      zGroupViTStage.forwardr)NF)rrrrrrMrlpropertyrrrrrrrrrr~rr!r!rxr"rs8" "rc sXeZdZ   d dedeedeedeeffdd Zdejd ejfd d Z Z S) rsNrgrpintermediate_size output_sizecspt||_t|j|_|dur|n|j}|dur|n|j}|dur&|n|}t |||_ t |||_ dSr) rkrlrgr hidden_act activation_fnrpr rrfc1fc2)rwrgrpr r!rxr!r"rl.s  zGroupViTMLP.__init__rrcCs"||}||}||}|Sr)r$r#r%)rwrr!r!r"r~>s   zGroupViTMLP.forwardr) rrrrrrMrlrrr~rr!r!rxr"rs-srscseZdZfddZZS)rcs t|dd}|ddSNrrH)rkr~rrrxr!r"r~Fs zGroupViTMixerMLP.forward)rrrr~rr!r!rxr"rEsrcseZdZdZfddZdejdedefddZ dd ejd e ejd e ejde ej de e de eje eje e ejff ddZ ZS)rmz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rr)rkrlrgrprnum_attention_heads num_headshead_dimrrUattention_dropoutrrrrrrout_projrvrxr!r"rlNs"   zGroupViTAttention.__init__rBrbszcCs ||||j|jddSr&)rr(r)rr])rwrBrr,r!r!r"_shapeas zGroupViTAttention._shapeNFrrrrzrrcCs|\}}}|du} |||j} | r*|||d|} |||d|} n|||d|} |||d|} ||jd|jf} || ||j| } | j| } | j| } | d}t | | dd}|||j||fkrt d||j||fd||dur||d||fkrt d|d||fd||||j|||}|||j||}|dur||d||fkrt d|d||fd||||j|||}|||j||}t jj|dd}|r |||j||}|||j||}nd}t jj||j|jd }t || }|||j||jfkr@t d ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x ChannelNr:rrHz$Attention weights should be of size z , but is z!Attention mask should be of size r)prz `attn_output` should be of size )rJrrUr-rrr(r)rrbmmrrrrr/rrrPr+)rwrrrrzrr,tgt_lenris_cross_attention query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputr!r!r"r~dsl       zGroupViTAttention.forward)NNNF)rrrrrlrrrMr-rrrrr~rr!r!rxr"rmKs* rmc sTeZdZdeffdd Z d dejdejdejdeed e ej f d d Z Z S) rrgcsRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSrh) rkrlrprrm self_attnrrorq layer_norm1rsrt layer_norm2rvrxr!r"rls   zGroupViTEncoderLayer.__init__FrrrrrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|S)aI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrr)r<r;r=rt)rwrrrrresidualr7rr!r!r"r~s"     zGroupViTEncoderLayer.forwardr) rrrrrlrrrrrrr~rr!r!rxr"rs rc@s$eZdZdZeZdZdZddZdS)GroupViTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. groupvitTcCs|jj}t|tjtjfr"|jjjd|d|j dur!|j j nt|tj r5|j j |jj d|jj }t|trX|jjjjd|dd|jjjjd|dddSt|tr|jj }|jdd|jjd|}|jd|}tjj|jj|dtjj|jj|dtjj|jj|dtjj|jj|ddSt|tr|jj }|jjdd|jjd|}d|jjd|}tjj|jj|dtjj|jj|ddSdS) zInitialize the weightsr=)meanstdNr.g{Gz?rrH)rB)rginitializer_rangerrrrweightdatanormal_biaszero_rofill_initializer_factorrrrrmrnum_hidden_layersinitrrrr+rsrpr$r%)rwmodule init_rangefactor in_proj_std out_proj_stdfc_stdr!r!r" _init_weightss8        z%GroupViTPreTrainedModel._init_weightsN) rrrrr config_classbase_model_prefixsupports_gradient_checkpointingrSr!r!r!r"r?s  r?aJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`GroupViTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`CLIPTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. aE Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`CLIPTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c sbeZdZdeddffdd Z   d dejdeedeed eede e e ff d d Z Z S) GroupViTVisionEncoderrgrNcs>t|_tfddttjD|_d|_ dS)Nc sFg|]}tj|j|j||dkrj|dnddqS)rr)rgrrrr)rdepthsnum_group_tokensnum_output_groups)rirr!r"r}sz2GroupViTVisionEncoder.__init__..F) rkrlrgrr r r rXstagesgradient_checkpointingrvrxrr"rlys    zGroupViTVisionEncoder.__init__routput_hidden_statesr return_dictc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}d}t|jD](\}} |r<||f}| |||} | d}| d}|rY| ddurY|| df}q1|ra||f}|sotdd|||fDSt|||dS)Nr!rrrHcs|] }|dur|VqdSrr!rvr!r!r"rz0GroupViTVisionEncoder.forward..last_hidden_staterrR)rgrr^use_return_dict enumerater\rr ) rwrr^rr_all_hidden_states all_groupingsrr[stage layer_outputsr!r!r"r~s.     zGroupViTVisionEncoder.forwardr)rrrrrlrrrrrrr r~rr!r!rxr"rWxs  rWcsteZdZdZdeffdd Z     ddeejdeejdee d ee d ee d e e e ff d d Z ZS)GroupViTTextEncoderz Transformer encoder consisting of `config.num_hidden_layers` self-attention layers. Each layer is a [`GroupViTEncoderLayer`]. Args: config: GroupViTTextConfig rgcs:t|_tfddtjD|_d|_dS)Ncrr!rrrr!r"rrz0GroupViTTextEncoder.__init__..F) rkrlrgrr r rKr r]rvrxrr"rls   zGroupViTTextEncoder.__init__Nrrrr^r_rc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD]1\} } |r<|| f}|jrM|jrM|| j | |||} n| | |||d} | d} |rb|| df}q1|rj|| f}|sxt dd| ||fDSt | ||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr!)rrrcsr`rr!rar!r!r"r rcz.GroupViTTextEncoder.forward..rd) rgrr^rfrgr r]r_gradient_checkpointing_func__call__rr ) rwrrrrr^r_encoder_statesall_attentionsridx encoder_layerrkr!r!r"r~sF&     zGroupViTTextEncoder.forward)NNNNN)rrrrrrlrrrrrrr r~rr!r!rxr"rls*  rlcseZdZdeffdd Zeeeeed      dde e j de e j de e j d e e d e e d e e d e eeffd dZZS)GroupViTTextTransformerrgcsHt||_|j}t||_t||_tj ||j d|_ |j |_ dSrh) rkrlrgrprrrlencoderrrorqfinal_layer_norm eos_token_idrrxr!r"rls    z GroupViTTextTransformer.__init__ output_typerTNrrrrr^r_rc Csj|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|}|d|d}|j||d}t||j |j d} |durLt ||j }|j ||| |||d} | d} | | } |jdkr| tj| jd| j d|jtj| j d jdd f} n| tj| jd| j d|jtj| j d |jkjdd f} |s| | f| d dSt| | | j| jd S)  Returns: NzYou have to specify input_idsr:)rrr)rrrrr^r_rrH)r>rrrre pooler_outputrrR)rgrr^rfrrJrrr r>rr rtrurvrrrDtorMargmaxr rrR) rwrrrrr^r_ input_shaperrencoder_outputsre pooled_outputr!r!r"r~ s\    zGroupViTTextTransformer.forwardNNNNNN)rrrrrlrGROUPVIT_TEXT_INPUTS_DOCSTRINGrr rrrrrrr~rr!r!rxr"rss2  rscseZdZeZdeffdd ZdejfddZddZ e e e e ed  dd eejd eejd eejdeedeedeedeee ffddZZS)GroupViTTextModelrgc"t|t||_|dSr)rkrlrs text_model post_initrvrxr!r"rlv   zGroupViTTextModel.__init__rcC |jjjSrrrrrr!r!r"get_input_embeddings| z&GroupViTTextModel.get_input_embeddingscCs||jj_dSrr)rwrr!r!r"set_input_embeddingssz&GroupViTTextModel.set_input_embeddingsrwNrrrrr^r_cCs|j||||||dS)aK Returns: Examples: ```python >>> from transformers import CLIPTokenizer, GroupViTTextModel >>> tokenizer = CLIPTokenizer.from_pretrained("nvidia/groupvit-gcc-yfcc") >>> model = GroupViTTextModel.from_pretrained("nvidia/groupvit-gcc-yfcc") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrrr^r_)r)rwrrrrr^r_r!r!r"r~szGroupViTTextModel.forwardr)rrrrrTrlrModulerrrrrr rrrrrrr~rr!r!rxr"rss8  rcsxeZdZdeffdd Zeeeeed    d de e j de e de e d e e d e eeff d d ZZS)GroupViTVisionTransformerrgcs@t||_|j}t||_t||_tj ||j d|_ dSrh) rkrlrgrprrrWrtrrorqrrrxr!r"rls   z"GroupViTVisionTransformer.__init__rwNrr^rr_rc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td||}|j||||d}|d}||}|jdd}|sO||f|ddSt |||j |j dS)ryNz You have to specify pixel_values)rr^rr_rrrrz) rgrr^rfrrrtrrAr rrR) rwrr^rr_rrrerr!r!r"r~s0    z!GroupViTVisionTransformer.forwardNNNN)rrrrrlr GROUPVIT_VISION_INPUTS_DOCSTRINGrr rrrrrrr~rr!r!rxr"rs&  rcseZdZeZdZdeffdd ZdefddZe e e e ed dde ejd e ed e ed e edeee ff d dZZS)GroupViTVisionModelrrgcrr)rkrlr vision_modelrrvrxr!r"rlrzGroupViTVisionModel.__init__rcCrr)rrrrr!r!r"rrz(GroupViTVisionModel.get_input_embeddingsrwNrr^r_cCs|j||||dS)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GroupViTVisionModel >>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc") >>> model = GroupViTVisionModel.from_pretrained("nvidia/groupvit-gcc-yfcc") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrr^r_)r)rwrrr^r_r!r!r"r~s zGroupViTVisionModel.forwardr)rrrrrTmain_input_namerlrrrrrr rrrrrrr~rr!r!rxr"rs,  rcsTeZdZeZdeffdd Zee      ddee j dee j dee j dee d ee d ee d e j fd d Z ee    ddee j dee d ee d ee d e j f ddZeeeeed         ddee jdee j dee j dee jdee dee d ee dee d ee d eeeffddZZS) GroupViTModelrgc s6t|t|jtstdt|jdt|jts(tdt|jd|j}|j}|j |_ |j |_ |j |_ |j |_ t||_t||_ttj|j |j ddt|j tjddtj|j |j dd|_ttj|j |j ddt|j tjddtj|j |j dd|_tt|jj|_|dS)NzOconfig.text_config is expected to be of type GroupViTTextConfig but is of type .zSconfig.vision_config is expected to be of type GroupViTVisionConfig but is of type T)rG)inplace) rkrlr text_configrrtype vision_configrprojection_dimprojection_intermediate_dimrptext_embed_dimvision_embed_dimrsrrrrrr BatchNorm1dReLUvisual_projectiontext_projectionrrrBrglogit_scale_init_value logit_scaler)rwrgrrrxr!r"rlsF          zGroupViTModel.__init__Nrrrrr^r_rc Csh|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||||d}|d}||} | S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`GroupViTTextModel`]. Examples: ```python >>> from transformers import CLIPTokenizer, GroupViTModel >>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc") >>> tokenizer = CLIPTokenizer.from_pretrained("nvidia/groupvit-gcc-yfcc") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```Nrr)rgrr^rfrr) rwrrrrr^r_ text_outputsr text_featuresr!r!r"get_text_featuresEs zGroupViTModel.get_text_featuresrcCsd|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||d}|d}||}|S)aH Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`GroupViTVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GroupViTModel >>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc") >>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nrr)rgrr^rfrr)rwrrr^r_vision_outputsrimage_featuresr!r!r"get_image_featurests z GroupViTModel.get_image_featuresrw return_lossoutput_segmentationc  CsP|dur|n|jj}|dur|n|jj}|rd}|dur|n|jj}| dur(| n|jj} |j|||| d} |j|||||| d} | d} || } | d} || } | | j ddd} | | j ddd} |j }t | | |}|}d}|r| d}||d|jd}|r| d }n| d }t||jd d}||j ddd}t || |}|| jdd| jddd d}||jd|jdd}t |||}||jd|jd|jd |jd }d}|rt|}| s|dur|||| | | | f}n||| | | | f}|dur|f|S|St||||| | | | d S) a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GroupViTModel >>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc") >>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```NTrrrr:rrrrH)rrrrrrrr)rgrrr^rfrrrrnormrexprmatmulr%rPrDrer\r(r)rwrrrrrrr^rr_rrrrrrr seg_logitsimage_group_embedsrRgroupinglogits_per_image_groupflatten_groupingroutputr!r!r"r~s'        zGroupViTModel.forwardrr) NNNNNNNNN)rrrrrTrlrrrrrrrrrrGROUPVIT_INPUTS_DOCSTRINGrrrrrr~rr!r!rxr"rs+ . 0    r)rFr:r)Lrcollections.abcrr dataclassesrtypingrrrrnumpyrNrtorch.utils.checkpointr activationsr modeling_attn_mask_utilsr r modeling_outputsr r modeling_utilsrutilsrrrrrconfiguration_groupvitrrr get_loggerrlogger_CHECKPOINT_FOR_DOC&GROUPVIT_PRETRAINED_MODEL_ARCHIVE_LISTrr#r(rMr9floatrrGrZrerrfrrrrrrrrsrrmrr?GROUPVIT_START_DOCSTRINGrrrrWrlrsrrrrr!r!r!r"sf     $ 072"F!^o2,  &:b_5:4