o h:@s`dZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZmZdd lmZmZdd lmZdd lmZmZmZmZmZd d lmZmZm Z e!e"Z#dZ$dgZ%de j&de j&fddZ'de j&de j&fddZ(eGdddeZ)eGdddeZ*eGdddeZ+Gddde j,Z-Gddde j,Z.Gd d!d!e j,Z/Gd"d#d#e j,Z0Gd$d%d%e j,Z1Gd&d'd'eZ2d(Z3d)Z4d*Z5d+Z6Gd,d-d-e j,Z7Gd.d/d/e j,Z8Gd0d1d1e2Z9Gd2d3d3e j,Z:Gd4d5d5e2Z;ee3Gd6d7d7e2Zedd>e2Z?dS)?z PyTorch CLIPSeg model.N) dataclass)AnyOptionalTupleUnion)nn)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) CLIPSegConfigCLIPSegTextConfigCLIPSegVisionConfigzCIDAS/clipseg-rd64-refinedlogitsreturncCstj|tjt||jdS)Ndevice)r functional cross_entropytorcharangelenr)rr!b/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/clipseg/modeling_clipseg.pycontrastive_loss5sr# similaritycCs t|}t|}||dS)Ng@)r#t)r$ caption_loss image_lossr!r!r" clipseg_loss:s  r(c@seZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed<dZeed <d eefd d ZdS) CLIPSegOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image:(`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegTextModel`]. image_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegVisionModel`]. text_model_output(`BaseModelOutputWithPooling`): The output of the [`CLIPSegTextModel`]. vision_model_output(`BaseModelOutputWithPooling`): The output of the [`CLIPSegVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrctfddDS)Nc3.|]}|dvr |nt|VqdS))r/r0Ngetattrto_tuple.0kselfr!r" `  z)CLIPSegOutput.to_tuple..tuplekeysr9r!r9r"r5_ zCLIPSegOutput.to_tuple)__name__ __module__ __qualname____doc__r*rr FloatTensor__annotations__r+r,r-r.r/r r0rrr5r!r!r!r"r)@s   r)c@sLeZdZUdZdZejed<dZe e ejed<dZ e e ejed<dS)CLIPSegDecoderOutputa Args: logits (`torch.FloatTensor` of shape `(batch_size, height, width)`): Classification scores for each pixel. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr hidden_states attentions) rArBrCrDrrrErFrHrrrIr!r!r!r"rGfs  rGc@sxeZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ e ed<dZeed<d eefd d ZdS) CLIPSegImageSegmentationOutputa, Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. ... vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPSegVisionModel`]. Nr*rconditional_embeddings pooled_outputr0decoder_outputrcr1)Nc3r2))r0rMNr3r6r9r!r"r;r<z:CLIPSegImageSegmentationOutput.to_tuple..r=r9r!r9r"r5r@z'CLIPSegImageSegmentationOutput.to_tuple)rArBrCrDr*rrrErFrrKrLr0r rMrGrrr5r!r!r!r"rJzs    rJcs@eZdZdeffdd ZddZdejdejfdd Z Z S) CLIPSegVisionEmbeddingsconfigcst||_|j|_|j|_|j|_tt |j|_ tj |j |j|j|jdd|_|j|jd|_|jd|_t|j|j|_|jdt |jddddS)NF) in_channels out_channels kernel_sizestridebiasr position_idsr persistent)super__init__rO hidden_size embed_dim image_size patch_sizer Parameterrrandnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferrexpandr:rO __class__r!r"r\s"  "z CLIPSegVisionEmbeddings.__init__cCst|dkr tdt|jd}|jjddjd|jj ||}t j j ||ddd d|jj |d|dj}t|jjdd|g}|S) NrUz#new_size should consist of 2 valuesg?rbicubicF)mode align_cornersr)r ValueErrorintrgrjweightTviewrOr]rr interpolatesqueezercat)r:new_sizenum_patches_one_directionabresultr!r!r"interpolate_position_embeddingss  z7CLIPSegVisionEmbeddings.interpolate_position_embeddings pixel_valuesrcCs|jd}||}|ddd}|j|dd}tj||gdd}|jd|jkrIt t |jdd}|| ||f}| |j}|S|||j}|S)NrrUrrXdim)shaperfflatten transposercrlrrzrhrtmathsqrtrtodtyperjrV)r:r batch_size patch_embeds class_embeds embeddings new_shaper!r!r"forwards   zCLIPSegVisionEmbeddings.forward) rArBrCrr\rrrETensorr __classcell__r!r!rnr"rNsrNc sXeZdZdeffdd Z   d deejdeejdeejdej fd d Z Z S) CLIPSegTextEmbeddingsrOcsRt|j}t|j||_t|j||_|j dt |j ddddS)NrVrWFrY) r[r\r]rri vocab_sizetoken_embeddingmax_position_embeddingsrjrkrrrlr:rOr^rnr!r"r\s  zCLIPSegTextEmbeddings.__init__N input_idsrV inputs_embedsrcCsb|dur |jdn|jd}|dur|jddd|f}|dur&||}||}||}|S)NrX)rrVrrj)r:rrVr seq_lengthposition_embeddingsrr!r!r"rs  zCLIPSegTextEmbeddings.forward)NNN) rArBrCrr\rr LongTensorrErrrr!r!rnr"rsrcseZdZdZfddZdejdedefddZ dd ejd e ejd e ejde e de eje eje e ejff ddZ Z S)CLIPSegAttentionz=Multi-headed attention from 'Attention Is All You Need' papercst||_|j|_|j|_|j|j|_|j|j|jkr-td|jd|jd|jd|_ |j |_ t |j|j|_t |j|j|_t |j|j|_t |j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).)r[r\rOr]r^num_attention_heads num_headshead_dimrsscaleattention_dropoutdropoutrLineark_projv_projq_projout_projrmrnr!r"r\s"   zCLIPSegAttention.__init__tensorseq_lenbszcCs ||||j|jddS)NrrU)rwrrr contiguous)r:rrrr!r!r"_shape s zCLIPSegAttention._shapeNFrHattention_maskcausal_attention_maskoutput_attentionsrcCs|\}}}|||j}|||d|} |||d|} ||jd|jf} ||||j| }| j| } | j| } | d} t || dd} | ||j|| fkrmt d||j|| fd| |dur||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } |dur||d|| fkrt d|d|| fd|| ||j|| |} | ||j|| } t jj| dd} |r| ||j|| }|||j|| } nd}t jj| |j|jd }t || }|||j||jfkr$t d ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x ChannelrXrrUz$Attention weights should be of size z , but is Nz!Attention mask should be of size r)ptrainingz `attn_output` should be of size )sizerrrrrrrrwrbmmrrsrrsoftmaxrrreshaper)r:rHrrrrtgt_lenr^ query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputr!r!r"r sd       zCLIPSegAttention.forward)NNF)rArBrCrDr\rrrtrrboolrrrr!r!rnr"rs$ rcs2eZdZfddZdejdejfddZZS) CLIPSegMLPcsDt||_t|j|_t|j|j |_ t|j |j|_ dSN) r[r\rOr hidden_act activation_fnrrr]intermediate_sizefc1fc2rmrnr!r"r\]s  zCLIPSegMLP.__init__rHrcCs"||}||}||}|Sr)rrr)r:rHr!r!r"rds   zCLIPSegMLP.forward)rArBrCr\rrrrr!r!rnr"r\s rc sTeZdZdeffdd Z d dejdejdejdeed e ej f d d Z Z S) CLIPSegEncoderLayerrOcRt|j|_t||_tj|j|jd|_ t ||_ tj|j|jd|_ dSN)eps r[r\r]r^r self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rmrnr!r"r\m   zCLIPSegEncoderLayer.__init__FrHrrrrcCsd|}||}|j||||d\}}||}|}||}||}||}|f}|r0||f7}|SaI Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rHrrr)rrrrr:rHrrrresidualroutputsr!r!r"rus"     zCLIPSegEncoderLayer.forwardF) rArBrCrr\rrrrrrErrr!r!rnr"rls rc@s$eZdZdZeZdZdZddZdS)CLIPSegPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. clipTcCsV|jj}t|tr"|jjjjd|dd|jjjjd|ddnt|t rW|jj}t j j|j d|j d|dt j j|jj|jj|dt j j|jj|jj|dnt|tr|jj}|j dd|jjd|}|j d|}t j j|jj|dt j j|jj|dt j j|jj|dt j j|jj|dn_t|tr|jj}|jjdd|jjd|}d|jjd|}t j j|jj|dt j j|jj|dn't|trt j j|jj|jd|jjdt j j|jj|jd|jjdt|t jr|j j!|jj"dt|t j#r'|j dur)|j j!dSdSdS) zInitialize the weightsgg{Gz?)meanstdr)rrUg?N)$rOinitializer_factor isinstancerrrudatanormal_rjrNrinitrcr^rfinitializer_rangernum_hidden_layersrrrrrr]rr CLIPSegModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimrrTzero_fill_r)r:modulefactor in_proj_std out_proj_stdfc_stdr!r!r" _init_weightssL       z$CLIPSegPreTrainedModel._init_weightsN) rArBrCrDr config_classbase_model_prefixsupports_gradient_checkpointingrr!r!r!r"rs  raI This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`CLIPSegConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. aE Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. csteZdZdZdeffdd Z     ddeejdeejdee d ee d ee d e e e ff d d Z ZS)CLIPSegEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPSegEncoderLayer`]. Args: config: CLIPSegConfig rOcs:t|_tfddtjD|_d|_dS)Ncg|]}tqSr!)rr7_rOr!r" ?z+CLIPSegEncoder.__init__..F) r[r\rOr ModuleListrangerlayersgradient_checkpointingrmrnrr"r\<s   zCLIPSegEncoder.__init__Nrrroutput_hidden_states return_dictrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|} t|jD]1\} } |r<|| f}|jrM|jrM|| j | |||} n| | |||d} | d} |rb|| df}q1|rj|| f}|sxt dd| ||fDSt | ||dS) a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr!)rrrcs|] }|dur|VqdSrr!r7vr!r!r"r;z)CLIPSegEncoder.forward..)last_hidden_staterHrI) rOrruse_return_dict enumeraterrr_gradient_checkpointing_func__call__r>r ) r:rrrrrrencoder_statesall_attentionsrHidx encoder_layer layer_outputsr!r!r"rBsF&     zCLIPSegEncoder.forwardNNNNN)rArBrCrDrr\rrrrrrr rrr!r!rnr"r3s*  rcseZdZdeffdd Zeeeeed      dde e j de e j de e j d e e d e e d e e d e eeffd dZZS)CLIPSegTextTransformerrOcsHt||_|j}t||_t||_tj ||j d|_ |j |_ dSr) r[r\rOr]rrrencoderrrrfinal_layer_norm eos_token_idrrnr!r"r\s    zCLIPSegTextTransformer.__init__ output_typerNrrrVrrrrc Csj|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|}|d|d}|j||d}t||j |j d} |durLt ||j }|j ||| |||d} | d} | | } |jdkr| tj| jd| j d|jtj| j d jdd f} n| tj| jd| j d|jtj| j d |jkjdd f} |s| | f| d dSt| | | j| jd S)  Returns: NzYou have to specify input_idsrX)rrVr)rrrrrrrrU)rrrrr pooler_outputrHrI)rOrrr rsrrwrr rrr rrrrrrrrtargmaxr rHrI) r:rrrVrrr input_shaperHrencoder_outputsrrLr!r!r"rs\    zCLIPSegTextTransformer.forwardNNNNNN)rArBrCrr\rCLIPSEG_TEXT_INPUTS_DOCSTRINGrr rrrrrrrrr!r!rnr"rs2  rcseZdZeZddgZdeffdd ZdejfddZ d d Z e e e eed  dd eejdeejdeejdeedeedeedeeeffddZZS)CLIPSegTextModelrrrOc"t|t||_|dSr)r[r\r text_model post_initrmrnr!r"r\   zCLIPSegTextModel.__init__rcC |jjjSrr#rrr9r!r!r"get_input_embeddings z%CLIPSegTextModel.get_input_embeddingscCs||jj_dSrr')r:valuer!r!r"set_input_embeddingssz%CLIPSegTextModel.set_input_embeddingsrNrrrVrrrcCs|j||||||dS)aM Returns: Examples: ```python >>> from transformers import AutoTokenizer, CLIPSegTextModel >>> tokenizer = AutoTokenizer.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegTextModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrVrrr)r#)r:rrrVrrrr!r!r"rszCLIPSegTextModel.forwardr)rArBrCrr_no_split_modulesr\rModuler(r+rr rr rrrrrrrrr!r!rnr"r!s:  r!csxeZdZdeffdd Zeeeeed    d de e j de e de e d e e d e eeff d d ZZS)CLIPSegVisionTransformerrOcsRt||_|j}t||_tj||jd|_ t ||_ tj||jd|_ dSr) r[r\rOr]rNrrrr pre_layrnormrrpost_layernormrrnr!r"r\.s   z!CLIPSegVisionTransformer.__init__rNrrrrrc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td||}||}|j||||d}|d}|dddddf}||}|sY||f|ddSt |||j |j dS)rNz You have to specify pixel_values)rrrrrrr) rOrrr rsrr0rr1r rHrI) r:rrrrrHrrrLr!r!r"r8s2   z CLIPSegVisionTransformer.forwardNNNN)rArBrCrr\rCLIPSEG_VISION_INPUTS_DOCSTRINGrr rrrErrrrrr!r!rnr"r/,s&  r/cseZdZeZdZdeffdd ZdejfddZ e e e e ed ddeejd eed eed eedeee ff d dZZS)CLIPSegVisionModelrrOcr"r)r[r\r/ vision_modelr$rmrnr!r"r\lr%zCLIPSegVisionModel.__init__rcCr&r)r5rrfr9r!r!r"r(rr)z'CLIPSegVisionModel.get_input_embeddingsrNrrrcCs|j||||dS)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegVisionModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegVisionModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrr)r5)r:rrrrr!r!r"rus zCLIPSegVisionModel.forwardr2)rArBrCrrmain_input_namer\rr.r(rr3rr rrrErrrrrr!r!rnr"r4hs,  r4csJeZdZeZdeffdd Zee      ddee j dee j dee j dee d ee d ee d e j fd d Z ee    ddee j dee d ee d ee d e j f ddZeeeeed        ddee jdee j dee j dee jdee dee d ee d ee d eeeffddZZS)rrOcst|t|jtstdt|jdt|jts(tdt|jd|j}|j}|j |_ |j |_ |j |_ t ||_t||_tj|j |j dd|_tj|j |j dd|_tt|jj|_|dS)NzNconfig.text_config is expected to be of type CLIPSegTextConfig but is of type .zRconfig.vision_config is expected to be of type CLIPSegVisionConfig but is of type F)rT)r[r\r text_configrrstype vision_configrprojection_dimr]rrrr#r/r5rrrrrarrrOlogit_scale_init_value logit_scaler$)r:rOr9r;rnr!r"r\s0      zCLIPSegModel.__init__NrrrVrrrrc Csh|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||||d}|d}||} | S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegTextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, CLIPSegModel >>> tokenizer = AutoTokenizer.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```Nr,r)rOrrr r#r) r:rrrVrrr text_outputsrL text_featuresr!r!r"get_text_featuress zCLIPSegModel.get_text_featuresrcCsd|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||d}|d}||}|S)aI Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPSegVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nr6r)rOrrr r5r)r:rrrrvision_outputsrLimage_featuresr!r!r"get_image_featuress zCLIPSegModel.get_image_featuresr return_lossc  Cs&|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||d} |j||||||d} | d} || } | d} || } | | jdddd} | | jdddd} |j } t | | | }| }d}|rst|}|s||| | | | f}|dur|f|S|St|||| | | | d S) a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPSegModel >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegModel.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```Nr6r,rrUrXT)rrkeepdim)r*r+r,r-r.r/r0)rOrrr r5r#rrnormr>exprmatmulr%r(r))r:rrrrVrErrrrBr?r.r-r>r,r+r*outputr!r!r"r sT&   zCLIPSegModel.forwardrr2)NNNNNNNN)rArBrCrrr\rr rrrrrErAr3rDCLIPSEG_INPUTS_DOCSTRINGrr)rrrrrr!r!rnr"rs  . 0   rc sXeZdZdZdeffdd Z d dejdejdejd ee d e ej f d d Z Z S)CLIPSegDecoderLayerz CLIPSeg decoder layer, which is identical to `CLIPSegEncoderLayer`, except that normalization is applied after self-attention/MLP, rather than before. rOcrrrrmrnr!r"r\rzCLIPSegDecoderLayer.__init__FrHrrrrcCsd|}|j||||d\}}||}||}|}||}||}||}|f}|r0||f7}|Sr)rrrrrr!r!r"rs"     zCLIPSegDecoderLayer.forwardr)rArBrCrDrr\rrrrrrErrr!r!rnr"rL~s rLc s\eZdZdeffdd Z   d deejdejdee d ee d ee f d d Z Z S)CLIPSegDecoderrOc sXtj|_tjj|_tjj|_j r`j j dj j df}t tj jjdddttjjjd|d|ddttjjdd|d|dd|_ntjjdj j j j d|_tj}tfd d t|D|_tj j_j_j_d _tfd d ttjD|_dS) Nrr)rRpaddingrUr)rRrS)rScsg|] }tjjjqSr!)rrr;r] reduce_dimrrr!r"rsz+CLIPSegDecoder.__init__..relucrr!)rLr)decoder_configr!r"rr)r[r\conditional_layerrrr<rPfilm_mulfilm_add"use_complex_transposed_convolutionr;r` SequentialrdReLUConvTranspose2dtransposed_convolutionr extract_layersrrreducescopydeepcopyr]decoder_num_attention_headsrdecoder_intermediate_sizerrr)r:rOtransposed_kernelsdepthrn)rOrRr"r\sB   (zCLIPSegDecoder.__init__NTrHrKrrrcCsn|rdnd}|r dnd}|ddd}d} tt||j|jD]O\} \} } } | dur1| | | } n| | } | |jkrR||| ddd||} | ddd} | | dd|d}|d} |re|| f7}|rn||df7}q| ddddddfddd} tt | j d}|j d}| || j d||} | | }|stdd|||fDSt|||d S) Nr!rXrrrU)rrrcsrrr!rr!r!r"r;rz)CLIPSegDecoder.forward..)rrHrI)r ziprr\rSrTpermuterUrtrrrrwrZryr>rG)r:rHrKrrrall_hidden_statesr activationsrJi activationlayerreducerrrrr!r!r"rsD  "  $ zCLIPSegDecoder.forward)NNT) rArBrCrr\rrrrrrrr!r!rnr"rMs .rMzn CLIPSeg model with a Transformer-based decoder on top for zero-shot and one-shot image segmentation. cseZdZeZdeffdd Z     ddedeej deej deej d eej f d d Z e e e eed           ddeejd eejd eejdeejdeej deejdeejdeedeedeedeeeffddZZS)CLIPSegForImageSegmentationrOcs:t|||_t||_|j|_t||_|dSr) r[r\rOrrr[rMdecoderr$rmrnr!r"r\$s    z$CLIPSegForImageSegmentation.__init__NrrrrVconditional_pixel_valuescCs|dur.t||krtdt|jj|||d}Wd|S1s'wY|S|durYt||kr 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoProcessor, CLIPSegForImageSegmentation >>> from PIL import Image >>> import requests >>> processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined") >>> model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["a cat", "a remote", "a blanket"] >>> inputs = processor(text=texts, images=[image] * len(texts), padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> print(logits.shape) torch.Size([3, 352, 352]) ```NTr6rrUcsg|]}|dqS)rr!)r7rgrHr!r"rsz7CLIPSegForImageSegmentation.forward..rrr)rrrrVrmzWMake sure to pass as many conditional embeddings as there are query images in the batchzcMake sure that the feature dimension of the conditional embeddings matches `config.projection_dim`.)rrr)r*rrKrLr0rM)rOr rrnrr5rrHr[r rrrIrorrsr<rlrrrrBCEWithLogitsLossrJ)r:rrrmrKrrVrprrrrBrLrfdecoder_outputsrr*loss_fnrJr!rqr"rNs|,    z#CLIPSegForImageSegmentation.forwardr) NNNNNNNNNN)rArBrCrrr\rtrrrrorrKrrJrrErrrrr)rrr!r!rnr"rksl      rk)@rDr]r dataclassesrtypingrrrrrtorch.utils.checkpointrrfr modeling_attn_mask_utilsr r modeling_outputsr r modeling_utilsrutilsrrrrrconfiguration_clipsegrrr get_loggerrAlogger_CHECKPOINT_FOR_DOC%CLIPSEG_PRETRAINED_MODEL_ARCHIVE_LISTrr#r(r)rGrJr.rNrrrrrCLIPSEG_START_DOCSTRINGr r3rKrrr!r/r4rrLrMrkr!r!r!r"sf     $>"i24  'aa7<4b9d