o hx@sdZddlZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZdd lmZmZdd lmZmZmZdd lmZmZmZmZmZm Z d d l!m"Z"m#Z#m$Z$m%Z%m&Z&e'e(Z)dZ*dZ+dZ,dZ-dZ.gdZ/dgZ0dgZ1dZ2dZ3e e&e$e%fZ4eGdddeZ5eGdddeZ6eGdddeZ7Gdddej8Z9Gdddej8Z:Gdd d ej8Z;Gd!d"d"ej8ZGd'd(d(ej8Z?Gd)d*d*ej8Z@Gd+d,d,ej8ZAGd-d.d.ej8ZBGd/d0d0ej8ZCd1ZDd2ZEd3ZFeFeEZGd4ZHeHeEZId5eEZJd6ZKeFeHeEeKZLd7eHeFd8eEZMd9ZNGd:d;d;eZOedd?d?eOZQed@eDjPdd=GdAdBdBeOZRedCeDjPdd=GdDdEdEeOZSedFeDjPdGd=GdHdIdIeOZTGdJdKdKej8ZUGdLdMdMej8ZVGdNdOdOej8ZWedPeDjPdQd=GdRdSdSeOZXGdTdUdUej8ZYGdVdWdWej8ZZGdXdYdYej8Z[GdZd[d[ej8Z\ed\eDjPdGd=eNGd]d^d^eOZ]dS)_z PyTorch FLAVA model.N) OrderedDict) dataclass)AnyDictListOptionalSetTupleUnion)nn)ACT2FN)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) FlavaConfigFlavaImageCodebookConfigFlavaImageConfigFlavaMultimodalConfigFlavaTextConfigzfacebook/flava-fullzfacebook/flava-image-codebookrrr)rg$(~k@c@seZdZUdZdZeejed<dZ ee ed<dZ eejed<dZ ee ed<dZ eejed<dZee ed<d eefd d ZdS) FlavaModelOutputa Output from FlavaModel containing embeddings and outputs from individual encoders. Note that `image_embeddings` and `text_embeddigns` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. Args: image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. Nimage_embeddings image_outputtext_embeddings text_outputmultimodal_embeddingsmultimodal_outputreturncstfddDS)Nc3s.|]}|dvr |nt|VqdS))r%r#r'Ngetattrto_tuple.0kself^/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/flava/modeling_flava.py es  z,FlavaModelOutput.to_tuple..tuplekeysr/r1r/r2r+ds zFlavaModelOutput.to_tuple)__name__ __module__ __qualname____doc__r"rtorch FloatTensor__annotations__r#rr$r%r&r'r rr+r1r1r1r2r!Es r!c@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<d efd d ZdS) FlavaLossesa"Class representing pretraining losses from FLAVA model Args: mim (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels` and `pixel_values` are present, `input_ids_masked` is absent and `mim_weight` > 0.: Masked Image Modeling loss as used in BeIT calculated only for unimodal image data. mlm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels` and `input_ids_masked` are present, `pixel_values` is absent and `mlm_weight` > 0.: Masked Language Modeling loss as used in BERT calculated only for unimodal text data. itm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `itm_labels`, `input_ids_masked`, `pixel_values` are present and `itm_weight` > 0.: Image Text Matching (ITM) loss calculated for paired image-text data. Note that ITM loss is calculated on masked pairs in FLAVA. global_contrastive (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `input_ids` and `pixel_values` are present and `global_contrastive_weight` > 0.: Contrastive loss for image-text similarity similar to CLIP but calculated globally for paired image-text data. This is calculated on unmasked images and texts. mmm_image (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_image_weight` > 0.: Masked Multimodal Modeling loss's image component calculated on paired image-text data. mmm_text (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_text_weight` > 0.: Masked Multimodal Modeling loss's text component calculated on paired image-text data. Nmimmlmitmglobal_contrastive mmm_imagemmm_textr(cCs(d}|D] }|durd}|Sq|S)NTF)values)r0all_nonevr1r1r2rFs zFlavaLosses.all_none)r7r8r9r:r?rr;r<r=r@rArBrCrDboolrFr1r1r1r2r>ks r>c@seZdZUdZdZeejed<dZ e ed<dZ eejed<dZ ee ed<dZeejed<dZee ed<dZeejed <dZee ed <dZeejed <dZee ed <dZeejed <dZee ed<dZeejed<dZee ed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dee fddZ!dS)FlavaForPreTrainingOutputa Output from FlavaForPreTraining containing embeddings, and outputs from individual encoders. Note that `image_embeddings` and `text_embeddings` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. Args: loss (`torch.FloatTensor`, *optional*, returned when `return_loss` is True): Total loss calculated for this model. loss_info (`FlavaLosses`): Detailed info for FLAVA Pretraining losses. Check `FlavaLosses` class description for the information on the keys. image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. image_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. image_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. text_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids_masked` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` are present): The output of the [`FlavaTextModel`]. multimodal_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_masked_output (`BaseModelOutputWithPooling`, returned when `input_ids_masked` and `pixel_values` are present): The output of the [`FlavaMultimodalModel`]. mim_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape `(total_masked_patches, image_vocab_size)` , *optional*, returned when `pixel_values` are present and `input_ids_masked` are not): The logits for MIM unimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mlm_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(total_masked_seq_length, text_vocab_size)`, *optional*, returned when `input_ids_masked` are present and `pixel_values` are not): The logits for MLM unimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. itm_logits (`torch.FloatTensor` of shape `(batch_size, 2)`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The logits for ITM loss. Note that ITM loss is calculated on masked pairs in FLAVA. mmm_image_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape`(total_masked_patches, image_vocab_size)`, *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM image multimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mmm_text_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(`(total_masked_seq_length, text_vocab_size)`), *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM text multimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. contrastive_logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeddings` and `text_embeddings` but passed through FLAVA's `image_projection` and `text_projection` layers respectively. This represents the image-text similarity scores. This is calculated on unmasked images and texts. contrastive_logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeddings` and `image_embeddings` but passed through FLAVA's `text_projection` and `image_projection` layers respectively. This is calculated on unmasked images and texts. Nloss loss_infor"r#r$r%r&r'image_masked_embeddingsimage_masked_outputtext_masked_embeddingstext_masked_outputmultimodal_masked_embeddingsmultimodal_masked_output mim_logits mlm_logits itm_logitscontrastive_logits_per_imagecontrastive_logits_per_textmmm_image_logitsmmm_text_logitsr(cs$gdtfddDS)N)r%r#r'rOrMrQc3s.|]}|vr |nt|VqdSNr)r,r0transformer_outputsr1r2r3s,z5FlavaForPreTrainingOutput.to_tuple..r4r/r1rZr2r+sz"FlavaForPreTrainingOutput.to_tuple)"r7r8r9r:rJrr;r<r=rKr>r"r#rr$r%r&r'rLrMrNrOrPrQrRrSrTrUrVrWrXr rr+r1r1r1r2rIs0 @ rIc sxeZdZdZddededdffdd Zd ejd e d e dejfd d Z  ddejde ej dedejfddZ ZS)FlavaImageEmbeddingszb Construct the CLS token, position and patch embeddings. Optionally, also the mask token. Fconfiguse_mask_tokenr(Ncst|p |j}ttdd|j|_|r#ttdd|jnd|_t |j |j |j |jd|_ |j j}ttd|d|j|_t|j|_||_dS)Nr) image_size patch_size num_channels embed_dim)super__init__ mask_tokenr Parameterr;zeros hidden_size cls_tokenPatchEmbeddingsr_r`rapatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropoutr])r0r]r^rl __class__r1r2rds    zFlavaImageEmbeddings.__init__ embeddingsheightwidthc Csp|jdd}|jjdd}||kr||kr|jS|jdddf}|jddddf}|jd}||jj} ||jj} | d| d} } tjj|dtt |tt || dddd| t || t |fdd d }t| |jd kst| |jdkrt d t| t| fd |jd |jdfd| dddd dd|}tj|d|fddS)a" This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. Source: https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/image_transformer.py#L174 rNrg?r bicubicF) scale_factormode align_cornerszNumber of patches for images (z/) don't match the shape of position embedding ()dim)shapermr]r`r functional interpolatereshapeintmathsqrtpermute ValueErrorviewr;cat unsqueeze) r0rsrtrunpatchnum_posclass_pos_embedpatch_pos_embedr num_h_patches num_w_patchesr1r1r2interpolate_pos_encoding s0    .$z-FlavaImageEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posrcCs|j\}}}}|j||d}|\}} } |durB|j|| d} |dkr0||dd}|d| } |d| | | }|j |dd} t j | |fdd}|r_|| |||}n||j }||}|S)N)rrvr r?rr~)rrksizereexpandrrrtype_asrir;rrrmrp)r0rrr batch_sizerartrursseq_len_ mask_tokensmask cls_tokensr1r1r2forward/s    zFlavaImageEmbeddings.forwardF)NF)r7r8r9r:rrHrdr;Tensorrrr BoolTensorr __classcell__r1r1rqr2r\s&r\c sheZdZdZ    ddedeeeeeffded effd d Zdd ej de dej fddZ Z S)rjz# Image to Patch Embedding. r r r_r`rarbcstt|tjjs||f}t|tjjs||f}|d|d|d|d}||_||_||_t j ||||d|_ dS)Nrr) kernel_sizestride) rcrd isinstance collectionsabcIterabler_r`rlr Conv2d projection)r0r_r`rarbrlrqr1r2rdXs  zPatchEmbeddings.__init__Frrr(c Csx|j\}}}}|s.||jdks||jdkr.td|d|d|jdd|jdd ||ddd}|S)NrrzInput image size (*z) doesn't match model (z).rw)rr_rrflatten transpose)r0rrrrartruxr1r1r2rkszPatchEmbeddings.forward)rrr r r) r7r8r9r:rr r rdr;rrHrrr1r1rqr2rjSs $rjcsPeZdZdZfddZ   d deejdeejdeejfdd ZZ S) FlavaTextEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrvF) persistenttoken_type_ids)dtype)rcrdr Embedding vocab_sizerh pad_token_idword_embeddingsmax_position_embeddingsrmtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsrnrorpr*rregister_bufferr;arangerrgrrlongr0r]rqr1r2rdzs  zFlavaTextEmbeddings.__init__N input_idsrrc Cs|}|d}|dur|jddd|f}|durAt|dr6|jddd|f}||d|}|}n tj|tj|jjd}| |}| |} || } |j dkr]| |} | | 7} | | } || } | S)Nrrr)rdevicer)rrhasattrrrr;rgrrrrrrmrrp) r0rrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expanded inputs_embedsrrsrmr1r1r2rs&       zFlavaTextEmbeddings.forward)NNN) r7r8r9r:rdrr;rrrr1r1rqr2rws rcseZdZdeddffdd ZdejdejfddZ   dd ejd eejd eejd e de e ejejfe ejff ddZ Z S)FlavaSelfAttentionr]r(Ncst|j|jdkr t|ds td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)rcrdrhnum_attention_headsrrrattention_head_size all_head_sizer Linearqkv_biasquerykeyvaluernattention_probs_dropout_probrprrqr1r2rds  zFlavaSelfAttention.__init__rcCs6|dd|j|jf}|j|}|ddddS)Nrvrrwrr )rrrrr)r0r new_x_shaper1r1r2transpose_for_scoress z'FlavaSelfAttention.transpose_for_scoresF hidden_statesattention_mask head_maskoutput_attentionscCs||}|||}|||}||}t||dd} | t|j } |dur4| |} t j j | dd} t j j | dd} | | } |durQ| |} t| |} | dddd} | dd|jf} | j| } |rz| | f} | S| f} | S)Nrvr|r~rrwrr )rrrrr;matmulrrrrr rsoftmaxrpr contiguousrrr)r0rrrrmixed_query_layer key_layer value_layer query_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr1r1r2rs*      zFlavaSelfAttention.forwardNNF)r7r8r9FlavaPossibleConfigsrdr;rrrrHr r rrr1r1rqr2rs"rcsFeZdZdZdeddffdd Zdejdejdejfd d ZZ S) FlavaSelfOutputz The residual connection is defined in FlavaLayer (same as ViTLayer) instead of here (as is the case with other models), due to the layernorm applied before each block. r]r(Ncs.tt|j|j|_t|j|_dSrY) rcrdr rrhdensernrorprrqr1r2rd zFlavaSelfOutput.__init__r input_tensorcC||}||}|SrYrrpr0rrr1r1r2r  zFlavaSelfOutput.forward) r7r8r9r:rrdr;rrrr1r1rqr2rs$rcseZdZdeddffdd ZdeeddfddZ   dd ej d e ej d e ej d e de e ej ej fe ej ff ddZZS)FlavaAttentionr]r(Ncs*tt||_t||_t|_dSrY)rcrdr attentionroutputset pruned_headsrrqr1r2rds    zFlavaAttention.__init__headscCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr~)lenrrrrrrrrrrrrunion)r0rindexr1r1r2 prune_headss zFlavaAttention.prune_headsFrrrrcCs8|j||||d}||d|}|f|dd}|SN)rrrrr)rr)r0rrrr self_outputsattention_outputrr1r1r2r s zFlavaAttention.forwardr)r7r8r9rrdrrrr;rrrHr r rrr1r1rqr2rs"rcs<eZdZdeddffdd ZdejdejfddZZS) FlavaIntermediater]r(NcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrY) rcrdr rrhintermediate_sizerr hidden_actstrr intermediate_act_fnrrqr1r2rd2s   zFlavaIntermediate.__init__rcCrrY)rr r0rr1r1r2r;rzFlavaIntermediate.forward r7r8r9rrdr;rrrr1r1rqr2r1s rcsBeZdZdeddffdd Zdejdejdejfdd ZZS) FlavaOutputr]r(Ncs.tt|j|j|_t|j|_ dSrY) rcrdr rr rhrrnrorprrqr1r2rdCrzFlavaOutput.__init__rrcCs ||}||}||}|SrYrrr1r1r2rIs  zFlavaOutput.forwardrr1r1rqr2rBs$rcsxeZdZdZdeddffdd Z   ddejd eejd eejd e de e ejejfe ejff d d Z Z S) FlavaLayerz?This corresponds to the Block class in the timm implementation.r]r(Ncsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dSNrr)rcrdchunk_size_feed_forward seq_len_dimrrr intermediaterrr rrhrlayernorm_beforelayernorm_afterrrqr1r2rdUs    zFlavaLayer.__init__Frrrrc Csb|j|||||d}|d}|dd}||}||}||}|||}|f|}|Sr)rrrrr) r0rrrrself_attention_outputsrr layer_outputr1r1r2ras     zFlavaLayer.forwardr)r7r8r9r:rrdr;rrrHr r rrr1r1rqr2rRs"rcsneZdZdeddffdd Z     ddejd eejd eejd ed ed ede e e ffddZ Z S) FlavaEncoderr]r(Ncs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr1)rr-rr]r1r2 sz)FlavaEncoder.__init__..F) rcrdr]r ModuleListrangenum_hidden_layerslayergradient_checkpointingrrqrr2rds   zFlavaEncoder.__init__FTrrrroutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD]:\} } |r||f}|dur$|| nd} |jr7|jr7|| j||| |} n| ||| |} | d}|rK|| df}q|rS||f}|satdd|||fDSt|||dS)Nr1rrcss|] }|dur|VqdSrYr1)r-rGr1r1r2r3z'FlavaEncoder.forward..)last_hidden_stater attentions) enumerater r!training_gradient_checkpointing_func__call__r5r) r0rrrrr"r#all_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputsr1r1r2rs4    zFlavaEncoder.forward)NNFFT)r7r8r9rrdr;rrrHr r5rrrr1r1rqr2rs,  rcs2eZdZdeffdd ZdejfddZZS) FlavaPoolerr]cs*tt|j|j|_t|_dSrY)rcrdr rrhrTanh activationrrqr1r2rds zFlavaPooler.__init__rcCs(|dddf}||}||}|SNr)rr3)r0rfirst_token_tensor pooled_outputr1r1r2rs  zFlavaPooler.forwardrr1r1rqr2r1sr1aD This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`{config}`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a; Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`FlavaImageProcessor.__call__`] for details. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). interpolate_pos_encoding (`bool`, *optional*): Whether to interpolate the pre-trained position encodings. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) z Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, image_num_patches + text_seq_len, hidden_size)`): The concatenated hidden states of unimodal encoders. z Args: skip_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder. Useful if multimodal encoding is not going to be used. a Args: input_ids_masked (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. These ones are the masked version of the original task to be used with MLM. Indices can be obtained using [`AutoTokenizer`] along with [`DataCollatorForMaskedLanguageModeling`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) a image_attention_mask (`torch.FloatTensor` of shape `({1})`, *optional*): Mask to avoid performing attention on padding token indices specifically for images. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) skip_unmasked_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder for unmasked inputs. FLAVA pretraining doesn't need unmasked multimodal embeddings or outputs as of now. mlm_labels (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Labels for computing the left-to-right language and multimodal masked modeling loss (next word prediction). Indices should be in `[-100, 0, ..., text_config.vocab_size - 1]` (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., text_config.vocab_size - 1]`. mim_labels (`torch.LongTensor` of shape `(batch_size, image_num_patches)`, *optional*): Labels for computing the image and multimodal masked modeling loss. Indices should be in `[-100, 0, ..., image_config.vocab_size - 1]`. Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., image_config.vocab_size - 1]`. If not passed, they are generated automatically using the image codebook assigned to the model. By default, it uses [`FlavaImageCodebook`]. See [`FlavaImageCodebook`] to understand how to generate mim_labels. itm_labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation of MMM and global contrastive losses as well. return_loss (`bool`, *optional*, default to None): Whether to return calculated loss or not. z Parameters: image_codebook ([`nn.Module`]): If passed, the image codebook will be set to this. Otherwise. it will be initialized using the image_codebook_config defined in the config first as the first parameter. c@s>eZdZdZeZdZdZdee j e j e j fddfddZ dS) FlavaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. flavaTmoduler(NcCst|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj rF|jjjd|jjd|j durD|jj|j  dSdSt|tj r[|j j |jjddSdS)zInitialize the weightsg)meanstdNr)rr rrweightdatanormal_r]initializer_rangerzero_rrrfill_)r0r9r1r1r2 _init_weights\s     z"FlavaPreTrainedModel._init_weights)r7r8r9r:r config_classbase_model_prefixsupports_gradient_checkpointingr r rrrrBr1r1r1r2r7Rs &r7zeThe bare FLAVA Image Model transformer outputting raw hidden-states without any specific head on top.rcseZdZeZdZdZddedeffdd Zde j fd d Z d e j fd d Z de eeefddfddZeedeeeeded        ddeejdeejdeedeejdeejdeedeedeedeeeffddZZ S) FlavaImageModelzflava.image_modelrTr]add_pooling_layercXt|||_t||_t||_tj|j |j d|_ |r#t |nd|_ |dSNr)rcrdr]r\rsrencoderr rrhr layernormr1pooler post_initr0r]rGrqr1r2rdws    zFlavaImageModel.__init__r(cC|jjSrYrsrkr/r1r1r2get_input_embeddingsz$FlavaImageModel.get_input_embeddingsrcC ||j_dSrYrPr0rr1r1r2set_input_embeddings z$FlavaImageModel.set_input_embeddingsheads_to_pruneNcC*|D]\}}|jj|j|qdSz Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel NitemsrJr rrr0rWr rr1r1r2 _prune_headszFlavaImageModel._prune_headsbatch_size, image_num_patchesvision) checkpoint output_typerCmodalityexpected_outputrrrrrr"r#c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|||jj}|j|||d} |j| |||||d} | d} | | } |j durT| | nd} |sb| | f| ddSt | | | j | j dS)Nz You have to specify pixel_values)rrrrrr"r#rrr% pooler_outputrr&)r]rr"use_return_dictr get_head_maskrrsrJrKrLrrr&) r0rrrrrrr"r#embedding_outputencoder_outputssequence_outputr6r1r1r2rs: zFlavaImageModel.forwardTNNNNNNNN)!r7r8r9rrCrDmain_input_namerHrdr ModulerQrUrrrr]rFLAVA_IMAGE_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr!_CONFIG_CLASS_FOR_IMAGE_MODEL_DOC_EXPECTED_IMAGE_OUTPUT_SHAPErr;rrr r5rrr1r1rqr2rFmsV     rFzdThe bare FLAVA Text Model transformer outputting raw hidden-states without any specific head on top.cseZdZeZdZddedeffdd Zdefdd Z d e j fd d Z d e eeefddfddZeedeeeed        ddeejdeejdeejdeejdeejdeedeedeedeeeffddZZS)FlavaTextModelzflava.text_modelTr]rGcrHrI)rcrdr]rrsrrJr rrhrrKr1rLrMrNrqr1r2rds    zFlavaTextModel.__init__r(cCrOrYrsrr/r1r1r2rQrRz#FlavaTextModel.get_input_embeddingsrcCrSrYrwrTr1r1r2rUrVz#FlavaTextModel.set_input_embeddingsrWNcCrXrYrZr\r1r1r2r]r^zFlavaTextModel._prune_headsbatch_size, text_seq_lengthrarbrCrrrrrrr"r#c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|} |dur6tj| |jd}| ||jj }| || |j} |j |||d} |j | | ||||d} | d} || } |jdurl|| nd}|sz| |f| ddSt| || j| jdS)NzYou have to specify input_idsr)rrrrerrrf)r]rr"rhrrr;onesrrirget_extended_attention_maskrsrJrKrLrrr&)r0rrrrrrr"r#rextended_attention_maskrjrkrlr6r1r1r2rsJ zFlavaTextModel.forwardrmrn)r7r8r9rrCrDrHrdrjrQr rprUrrrr]rFLAVA_TEXT_INPUTS_DOCSTRINGrrrrsr _CONFIG_CLASS_FOR_TEXT_MODEL_DOCrr;rr r5rrr1r1rqr2rvsP    rvzjThe bare FLAVA Multimodal Model transformer outputting raw hidden-states without any specific head on top.cseZdZeZdZdZddeffdd Zdee e e fdd fd d Z e e d eeeed  ddejdeejdeejdeedeedeedeeeffddZZS)FlavaMultimodalModelzflava.multimodal_modelrTr]csvt|||_|jj|_|jrttdd|j|_ t ||_ tj |j|j d|_|r2t|nd|_|dSr)rcrdr] use_cls_tokenr rfr;rgrhrirrJrrrKr1rLrMrNrqr1r2rdCs    zFlavaMultimodalModel.__init__rWr(NcCrXrYrZr\r1r1r2r]Qr^z!FlavaMultimodalModel._prune_heads,batch_size, image_num_patches + text_seq_lenryrrrr"r#cCs&|dur|n|jj}|dur|n|jj}|dur|n|jj}|\}}} |jr=|j|dd} tj | |fdd}|d7}|durKtj ||f|j d}| ||jj }||||f|j } |j|| ||||d} | d} || } |jdur{|| nd}|s| |f| ddSt| || j| jdS)Nrvrr~rzrerrf)r]rr"rhrrrirr;rr{rrirr|rJrKrLrrr&)r0rrrrr"r#rrrrr}rkrlr6r1r1r2rYsD  zFlavaMultimodalModel.forwardrm)NNNNN)r7r8r9rrCrDrordrrrr]r!FLAVA_MULTIMODAL_INPUTS_DOCSTRINGrrrrsr&_CONFIG_CLASS_FOR_MULTIMODAL_MODEL_DOCr;rrrHr r5rrr1r1rqr2r9sD rz_The bare FLAVA Model transformer outputting raw hidden-states without any specific head on top.rcseZdZeZdeffdd Zeed       dde e j de e j de e j d e e j d e e d e e d e e d e j fddZeed        dde e j de e jde e de e j de e j d e e d e e d e e d e j fddZeedeeed           d de e jde e j de e j de e j de e j d e e jde e j de e d e e d e d e e d eeeffddZZS)! FlavaModelr]cs0t|t|jtstdt|jdt|jts(tdt|jdt|j t s;tddt|j d|j}|j}|j }|j |_ |j |_ |j |_|j |_t||_t||_t||_t|j|j |_t|j |j |_tt|jj|_t|j|j|_ t|j |j|_!|"dS)NzLconfig.text_config is expected to be of type FlavaTextConfig but is of type rzNconfig.image_config is expected to be of type FlavaImageConfig but is of type zMconfig.multimodal_config is expected to be of type FlavaMultimodalConfig but z is of type )#rcrdr text_configrrtype image_configrmultimodal_configrprojection_dimrhtext_hidden_sizeimage_hidden_sizemm_hidden_sizerv text_modelrF image_modelrmultimodal_modelr rimage_projectiontext_projectionrfr;tensorr]logit_scale_init_value logit_scaleimage_to_mm_projectiontext_to_mm_projectionrM)r0r]rrrrqr1r2rdsF        zFlavaModel.__init__rxNrrrrrr"r#r(c Cs8dt|j|||||||d}|d} || } | S)Na Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`FlavaTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], max_length=77, padding="max_length", return_tensors="pt" ... ) >>> text_features = model.get_text_features(**inputs) ```)rrrrrr"r#r)rrrsrr) r0rrrrrr"r# text_outputsr6 text_featuresr1r1r2get_text_featuress  zFlavaModel.get_text_featuresr_rrrrc Cs:dt|j||||||||d} | d} || } | S)Na Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`FlavaImageModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```)rrrrrr"rr#r)rrrsrr) r0rrrrrrr"r# image_outputsr6image_featuresr1r1r2get_image_featuress  zFlavaModel.get_image_featuresrrbrCTimage_attention_maskskip_multimodal_encoderc  Cs*| dur| n|jj} | stdd} d} d}d}|dur7|j|||| | | d}|d|d} } || d}d}d}d}d}|dur_|j||||| | | d}|d|d}}||d}d}d}|dur|dur|stj||gdd }|j || d }|d}| s| |||||fSt | |||||d S) a  Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=["a photo of a cat"], images=image, return_tensors="pt", padding=True) >>> outputs = model(**inputs) >>> image_embeddings = outputs.image_embeddings >>> text_embeddings = outputs.text_embeddings >>> multimodal_embeddings = outputs.multimodal_embeddings >>> outputs.image_embeddings.shape torch.Size([1, 197, 768]) >>> text_embeddings.shape torch.Size([1, 7, 768]) >>> multimodal_embeddings.shape torch.Size([1, 205, 768]) ``` NzRFLAVA model requires hidden states to work. Please set `output_hidden_states=True`)rrrrr"r#rrwrv)rrrrrr"r#rr~)r#)r"r#r$r%r&r') r]r#rrrrrr;rrr!)r0rrrrrrrrrr"r#r" image_statesimage_mm_projectionr#r$ text_statestext_mm_projectionr%r&r'multimodal_inputr1r1r2r+sp5  zFlavaModel.forward)NNNNNNNrn) NNNNNNNNNTN)r7r8r9rrCrdrr~rrrr;rrHr<rrqrrFLAVA_MODEL_INPUTS_DOCSTRINGrr! LongTensorr r rrrr1r1rqr2rs + +  1      rcs<eZdZdedeffdd ZdejdejfddZZS) FlavaImageCodebookResPathin_sizeout_sizec st|d}t}t|d<tj||ddd|d<t|d<tj||ddd|d<t|d <tj||ddd|d <t|d <tj||dd d|d <t||_dS)Nrelu_1r rrpaddingconv_1relu_2conv_2relu_3conv_3relu_4rconv_4)rcrdrr ReLUr Sequentialpath)r0rrkwargshid_sizerrqr1r2rds     z"FlavaImageCodebookResPath.__init__rr(cC ||SrY)rr0rr1r1r2r z!FlavaImageCodebookResPath.forward r7r8r9rrdr;rrrr1r1rqr2rsrcs@eZdZdededeffdd Zdejdejfdd ZZS) FlavaImageCodebookBlockrr num_layersc sPtd|d|_||krtj||ddd|_nt|_t|||_dS)Nrrwrr) rcrd post_gainr rid_pathIdentityrres_path)r0rrrrrqr1r2rds  z FlavaImageCodebookBlock.__init__rr(cCs|||j||SrY)rrrrr1r1r2rszFlavaImageCodebookBlock.forwardrr1r1rqr2rs rc sJeZdZd dededededef fdd Zd ejd ejfd d ZZ S)FlavaImageCodebookLayerGroupT num_blocksrrruse_poolcstt}t|D]!}|dkr t||||d|d<q t||||d|d<q |r8tjdd|d<t||_dS)Nrblock_rrw)rpool) rcrdrrrr MaxPool2drgroup)r0rrrrrblocksr-rqr1r2rds  z%FlavaImageCodebookLayerGroup.__init__rr(cCrrY)rrr1r1r2rrz$FlavaImageCodebookLayerGroup.forwardrm) r7r8r9rrHrdr;rrrr1r1rqr2rs$ra" The FLAVA's image codebook model inspired from DALL-E's original encoder. Outputs raw hidden states and can be used to generate image tokens for an image based on DALL-E's vocab. Used to generate labels for MIM. Use `get_codebook_indices` to get image tokens for an image. rcsxeZdZdZeZdZdZdedeffdd Z de j de j fd d Z de j de j fd d Z de jde j fd dZZS)FlavaImageCodebookrFr]rc sdt|||_|j|_|j|_|j|_|j|_|j|_|j|j}t}t |d<t j d|j|jddd|d<t}t j |jd|jddd|d <t |j|d|jd|j|d <t |j|d|jd |j|d <t |j|d |jd |j|d<t |j|d |jd|jdd|d<t ||d<t ||_||jjr|D]}d|_qdSdS)Nrelurrrconvr inputgroup_1rwgroup_2rgroup_3F)rgroup_4r)rcrdr] num_groupsinput_channelsnum_blocks_per_grouprhrrr rrrrrrMfreeze parameters requires_grad)r0r]rr output_blocksrparamrqr1r2rdsB      zFlavaImageCodebook.__init__r(cCs"dt||}tj|ddS)Na Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("{0}") >>> image_processor = AutoImageProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model.get_codebook_indices(**inputs) ``` r)axis)rr_CHECKPOINT_FOR_CODEBOOK_DOCrr;argmaxr0rz_logitsr1r1r2get_codebook_indicess z'FlavaImageCodebook.get_codebook_indicescCs||}tjdd|S)Nrr~)rr Softmaxrr1r1r2get_codebook_probs2s z%FlavaImageCodebook.get_codebook_probscCs`dtt|jdkrtd|jd|jd|jkr+td|jdd|j||S)Na Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("{0}") >>> image_processor = AutoImageProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model(**inputs) >>> print(outputs.shape) (1, 196) ``` rz input shape z is not 4drz input has z channels but model built for )rrrrrrrr)r0rr1r1r2r6s zFlavaImageCodebook.forward)r7r8r9rDrrCrorErrdr;rrrr<rrr1r1rqr2rs ,rc$eZdZfddZddZZS)FlavaPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSrI) rcrdr rrhrrr r r transform_act_fnrrrrqr1r2rdZs  z%FlavaPredictionHeadTransform.__init__cCs"||}||}||}|SrY)rrrr r1r1r2rcs   z$FlavaPredictionHeadTransform.forwardr7r8r9rdrrr1r1rqr2rYs  rcs&eZdZdfdd ZddZZS)FlavaMaskedPredictionHeadNcsbt||_t||_tj|j|jdd|_ t t |j|_ |dur*||j _|j |j _ dS)NFr)rcrdr]r transformr rrhrdecoderrfr;rgrr<)r0r]r<rqr1r2rdks  z"FlavaMaskedPredictionHead.__init__cCrrY)rrrr1r1r2rw  z!FlavaMaskedPredictionHead.forwardrYrr1r1rqr2rjs rcr) FlavaITMHeadcs.t||_t||_t|jd|_dS)Nrw) rcrdr]r1rLr rrhseq_relationshiprrqr1r2rd~s  zFlavaITMHead.__init__cCrrY)rLrrr1r1r2rrzFlavaITMHead.forwardrr1r1rqr2r}s rcr)FlavaGlobalContrastiveHeadcst||_|j|_dSrY)rcrdr]global_backprop_contrastiverrqr1r2rds  z#FlavaGlobalContrastiveHead.__init__c s2t|}tjrtjs!tjdjd}g}g}nQd}tj} |j r?tjj j }tjj j }n$fddt | D}fddt | D}tj |tj ||tjtj|jd}t|}t|}t|dd|} t|dd|} | | |fS)Nrrzcg|]}tqSr1r; zeros_liker)r$r1r2rz6FlavaGlobalContrastiveHead.forward..crr1rr)r"r1r2rrr)r;exp distributed is_availableis_initializedrrrget_world_sizerr r all_gatherrget_rankrrr) r0r"r$r temperaturelabelsimage_embeddings_alltext_embeddings_alllocal_batch_size world_sizelogits_per_imagelogits_per_textr1)r"r$r2rs,      z"FlavaGlobalContrastiveHead.forwardrr1r1rqr2rs rzk The FLAVA model for pretraining which outputs losses, embeddings, logits and transformer outputs. c)s:eZdZgdZd"dedeejffdd Zde j fdd Z e e d d eeed                 d#dee jdee jdee jdee jdee j dee j dee j dee jdee j dedee j dee j dee j deededeedeedeee j eff$d d!ZZS)$FlavaForPreTraining)zmmm_text_head.decoder.biaszmmm_image_head.decoder.biaszmlm_head.decoder.biaszmim_head.decoder.biasNr]image_codebookcst|t||_||_|jdur|jrt|j|_t|j |_ t|j |_ t ||_t|j |_t|j |_t||_|j j|_|j j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|dSrY)rcrdrr8r init_codebookrimage_codebook_configrrmim_headrmlm_headritm_headmmm_image_head mmm_text_headrglobal_contrastive_headrimage_vocab_sizetext_vocab_size mlm_weight mim_weightglobal_contrastive_weightce_ignore_index itm_weightmmm_image_weightmmm_text_weight skip_unmasked_multimodal_encoderrM)r0r]rrqr1r2rds,            zFlavaForPreTraining.__init__rcCs"|dkr||dd}|S)Nrwrrv)rrrrr1r1r2 _resize_to_2ds z!FlavaForPreTraining._resize_to_2dzbatch_size, text_seq_lenr_rTrinput_ids_maskedrcodebook_pixel_valuesrrrrrr mlm_labels mim_labels itm_labelsrr"r# return_lossr(c6Cs|dur|n|jj}|dur|n|jj}| dur| n|j} |dur,|dur,td|}|j|||||| | ||dd }|j||||| |||dd }d}|j}|j}|j}|j}|j }d}}}}}}} d}!}"}#}$d}%}&}'|dus~|dur| dur|r|j durt d|durt d|j |} |jdkr |dur |dur |}(| dur|| } ||}|j| |d<|(dd| d  dddf}(| |j})| |)}*|(|)ddf}(||(}!|rtj|!d |j|*d }||j9}n||(}!|jdkrj|durj|durj|}+| dure|| } |+dd| d  dddf}+| |j})| |)},|+|)ddf}+||+}"|rdtj|"d |j|,d }||j9}n||+}"|jdkr|dur||}%| dur| d}-t|- |-|-!dg}|rtj|%| } | |j9} |dur||}| dur| |} | dur| |} |dur2|j"dkr2|}(|d d }.|(ddd d |.ddf}(|dur|(|}(| dur-|| } ||}|j| |d<| |j})| |)}*|(|)ddf}(|#|(}$|r,tj|$d |j|*d }||j"9}n|#|(}$|dur|j$dkr|}+|+dd|d  dddf}+|durY|+|}+| dur|| } | |j})| |)},|+|)ddf}+|%|+}#|rtj|#d |j|,d }||j$9}n|%|+}#|dur|dur|j&dkr|j'|dddddf}/tjj(|/d d }/|j)|dddddf}0tjj(|0d d }0|jj*j+,t-t.|/|0|/|jj*\}&}'}1|dur|&|}&|'|}'|1|}1|rtj|&|1}2tj|'|1}3|2|3d }||j&9}t0||| |||d }4|r4|41s4t2dd|43D}|s||j4durC|j45nd||j6durP|j65nd|j |j7dur^|j75nd||j4durk|j45nd||j6durx|j65nd||j7dur|j75nd|!|"|%|&|&|$|#f}5|r|41s||4f|5}5t8dd|5DSt9d&id|d|4d|d|j4d|d|j6d|j d|j7d|d|j4d|d|j6d|d|j7d|!d |"d!|%d"|&d#|'d$|$d%|#S)'ai Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import FlavaForPreTraining, AutoProcessor >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> model = FlavaForPreTraining.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> text = ["a photo of a cat"] >>> inputs = processor( ... images=[image], ... text=text, ... return_masks=True, ... return_codebook_pixels=True, ... padding=True, ... max_length=77, ... return_tensors="pt", ... ) >>> output = model(**inputs) ``` Return: Nz`input_ids_masked` isn't passed which means MLM loss won't be calculated correctlySetting it to `input_ids` so that model can work. Please pass it if this is unintentional. This is usually OKAY if you are doing inference on unmasked text...T) rrrrrrrrr"r#) rrrrrrrr"r#z`return_loss` is set to True but the image codebook is not initialized and no `mim_labels` have been passed. Reinstantiate the model with `init_codebook` set to True or pass in your custom `mim_labels`z`codebook_pixel_value` are required to generate `mim_labels` if loss is expected. Call `AutoProcessor` with `return_codebook_pixels` set to Truerrrvrwr~)r?r@rArBrCrDcss |] }|dur |ndVqdSr4r1)r-rJr1r1r2r3sz.FlavaForPreTraining.forward..css|] }|dur|VqdSrYr1)r-rr1r1r2r3r$rJrKr"r#r$r%r&r'rLrMrNrOrPrQrRrSrTrUrVrWrXr1):r]rhr!rloggerwarningr8r"r$r&r RuntimeErrorrrrrrnerr r r cross_entropyrrrr rrr r;whereanynewrrrrrr normalizerrr=clamp_LOGIT_SCALE_CLAMP_MINLOGIT_SCALE_CLAMP_MAXrr>rFsumrEr#r+r%r'r5rI)6r0rrrrrrrrrrrrr rr"r#r! flava_outputflava_masked_outputpos_maskr"r$rLrNrP total_lossmim_lossmlm_loss mmm_text_lossmmm_image_lossgc_lossitm_lossrRrSrXrWrTrrsequence_for_image masked_tokensmim_labels_filteredsequence_for_textmlm_labels_filtered pos_pairs end_indextext_embeddingimage_embedding gc_labels gc_loss_image gc_loss_text flava_lossesrr1r1r2rs8         "       "                   "                    zFlavaForPreTraining.forwardrY)NNNNNNNNNNNNNNTNN)r7r8r9_tied_weights_keysrrr rprdr;rrr"FLAVA_PRETRAINING_INPUTS_DOCSTRINGrrrrIrr<rHr r rrr1r1rqr2rs|       r)^r:rrr dataclassesrtypingrrrrrr r r;torch.utils.checkpointr activationsr modeling_outputsrrmodeling_utilsrrrutilsrrrrrrconfiguration_flavarrrrr get_loggerr7r"rsrrtrrru#FLAVA_PRETRAINED_MODEL_ARCHIVE_LIST,FLAVA_CODEBOOK_PRETRAINED_MODEL_ARCHIVE_LISTr,r-rr!r>rIrpr\rjrrrrrrrrr1FLAVA_START_DOCSTRINGFLAVA_INPUTS_DOCSTRING_COMMON!FLAVA_IMAGE_INPUTS_DOCSTRING_BASErq FLAVA_TEXT_INPUTS_DOCSTRING_BASEr~r!FLAVA_MODEL_INPUTS_DOCSTRING_BASErrG'FLAVA_PRETRAINING_START_DOCSTRING_EXTRAr7rrrFrvrrrrrrrrrrrr1r1r1r2s  $    %$e]$9E*.3       +/ _ e ^  u (