o h@sdZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!m"Z"m#Z#d dl$m%Z%e"&e'Z(dZ)dZ*dgZ+eGdddeZ,Gddde j-Z.Gddde j-Z/Gddde j-Z0Gddde j-Z1Gddde j-Z2Gddde j-Z3Gdd d e j-Z4Gd!d"d"e j-Z5Gd#d$d$e j-Z6Gd%d&d&e j-Z7Gd'd(d(eZ8d)Z9d*Z:d+Z;e d,e9Gd-d.d.e8ZGd4d5d5e j-Z?Gd6d7d7e j-Z@e d8e9Gd9d:d:e8ZAe d;e9Gde;Gd?d@d@e8ZCe dAe9GdBdCdCe8ZDdS)Dz PyTorch ViLT model.N) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesmeshgridprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) ViltConfigrzdandelin/vilt-b32-mlmc@sfeZdZUdZdZeejed<dZ ejed<dZ ee e ejed<dZ ee e ejed<dS)(ViltForImagesAndTextClassificationOutputa Class for outputs of [`ViltForImagesAndTextClassification`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the attention weights of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosslogits hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrr'r'\/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/vilt/modeling_vilt.pyr9s rcs6eZdZdZfddZd ddZ d dd ZZS) ViltEmbeddingsz Construct the text and patch embeddings. Text embeddings are equivalent to BERT embeddings. Patch embeddings are equivalent to ViT embeddings. cstt||_ttdd|j|_ t ||_ |j j }ttd|d|j|_ t|j|j|_t|j|_||_dSNr)super__init__TextEmbeddingstext_embeddingsr Parameterr$zeros hidden_size cls_tokenViltPatchEmbeddingspatch_embeddings num_patchesposition_embeddings Embeddingmodality_type_vocab_sizetoken_type_embeddingsDropouthidden_dropout_probdropoutconfig)selfr=r5 __class__r'r(r,\s    zViltEmbeddings.__init__cs|jjjj\}}}}||}|dddddddf}tjj||jd|jdfd}|dddfj dddddf} |dddfj dddddf} |j\} } |j j |j j } |j ddddddfddd| | | tjfddt| | Ddd}|ddd}|ddd}tjtt|jd t|jd d d d dj|jd }|ddddddddf}||jd|jdd d d }|dd}|d}dksdustts| | }|n | | }t||jddd|jdddddf}fdd|D}fdd|D}dd|D}dd|D}fdd|D}g}t t|||D]B\}\}}}|dkrut!t"|}|#|||qStj!t"||dd}|#tj|||||gddqStj|dd}||dddf|dddff| d | }||dddf|dddff| d }||dddf|dddff| d d}||dddf|dddff| d | }|j$| d d }tj||fdd}tj|j dddddfdddddf| d d |fdd}||}|%|}tjt"|jdd||gdd}|||fffS)Nr sizerrdimc sBg|]\}}tjtjj||fdddd|d|fqS)bilinearT)rDmode align_cornersr)r functionalpad interpolate).0hw)height spatial_poswidthr'r( xs z/ViltEmbeddings.visual_embed..ij)indexingdeviceF)as_tuplec$g|]}dddf|kqSNrr'rMu) valid_idxr'r(rS$cr[r\r'r]) non_valid_idxr'r(rSr`cSg|]}|dqSrrCrMvr'r'r(rScSrbrcrCrdr'r'r(rSrfcsg|]}|qSr'r'rdmax_image_lengthr'r(rST) replacement)&r4 projectionweightshapefloatrrJrLlongsumr= image_size patch_sizer6 transposeviewr$catzipflattenstackrarangetorYexpand isinstanceintmaxminnonzerounique enumerate multinomialonesappendr2r<)r> pixel_values pixel_maskrh_phpwxx_maskx_hx_w batch_size num_channels patch_dim pos_embed patch_indexeffective_resolution unique_rows valid_row_idxnon_valid_row_idx valid_numsnon_valid_numspad_numsselectirenvp valid_choice pad_choice cls_tokensr')rPrhrarQr_rRr( visual_embedksx  $$$0  &     (.,..8 &zViltEmbeddings.visual_embedrc  Cs|j|||d} |dur|j|||jjd\}} } n|d} |dur%d}| |tj|tj| j d} ||tj | |tj| j d}tj | |gdd} tj || gdd} | | fS)N) input_idstoken_type_ids inputs_embedsrgrdtyperYrE) r.rr=rhrwr9r$ zeros_likerorY full_likeru)r>rattention_maskrrrr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmasksr'r'r(forwards&   zViltEmbeddings.forward)rA)r)r r!r"r#r,rr __classcell__r'r'r?r(r)Ss   ar)cs*eZdZdZfddZdddZZS)r-zGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrUF) persistentr)r)r+r,rr7 vocab_sizer1 pad_token_idword_embeddingsmax_position_embeddingsr6type_vocab_sizer9 LayerNormlayer_norm_epsr:r;r<getattrrregister_bufferr$ryr{r0rrDror>r=r?r'r(r,s  zTextEmbeddings.__init__Nc Cs|dur |}n|dd}|d}|dur$|jddd|f}|durNt|drC|jddd|f}||d|}|}n tj|tj|jjd}|durW| |}| |} || } |j dkrn| |} | | 7} | | } || } | S)NrUrrrrr)rDrhasattrrr{r$r0rorYrr9rr6rr<) r>rrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr9rr6r'r'r(rs,        zTextEmbeddings.forward)NNNNr r!r"r#r,rrr'r'r?r(r-s r-cs(eZdZdZfddZddZZS)r3z# Image to Patch Embedding. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizestride)r+r,rqrrrr1r| collectionsabcIterabler5rConv2drk)r>r=rqrrrr1r5r?r'r(r,+s  zViltPatchEmbeddings.__init__cCs.|j\}}}}||jkrtd||}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.)rmr ValueErrorrk)r>rrrrPrRrr'r'r(r:s  zViltPatchEmbeddings.forwardrr'r'r?r(r3&s r3c.eZdZfddZddZd ddZZS) ViltSelfAttentioncst|j|jdkr t|ds td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r+r,r1num_attention_headsrrr}attention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer:attention_probs_dropout_probr<rr?r'r(r,Es  zViltSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrUrrBrr )rDrrrtpermute)r>r new_x_shaper'r'r(transpose_for_scoresWs z&ViltSelfAttention.transpose_for_scoresNFcCs||}|||}|||}||}t||dd} | t|j } |dur4| |} t j dd| } | | } |durI| |} t| |} | dddd} | dd|jf} | j| } |rr| | f} | S| f} | S)NrUrTrErrBrr )rrrrr$matmulrsmathsqrtrrSoftmaxr<r contiguousrDrrt)r>rr head_maskoutput_attentionsmixed_query_layer key_layer value_layer query_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr'r'r(r\s(      zViltSelfAttention.forwardNNF)r r!r"r,rrrr'r'r?r(rDs rcsFeZdZdZdeddffdd Zdejdejdejfd d ZZ S) ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r=returnNcs.tt|j|j|_t|j|_dSN) r+r,rrr1denser:r;r<rr?r'r(r, zViltSelfOutput.__init__r input_tensorcC||}||}|Srrr<r>rrr'r'r(r  zViltSelfOutput.forward) r r!r"r#rr,r$Tensorrrr'r'r?r(rs$rcr) ViltAttentioncs*tt||_t||_t|_dSr)r+r,r attentionroutputset pruned_headsrr?r'r(r,s    zViltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrE)lenrrrrrrrrrrrrunion)r>headsindexr'r'r( prune_headss zViltAttention.prune_headsNFcCs6|||||}||d|}|f|dd}|S)Nrr)rr)r>rrrr self_outputsattention_outputrr'r'r(rszViltAttention.forwardr)r r!r"r,rrrr'r'r?r(rs rcs<eZdZdeddffdd ZdejdejfddZZS) ViltIntermediater=rNcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r+r,rrr1intermediate_sizerr| hidden_actstrr intermediate_act_fnrr?r'r(r,s   zViltIntermediate.__init__rcCrr)rrr>rr'r'r(rrzViltIntermediate.forward r r!r"rr,r$rrrr'r'r?r(r sr csBeZdZdeddffdd Zdejdejdejfdd ZZS) ViltOutputr=rNcs.tt|j|j|_t|j|_ dSr) r+r,rrr r1rr:r;r<rr?r'r(r,rzViltOutput.__init__rrcCs ||}||}||}|Srrrr'r'r(rs  zViltOutput.forwardrr'r'r?r(rs$rcs*eZdZdZfddZdddZZS) ViltLayerz?This corresponds to the Block class in the timm implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nrr)r+r,chunk_size_feed_forward seq_len_dimrrr  intermediaterrrrr1rlayernorm_beforelayernorm_afterrr?r'r(r,s    zViltLayer.__init__NFc Csj|j|||||d}|d}|dd}|||j}||}||}|||}|f|}|S)N)rrr)rrrzrYrrr) r>rrrrself_attention_outputsr r layer_outputr'r'r(rs     zViltLayer.forwardrrr'r'r?r(rs  rcs0eZdZfddZ     dddZZS) ViltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr')r)rMrr=r'r(rSriz(ViltEncoder.__init__..F) r+r,r=r ModuleListrangenum_hidden_layerslayergradient_checkpointingrr?rr(r,s   zViltEncoder.__init__NFTc Cs|rdnd}|r dnd}t|jD]:\} } |r||f}|dur$|| nd} |jr7|jr7|| j||| |} n| ||| |} | d}|rK|| df}q|rS||f}|satdd|||fDSt|||dS)Nr'rrcss|] }|dur|VqdSrr'rdr'r'r( )sz&ViltEncoder.forward..)last_hidden_staterr)rr r!training_gradient_checkpointing_func__call__tupler ) r>rrrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputsr'r'r(rs8    zViltEncoder.forward)NNFFTr r!r"r,rrr'r'r?r(rs  rc@s,eZdZdZeZdZdZddgZddZ dS) ViltPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. viltTr)rcCst|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj rF|jjjd|jjd|j durD|jj|j  dSdSt|tj r[|j j |jjddSdS)zInitialize the weightsg)meanstdNg?)r|rrrrldatanormal_r=initializer_rangerzero_r7rrfill_)r>moduler'r'r( _init_weights<s     z!ViltPreTrainedModel._init_weightsN) r r!r"r#r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modulesr:r'r'r'r(r01s r0aH This model is a PyTorch `torch.nn.Module `_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ViltConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViltImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_images, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViltImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, num_images, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. image_embeds (`torch.FloatTensor` of shape `(batch_size, num_images, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare ViLT Model transformer outputting raw hidden-states without any specific head on top.c seZdZdfdd ZddZddZdd Zeee e e d  dd e e jd e e jde e jde e jde e jde e jde e jde e jde ede ede ede edee ee jffddZZS) ViltModelTcsXt|||_t||_t||_tj|j |j d|_ |r#t |nd|_ |dSNr)r+r,r=r)rrencoderrrr1r layernorm ViltPoolerpooler post_init)r>r=add_pooling_layerr?r'r(r,s    zViltModel.__init__cCs |jjjSrrr.rr>r'r'r(get_input_embeddingss zViltModel.get_input_embeddingscCs||jj_dSrrG)r>rr'r'r(set_input_embeddingsszViltModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrAr rr)r>heads_to_pruner rr'r'r( _prune_headsszViltModel._prune_heads output_typer;Nrrrrrrrrrrr(r)rc  Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd| \}}|durU|jn|j}|duretj ||f|d}|durq|durqtd|dur}|dur}td|dur|j dn|j d}||krtd |durtj ||jj |jj f|d}| ||jj }|j|||||||| d \}}||| }|j|||| | | d }|d}||}|jdur||nd}| s||f|d dSt|||j|jd S)a Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerUz5You have to specify either input_ids or inputs_embedsrXzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrr(r)r)r# pooler_outputrr)r=rr(use_return_dictr%warn_if_padding_and_no_attention_maskrDrYr$rrmrq get_head_maskrrget_extended_attention_maskrArBrDr rr)r>rrrrrrrrrrr(r)rtext_batch_sizerrYimage_batch_sizeembedding_outputextended_attention_maskencoder_outputssequence_output pooled_outputr'r'r(rsp'    zViltModel.forward)T NNNNNNNNNNNN)r r!r"r,rIrJrMrVILT_INPUTS_DOCSTRINGrr _CONFIG_FOR_DOCrr$ LongTensorr%r}boolrrrrr'r'r?r(r?s\       r?c$eZdZfddZddZZS)rCcs*tt|j|j|_t|_dSr)r+r,rrr1rTanh activationrr?r'r(r,]s zViltPooler.__init__cCs(|dddf}||}||}|Sr\)rrc)r>rfirst_token_tensorr[r'r'r(rbs  zViltPooler.forwardr/r'r'r?r(rC\s rCzU ViLT Model with a language modeling head on top as done during pretraining. c seZdZddgZfddZddZddZee d e e e d  dd e ejd e ejde ejde ejde ejde ejde ejde ejde ejde ede ede edee eejffddZZS)ViltForMaskedLMzmlm_score.decoder.weightzmlm_score.decoder.biascs,t|t||_t||_|dSr)r+r,r?r1 ViltMLMHead mlm_scorerErr?r'r(r,ts    zViltForMaskedLM.__init__cCs|jjSrrgdecoderrHr'r'r(get_output_embeddings}sz%ViltForMaskedLM.get_output_embeddingscCs ||j_dSrrh)r>new_embeddingsr'r'r(set_output_embeddingss z%ViltForMaskedLM.set_output_embeddingszbatch_size, sequence_lengthrNNrrrrrrrrlabelsrr(r)rc  Cs| dur| n|jj} |j||||||||| | | d } | dd\}}|dur+|jdn|jd}|ddd|f|dd|df}}||}d}| durgt}| |j} ||d|jj | d}| s}|f| dd}|dur{|f|S|St ||| j | j dS)a labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrrrrrrr(r)rBrrUrrrr) r=rQr1rmrgrrzrYrtrr rr)r>rrrrrrrrrmrr(r)rrZr[ text_seq_len text_featuresr mlm_logitsmasked_lm_lossloss_fctrr'r'r(rs@I*  zViltForMaskedLM.forwardr\)r r!r"_tied_weights_keysr,rjrlrr]formatrr r^rr$r_r%r`rrrrr'r'r?r(reks\         recra)ViltPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSr@) r+r,rrr1rr|r rr transform_act_fnrrrr?r'r(r,s  z$ViltPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rrxrrr'r'r(rs   z#ViltPredictionHeadTransform.forwardr/r'r'r?r(rws  rwcs&eZdZdfdd ZddZZS)rfNcsbt||_t||_tj|j|jdd|_ t t |j|_ |dur*||j _|j |j _ dS)NFr)r+r,r=rw transformrrr1rrir/r$r0rrl)r>r=rlr?r'r(r,s  zViltMLMHead.__init__cCrr)ryri)r>rr'r'r(rs  zViltMLMHead.forwardrr/r'r'r?r(rfs rfz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. c eZdZfddZeeeeed            dde e j de e j de e j de e j d e e j d e e j d e e j d e e j d e e j de e de e de e deeee j ffddZZS)ViltForQuestionAnsweringc slt||j|_t||_tt|j|jdt |jdt t|jd|j|_ | dS)NrB) r+r, num_labelsr?r1r Sequentialrr1rGELU classifierrErr?r'r(r,"s   z!ViltForQuestionAnswering.__init__rNNrrrrrrrrrmrr(r)rc  Cs| dur| n|jj} |j||||||||| | | d } | r| jn| d}||}d}| dur@| |j} tj || | j d}| sV|f| dd}|durT|f|S|St ||| j | j dS)a labels (`torch.FloatTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the visual question answering loss. This tensor must be either a one-hot encoding of all answers that are applicable for a given example in the batch, or a soft encoding indicating which answers are applicable, where 1.0 is the highest score. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrnrrBro)r=rQr1rPrrzrYrrJ binary_cross_entropy_with_logitsrmrrrr>rrrrrrrrrmrr(r)rrPrrrr'r'r(r3s:1  z ViltForQuestionAnswering.forwardr\r r!r"r,rr]rrr^rr$r_r%r`rrrrr'r'r?r(r{sV       r{z Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. c rz)ViltForImageAndTextRetrievalcs2t|t||_t|jd|_|dSr*) r+r,r?r1rrr1 rank_outputrErr?r'r(r,s   z%ViltForImageAndTextRetrieval.__init__rNNrrrrrrrrrmrr(r)rc  Cs| dur| n|jj} |j||||||||| | | d } | r| jn| d}||}d}| dur8| |j} td| sN|f| dd}|durL|f|S|St||| j | j dS)a' labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NrnrzTraining is not yet supported.rBro) r=rQr1rPrrzrYNotImplementedErrorrrrrr'r'r(rs:-  z$ViltForImageAndTextRetrieval.forwardr\rr'r'r?r(rsV       rzq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. c rz)"ViltForImagesAndTextClassificationc svt||j|_t||_|j}tt|j ||j |t |j |t t|j ||j|_ | dSr)r+r,r|r?r1 num_imagesrr}rr1rr~rrE)r>r=rr?r'r(r,s   z+ViltForImagesAndTextClassification.__init__rNNrrrrrrrrrmrr(r)rc CsD| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur,|jdkr,|d}|dur:|jdkr:|d}|durC|jdnd} | durT|durR|jdnd} | |jjkr^tdg}| rdgnd}| rjgnd}t | D]l}|j ||||dur|dd|ddddddfnd|dur|dd|ddddfnd|||dur|dd|ddddfnd|d| | | d }| r|j n|d}| || r| |j | r| |jqptj|dd}||}d}| durt}| |j} ||d|j| d}| s|||f}|dur|f|S|St||||d S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrr z\Make sure to match the number of images in the model with the number of images in the input.) rrrrrrrrrr(r)rUrEro)r=rr(rQndim unsqueezermrrrr1rPrrrr$rurrrzrYrtr|r)r>rrrrrrrrrmrr(r)rpooler_outputsrrrrrPr[rrrtrr'r'r(r sp/      ,&&       z*ViltForImagesAndTextClassification.forwardr\)r r!r"r,rr]rrr^rr$r_r%r`rrrrr'r'r?r(rsV       rz ViLT Model with a token classification head on top (a linear layer on top of the final hidden-states of the text tokens) e.g. for Named-Entity-Recognition (NER) tasks. c rz)ViltForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dS)NF)rF) r+r,r|r?r1rr:r;r<rr1rrErr?r'r(r,s  z#ViltForTokenClassification.__init__rNNrrrrrrrrrmrr(r)rc  Cs| dur| n|jj} |j||||||||| | | d } | d}|dur'|jdn|jd}||}||ddd|f}d}| durZt}| |j} || d|j | d}| sp|f| dd}|durn|f|S|St ||| j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Returns: NrnrrrUrBro)r=rQr1rmr<rrrzrYrtr|rrr)r>rrrrrrrrrmrr(r)rrZtext_input_sizerrrtrr'r'r(rs@  z"ViltForTokenClassification.forwardr\)r r!r"r,rr]rrr^rr$r_r%r`rrrrr'r'r?r(r~sV       r)Er#collections.abcrr dataclassesrtypingrrrrr$torch.utils.checkpointrtorch.nnr activationsr modeling_outputsr r r rrrmodeling_utilsr pytorch_utilsrrrutilsrrrrconfiguration_viltr get_loggerr loggerr^_CHECKPOINT_FOR_DOC"VILT_PRETRAINED_MODEL_ARCHIVE_LISTrModuler)r-r3rrrr rrrr0VILT_START_DOCSTRINGr]4VILT_IMAGES_AND_TEXT_CLASSIFICATION_INPUTS_DOCSTRINGr?rCrerwrfr{rrrr'r'r'r(s        9=#&5 78j_