o hj@s$dZddlmZddlmZmZmZmZddlZddl Zddlm Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZmZd d lmZmZddlmZeeZdZ gdZ!eGdddeZ"Gddde j#Z$dZ%ede%Gddde Z&dZ'ede%Gddde&Z(dS)z PyTorch Llava model.) dataclass)ListOptionalTupleUnionN)nn)PreTrainedModel)ACT2FN)Cache) ModelOutput)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) AutoModelAutoModelForCausalLM) LlavaConfigr)zllava-hf/llava-1.5-7b-hfzllava-hf/llava-1.5-13b-hfzllava-hf/bakLlava-v1-hfc@seZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed<dS) LlavaCausalLMOutputWithPasta Base class for Llava causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlosslogitspast_key_values hidden_states attentionsimage_hidden_states)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrrrr$r$^/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/llava/modeling_llava.pyr1s  rcs*eZdZdeffdd ZddZZS)LlavaMultiModalProjectorconfigcsNttj|jj|jjdd|_t|j |_ tj|jj|jjdd|_ dS)NT)bias) super__init__rLinear vision_config hidden_size text_configlinear_1r projector_hidden_actactlinear_2selfr' __class__r$r%r*]s  z!LlavaMultiModalProjector.__init__cCs"||}||}||}|SN)r/r1r2)r4image_featuresrr$r$r%forwardds   z LlavaMultiModalProjector.forward)rrrrr*r9 __classcell__r$r$r5r%r&\sr&ac This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`LlavaConfig`] or [`LlavaVisionConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zSThe bare LLaMA Model outputting raw hidden-states without any specific head on top.c@s*eZdZeZdZdZdgZdZddZ dS)LlavaPreTrainedModelmodelTLlavaVisionAttentioncCst|jdr |jjn|jjj}t|dr|jjjd|dt|tj tj fr>|j jjd|d|j dur<|j j dSdSt|tjr]|j jjd|d|jdur_|j j|j dSdSdS)Ninitializer_rangeclass_embeddingg)meanstd)hasattrr'r>r.r?datanormal_ isinstancerr+Conv2dweightr(zero_ Embedding padding_idx)r4modulerAr$r$r% _init_weightss"      z"LlavaPreTrainedModel._init_weightsN) rrrr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_2rLr$r$r$r%r;|s r;a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)): The tensors corresponding to the input images. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details ([]`LlavaProcessor`] uses [`CLIPImageProcessor`] for processing images). attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zIThe LLAVA model which consists of a vision backbone and a language model.c!sHeZdZdeffdd ZddZddZdd Zd d Zd d Z ddZ ddZ d-de e dejfddZddZeeeeed             d.dejdejde ejde ejde eejde ejd e e d!e ed"e ejd#e ed$e ed%e ed&e edeeeffd'd(Z d/d)d*Z!d+d,Z"Z#S)0LlavaForConditionalGenerationr'csft|t|j|_t||_|j|_t j|j |j d|_ |j jdur*|j jnd|_|dS)N)attn_implementation)r)r*r from_configr, vision_towerr&multi_modal_projector vocab_sizerr._attn_implementationlanguage_modelr' pad_token_id post_initr3r5r$r%r*s   z&LlavaForConditionalGeneration.__init__cC |jSr7)rZget_input_embeddingsr4r$r$r%r^ z2LlavaForConditionalGeneration.get_input_embeddingscC|j|dSr7)rZset_input_embeddings)r4valuer$r$r%rbz2LlavaForConditionalGeneration.set_input_embeddingscCr]r7)rZget_output_embeddingsr_r$r$r%rer`z3LlavaForConditionalGeneration.get_output_embeddingscCrar7)rZset_output_embeddings)r4new_embeddingsr$r$r%rfrdz3LlavaForConditionalGeneration.set_output_embeddingscCrar7)rZ set_decoder)r4decoderr$r$r%rhrdz)LlavaForConditionalGeneration.set_decodercCr]r7)rZ get_decoderr_r$r$r%rjr`z)LlavaForConditionalGeneration.get_decodercCr]r7)rZ tie_weightsr_r$r$r%rkr`z)LlavaForConditionalGeneration.tie_weightsNnew_num_tokensreturncCs0|j||}|j|jj_|j|j_|j|_|Sr7)rZresize_token_embeddingsnum_embeddingsr'r.rX)r4rlpad_to_multiple_of model_embedsr$r$r%rns   z5LlavaForConditionalGeneration.resize_token_embeddingscCs|j\}}}|j\} } t|dddft|jk } ||jjk} tj| dd} | |d| }t||jjk\}}t | |dddd}|d|dddf}| re||dddf7}|||f}tj | |||j |j d}tj | ||j |j d}|||f|||f<|||f|||f<tj |dkdd}|| dd|dddfkM}||jddkrtdt| d|d|d|||<||O}| dd|dkd}|||fS) NrTdimrdtypedevicerzIThe input provided to the model are wrong. The number of image tokens is z1 while the number of image given to the model is z=. This prevents correct indexing and breaks batch generation.)shaper!sumtensorr[r'image_token_indexmaxwherecumsumzerosrurvallnumel ValueError contiguousreshape masked_fill_)r4r8 inputs_embeds input_idsattention_mask position_ids num_imagesnum_image_patches embed_dim batch_sizesequence_length left_paddingspecial_image_token_masknum_special_image_tokens max_embed_dim batch_indicesnon_image_indicesnew_token_positions nb_image_padtext_to_overwritefinal_embeddingfinal_attention_maskimage_to_overwriter$r$r%$_merge_input_ids_with_image_featuress@  $   " zBLlavaForConditionalGeneration._merge_input_ids_with_image_features) output_typerMr pixel_valuesrrrrvision_feature_layervision_feature_select_strategylabels use_cacheoutput_attentionsoutput_hidden_states return_dictc Cs"| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|dur.|n|jj}|dur||}|dur|jddkr|j|dd}|j |}|dkrb|ddddf}n|dkri|}n t d|jj| |}| |||||\}}}| durt ||jjt j} n^|dur|dur|jddkr|ddddddddf}t |dk\}}|jd d}t j|jd||jdf|j|jd }d|||f<t j||fdd }t j|dd d d}|j||||| | | | d }|d}d}| durm|dur?|d ddf}|d dd ddf||jdk}| d ddf|| jdk}n|d dd ddf}| d ddf}t}||d |d |d |j}| s|f|dd}|dur|f|S|St|||j |j |j!dS)ag Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Returns: Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, LlavaForConditionalGeneration >>> model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> prompt = "\nUSER: What's the content of the image?\nASSISTANT:" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=text, images=image, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "There seems to be a stop sign" ```NrT)rdefaultfullz$Unexpected select feature strategy: rrTrtrr)rrrrrrrr.)rrrrr)"r'rruse_return_dictrrr^rwrVrrrWrr! full_like ignore_indextolongr|onesrurvcatrx unsqueezerZrrCrossEntropyLossviewsizerrr)r4rrrrrrrrrrrrr image_outputsselected_image_featurer8first_layer_past_key_value batch_indexnon_attended_tokens target_seqlenextended_attention_maskoutputsrrshift_attention_mask shift_logits shift_labelsloss_fctoutputr$r$r%r9Bs1           *& z%LlavaForConditionalGeneration.forwardc Ks|durzt|tr|}|j}n |ddjd}}|dur;|jd|jdkr;|dd|jd| df}n'||jdkrM|dd|df}n|jj|vrb|dd|jdddf}||krz|durz|dd||jd df}|dd} |dur| dur| dd} | |dkd|r| dd|jd df} |dur|durd|i} nd|i} | | ||d||d | S) NrrrrrTrrr)rrrrr) rEr get_seq_length seen_tokensrwr'rzgetrr}rupdate) r4rrrrrkwargs cache_length past_lengthr model_inputsr$r$r%prepare_inputs_for_generations> "     z;LlavaForConditionalGeneration.prepare_inputs_for_generationcOs|jj|i|Sr7)rZ_reorder_cache)r4argsrr$r$r%rsz,LlavaForConditionalGeneration._reorder_cache)NN) NNNNNNNNNNNNN)NNNN)$rrrrr*r^rbrerfrhrjrkrintrrIrnrrLLAVA_INPUTS_DOCSTRINGrr_CONFIG_FOR_DOCr! LongTensorr"Tensorrstrboolrrr9rrr:r$r$r5r%rRsv 4         5rR))r dataclassesrtypingrrrrr!torch.utils.checkpointrr activationsr cache_utilsr modeling_outputsr utilsr rrrautorrconfiguration_llavar get_loggerrloggerr#LLAVA_PRETRAINED_MODEL_ARCHIVE_LISTrModuler&LLAVA_START_DOCSTRINGr;rrRr$r$r$r%s>        )B