o h @sdZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z mZddlmZmZmZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)ddl*m+Z+e&,e-Z.dZ/dZ0gdZ1dZ2dZ3dgZ4eGdddeZ5dTde j de6de7de j fddZ8Gdd d ej9Z:Gd!d"d"ej9Z;Gd#d$d$ej9ZGd)d*d*ej9Z?Gd+d,d,ej9Z@Gd-d.d.ej9ZAGd/d0d0ej9ZBGd1d2d2ej9ZCGd3d4d4ej9ZDGd5d6d6eZEd7ZFd8ZGe$d9eFGd:d;d;eEZHGdeFGd?d@d@eEZJe$dAeFGdBdCdCeEZKGdDdEdEej9ZLGdFdGdGej9ZMGdHdIdIej9ZNGdJdKdKej9ZOGdLdMdMej9ZPe$dNeFGdOdPdPeEZQe$dQeFGdRdSdSeEe)ZRdS)Uz PyTorch BEiT model.N) dataclass)ListOptionalTupleUnion)Tensornn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedLMOutputSemanticSegmenterOutput)PreTrainedModel) find_pruneable_heads_and_indicesmeshgridprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) BackboneMixin) BeitConfigrz%microsoft/beit-base-patch16-224-pt22k)rizmicrosoft/beit-base-patch16-224ztabby, tabby catc@seZdZdZdS)BeitModelOutputWithPoolinga Class for outputs of [`BeitModel`]. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N)__name__ __module__ __qualname____doc__r&r&\/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/beit/modeling_beit.pyr!Esr!Finput drop_probtrainingreturncCsd|dks|s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. r(rrr)dtypedevice)shapendimtorchrandr.r/floor_div)r)r*r+ keep_probr0 random_tensoroutputr&r&r' drop_path_s r9csTeZdZdZd deeddffdd Zdejdejfdd Z de fd d Z Z S) BeitDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nr*r,cst||_dSN)super__init__r*)selfr* __class__r&r'r=vs  zBeitDropPath.__init__ hidden_statescCst||j|jSr;)r9r*r+r>rAr&r&r'forwardzszBeitDropPath.forwardcCs d|jS)Nzp={})formatr*r>r&r&r' extra_repr}s zBeitDropPath.extra_reprr;) r"r#r$r%rfloatr=r2rrCstrrF __classcell__r&r&r?r'r:ss r:csLeZdZdZdeddffdd Zd dejdeej dejfd d Z Z S) BeitEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. configr,Ncstttdd|j|_|jr!ttdd|j|_ nd|_ t ||_ |j j }|j r?ttd|d|j|_nd|_t|j|_dS)Nr)r<r=r Parameterr2zeros hidden_size cls_tokenuse_mask_token mask_tokenBeitPatchEmbeddingspatch_embeddings num_patches use_absolute_position_embeddingsposition_embeddingsDropouthidden_dropout_probdropout)r>rKrTr?r&r'r=s  zBeitEmbeddings.__init__ pixel_valuesbool_masked_posc Cs|||jdur|jddddddfnd\}\}}|\}}}|durB|j||d} |d| } |d| | | }|j|dd} |jdur_| |jddddddf} tj | |fdd}| |}|||ffS)Nrdim) rSrVsizerQexpand unsqueezetype_asrOr2catrY) r>rZr[ embeddings patch_height patch_width batch_sizeseq_len_ mask_tokensw cls_tokensr&r&r'rCs*     zBeitEmbeddings.forwardr;) r"r#r$r%rr=r2rr BoolTensorrCrIr&r&r?r'rJs*rJcsBeZdZdZfddZd dejdeejdejfdd ZZ S) rRz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r|n||f}t|tjj r)|n||f}|d|d|d|d}|d|d|d|df}||_||_||_||_ ||_ t j ||||d|_dS)Nrr kernel_sizestride)r<r= image_size patch_size num_channelsrN isinstance collectionsabcIterablerT patch_shaperConv2d projection)r>rKrqrrrsrNrTrxr?r&r'r=s   zBeitPatchEmbeddings.__init__NrZposition_embeddingr,c Cs|j\}}}}||jkrtd||}|jd|jd}} |durG|d|jd|jdddddd}tjj ||| fdd}||}| d dd}||| ffS) NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r rrr\bicubic)r_mode) r0rs ValueErrorrzviewrxpermuter functional interpolateflatten transpose) r>rZr{rgrsheightwidthrdrerfr&r&r'rCs"    zBeitPatchEmbeddings.forwardr;) r"r#r$r%r=r2rrrCrIr&r&r?r'rRs *rRceZdZddedeeddffdd ZddZ  dd ej d eej d e d edde e ej e ej ej fff ddZ ZS)BeitSelfAttentionNrK window_sizer,cst|j|jdkr t|ds td|jfd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t j |j|jdd|_ t |j|j|_ t |j|_|rct||d|_dSd|_dS) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .F)biasr)r<r=rNnum_attention_headshasattrrintattention_head_size all_head_sizerLinearquerykeyvaluerWattention_probs_dropout_probrYBeitRelativePositionBiasrelative_position_biasr>rKrr?r&r'r=s"   zBeitSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nr\rr|rr )r_rrrr)r>x new_x_shaper&r&r'transpose_for_scoress z&BeitSelfAttention.transpose_for_scoresFrA head_maskoutput_attentionsrrcCs ||}|||}|||}||}t||dd} | t|j } |j dur:| | d} |durB| |} t j j| dd} || } |durW| |} t| |} | dddd} | dd|jf} | j| } |r| | f} | S| f} | S)Nr\rr]r|rr )rrrrr2matmulrmathsqrtrrrarrsoftmaxrYr contiguousr_rr)r>rArrrmixed_query_layer key_layer value_layer query_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr&r&r'rCs,       zBeitSelfAttention.forwardr;NFN)r"r#r$rrtupler=rr2rboolrrrCrIr&r&r?r'rs" rcsHeZdZdZdeddffdd Zd dejdejdejfd d ZZ S) BeitSelfOutputz The residual connection is defined in BeitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rKr,Ncs.tt|j|j|_t|j|_dSr;) r<r=rrrNdenserWrXrYr>rKr?r&r'r=4 zBeitSelfOutput.__init__rA input_tensorcC||}||}|Sr;rrY)r>rArgammar&r&r'rC9  zBeitSelfOutput.forwardr;) r"r#r$r%rr=r2rrCrIr&r&r?r'r.s&rcr) BeitAttentionNrKrr,cs.tt||d|_t||_t|_dS)Nr)r<r=r attentionrr8set pruned_headsrr?r&r'r=As   zBeitAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr])lenrrrrrrrrrr8rrunion)r>headsindexr&r&r' prune_headsGs zBeitAttention.prune_headsFrArrrrcCs6|||||}||d|}|f|dd}|S)Nrr)rr8)r>rArrr self_outputsattention_outputrr&r&r'rCYszBeitAttention.forwardr;r)r"r#r$rrrr=rr2rrrrrCrIr&r&r?r'r@s" rc<eZdZdeddffdd ZdejdejfddZZS) BeitIntermediaterKr,NcsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr;) r<r=rrrNintermediate_sizerrt hidden_actrHr intermediate_act_fnrr?r&r'r=is   zBeitIntermediate.__init__rAcCrr;)rrrBr&r&r'rCqrzBeitIntermediate.forward r"r#r$rr=r2rrCrIr&r&r?r'rhsrcr) BeitOutputrKr,Ncs.tt|j|j|_t|j|_ dSr;) r<r=rrrrNrrWrXrYrr?r&r'r=yrzBeitOutput.__init__rAcCrr;rrBr&r&r'rC~rzBeitOutput.forwardrr&r&r?r'rxsrcseZdZdZddedeededdffdd Z  dd e j d ee j d e dedde e e j e e j e j fff ddZZS) BeitLayerz?This corresponds to the Block class in the timm implementation.Nr(rKrdrop_path_rater,cst|j|_d|_t||d|_t||_t||_ t j |j |j d|_|dkr/t|nt |_t j |j |j d|_|j}|dkrct j|t|j dd|_t j|t|j dd|_dSd\|_|_dS) Nrrepsr(rT) requires_grad)NN)r<r=chunk_size_feed_forward seq_len_dimrrr intermediaterr8r LayerNormrNlayer_norm_epslayernorm_beforer:Identityr9layernorm_afterlayer_scale_init_valuerLr2oneslambda_1lambda_2)r>rKrr init_valuesr?r&r'r=s    zBeitLayer.__init__FrArrrrc Cs|j|||||d}|d}|dd}|jdur |j|}|||}||}||}||}|jdur@|j|}|||}|f|}|S)N)rrrr)rrrr9rrr8r) r>rArrrself_attention_outputsrr layer_outputr&r&r'rCs&         zBeitLayer.forward)Nr(r)r"r#r$r%rrrrGr=r2rrrrrCrIr&r&r?r'rs"$rcs:eZdZdededdffdd ZdejfddZZ S) rrKrr,Nc st||_d|ddd|ddd|_tt|j|j|_ t |d}t |d}t t ||gdd}t |d}|dddddf|dddddf}|ddd}|dddddf|dd7<|dddddf|dd7<|dddddfd|dd9<tj|d|ddfd|jd}|d|ddddf<|jd|dddf<|jd|dddf<|jd|d <|jd |d d dS) Nr|rrr ij)indexing)r_r.r\)rrrelative_position_indexF) persistent)r<r=rnum_relative_distancerrLr2rMrrelative_position_bias_tablearangestackrrrrr.sumregister_buffer) r>rKrcoords_hcoords_wcoordscoords_flattenrelative_coordsrr?r&r'r=s. & ,&&*z!BeitRelativePositionBias.__init__cCsV|j|jd|jd|jdd|jd|jddd}|dddS)Nr\rrr|)rrrrrr)r>rr&r&r'rCs.z BeitRelativePositionBias.forward) r"r#r$rrr=r2rrCrIr&r&r?r'rsrcsleZdZddedeeddffdd Z    dd ejd eejd e d e d e de ee ff ddZ Z S) BeitEncoderNrKrr,csvt|_jrtd|_nd|_ddtdjj Dt fddt j D|_ d|_dS)NrcSsg|]}|qSr&)item.0rr&r&r' sz(BeitEncoder.__init__..rcs(g|]}tjr nd|dqS)N)rr)ruse_relative_position_biasrirKdprrr&r'rs F)r<r=rK!use_shared_relative_position_biasrrr2linspacernum_hidden_layersr ModuleListrangelayergradient_checkpointingrr?rr'r=s  zBeitEncoder.__init__FTrArroutput_hidden_states return_dictc Cs|rdnd}|r dnd}t|jD]D\}} |r||f}|dur$||nd} |jr6|jr6|| j|| |} n|jdur?|nd} | || || } | d}|rU|| df}q|r]||f}|sktdd|||fDSt|||dS)Nr&rrcss|] }|dur|VqdSr;r&)rvr&r&r' &sz&BeitEncoder.forward..)last_hidden_staterA attentions) enumeraterrr+_gradient_checkpointing_func__call__rrr) r>rArrrrall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputsrr&r&r'rCs:     zBeitEncoder.forwardr;)NFFT)r"r#r$rrrr=r2rrrrrCrIr&r&r?r'rs&  rc@s(eZdZdZeZdZdZdZddZ dS)BeitPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. beitrZTcCst|tjtjtjfr%|jjjd|jj d|j dur#|j j dSdSt|tj rH|jjjd|jj d|j durF|jj|j  dSdSt|tjr]|j j |jjddSdS)zInitialize the weightsr()meanstdNg?)rtrrryConvTranspose2dweightdatanormal_rKinitializer_rangerzero_ Embedding padding_idxrfill_)r>moduler&r&r' _init_weights9s     z!BeitPreTrainedModel._init_weightsN) r"r#r$r%r config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrr&r&r&r'r.s raF This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`BeitConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aL Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`BeitImageProcessor.__call__`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare Beit Model transformer outputting raw hidden-states without any specific head on top.cseZdZddededdffdd Zdd Zd d Zee e e e e d ed       ddeejdeejdeejdeedeedeedeee ffddZZS) BeitModelTrKadd_pooling_layerr,Ncspt|||_t||_t||jjjd|_|j rt nt j |j |jd|_|r/t|nd|_|dS)Nrr)r<r=rKrJrdrrSrxencoderuse_mean_poolingrrrrNr layernorm BeitPoolerpooler post_init)r>rKr$r?r&r'r=qs   zBeitModel.__init__cC|jjSr;rdrSrEr&r&r'get_input_embeddingszBeitModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr%rrr)r>heads_to_prunerrr&r&r' _prune_headsszBeitModel._prune_headsvision) checkpoint output_typermodalityexpected_outputrZr[rrrrcCs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td|||jj}|||\}\}} |j|||||d} | d} | | } |j durU| | nd} |sl| dura| | fn| f} | | ddSt | | | j | j dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_valuesrrrrrr)r pooler_outputrAr)rKrruse_return_dictr get_head_maskrrdr%r'r)r!rAr)r>rZr[rrrrembedding_outputrerfencoder_outputssequence_output pooled_output head_outputsr&r&r'rCs6 zBeitModel.forward)TNNNNNN)r"r#r$rrr=r-r1rBEIT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr!_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr2rrmrrrCrIr&r&r?r'r#lsB  r#cr) r(rKr,Ncs2t|jrtj|j|jd|_dSd|_dS)Nr)r<r=r&rrrNrr'rr?r&r'r=s zBeitPooler.__init__rAcCsL|jdur|ddddddf}||d}|S|dddf}|S)Nrr)r'r)r>rA patch_tokensr>r&r&r'rCs zBeitPooler.forwardrr&r&r?r'r(sr(aBeit Model transformer with a 'language' modeling head on top. BEiT does masked image modeling by predicting visual tokens of a Vector-Quantize Variational Autoencoder (VQ-VAE), whereas other vision models like ViT and DeiT predict RGB pixel values. As a result, this class is incompatible with [`AutoModelForMaskedImageModeling`], so you will need to use [`BeitForMaskedImageModeling`] directly if you wish to do masked image modeling with BEiT.cseZdZdeddffdd Zeeeee d       dde e j de e j d e e j d e e j d e ed e ed e edeeeffddZZS)BeitForMaskedImageModelingrKr,NcsTt||j|_t|dd|_tj|j|jd|_ t |j|j |_ | dS)NFr$r)r<r= num_labelsr#rrrrNrr'r vocab_sizelm_headr*rr?r&r'r=s  z#BeitForMaskedImageModeling.__init__r4rrZr[rlabelsrrrcCs|dur|n|jj}|j||||||d}|d} || } || ddddf} d} |dur;t} | | ||} |sQ| f|dd} | durO| f| S| St| | |j|jdS)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, BeitForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> model = BeitForMaskedImageModeling.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, logits = outputs.loss, outputs.logits >>> list(logits.shape) [1, 196, 8192] ```N)r[rrrrrrlosslogitsrAr) rKr9rr'rJr rrAr)r>rZr[rrLrrrrr=prediction_scoresmasked_lm_lossloss_fctr8r&r&r'rCs2/ z"BeitForMaskedImageModeling.forward)NNNNNNN)r"r#r$rr=rrArrrCrr2rrmrrrrCrIr&r&r?r'rFs8  rFz Beit Model transformer with an image classification head on top (a linear layer on top of the average of the final hidden states of the patch tokens) e.g. for ImageNet. cseZdZdeddffdd Zeeeee e e d      dde e jde e jd e e jd e ed e ed e edeee ffd dZZS)BeitForImageClassificationrKr,NcsRt||j|_t|dd|_|jdkrt|j|jnt|_ | dS)NTrGr) r<r=rHr#rrrrNr classifierr*rr?r&r'r=Hs $ z#BeitForImageClassification.__init__)r3r4rr6rZrrLrrrc Csj|dur|n|jj}|j|||||d}|r|jn|d}||} d} |dur|jjdurR|jdkr8d|j_n|jdkrN|jtj ksI|jtj krNd|j_nd|j_|jjdkrpt } |jdkrj| | | } n+| | |} n%|jjdkrt } | | d|j|d} n|jjdkrt} | | |} |s| f|dd} | dur| f| S| St| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr7r regressionsingle_label_classificationmulti_label_classificationr\r|rM)rKr9rr8rT problem_typerHr.r2longrr squeezer rr rrAr) r>rZrrLrrrrr>rOrNrRr8r&r&r'rCTsL    "       z"BeitForImageClassification.forwardr@)r"r#r$rr=rrAr_IMAGE_CLASS_CHECKPOINTrrC_IMAGE_CLASS_EXPECTED_OUTPUTrr2rrrrrCrIr&r&r?r'rS@s<  rScseZdZdZ   ddededeeeeeffdeeeeefefd ed eeeeeffd d ffd d Z de j d e j fddZ Z S)BeitConvModuleaD A convolutional block that bundles conv/norm/activation layers. This block simplifies the usage of convolution layers, which are commonly used with a norm layer (e.g., BatchNorm) and activation layer (e.g., ReLU). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rFr in_channels out_channelsropaddingrdilationr,Ncs<ttj||||||d|_t||_t|_dS)N)r^r_ror`rra) r<r=rryconv BatchNorm2dbnReLU activation)r>r^r_ror`rrar?r&r'r=s  zBeitConvModule.__init__r)cCs"||}||}||}|Sr;)rbrdrf)r>r)r8r&r&r'rCs   zBeitConvModule.forward)rFr)r"r#r$r%rrrrHrr=r2rrCrIr&r&r?r'r]s* r]csDeZdZdedededdffdd Zdejdejfd d ZZS) BeitPyramidPoolingBlock pool_scaler^channelsr,NcsLtt|t||ddg|_t|jD] \}}|t||qdS)Nrro) r<r=rAdaptiveAvgPool2dr]layersr add_modulerH)r>rhr^rirrr?r&r'r=s  z BeitPyramidPoolingBlock.__init__r)cCs|}|jD]}||}q|Sr;)rl)r>r) hidden_staterr&r&r'rCs  zBeitPyramidPoolingBlock.forward) r"r#r$rr=r2rrCrIr&r&r?r'rgs rgc sXeZdZdZdeedfdedededdf fd d Zd ej de ej fd d Z Z S)BeitPyramidPoolingModulea Pyramid Pooling Module (PPM) used in PSPNet. Args: pool_scales (tuple[int]): Pooling scales used in Pooling Pyramid Module. in_channels (int): Input channels. channels (int): Channels after modules, before conv_seg. align_corners (bool): align_corners argument of F.interpolate. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. pool_scales.r^ri align_cornersr,Ncsht||_||_||_||_g|_t|D]\}}t|||d}|j || t ||qdS)N)rhr^ri) r<r=rprqr^riblocksrrgappendrmrH)r>rpr^rirqrrhblockr?r&r'r=s  z!BeitPyramidPoolingModule.__init__rcCsHg}|jD]}||}tjj||ddd|jd}||q|S)Nr|bilinearr_r~rq)rrrrrr_rqrs)r>rppm_outsppmppm_outupsampled_ppm_outr&r&r'rCs  z BeitPyramidPoolingModule.forward) r"r#r$r%rrrr=r2rrrCrIr&r&r?r'ros* " rocsHeZdZdZdeddffdd ZddZd ejdejfd d Z Z S) BeitUperHeadz Unified Perceptual Parsing for Scene Understanding. This head is the implementation of [UPerNet](https://arxiv.org/abs/1807.10221). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rKr,Ncst|j|_|jgd|_|j|_d|_tj|j|j dd|_ t |j|jd|j|jd|_ t |jdt|j|j|jddd|_t|_t|_|jddD] }t ||jdd}t |j|jddd}|j||j|qWt t|j|j|jddd|_dS) NFrrjr\)rqr ror`)r<r=rprNr^rirqrryrHrTro psp_modulesr]r bottleneckr lateral_convs fpn_convsrsfpn_bottleneck)r>rKr^l_convfpn_convr?r&r'r=s>     zBeitUperHead.__init__cCs:|d}|g}|||tj|dd}||}|S)Nr\rr])extendr~r2rcr)r>inputsrpsp_outsr8r&r&r' psp_forward"s  zBeitUperHead.psp_forwardencoder_hidden_statescsfddtjDt}t|dddD]$}|djdd}|dtjj ||dj d|d<q fd dt|dD}|dt|dddD]}tjj |||djdddj d||<qbt j |dd } |}|}|S) Ncsg|] \}}||qSr&r&)rr lateral_conv)rr&r'r-sz(BeitUperHead.forward..rrr\r|rurvcsg|] }j||qSr&)rr)lateralsr>r&r'r:sr])rrrsrrrr0rrrrqr2rcrrT)r>rused_backbone_levelsr prev_shapefpn_outsr8r&)rrr>r'rC+s$   zBeitUperHead.forward) r"r#r$r%rr=rr2rrCrIr&r&r?r'r{s & r{c s`eZdZdZ ddedededeeeeeffd d f fd d Zd e j d e j fddZ Z S) BeitFCNHeada Fully Convolution Networks for Semantic Segmentation. This head is implemented of [FCNNet](https://arxiv.org/abs/1411.4038>). Args: config (BeitConfig): Configuration. in_channels kernel_size (int): The kernel size for convs in the head. Default: 3. dilation (int): The dilation rate for convs in the head. Default: 1. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. r|r rrKin_indexrorar,Nc st|j|_|j|_|j|_|j|_ ||_ |d|}g}| t |j|j|||dt |jdD]}| t |j|j|||dq5|jdkrQt|_ntj||_|j rjt |j|j|j||dd|_tj|j|jdd|_dS)Nr|)ror`rarrr}rj)r<r=rNr^auxiliary_channelsriauxiliary_num_convs num_convsauxiliary_concat_input concat_inputrrsr]rrrconvs Sequentialconv_catryrHrT)r>rKrrora conv_paddingrrr?r&r'r=Xs6     zBeitFCNHead.__init__rcCs@||j}||}|jr|tj||gdd}||}|S)Nrr])rrrrr2rcrT)r>rrAr8r&r&r'rCzs   zBeitFCNHead.forward)r|r r) r"r#r$r%rrrrr=r2rrCrIr&r&r?r'rIs"rzf Beit Model transformer with a semantic segmentation head on top e.g. for ADE20k, CityScapes. cseZdZdeddffdd ZddZeeee e d      dd e e j d e e j d e e j d e ed e ede edeee ffddZZS)BeitForSemanticSegmentationrKr,Nc st||j|_t|dd|_t|jjdkrtdt t j |j |j dddt |j t t j |j |j ddd|_t t j |j |j ddd|_t |_t jddd|_t||_|jrft|nd|_|dS)NFrGr|zBeitForSemanticSegmentation requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.r|rn)r<r=rHr#rrrK out_indicesrrrrrNrcGELUfpn1fpn2rfpn3 MaxPool2dfpn4r{ decode_headuse_auxiliary_headrauxiliary_headr*rr?r&r'r=s*     z$BeitForSemanticSegmentation.__init__c Cstjj||jddddd}|dur"tjj||jddddd}t|jjd}|||}|}|durA|||} ||jj| 7}|S)NrruFrv) ignore_index)rrrr0r rKsemantic_loss_ignore_indexauxiliary_loss_weight) r>rOauxiliary_logitsrLupsampled_logitsupsampled_auxiliary_logitsrR main_lossrNauxiliary_lossr&r&r' compute_losss  z(BeitForSemanticSegmentation.compute_lossrKrZrrLrrrcs||dur|njj}|dur|njj}j|||d|d}|r#|jn|d}fddt|D} |jdjjjjfdd| D} j j j j g} t t| D] } | | | | | | <qY| } d} jdurw| } d}|durjjdkrtd | | |}|s|r| f|dd}n | f|d d}|dur|f|S|St|| |r|jnd|jd S) aV labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, BeitForSemanticSegmentation >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> model = BeitForSemanticSegmentation.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```NTr7rcs$g|]\}}|djjvr|qSr-)rKr)ridxfeaturerEr&r'rs$z7BeitForSemanticSegmentation.forward..rcs<g|]}|ddddddfddddqS)Nrrr|r\)rreshaper)rgpatch_resolutionr&r'rs0z/The number of labels should be greater than oner|rM)rKr9rrrArr0rqrrrrrrrrrrrHrrrr)r>rZrrLrrrrrfeaturesopsrrOrrNr8r&)rgrr>r'rCsP$       z#BeitForSemanticSegmentation.forwardr@)r"r#r$rr=rrrArrrCrr2rrrrrCrIr&r&r?r'rs4   rzM BEiT backbone, to be used with frameworks like DETR and MaskFormer. c sjeZdZfddZddZeeeee d   dde de e d e e d e e d ef d d Z ZS) BeitBackbonec sttfddtjdD|_t|_t|jj j d|_ j rut |jjdkr9tdj}ttj||dddtj|jd ttj||ddd|_ttj||ddd|_t|_tjddd|_|dS) Ncsg|]}jqSr&)rN)rrirKr&r'r&sz)BeitBackbone.__init__..rrr|zBeitBackbone requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.r|rnr)r<r=_init_backbonerr num_featuresrJrdrrSrxr%add_fpnrrKrrrNrrrrcbatch_norm_epsrrrrrrrr*)r>rKrNr?rr'r="s*     zBeitBackbone.__init__cCr+r;r,rEr&r&r'r-@r.z!BeitBackbone.get_input_embeddingsrKNrZrrrr,cCs||dur|n|jj}|dur|n|jj}|dur|n|jj}|jd}||\}\}}|j|d||d} |r:| jn| d} d} t|j | D].\} } | |j vrt|jj ro| ddddddf} | ddd} | |d||} | | f7} qF|jjr|| d|| d|| d|| d g} t| } |s|r| f| dd}|S| f| dd}|St| |r| jnd| jd S) aL Returns: Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224") >>> model = AutoBackbone.from_pretrained( ... "microsoft/beit-base-patch16-224", out_features=["stage1", "stage2", "stage3", "stage4"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```NrT)rrrrr&r|r\r ) feature_mapsrAr)rKr9rrr0rdr%rAzip stage_names out_featuresreshape_hidden_statesrrrrrrrrrr)r>rZrrrrgr;rerfrrArstagernr8r&r&r'rCCsJ#        zBeitBackbone.forward)NNN)r"r#r$r=r-rrArrrCrrrrCrIr&r&r?r'rs&  r)r(F)Sr%collections.abcrur dataclassesrtypingrrrrr2torch.utils.checkpointrrtorch.nnr r r activationsr modeling_outputsrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrrrutils.backbone_utilsrconfiguration_beitr get_loggerr"loggerrCrBrDr[r\"BEIT_PRETRAINED_MODEL_ARCHIVE_LISTr!rGrr9Moduler:rJrRrrrrrrrrrBEIT_START_DOCSTRINGrAr#r(rFrSr]rgror{rrrr&r&r&r's        -1M( =&F Y]R%%U;