o h;{ @s,dZddlZddlmZmZmZmZddlZddlZddlm Z ddl m Z ddl m Z mZddlmZdd lmZmZdd lmZmZmZmZmZd d lmZeeZd ZdZ gdZ!dgZ"d>dej#de$de%dej#fddZ&Gddde j'Z(Gddde j'Z)Gddde j'Z*Gddde j'Z+Gd d!d!e j'Z,Gd"d#d#e j'Z-Gd$d%d%e j'Z.Gd&d'd'e j'Z/Gd(d)d)e j'Z0Gd*d+d+eZ1d,Z2d-Z3ed.e2Gd/d0d0e1Z4Gd1d2d2e j'Z5Gd3d4d4e j'Z6Gd5d6d6e j'Z7Gd7d8d8e j'Z8Gd9d:d:e j'Z9ed;e2Gd||}|j\}}}}|ddd}||}|||fS)Nr@r)rFrflatten transposerH)r/ pixel_values embeddings_heightwidthr'r'r(r4ks   z"GLPNOverlapPatchEmbeddings.forwardr7r8r9r:r.r4r>r'r'r0r(r?\s  r?cs4eZdZdZfddZddZ d ddZZS) GLPNEfficientSelfAttentionzSegFormer's efficient self-attention mechanism. Employs the sequence reduction process introduced in the [PvT paper](https://arxiv.org/abs/2102.12122).cst||_||_|j|jdkr td|jd|jdt|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_||_|dkrktj||||d|_t||_dSdS)NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r)rBrC)r-r.rKnum_attention_heads ValueErrorintattention_head_size all_head_sizerLinearquerykeyvalueDropoutattention_probs_dropout_probdropoutsr_ratiorEsrrGrHr/configrKrVsequence_reduction_ratior0r'r(r.zs,  z#GLPNEfficientSelfAttention.__init__cCs6|dd|j|jf}||}|ddddS)Nrr@rr)sizerVrYviewpermute)r/r3 new_shaper'r'r(transpose_for_scoress z/GLPNEfficientSelfAttention.transpose_for_scoresFcCs&|||}|jdkr6|j\}}}|ddd||||}||}|||dddd}||}|||} || |} t || dd} | t |j} tjj| dd} || } t | | } | dddd} | dd|jf}| |} |r| | f}|S| f}|S)Nrrr@rgdimr)rlr\rbrrjreshapercrHr]r^r matmulrMmathsqrtrYr functionalsoftmaxra contiguousrhrZri)r/r3rQrRoutput_attentions query_layer batch_sizeseq_lenrJ key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputsr'r'r(r4s*        z"GLPNEfficientSelfAttention.forwardF)r7r8r9r:r.rlr4r>r'r'r0r(rTvs   rTcs$eZdZfddZddZZS)GLPNSelfOutputcs*tt|||_t|j|_dSr,)r-r.rr[denser_hidden_dropout_probra)r/rerKr0r'r(r.s zGLPNSelfOutput.__init__cCs||}||}|Sr,)rra)r/r3 input_tensorr'r'r(r4s  zGLPNSelfOutput.forwardr7r8r9r.r4r>r'r'r0r(rs rcs.eZdZfddZddZdddZZS) GLPNAttentioncs6tt||||d|_t||d|_t|_dS)N)rerKrVrf)rK)r-r.rTr/rr&set pruned_headsrdr0r'r(r.s  zGLPNAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrn)lenr r/rVrYrr r\r]r^r&rrZunion)r/headsindexr'r'r( prune_headss zGLPNAttention.prune_headsFcCs6|||||}||d|}|f|dd}|S)Nrr)r/r&)r/r3rQrRrw self_outputsattention_outputrr'r'r(r4szGLPNAttention.forwardr)r7r8r9r.rr4r>r'r'r0r(rs  rc&eZdZdfdd ZddZZS) GLPNDWConvc s(ttj||dddd|d|_dS)NrrT)biasgroups)r-r.rrEdwconv)r/ror0r'r(r.s zGLPNDWConv.__init__cCsD|j\}}}|dd||||}||}|ddd}|S)Nrr@)rrMrirrL)r/r3rQrRryrzrJr'r'r(r4s  zGLPNDWConv.forward)rrr'r'r0r(rsrcr) GLPNMixFFNNcslt|p|}t|||_t||_t|jt r"t |j|_ n|j|_ t|||_ t |j|_dSr,)r-r.rr[dense1rr isinstance hidden_actr=rintermediate_act_fndense2r_rra)r/re in_featureshidden_features out_featuresr0r'r(r. s   zGLPNMixFFN.__init__cCsD||}||||}||}||}||}||}|Sr,)rrrrar)r/r3rQrRr'r'r(r4s     zGLPNMixFFN.forward)NNrr'r'r0r(r s rcs*eZdZdZfddZdddZZS) GLPNLayerzCThis corresponds to the Block class in the original implementation.csntt||_t||||d|_|dkrt|nt|_ t||_ t ||}t |||d|_ dS)N)rKrVrfr)rr)r-r.rrG layer_norm_1r attentionr*Identityr) layer_norm_2rXrmlp)r/rerKrVr)rf mlp_ratiomlp_hidden_sizer0r'r(r.&s    zGLPNLayer.__init__Fc Csr|j|||||d}|d}|dd}||}||}|||||}||}||} | f|}|S)N)rwrr)rrr)rr) r/r3rQrRrwself_attention_outputsrr mlp_output layer_outputr'r'r(r44s    zGLPNLayer.forwardrrSr'r'r0r(r#s rcs,eZdZfddZ   dddZZS) GLPNEncoderc sTt|_ddtdjtjD}g}tj D]"}| t j |j ||dkr3jnj|dj|dqt||_g}d}tj D]@}g}|dkrb|j|d7}tj|D]}| tj|j||||j|j|dqi| t|qQt||_tfddtj D|_dS)NcSsg|]}|qSr')item).0xr'r'r( Tsz(GLPNEncoder.__init__..rr)rIrCrJrK)rKrVr)rfrcsg|] }tj|qSr')rrG hidden_sizes)rirer'r(r|s)r-r.rer linspacedrop_path_ratesumdepthsrangenum_encoder_blocksappendr? patch_sizesstridesrJrr ModuleListpatch_embeddingsrrV sr_ratios mlp_ratiosblockrH) r/redprrOrblockscurlayersjr0rr(r.OsH      zGLPNEncoder.__init__FTcCs|rdnd}|r dnd}|jd}|}tt|j|j|jD]H\} } | \} } } | |\}}}t| D]\}}|||||}|d}|rJ||df}q2| |}||||ddddd}|rf||f}q|sut dd|||fDSt |||d S) Nr'rrrgrr@css|] }|dur|VqdSr,r')rvr'r'r( sz&GLPNEncoder.forward..last_hidden_stater3 attentions) r enumerateziprrrHrprjrvtupler )r/rNrwoutput_hidden_states return_dictall_hidden_statesall_self_attentionsryr3idxrembedding_layer block_layer norm_layerrQrRrblk layer_outputsr'r'r(r4s2      zGLPNEncoder.forward)FFTrr'r'r0r(rNs  3rc@s$eZdZdZeZdZdZddZdS)GLPNPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. glpnrNcCst|tjtjfr#|jjjd|jjd|j dur!|j j dSdSt|tj rF|jjjd|jjd|j durD|jj|j  dSdSt|tj r[|j j |jjddSdS)zInitialize the weightsr)meanstdNg?)rrr[rEweightdatanormal_reinitializer_rangerzero_ Embedding padding_idxrGfill_)r/moduler'r'r( _init_weightss     z!GLPNPreTrainedModel._init_weightsN) r7r8r9r:r config_classbase_model_prefixmain_input_namerr'r'r'r(rs  raG This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`GLPNConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aG Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`GLPNImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zfThe bare GLPN encoder (Mix-Transformer) outputting raw hidden-states without any specific head on top.cseZdZfddZddZeedee e e de d   dd e jd eed eed eed eee ff ddZZS) GLPNModelcs(t|||_t||_|dSr,)r-r.rerencoder post_initr/rer0r'r(r.s   zGLPNModel.__init__cCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrlayerrr)r/heads_to_prunerrr'r'r( _prune_headsszGLPNModel._prune_headsz(batch_size, sequence_length)vision) checkpoint output_typermodalityexpected_outputNrNrwrrrcCs~|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||d}|d}|s6|f|ddSt||j|jdS)Nrwrrrrr)rerwruse_return_dictrr r3r)r/rNrwrrencoder_outputssequence_outputr'r'r(r4s$zGLPNModel.forward)NNN)r7r8r9r.rrGLPN_INPUTS_DOCSTRINGr5r_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPEr FloatTensorrboolrrr4r>r'r'r0r(rs2    rc*eZdZdZdfdd ZddZZS)GLPNSelectiveFeatureFusionz Selective Feature Fusion module, as explained in the [paper](https://arxiv.org/abs/2201.07436) (section 3.4). This module adaptively selects and integrates local and global features by attaining an attention map for each feature. @c stttjt|d|ddddt|t|_ttj|t|dddddtt|dt|_ tjt|dddddd|_ t |_ dS)Nr@rr) in_channels out_channelsrBrCrD) r-r.r SequentialrErX BatchNorm2dReLUconvolutional_layer1convolutional_layer2convolutional_layer3Sigmoidsigmoid)r/ in_channelr0r'r(r.%s z#GLPNSelectiveFeatureFusion.__init__cCstj||fdd}||}||}||}||}||dddddddfd||dddddddfd}|S)Nrrnr)r catr r r r  unsqueeze)r/local_featuresglobal_featuresfeaturesattnhybrid_featuresr'r'r(r4:s    (z"GLPNSelectiveFeatureFusion.forward)rrSr'r'r0r(rsrcs&eZdZfddZdddZZS)GLPNDecoderStagecsPt||k}|stj||ddnt|_t||_tjdddd|_ dS)Nr)rBr@bilinearF scale_factormode align_corners) r-r.rrEr convolutionrfusionUpsampleupsample)r/rr should_skipr0r'r(r.Ls  zGLPNDecoderStage.__init__NcCs,||}|dur|||}||}|Sr,)rrr)r/ hidden_stateresidualr'r'r(r4Ss   zGLPNDecoderStage.forwardr,rr'r'r0r(rKs rcs:eZdZfddZdeejdeejfddZZS) GLPNDecodercs\t|jddd}|jtfdd|D|_d|jd_tjdddd|_ dS) Nrgcsg|]}t|qSr')r)rrKrr'r(rgsz(GLPNDecoder.__init__..rr@rFr) r-r.rdecoder_hidden_sizerrstagesrrfinal_upsample)r/rereserved_hidden_sizesr0r$r(r.`s  zGLPNDecoder.__init__r3rcCsNg}d}t|ddd|jD]\}}|||}||q|||d<|S)Nrg)rr&rr')r/r3stage_hidden_statesstage_hidden_stater!stager'r'r(r4ns  zGLPNDecoder.forward r7r8r9r.rr r<r4r>r'r'r0r(r#_s &r#cr) SiLogLossz Implements the Scale-invariant log scale loss [Eigen et al., 2014](https://arxiv.org/abs/1406.2283). $$L=\frac{1}{n} \sum_{i} d_{i}^{2}-\frac{1}{2 n^{2}}\left(\sum_{i} d_{i}^{2}\right)$$ where $d_{i}=\log y_{i}-\log y_{i}^{*}$. ?cr+r,)r-r.lambd)r/r/r0r'r(r.r2zSiLogLoss.__init__cCsX|dk}t||t||}tt|d|jt|d}|S)Nrr@)detachr logrspowrr/)r/predtarget valid_maskdiff_loglossr'r'r(r4s ,zSiLogLoss.forward)r.rSr'r'r0r(r-zsr-cs6eZdZfddZdeejdejfddZZS)GLPNDepthEstimationHeadc sRt||_|j}ttj||ddddtjddtj|ddddd|_dS)NrrrAF)inplace) r-r.rer%rrrErhead)r/rechannelsr0r'r(r.s   z GLPNDepthEstimationHead.__init__r3rcCs8||jj}||}t||jj}|jdd}|S)Nrrn)re head_in_indexr:r r  max_depthsqueeze)r/r3predicted_depthr'r'r(r4s   zGLPNDepthEstimationHead.forwardr,r'r'r0r(r8s " r8z]GLPN Model transformer with a lightweight depth estimation head on top e.g. for KITTI, NYUv2.cseZdZfddZeedeee d    dde j de e j de e d e e d e e d eee jeff d d ZZS)GLPNForDepthEstimationcs6t|t||_t||_t||_|dSr,) r-r.rrr#decoderr8r:rrr0r'r(r.s     zGLPNForDepthEstimation.__init__zbatch_size, sequence_length)rrNrNlabelsrwrrrc Cs|dur|n|jj}|dur|n|jj}|j||d|d}|r"|jn|d}||}||} d} |dur>t} | | |} |s`|rL| f|dd} n | f|dd} | dur^| f| S| St| | |rh|jnd|j dS)a labels (`torch.FloatTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth depth estimation maps for computing the loss. Returns: Examples: ```python >>> from transformers import AutoImageProcessor, GLPNForDepthEstimation >>> import torch >>> import numpy as np >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("vinvino02/glpn-kitti") >>> model = GLPNForDepthEstimation.from_pretrained("vinvino02/glpn-kitti") >>> # prepare image for the model >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... predicted_depth = outputs.predicted_depth >>> # interpolate to original size >>> prediction = torch.nn.functional.interpolate( ... predicted_depth.unsqueeze(1), ... size=image.size[::-1], ... mode="bicubic", ... align_corners=False, ... ) >>> # visualize the prediction >>> output = prediction.squeeze().cpu().numpy() >>> formatted = (output * 255 / np.max(output)).astype("uint8") >>> depth = Image.fromarray(formatted) ```NTrrr@)r7r?r3r) rerrrr3rAr:r-r r) r/rNrBrwrrrr3outr?r7loss_fctr&r'r'r(r4s63    zGLPNForDepthEstimation.forward)NNNN)r7r8r9r.rrr5rr rr rrrrrr<r4r>r'r'r0r(r@s*  r@)rF);r:rrtypingrrrrr torch.utils.checkpointr activationsrmodeling_outputsr r modeling_utilsr pytorch_utilsr r utilsrrrrrconfiguration_glpnr get_loggerr7loggerrrr"GLPN_PRETRAINED_MODEL_ARCHIVE_LISTr<r;rr)Moduler*r?rTrrrrrrrGLPN_START_DOCSTRINGrrrrr#r-r8r@r'r'r'r(s\      R '+X <,