o h@sZdZddlZddlZddlmZmZmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZmZdd lmZmZmZmZmZmZdd lmZmZmZmZdd l m!Z!d d l"m#Z#e!$e%Z&dZ'dZ(ddgZ)Gddde j*Z+d=ddZ,Gddde j*Z-Gddde j*Z.Gddde j*Z/Gddde j*Z0Gddde j*Z1Gdd d e j*Z2Gd!d"d"e j*Z3Gd#d$d$e j*Z4Gd%d&d&e j*Z5Gd'd(d(e j*Z6Gd)d*d*e j*Z7Gd+d,d,e j*Z8Gd-d.d.eZ9d/Z:d0Z;ed1e:Gd2d3d3e9Zed:e:Gd;d<dz PyTorch MarkupLM model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)add_start_docstrings%add_start_docstrings_to_model_forwardreplace_return_docstrings))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModelapply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)logging)MarkupLMConfigzmicrosoft/markuplm-baserzmicrosoft/markuplm-largecs*eZdZdZfddZdddZZS)XPathEmbeddingszConstruct the embeddings from xpath tags and subscripts. We drop tree-id in this version, as its info can be covered by xpath. cstt|j|_tj|jj|_t j |_ t |_ tj|jdj|_tdjj|_tfddt|jD|_tfddt|jD|_dS)Ncg|] }tjjqS)r Embeddingmax_xpath_tag_unit_embeddingsxpath_unit_hidden_size.0_configrd/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py Pz,XPathEmbeddings.__init__..crr)rrmax_xpath_subs_unit_embeddingsr!r"r%rr'r(Wr))superr__init__ max_depthrLinearr! hidden_sizexpath_unitseq2_embeddingsDropouthidden_dropout_probdropoutReLU activationxpath_unitseq2_inner inner2emb ModuleListrangexpath_tag_sub_embeddingsxpath_subs_sub_embeddingsselfr& __class__r%r'r,Cs"    zXPathEmbeddings.__init__Nc Csg}g}t|jD](}||j||dddd|f||j||dddd|fq tj|dd}tj|dd}||}||| | |}|S)Ndim) r9r-appendr:r;torchcatr7r3r5r6)r=xpath_tags_seqxpath_subs_seqxpath_tags_embeddingsxpath_subs_embeddingsixpath_embeddingsrrr'forward]s&(zXPathEmbeddings.forward)NN)__name__ __module__ __qualname____doc__r,rL __classcell__rrr>r'r=s rcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rrA)neintrDcumsumtype_aslong) input_ids padding_idxpast_key_values_lengthmaskincremental_indicesrrr'"create_position_ids_from_input_idsps  r\cs@eZdZdZfddZddZ       d dd ZZS) MarkupLMEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cstt|||_tj|j|j|jd|_ t|j |j|_ |j |_ t ||_t|j|j|_tj|j|jd|_t|j|_|jdt|j ddd|j|_tj|j |j|jd|_ dS)N)rXeps position_ids)rr@F) persistent)r+r]r,r&rr vocab_sizer/ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsr-rrKtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsr1r2r3register_bufferrDarangeexpandrXr<r>rr'r,s    zMarkupLMEmbeddings.__init__cCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr@rdtypedevicer)sizerDrlrXrVrp unsqueezerm)r= inputs_embeds input_shapesequence_lengthr`rrr'&create_position_ids_from_inputs_embedss  z9MarkupLMEmbeddings.create_position_ids_from_inputs_embedsNrcCs<|dur |}n|dd}|dur|jn|j} |dur0|dur+t||j|}n||}|dur=tj|tj| d}|durF||}|dur_|j j tj t t ||jgtj| d}|durx|j jtj t t ||jgtj| d}|} ||} ||} |||} | | | | }||}||}|S)Nr@rn)rqrpr\rXrvrDzerosrVrdr& tag_pad_idonestuplelistr- subs_pad_idrfrhrKrir3)r=rWrFrGtoken_type_idsr`rsrYrtrpwords_embeddingsrfrhrK embeddingsrrr'rLs8          zMarkupLMEmbeddings.forward)NNNNNNr)rMrNrOrPr,rvrLrQrrr>r'r]s r]c8eZdZfddZdejdejdejfddZZS)MarkupLMSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr^) r+r,rr.r/denserirjr1r2r3r<r>rr'r, zMarkupLMSelfOutput.__init__ hidden_states input_tensorreturncC&||}||}|||}|SNrr3rir=rrrrr'rL  zMarkupLMSelfOutput.forwardrMrNrOr,rDTensorrLrQrrr>r'r $rc2eZdZfddZdejdejfddZZS)MarkupLMIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r+r,rr.r/intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr<r>rr'r,s   zMarkupLMIntermediate.__init__rrcC||}||}|Sr)rrr=rrrr'rL  zMarkupLMIntermediate.forwardrrrr>r'rs rcr)MarkupLMOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r+r,rr.rr/rrirjr1r2r3r<r>rr'r,rzMarkupLMOutput.__init__rrrcCrrrrrrr'rL rzMarkupLMOutput.forwardrrrr>r'rrrcr)MarkupLMPoolercs*tt|j|j|_t|_dSr)r+r,rr.r/rTanhr5r<r>rr'r,s zMarkupLMPooler.__init__rrcCs(|dddf}||}||}|S)Nr)rr5)r=rfirst_token_tensor pooled_outputrrr'rLs  zMarkupLMPooler.forwardrrrr>r'rs rcr)MarkupLMPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSr) r+r,rr.r/rrrrr transform_act_fnrirjr<r>rr'r,"s  z(MarkupLMPredictionHeadTransform.__init__rrcCs"||}||}||}|Sr)rrrirrrr'rL+s   z'MarkupLMPredictionHeadTransform.forwardrrrr>r'r!s  rcs$eZdZfddZddZZS)MarkupLMLMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dS)NF)bias) r+r,r transformrr.r/rbdecoder ParameterrDrwrr<r>rr'r,4s  z!MarkupLMLMPredictionHead.__init__cCrr)rrrrrr'rLArz MarkupLMLMPredictionHead.forward)rMrNrOr,rLrQrrr>r'r3s  rcr)MarkupLMOnlyMLMHeadcstt||_dSr)r+r,r predictionsr<r>rr'r,Is zMarkupLMOnlyMLMHead.__init__sequence_outputrcCs||}|Sr)r)r=rprediction_scoresrrr'rLMs zMarkupLMOnlyMLMHead.forwardrrrr>r'rHs rcseZdZdfdd ZdejdejfddZ      dd ejd eejd eejd eejd eejdee e ejdee de ejfddZ Z S)MarkupLMSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|p\t|dd|_|jdksh|jd kry|j|_t d |jd |j|_|j|_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()position_embedding_typeabsolute relative_keyrelative_key_queryr)r+r,r/num_attention_headshasattr ValueErrorrSattention_head_size all_head_sizerr.querykeyvaluer1attention_probs_dropout_probr3getattrrrerdistance_embedding is_decoderr=r&rr>rr'r,Ts*   zMarkupLMSelfAttention.__init__xrcCs6|dd|j|jf}||}|ddddS)Nr@rrrr )rqrrviewpermute)r=r new_x_shaperrr'transpose_for_scoresns z*MarkupLMSelfAttention.transpose_for_scoresFrattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionscCs||}|du} | r|dur|d} |d} |}nP| r/|||} |||} |}n;|durZ|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} |du} |jrz| | f}t| | dd}|j dks|j dkr | j d| j d}}| rtj |dtj |j d dd}ntj|tj |j d dd}tj|tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}||}n|j dkr td | |}td | |}|||}|t|j}|dur||}tjj|dd}||}|dur0||}t|| }|dddd }|dd|jf}||}|rX||fn|f}|jrd||f}|S)NrrrrAr@rrrnrozbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rrrrrDrErmatmul transposershapetensorrVrprrlrretoroeinsummathsqrtrr functionalsoftmaxr3r contiguousrqr)r=rrrrrrrmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputsrrr'rLssn             zMarkupLMSelfAttention.forwardrNNNNNF) rMrNrOr,rDrrr FloatTensorrboolrLrQrrr>r'rSs4 rcseZdZdfdd ZddZ      ddejdeejd eejd eejd eejd ee e ejd ee de ejfddZ Z S)MarkupLMAttentionNcs.tt||d|_t||_t|_dS)Nr)r+r,rr=routputset pruned_headsrr>rr'r,s   zMarkupLMAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrA)lenrr=rrrrrrrrrrunion)r=headsindexrrr' prune_headss zMarkupLMAttention.prune_headsFrrrrrrrrc Cs<||||||||}||d|} | f|dd} | S)Nrr)r=r) r=rrrrrrr self_outputsattention_outputrrrr'rLs  zMarkupLMAttention.forwardrr) rMrNrOr,rrDrrrrrrLrQrrr>r'rs4 rcseZdZfddZ      ddejdeejdeejdeejd eejd eeeejd ee d eejfd dZ ddZ Z S) MarkupLMLayercsrt|j|_d|_t||_|j|_|j|_|jr-|js&t|dt|dd|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedrr)r+r,chunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrcrossattentionr intermediaterrr<r>rr'r, s   zMarkupLMLayer.__init__NFrrrrrrrrc Cs|dur |ddnd}|j|||||d} | d} |jr(| dd} | d} n| dd} d} |jro|durot|dsDtd|d|durN|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) Nr)rrrrr@rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrrrrfeed_forward_chunkrr)r=rrrrrrrself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputrrr'rLsP         zMarkupLMLayer.forwardcCs||}|||}|Sr)rr)r=rintermediate_outputrrrr'r\s  z MarkupLMLayer.feed_forward_chunkr) rMrNrOr,rDrrrrrrLrrQrrr>r'r s4  ArcseZdZfddZ         ddejdeejdeejd eejd eejd eeeejd ee d ee dee dee de eeje ffddZ Z S)MarkupLMEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr)rr"r%rr'r(gsz,MarkupLMEncoder.__init__..F) r+r,r&rr8r9num_hidden_layerslayergradient_checkpointingr<r>r%r'r,ds   zMarkupLMEncoder.__init__NFTrrrrrpast_key_valuesrroutput_hidden_states return_dictrc  Cs^| rdnd} |r dnd} |r|jjrdnd} |jr%|jr%|r%tdd}|r)dnd}t|jD]^\}}| r;| |f} |durC||nd}|durM||nd}|jrc|jrc||j |||||||}n ||||||||}|d}|rz||df7}|r| |df} |jjr| |df} q0| r| |f} | st dd ||| | | fDSt ||| | | d S) NrzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frr@rrcss|] }|dur|VqdSrr)r#vrrr' sz*MarkupLMEncoder.forward..)last_hidden_staterr attentionscross_attentions) r&rr traininglogger warning_once enumerater _gradient_checkpointing_func__call__rzr)r=rrrrrrrrrrall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cacherJ layer_modulelayer_head_maskr layer_outputsrrr'rLjsz         zMarkupLMEncoder.forward) NNNNNNFFT)rMrNrOr,rDrrrrrrrrLrQrrr>r'r csD      r csLeZdZdZeZeZdZddZ e de e e ejfffdd ZZS)MarkupLMPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. markuplmcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsg)meanstdN?)rrr.weightdatanormal_r&initializer_rangerzero_rrXrifill_)r=modulerrr' _init_weightss      z%MarkupLMPreTrainedModel._init_weightspretrained_model_name_or_pathcstt|j|g|Ri|Sr)r+r#from_pretrained)clsr0 model_argskwargsr>rr'r1s z'MarkupLMPreTrainedModel.from_pretrained)rMrNrOrPr config_class&MARKUPLM_PRETRAINED_MODEL_ARCHIVE_LISTpretrained_model_archive_mapbase_model_prefixr/ classmethodrrrosPathLiker1rQrrr>r'r#s*r#aK This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MarkupLMConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) xpath_tags_seq (`torch.LongTensor` of shape `({0}, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `({0}, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: `1` for tokens that are NOT MASKED, `0` for MASKED tokens. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: `0` corresponds to a *sentence A* token, `1` corresponds to a *sentence B* token [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: `1` indicates the head is **not masked**, `0` indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): If set to `True`, the attentions tensors of all attention layers are returned. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): If set to `True`, the hidden states of all layers are returned. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): If set to `True`, the model will return a [`~file_utils.ModelOutput`] instead of a plain tuple. zbThe bare MarkupLM Model transformer outputting raw hidden-states without any specific head on top.cseZdZdfdd ZddZddZdd Zee d e e e d  d d e ejde ejde ejde ejde ejde ejde ejde ejde ede ede edeee ffddZ d!ddZddZZS)" MarkupLMModelTcsDt|||_t||_t||_|rt|nd|_| dSr) r+r,r&r]rr encoderrpooler post_init)r=r&add_pooling_layerr>rr'r,$s    zMarkupLMModel.__init__cCs|jjSrrrd)r=rrr'get_input_embeddings0sz"MarkupLMModel.get_input_embeddingscCs ||j_dSrrA)r=rrrr'set_input_embeddings3s z"MarkupLMModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr=r rr)r=heads_to_pruner rrrr' _prune_heads6szMarkupLMModel._prune_headsbatch_size, sequence_length output_typer5NrWrFrGrr}r`rrsrrrrc Cs| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd|durQ|jn|j} |dur_tj | | d}|durltj | tj | d}| d d}|j |jd }d |d }|dur|dkr| d  d  d d}||jjdddd}n|dkr| d d d}|j t|jd }ndg|jj}|j||||||d }|j|||| | | d}|d }|jdur||nd}| s||f|ddSt|||j|j|jdS)a` Returns: Examples: ```python >>> from transformers import AutoProcessor, MarkupLMModel >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = MarkupLMModel.from_pretrained("microsoft/markuplm-base") >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 768] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer@z5You have to specify either input_ids or inputs_embeds)rprnrrrr'gr)rWrFrGr`r}rs)rrrr)r pooler_outputrrr)r&rruse_return_dictr%warn_if_padding_and_no_attention_maskrqrprDryrwrVrrrrorBrmr next parametersrr=r>rrrr)r=rWrFrGrr}r`rrsrrrrtrpextended_attention_maskembedding_outputencoder_outputsrrrrr'rL>sn$     zMarkupLMModel.forwardc Ksv|j}|dur ||}|dur4|ddjd}|jd|kr#|}n|jdd}|dd|df}||||dS)Nrrr)rWrrr)rnew_ones) r=rWrrr model_kwargsrt past_lengthremove_prefix_lengthrrr'prepare_inputs_for_generations z+MarkupLMModel.prepare_inputs_for_generationcs.d}|D]}|tfdd|Df7}q|S)Nrc3s$|] }|d|jVqdS)rN) index_selectrrp)r# past_statebeam_idxrr'rs"z/MarkupLMModel._reorder_cache..)rz)r=rrZreordered_past layer_pastrrYr'_reorder_caches zMarkupLMModel._reorder_cache)T) NNNNNNNNNNN)NNT)rMrNrOr,rBrCrFr MARKUPLM_INPUTS_DOCSTRINGformatr r_CONFIG_FOR_DOCrrD LongTensorrrrrrLrVr]rQrrr>r'r<s\        i r<z MarkupLM Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). c!seZdZfddZeedeee d             dde e j de e j de e j d e e j d e e j d e e j d e e j d e e j de e j de e j de e de e de e deee j effddZZS)MarkupLMForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSNF)r@) r+r, num_labelsr<r$rr.r/ qa_outputsr?r<r>rr'r,s  z%MarkupLMForQuestionAnswering.__init__rGrHNrWrFrGrr}r`rrsstart_positions end_positionsrrrrc CsL| dur| n|jj} |j||||||||| | | d }|d}||}|jddd\}}|d}|d}d}| dur| durt| dkrQ| d} t| dkr^| d} |d}| d|| d|t |d}||| }||| }||d}| s||f|dd}|dur|f|S|St ||||j |j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: Examples: ```python >>> from transformers import AutoProcessor, MarkupLMForQuestionAnswering >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> model = MarkupLMForQuestionAnswering.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> html_string = " My name is Niels " >>> question = "What's his name?" >>> encoding = processor(html_string, questions=question, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> processor.decode(predict_answer_tokens).strip() 'Niels' ```N rFrGrr}r`rrsrrrrrr@rA) ignore_indexr)loss start_logits end_logitsrr)r&rKr$resplitsqueezerrrqclamp_rrrr)r=rWrFrGrr}r`rrsrfrgrrrrrlogitsrkrl total_loss ignored_indexloss_fct start_lossend_lossrrrr'rLsT6          z$MarkupLMForQuestionAnswering.forward) NNNNNNNNNNNNN)rMrNrOr,r r^r_r rr`rrDrrrrrLrQrrr>r'rbs\      rbz9MarkupLM Model with a `token_classification` head on top.ceZdZfddZeedeee d            dde e j de e j de e j d e e j d e e j d e e j d e e j d e e j de e j de e de e de e deee j effddZZS)MarkupLMForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSrc)r+r,rdr<r$classifier_dropoutr2rr1r3r.r/ classifierr?r=r&rxr>rr'r,Os   z'MarkupLMForTokenClassification.__init__rGrHNrWrFrGrr}r`rrslabelsrrrrc  Cs| dur| n|jj} |j||||||||| | | d } | d}||}d}| dur:t}||d|jj| d}| sP|f| dd}|durN|f|S|St||| j| j dS)a  labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Returns: Examples: ```python >>> from transformers import AutoProcessor, AutoModelForTokenClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> processor.parse_html = False >>> model = AutoModelForTokenClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> nodes = ["hello", "world"] >>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span"] >>> node_labels = [1, 2] >>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```Nrhrr@rrjrprr) r&rKr$ryrrrdrrr)r=rWrFrGrr}r`rrsr{rrrrrrrjrsrrrr'rL]s@, z&MarkupLMForTokenClassification.forward NNNNNNNNNNNN)rMrNrOr,r r^r_r rr`rrDrrrrrLrQrrr>r'rwLsV        rwz MarkupLM Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. crv)!MarkupLMForSequenceClassificationcsdt||j|_||_t||_|jdur|jn|j}t ||_ t |j |j|_ |dSr)r+r,rdr&r<r$rxr2rr1r3r.r/ryr?rzr>rr'r,s    z*MarkupLMForSequenceClassification.__init__rGrHNrWrFrGrr}r`rrsr{rrrrc  Csv| dur| n|jj} |j||||||||| | | d } | d}||}||}d}| dur|jjdurX|jdkr>d|j_n|jdkrT| jtj ksO| jtj krTd|j_nd|j_|jjdkrvt }|jdkrp|| | }n+||| }n%|jjdkrt }||d|j| d}n|jjdkrt}||| }| s|f| dd}|dur|f|S|St||| j| jd S) a& labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoProcessor, AutoModelForSequenceClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```Nrhr regressionsingle_label_classificationmulti_label_classificationr@rr|)r&rKr$r3ry problem_typerdrorDrVrSrrnrrrrrr)r=rWrFrGrr}r`rrsr{rrrrrrprjrsrrrr'rLsZ+     "       z)MarkupLMForSequenceClassification.forwardr})rMrNrOr,r r^r_r rr`rrDrrrrrLrQrrr>r'r~sV       r~)r)@rPrr:typingrrrrDtorch.utils.checkpointrtorch.nnrrr activationsr file_utilsr r r modeling_outputsrrrrrrmodeling_utilsrrrrutilsrconfiguration_markuplmr get_loggerrMr_CHECKPOINT_FOR_DOCr`r6Modulerr\r]rrrrrrrrrrr r#MARKUPLM_START_DOCSTRINGr^r<rbrwr~rrrr'sp       3c 2W]" 1-wc