o h$@shdZddlZddlmZmZmZddlmZddlm Z m Z m Z m Z m Z ddlmZGdd d eZdS) z Processor class for LayoutXLM. N)ListOptionalUnion)ProcessorMixin) BatchEncodingPaddingStrategyPreTokenizedInput TextInputTruncationStrategy) TensorTypec(sreZdZdZddgZdZdZd-fdd Z          d.d ee e e e e e fd e ee e e fdee e e e e e e fde ee e e e e fdedeeeefdeeeefde e de de e de ede edededededede eeefdef&dd Zd!d"Zd#d$Zd%d&Zed'd(Zed)d*Zed+d,ZZS)/LayoutXLMProcessoran Constructs a LayoutXLM processor which combines a LayoutXLM image processor and a LayoutXLM tokenizer into a single processor. [`LayoutXLMProcessor`] offers all the functionalities you need to prepare data for the model. It first uses [`LayoutLMv2ImageProcessor`] to resize document images to a fixed size, and optionally applies OCR to get words and normalized bounding boxes. These are then provided to [`LayoutXLMTokenizer`] or [`LayoutXLMTokenizerFast`], which turns the words and bounding boxes into token-level `input_ids`, `attention_mask`, `token_type_ids`, `bbox`. Optionally, one can provide integer `word_labels`, which are turned into token-level `labels` for token classification tasks (such as FUNSD, CORD). Args: image_processor (`LayoutLMv2ImageProcessor`, *optional*): An instance of [`LayoutLMv2ImageProcessor`]. The image processor is a required input. tokenizer (`LayoutXLMTokenizer` or `LayoutXLMTokenizerFast`, *optional*): An instance of [`LayoutXLMTokenizer`] or [`LayoutXLMTokenizerFast`]. The tokenizer is a required input. image_processor tokenizerLayoutLMv2ImageProcessor)LayoutXLMTokenizerLayoutXLMTokenizerFastNc s`d|vrtdt|d}|dur|n|}|durtd|dur'tdt||dS)Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.)warningswarn FutureWarningpop ValueErrorsuper__init__)selfrrkwargsr __class__h/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/layoutxlm/processing_layoutxlm.pyr2s zLayoutXLMProcessor.__init__TFrtext text_pairboxes word_labelsadd_special_tokenspadding truncation max_lengthstridepad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosereturn_tensorsreturncKs\|jjr |dur td|jjr|durtd|dur$|dur$td|j||d}|durC|jjrC|durCt|tr?|g}|d}|jdid |durN|n|dd |durY|ndd |durb|n|d d |d |d|d|d| d| d| d| d| d|d|d|d|d|d||}|d}|dur|||d}||d<|S)a This method first forwards the `images` argument to [`~LayoutLMv2ImagePrpcessor.__call__`]. In case [`LayoutLMv2ImagePrpcessor`] was initialized with `apply_ocr` set to `True`, it passes the obtained words and bounding boxes along with the additional arguments to [`~LayoutXLMTokenizer.__call__`] and returns the output, together with resized `images`. In case [`LayoutLMv2ImagePrpcessor`] was initialized with `apply_ocr` set to `False`, it passes the words (`text`/``text_pair`) and `boxes` specified by the user along with the additional arguments to [`~LayoutXLMTokenizer.__call__`] and returns the output, together with resized `images``. Please refer to the docstring of the above two methods for more information. NzdYou cannot provide bounding boxes if you initialized the image processor with apply_ocr set to True.zaYou cannot provide word labels if you initialized the image processor with apply_ocr set to True.TFzKYou cannot return overflowing tokens without returning the offsets mapping.)imagesr2wordsr!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2 pixel_valuesoverflow_to_sample_mappingimager)r apply_ocrr isinstancestrrrget_overflowing_images)rr4r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2rfeaturesencoded_inputsrrr __call__Csz"        zLayoutXLMProcessor.__call__cCsLg}|D] }|||qt|t|kr$tdt|dt||S)Nz`Expected length of images to be the same as the length of `overflow_to_sample_mapping`, but got z and )appendlenr)rr4r7images_with_overflow sample_idxrrr r<sz)LayoutXLMProcessor.get_overflowing_imagescO|jj|i|S)z This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoderargsrrrr rEzLayoutXLMProcessor.batch_decodecOrD)z This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoderFrrr rIrHzLayoutXLMProcessor.decodecCsgdS)N) input_idsbboxattention_maskr8rrrrr model_input_namessz$LayoutXLMProcessor.model_input_namescCtdt|jS)Nzg`feature_extractor_class` is deprecated and will be removed in v5. Use `image_processor_class` instead.)rrrimage_processor_classrMrrr feature_extractor_class z*LayoutXLMProcessor.feature_extractor_classcCrO)Nz[`feature_extractor` is deprecated and will be removed in v5. Use `image_processor` instead.)rrrrrMrrr rrRz$LayoutXLMProcessor.feature_extractor)NN)NNNNTFNNrNNNFFFFTN)__name__ __module__ __qualname____doc__ attributesrPtokenizer_classrrr r rrintboolr;rr r rr?r<rErIpropertyrNrQr __classcell__rrrr r s       W  r )rVrtypingrrrprocessing_utilsrtokenization_utils_baserrr r r utilsr r rrrr s