o h@s@dZddlZddlZddlmZddlmZGdddeZdS)z Processor class for Donut. N)contextmanager)ProcessorMixincsxeZdZdZddgZdZdZdfdd Zd d Zd d Z d dZ e ddZ dddZ eddZeddZZS)DonutProcessora Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single processor. [`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and [`~DonutProcessor.decode`] for more information. Args: image_processor ([`DonutImageProcessor`], *optional*): An instance of [`DonutImageProcessor`]. The image processor is a required input. tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`], *optional*): An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input. image_processor tokenizerAutoImageProcessor AutoTokenizerNc srd}d|vrtdt|d}|dur|n|}|dur!td|dur)tdt|||j|_d|_ dS)Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.F) warningswarn FutureWarningpop ValueErrorsuper__init__rcurrent_processor_in_target_context_manager)selfrrkwargsr  __class__`/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/donut/processing_donut.pyr-s  zDonutProcessor.__init__cOs|jr |j|i|S|dd}|dd}t|dkr'|d}|dd}|dur3|dur3td|durC|j|g|Ri|}|durP|j|fi|}|durV|S|dur\|S|d|d<|S) a When used in normal mode, this method forwards all its arguments to AutoImageProcessor's [`~AutoImageProcessor.__call__`] and returns its output. If used in the context [`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's [`~DonutTokenizer.__call__`]. Please refer to the doctsring of the above two methods for more information. imagesNtextrzBYou need to specify either an `images` or `text` input to process. input_idslabels)rrrlenrrr)rargsrrrinputs encodingsrrr__call__As&     zDonutProcessor.__call__cO|jj|i|S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoderr rrrrr%bzDonutProcessor.batch_decodecOr$)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoder&rrrr(ir'zDonutProcessor.decodeccs2tdd|_|j|_dV|j|_d|_dS)z Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR. z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r r rrrrrrrras_target_processorps z"DonutProcessor.as_target_processorFcCs|dur |j}i}|rtd|tj}|durn|d}t|}td|d|tj}|}|dur?||d}n|}t|} t|} t| d| |tj} | dur| d} d| vrd| vr|j | d |d } | rt | dkr| d } | ||<n?g||<| d D]%} | } | |vr| d d kr| dddkr| dd} || | qt ||dkr||d ||<|| |t |d}|ddd kr|g|j |ddd |d S|s t |r|r|gS|S|rgSd|iS)zS Convert a (generated) token sequence into an ordered JSON format. Nz rzz(.*?)z text_sequence)rget_added_vocabresearch IGNORECASEgroupescapereplacestrip token2jsonrsplitappendfind)rtokensr-r.output start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafrrrr;sR      $ &zDonutProcessor.token2jsoncCtdt|jS)Nzg`feature_extractor_class` is deprecated and will be removed in v5. Use `image_processor_class` instead.)r r r image_processor_classr)rrrfeature_extractor_class z&DonutProcessor.feature_extractor_classcCrJ)Nz[`feature_extractor` is deprecated and will be removed in v5. Use `image_processor` instead.)r r r rr)rrrr rMz DonutProcessor.feature_extractor)NN)FN)__name__ __module__ __qualname____doc__ attributesrKtokenizer_classrr#r%r(rr*r;propertyrLr __classcell__rrrrrs !  4 r)rQr4r contextlibrprocessing_utilsrrrrrrs