o
h@t
@ sP d Z ddlZddlZddlZddlmZmZmZmZ ddl m
Z
ddl m Z m
Z
ddlmZ ddlmZ dd lmZmZmZmZ dd
lmZ eeeeef eeeeeef eeeeef eeeeeef f ZG d d d eZ d
eeeeef dedeeef fddZ dededefddZ dd Z dd Z dd Z!d d d Z"dS ) z Processor class for KOSMOS-2. N)ListOptionalTupleUnion ) BatchFeature)
ImageInput
is_batched)ProcessorMixin)
AddedToken)
BatchEncodingPaddingStrategy TextInputTruncationStrategy)
TensorTypec " s e Zd ZdZddgZdZdZd3 fdd Z
d4d
ede e
e e
f de de
e de
e dedede eeef de eeef de
e de
e de
e dedede
e eef d ef d d Zd d Zd!d" Z
d5d#e e
e e
f d
ede de
e d e ee e f f
d$d%Zd&d' Zd(d) Zd6d*d+Ze d,d- Z dede e e e e e e f d efd.d/Z d0e e eef e e e e e f f d e eef fd1d2Z! Z"S )7Kosmos2Processora,
Constructs an KOSMOS-2 processor which wraps a KOSMOS-2 image processor and a KOSMOS-2 tokenizer into a single
processor.
[`Kosmos2Processor`] offers all the functionalities of [`CLIPImageProcessor`] and some functionalities of
[`XLMRobertaTokenizerFast`]. See the docstring of [`~Kosmos2Processor.__call__`] and [`~Kosmos2Processor.decode`]
for more information.
Args:
image_processor (`CLIPImageProcessor`):
An instance of [`CLIPImageProcessor`]. The image processor is a required input.
tokenizer (`XLMRobertaTokenizerFast`):
An instance of ['XLMRobertaTokenizerFast`]. The tokenizer is a required input.
num_patch_index_tokens (`int`, *optional*, defaults to 1024):
The number of tokens that represent patch indices.
image_processor tokenizerCLIPImageProcessor)XLMRobertaTokenizerXLMRobertaTokenizerFast c s d|_ d| _d| _d| _d| _d| _d| _d| _d | _d
| _ d | _
d | _ | j| j| j| j| j| j| j| j| j | j
| j g | _ || _
d
d t| j
D }g }| j | D ]
}|t|dddd qQ|| t || d S )NFzzzzzzz z z c S s" g | ]
}d t |d dqS )
)strzfill.0x r! d/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/models/kosmos2/processing_kosmos2.py
` s " z-Kosmos2Processor.__init__..T)lstriprstrip
normalized)return_token_type_ids eod_token boi_token eoi_token eoc_token eol_token bop_token eop_token boo_token eoo_token dom_token grd_token
tag_tokensnum_patch_index_tokensrangeappendr
add_tokenssuper__init__)selfr r r4 patch_index_tokens
tokens_to_addtoken __class__r! r" r9 < s>
zKosmos2Processor.__init__N@ TFimagestextbboxesnum_image_tokensfirst_image_token_idadd_special_tokens
add_eos_tokenpadding
truncation
max_lengthpad_to_multiple_ofreturn_attention_mask
return_lengthverbosereturn_tensorsreturnc ! sh |du r |du r t dt }|dur j||d}|| |duruj||||d}|rL|sLt|tr>jj | }nt|t rLfdd|D }jd ||oS||oX|du | |
|du r`| n| | ||du ri|ndd |}|| |dur|dur|du rjj
d }|}t |d }t t ||| }d gdg| d g }g }g }|d
}t|tr|g}|d g|d < |D ]5}|d| | ||| d }|
| t|} |rd g| } | d gt|t| 7 } |
| qt|t rtd d t|jD d
d d} | d \} } | d \} } jd || g|o!||| |
| |dd|}t|jd | krjjdkra fdd|D } fdd|D } fdd|d D |d < n'jjdkr fdd|D } fdd|D } fdd|d D |d < t|tr|du r|d }|d d |d < |d }|t||d |d|d |S ) a
This method uses [`CLIPImageProcessor.__call__`] method to prepare image(s) for the model, and
[`XLMRobertaTokenizerFast.__call__`] to prepare text for the model.
Please refer to the docstring of the above two methods for more information.
The rest of this documentation shows the arguments specific to `Kosmos2Processor`.
Args:
bboxes (`Union[List[Tuple[int]], List[Tuple[float]], List[List[Tuple[int]]], List[List[Tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, defaults to 64):
The number of (consecutive) places that are used to mark the placeholders to store image information.
This should be the same as `latent_query_num` in the instance of `Kosmos2Config` you are using.
first_image_token_id (`int`, *optional*):
The token id that will be used for the first place of the subsequence that is reserved to store image
information. If unset, will default to `self.tokenizer.unk_token_id + 1`.
add_eos_token (`bool`, defaults to `False`):
Whether or not to include `EOS` token id in the encoding when `add_special_tokens=True`.
Nz*You have to specify either images or text.)rO )rD c s g | ]
} j j | qS r! )r bos_token)r s)r: r! r" r# z-Kosmos2Processor.__call__..) rB rF rH rI rJ rK rL rN rO r input_idsattention_maskc S s g | ]
\}}|t |fqS r! len)r idxr r! r! r" r# rS c S s | d S Nr! )r r! r! r" s z+Kosmos2Processor.__call__..)keyr[ )rB rF rH rI rJ rK rN rO rightc s& g | ]}|j jg t| qS r! r pad_token_idrX r max_len_paddedr: r! r" r# & c " g | ]
}|d g t | qS r rW r rb r! r" r# c rd re rW r rf r! r" r# rg leftc s& g | ]}j jg t| | qS r! r_ r ra r! r" r# rc c " g | ]
}d g t | | qS re rW r rf r! r" r# rg c ri re rW r rf r! r" r# rg )rU rV image_embeds_position_mask)data tensor_typer! )
ValueErrorr r updatepreprocess_examples
isinstancer r rQ list unk_token_idintr5 r6 copyrX sorted enumeraterU padding_sider )!r: rA rB rC rD rE rF rG rH rI rJ rK rL rM rN rO kwargsencodingimage_encoding
text_encodingwith_bos start_indeximage_token_ids base_image_embeds_position_maskrU rj
all_input_idstext_idsmask
sorted_length_min_len_not_paddedrY r! ra r" __call__i s '
zKosmos2Processor.__call__c C s |du rdS t |tstd|D ];}|du rqt |ts |g}|D ])}t |trGt|dkr8tdd |D sKt|dkrGtdd |D sKtdq"qdS ) a
Check `bboxes` for a single text example. It could be
- `None`: no bounding box associated to a text.
- A list with each element being the bounding boxes associated to one ` ... ` pair found
in a text. This could be:
- `None`: no bounding box associated to a ` ... ` pair.
- A tuple of 2 integers: A single bounding box specified by patch indices.
- A tuple of 4 float point number: A single bounding box specified by (normalized) coordinates.
- A list containing the above 2 tuple types: Multiple bounding boxes for a
` ... ` pair.
Nz@`bboxes` (for a single text example) should be `None` or a list. c s | ]}t |tV qd S N)rp rs r r! r! r" ) zAKosmos2Processor._check_bboxes_for_single_text..r c s r r )rp floatr r! r! r" r * r a' Each element in `bboxes` (for a single text example) should be either `None`, a tuple containing 2 integers or 4 float point numbers, or a list containing such tuples. Also make sure the arguments `texts` and `bboxes` passed to `preprocess_text` are both in batches or both for a single example.)rp rq rm tuplerX all)r: rC bboxelementr! r! r" _check_bboxes_for_single_text s&
z.Kosmos2Processor._check_bboxes_for_single_textc C s. | }|d ur| d| }| ||}|S )N )strip_insert_patch_index_tokens)r: rB imagerC img_info_tokensr! r! r" _preprocess_single_example3 s
z+Kosmos2Processor._preprocess_single_exampletextsc s@ j g| }dj g| jg d}t|tr d}|g}|du r+dgt| }nt|s2|g}t|t|krItdt| dt| d|sT| |g}n |durlt|t satd|D ]}| qcndgt| }t|t|krtd t| dt| d fd
d t
|||D }|s|d }|S )
a- Add image and bounding box information to `texts` as image and patch index tokens.
Args:
texts (`Union[TextInput, List[TextInput]]`): The texts to be processed.
images (`ImageInput`, *optional*): The images associated to `texts`.
bboxes (`Union[List[Tuple[int]], List[Tuple[float]], List[List[Tuple[int]]], List[List[Tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional*, defaults to 64):
The number of image tokens (used as latent queries). This should corresponds to the `latent_query_num`
attribute in `Kosmos2Config`.
Returns:
`Union[TextInput, List[TextInput]]`: The processed texts with image and patch index tokens.
r TFNzGThe number of examples in `texts` and `images` should be the same. Got v.s. instead.zS`bboxes` should be `None` or a list (as a batch) when `texts` is passed as a batch.zGThe number of examples in `texts` and `bboxes` should be the same. Got c s" g | ]
\}}} ||| qS r! )r )r rB r r r r: r! r" r# u s z8Kosmos2Processor.preprocess_examples..r ) r) joinr* rp r rX r rm r rq zip) r: r rA rC rD
img_tokensbatchedr resultr! r r" ro = sD
z$Kosmos2Processor.preprocess_examplesc O | j j|i |S )z
This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please
refer to the docstring of this method for more information.
)r batch_decoder: argsrx r! r! r" r z Kosmos2Processor.batch_decodec O r )z
This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to
the docstring of this method for more information.
)r decoder r! r! r" r r zKosmos2Processor.decodec C s | | jd }|rt|S |S rZ )splitr* +clean_text_and_extract_entities_with_bboxes)r: rB cleanup_and_extractcaptionr! r! r" post_process_generation s z(Kosmos2Processor.post_process_generationc C s" | j j}| jj}tt|| S r )r model_input_namesr rq dictfromkeys)r: tokenizer_input_namesimage_processor_input_namesr! r! r" r s z"Kosmos2Processor.model_input_namesc C sP |d u s
t |dkr |S ttjd|d}t |t |kr,tdt | dt | dd}g }t||D ]\\}}| \}} |||| | }|d u rOq5t|t rW|g}g }
t
dd |D sftd |D ]} | | \} }
|
| d
|
qht |
dkrq5d |
}|d | d
q5|t |k r|||d d |}|S )Nr z.+?)stringzuThe number of elements in `bboxes` should be the same as the number of ` ... ` pairs in `text`. Got r r c s s | ]}|d uV qd S r r! )r boxr! r! r" r s z>Kosmos2Processor._insert_patch_index_tokens..zTThe multiple bounding boxes for a single phrase should not contain any `None` value.r z z )
rX rq refinditerrm r spanr6 rp r r #_convert_bbox_to_patch_index_tokensr )r: rB rC matched_phrasescurr_posbuffermatchedr r endpatch_index_stringsr
patch_index_1
patch_index_2 position_strr! r! r" r sB
z+Kosmos2Processor._insert_patch_index_tokensr c C sh t |dkr |\}}ntt| j}t||\}}dt|d d}dt|d d}||fS )Nr r r r )rX rs mathsqrtr4 coordinate_to_patch_indexr r )r: r idx_1idx_2num_patches_per_sidetoken_1token_2r! r! r" r s
z4Kosmos2Processor._convert_bbox_to_patch_index_tokens)r )NNNr@ NTFFNNNNFTN)NNr@ )T)#__name__
__module__ __qualname____doc__
attributesimage_processor_classtokenizer_classr9 r r r r BboxInputr rs boolr r
r r r r r r ro r r r propertyr r r r r
__classcell__r! r! r> r" r & s /
(#
C
.-
r r r rP c C s | \}}}}||kr||kst dt|| }t|| }t|| d }t|| d } || | }
| | | } |
| fS )a Convert a bounding box to a pair of patch indices.
Args:
bbox (`Tuple[float, float, float, float]`):
The 4 coordinates of the bounding box, with the format being (x1, y1, x2, y2) specifying the upper-left and
lower-right corners of the box. It should have x2 > x1 and y2 > y1.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`Tuple[int, int]`: A pair of patch indices representing the upper-left patch and lower-right patch.
zTThe coordinates in `bbox` should be `(x1, y1, x2, y2)` with `x2 > x1` and `y2 > y1`.rT )rm r floorceil) r r x1y1x2y2ul_xul_ylr_xlr_yul_idxlr_idxr! r! r" r s r r r c C s d| }| | }| | }|| }|| }| |kr-|| }|| } || | }
|| | } n=||ks5||krJ|| }|| } || | }
|| | } n || |d }|| |d } || |d }
|| |d } || |
| fS )a
Given a grid of length `num_patches_per_side` and the indices of the upper-left and lower-right corners of a
bounding box, returns the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
Args:
ul_idx (`int`): the index of the grid cell that corresponds to the upper-left corner of the bounding box.
lr_idx (`int`): the index of the grid cell that corresponds to the lower-right corner of the bounding box.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`Tuple[float]`: the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
g ?r r! ) r r r cell_sizer r r r r r r r r! r! r" patch_index_to_coordinate s( r c C s4 d}t || }g }|D ]}|d}| \}}}|s,d}|dd |dd f}|d} g }
| D ];} t d| } t d| dd }
| rp|
rp|r_|
t| dt|
df q5|
t| dt|
df q5|r|||||
f q |
D ]}d|d d |d d
}||||gf q~q |S ) a Extract entities contained in `text`. The bounding bboxes is given in the form of patch indices.
This functioin is only intended to be used within `clean_text_and_extract_entities_with_bboxes` where further
processing happens, including converting to normalized coordinates and whitespace character cleaning up.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> entities = extract_entities_with_patch_indices(text)
>>> entities
[(' a snowman', (31, 41), [(44, 863)]), (' a fire', (130, 137), [(5, 911)])]
```z(?:(([^<]+)))?r Nr r zrT r z>r N)rX r sub)r rB entity_namestartr adjusted_start adjusted_endadjusted_entityr! r! r" adjust_entity_positionsa s
r c C s | }t| t| }g }|D ]5\}\}}}t|t| } t|t| }
|| | }|| |
}| }||||f|f q||fS )z9Remove the spaces around the text and the entities in it.)r rX r$ r% r6 ) rB entitiesnew_textleading_spaces new_entitiesr r r rC entity_name_leading_spacesentity_name_trailing_spacesr! r! r" _cleanup_spacesk s r c
sp t dd| }t| }g }|D ]#}|dd |d }}t|| } fdd|D } ||| f qt||S )a Remove the tag tokens from `text`, extract entities in it with some cleaning up of white characters.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> clean_text, entities = clean_text_and_extract_entities_with_bboxes(text)
>>> clean_text
'An image of a snowman warming himself by a fire.'
>>> entities
[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]
```r r r r c s g | ] }t |d |d qS )r rT )r )r r r r! r" r# s z?clean_text_and_extract_entities_with_bboxes..)r r r r r6 r )
rB r processed_textr r itemr rC r bboxes_in_coordsr! r r" r s
r )r )#r rt r r typingr r r r image_processing_utilsr image_utilsr r processing_utilsr
tokenization_utilsr tokenization_utils_baser r
r r utilsr rs r r r r r r r r r r! r! r! r" s8 *8 -: