o hO@s8ddlmZddlmZmZmZmZddlZeGdddZ ddeej deej eefdej d e d ee f d d Z ddeej deej eefdej d e d ee f d dZ ddej dejdee fddZddej dejdee fddZ ddeej eefdejdejd e d ee deej f ddZdS)) dataclass)ListOptionalTupleUnionNc@seZdZUdZeed<eed<d"dedeefddZ d#ded ed ed e j d e e j d fdee j f ddZ d"de j d ed e j d eede j f ddZe  d$de jd e j d e j dedeef ddZed"de j d e j deefddZede j de j de eeffd d!ZdS)%AttentionMaskConvertera9 A utility attention mask class that allows one to: - Create a causal 4d mask - Create a causal 4d mask with slided window - Convert a 2d attention mask (batch_size, query_length) to a 4d attention mask (batch_size, 1, query_length, key_value_length) that can be multiplied with attention scores Examples: ```python >>> import torch >>> from transformers.modeling_attn_mask_utils import AttentionMaskConverter >>> converter = AttentionMaskConverter(True) >>> converter.to_4d(torch.tensor([[0, 0, 0, 1, 1]]), 5, key_value_length=5, dtype=torch.float32) tensor([[[[-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, 0.0000e+00, -3.4028e+38], [-3.4028e+38, -3.4028e+38, -3.4028e+38, 0.0000e+00, 0.0000e+00]]]]) ``` Parameters: is_causal (`bool`): Whether the attention mask should be a uni-directional (causal) or bi-directional mask. sliding_window (`int`, *optional*): Optionally, the sliding window masks can be created if `sliding_window` is defined to a positive integer. is_causalsliding_windowNcCs:||_||_|jdur|jdkrtd|jddSdS)NrzaMake sure that when passing `sliding_window` that its value is a strictly positive integer, not ``)rr ValueError)selfrr r [/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/modeling_attn_mask_utils.py__init__7s zAttentionMaskConverter.__init__cpu batch_size query_lengthkey_value_lengthdtypedevicestrreturnc Cs\|js td|jd||f}||}d}|ddks!|jdur,|j|||||jd}|S)z Creates a causal 4D mask of (bsz, head_dim=1, query_length, key_value_length) shape and adds large negative bias to upper right hand triangular matrix (causal mask). z"Please use `to_causal_4d` only if z has `is_causal` set to True.Nrpast_key_values_lengthr )rr __class__r _make_causal_mask) r rrrrr input_shapercausal_4d_maskr r r to_causal_4d@s z#AttentionMaskConverter.to_causal_4dattention_mask_2dc Cs|jd|f}d}|ddks|jdur0|jr0|durtd||}|j|||j||jd}n |jdur9td|j|||dd|j}|durW| | t |j }|} | S) a Converts 2D attention mask to 4D attention mask by expanding mask to (bsz, head_dim=1, query_length, key_value_length) shape and by adding a large negative bias to not-attended positions. If attention_mask is causal, a causal mask will be added. rNrrzpThis attention mask converter is causal. Make sure to pass `key_value_length` to correctly create a causal mask.rz?Sliding window is currently only implemented for causal masking)tgt_len)shaper rr rrNotImplementedError _expand_maskto masked_fillbooltorchfinfomin) r r!rrrrrrexpanded_attn_maskexpanded_4d_maskr r rto_4das0  zAttentionMaskConverter.to_4drinput_ids_shaperc Cs|\}}tj||ft|j|d}tj|d|d}|||d|ddkd||}|dkrFtj tj ||||d|gdd}|durk||d} dtj tj |tj d| d } || t|j|ddddddf|d|||S) zJ Make causal mask used for bi-directional self-attention. )rrrrrr)dimN)r)diagonal)r)fullr*r+arangesize masked_fill_viewr&catzerostriu ones_likeintr(expand) r/rrrr bszr"mask mask_condr2 context_maskr r rrs "   (z(AttentionMaskConverter._make_causal_maskr?r"cCsj|\}}|dur |n|}|ddddddf|d|||}d|}||tjt|jS)zg Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nrg?)r5r=r&r'r)r(r*r+)r?rr"r>src_len expanded_mask inverted_maskr r rr%s *z#AttentionMaskConverter._expand_maskrCattention_maskunmasked_valuec Cs&t|jddd}tj||ddd}t|dkd}|jddkr(|S||}t|}t|d}|| dd}d|||k<| dkr|jd}|dkrc|dddfd|f} n*|ddddft|ddddf|dddddff} n |dddf|f} ||| <|S)aC Attend to all tokens in masked rows from the expanded attention mask, for example the relevant first rows when using left padding. This is required by F.scaled_dot_product_attention memory-efficient attention path. Details: https://github.com/pytorch/pytorch/issues/110213 `expanded_mask` is [bsz, num_masks, tgt_seq_len, src_seq_len] or [bsz, tgt_seq_len, src_seq_len]. `attention_mask` is [bsz, src_seq_len]. The dimension num_masks of `expanded_mask` is most often 1, but it can also be the number of heads in the case of alibi attention bias. For example, if `attention_mask` is ``` [[0, 0, 1], [1, 1, 1], [0, 1, 1]] ``` and `expanded_mask` is (e.g. here left-padding case) ``` [[[[0, 0, 0], [0, 0, 0], [0, 0, 1]]], [[[1, 0, 0], [1, 1, 0], [1, 1, 1]]], [[[0, 0, 0], [0, 1, 0], [0, 1, 1]]]] ``` then the modified `expanded_mask` will be ``` [[[[1, 1, 1], <-- modified [1, 1, 1], <-- modified [0, 0, 1]]], [[[1, 0, 0], [1, 1, 0], [1, 1, 1]]], [[[1, 1, 1], <-- modified [0, 1, 0], [0, 1, 1]]]] ``` rrrT)keepdimN) r)r4r#argmaxrwheremax unsqueezerepeatr5r1) rCrErFtmpindicesleft_masked_rowsmax_len range_tensor num_masks mask_slicer r r_unmask_unattendeds*2    z)AttentionMaskConverter._unmask_unattendedN)rrN)__name__ __module__ __qualname____doc__r(__annotations__r<rrr)rrrTensorr r. staticmethodSizerr%floatrUr r r rrsn   & . "  rrEr inputs_embedsrr cCs`td|d}|d|}|dur|j||d||jd}|S|j|d|d||j|jd}|S)a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)` Args: attention_mask (`torch.Tensor` or `None`): A 2D attention mask of shape `(batch_size, key_value_length)` input_shape (`tuple(int)` or `list(int)` or `torch.Size`): The input shape should be a tuple that defines `(batch_size, query_length)`. inputs_embeds (`torch.Tensor`): The embedded inputs as a torch Tensor. past_key_values_length (`int`): The length of the key value cache. sliding_window (`int`, *optional*): If the model uses windowed attention, a sliding window should be passed. Trr rN)rrrr0)rr.rr r)rErrarr attn_mask_converterrr r r!_prepare_4d_causal_attention_masks  rdc Cstd|d}|d|}|\}}tj} |dur3t|dkr2| r#n!|dkr*d}n||kr1d}n n|dkr>||kr>d}n| rDtd|durLd} | S|durb|j|d|d||j|jd} | S|j ||d|j|d } |dkrztj | |d d } | S) a Prepares the correct `attn_mask` argument to be used by `torch.nn.functional.scaled_dot_product_attention`. In case no token is masked in the `attention_mask` argument, we simply set it to `None` for the cases `query_length == 1` and `key_value_length == query_length`, and rely instead on SDPA `is_causal` argument to use causal/non-causal masks, allowing to dispatch to the flash attention kernel (that can otherwise not be used if a custom `attn_mask` is passed). TrbrNrzAttention using SDPA can not be traced with torch.jit.trace when no attention_mask is provided. To solve this issue, please either load your model with the argument `attn_implementation="eager"` or pass an attention_mask input when tracing the model.rr0)rrg)rF) rr)jit is_tracingallr r rrr.rU) rErrarr rcrrrrfr-r r r*_prepare_4d_causal_attention_mask_for_sdpa>sN    rhr?rr"cCstj|||dS) Creates a non-causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)` Args: mask (`torch.Tensor` or `None`): A 2D attention mask of shape `(batch_size, key_value_length)` dtype (`torch.dtype`): The torch dtype the created mask shall have. tgt_len (`int`): The target length or query length the created mask shall have. r?rr")rr%rjr r r_prepare_4d_attention_masks rkcCsr|j\}}|dur |n|}tj}t|dkr1|rdS|dkr#dS||kr)dStj|||dStj|||dS)riNrrj)r#r)rerfrgrr%)r?rr"rrrfr r r#_prepare_4d_attention_mask_for_sdpas  rlrrcCs8td|d}||d}|j|d|d|||d}|S)a/ Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` Args: input_shape (`tuple(int)` or `list(int)` or `torch.Size`): The input shape should be a tuple that defines `(batch_size, query_length)`. dtype (`torch.dtype`): The torch dtype the created mask shall have. device (`int`): The torch device the created mask shall have. sliding_window (`int`, *optional*): If the model uses windowed attention, a sliding window should be passed. Trbrrr0)rr )rrrrr rcrrEr r r _create_4d_causal_attention_masks  rmrVrW) dataclassesrtypingrrrrr)rr]r_r<rdrhrrkrlrrmr r r rs`  . G *