o h:@sVddlmZmZmZmZmZddlZGdddZGdddeZGdddeZ dS) )AnyDictListOptionalTupleNc @seZdZdZ ddejdejdedeee e fde ejejff dd Z ddeedefd d Z deefd dZddedeedefddZdS)Cachezf Base, abstract class for all caches. The actual data structure is specific to each subclass. N key_states value_states layer_idx cache_kwargsreturncCtd)a Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`Dict[str, Any]`, `optional`): Additional arguments for the cache subclass. These are specific to each subclass and allow new types of cache to be created. Return: A tuple containing the updated key and value states. z.Make sure to implement `update` in a subclass.NotImplementedErrorselfrr r r rN/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/cache_utils.pyupdate sz Cache.updatercCr )YReturns the sequence length of the cached states. A layer index can be optionally passed.z6Make sure to implement `get_seq_length` in a subclass.rrr rrrget_seq_length%zCache.get_seq_lengthcCr )zJReturns the maximum sequence length of the cached states, if there is any.z6Make sure to implement `get_max_length` in a subclass.rrrrrget_max_length)rzCache.get_max_lengthnew_seq_lengthcCs2|}||}|dur|||kr||S|S)zTGiven the sequence length of the new inputs, returns the usable length of the cache.N)rr)rrr max_lengthprevious_seq_lengthrrrget_usable_length-s  zCache.get_usable_lengthNr)__name__ __module__ __qualname____doc__torchTensorintrrstrrrrrrrrrrrrs"   rc @seZdZdZdddZdedeeej fddZ d d Z d d Z d d ej dej dede eeefdeej ej ff ddZd!de edefddZde efddZdejfddZdeeej eej ffddZed de eeejddfddZdS)" DynamicCachea A cache that grows dynamically as more tokens are generated. This is the default for generative models. It stores the Key and Value states as a list of tensors, one for each layer. The expected shape for each tensor is `[batch_size, num_heads, seq_len, head_dim]`. r NcCsg|_g|_d|_dSNr) key_cache value_cache seen_tokensrrrr__init__As zDynamicCache.__init__r cCs8|t|kr|j||j|fStdt|d|)z Support for backwards-compatible `past_key_value` indexing, e.g. `past_key_value[0][0].shape[2]` to get the sequence length. zCache only has z. layers, attempted to access layer with index )lenr+r,KeyErrorrrrr __getitem__Fs zDynamicCache.__getitem__ccs.tt|D] }|j||j|fVqdS)z Support for backwards-compatible `past_key_value` iteration, e.g. `for x in past_key_value:` to iterate over keys and values Nranger/r+r,rrrr__iter__PszDynamicCache.__iter__cCs t|jS)z Support for backwards-compatible `past_key_value` length, e.g. `len(past_key_value)`. This value corresponds to the number of layers in the model. )r/r+rrrr__len__Xs zDynamicCache.__len__rr r cCs|dkr|j|jd7_t|j|kr"|j||j|ntj|j||gdd|j|<tj|j||gdd|j|<|j||j|fS)a Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`Dict[str, Any]`, `optional`): Additional arguments for the cache subclass. No additional arguments are used in `DynamicCache`. Return: A tuple containing the updated key and value states. rdim)r-shaper/r+appendr,r%catrrrrr_s zDynamicCache.updatercC"t|j|kr dS|j|jdSrrr6r/r+r9rrrrrszDynamicCache.get_seq_lengthcCsdS)zfReturns the maximum sequence length of the cached states. DynamicCache does not have a maximum length.NrrrrrrszDynamicCache.get_max_lengthbeam_idxcCltt|jD],}|j|j}|j|d|||j|<|j|j}|j|d|||j|<qdSzDReorders the cache for beam search, given the selected beam indices.rNr3r/r+device index_selecttor,rr?r rCrrr reorder_cache    zDynamicCache.reorder_cachecCs4d}tt|D]}||j||j|ff7}q|S)zXConverts the `DynamicCache` instance into the its equivalent in the legacy cache format.rr2)r legacy_cacher rrrto_legacy_cacheszDynamicCache.to_legacy_cachepast_key_valuescCs>|}|durtt|D]}||\}}||||q |S)zNConverts a cache in the legacy cache format into an equivalent `DynamicCache`.N)r3r/r)clsrKcacher rr rrrfrom_legacy_caches  zDynamicCache.from_legacy_cache)r Nrr )r!r"r#r$r.r'rrr%r&r1r4r5rrr(rrrr LongTensorrGrJ classmethod FloatTensorrNrrrrr)9s0    %"(r)c @seZdZdZdededdfddZedd Zd ej d ej d ej dej fd dZ d ej d ej d ej de ej ej ffddZ dde edefddZde efddZ dd ej dej dede eeefde ej ej ff ddZdejfddZdS) SinkCachea A cache that as described in the [Attention Sinks paper](https://arxiv.org/abs/2309.17453). It allows the model to generate beyond the length of its context window, without losing fluency in the conversation. As it discards past tokens, the model will lose the ability to generate tokens that depend on the context that was discarded. It stores the Key and Value states as a list of tensors, one for each layer. The expected shape for each tensor is `[batch_size, num_heads, seq_len, head_dim]`. Parameters: window_length (`int`): The length of the context window. num_sink_tokens (`int`): The number of sink tokens. See the original paper for more information. window_lengthnum_sink_tokensr NcCs(g|_g|_||_||_i|_d|_dSr*)r+r,rSrT cos_sin_cacher-)rrSrTrrrr.s  zSinkCache.__init__cCsH|dd|jddf}|d|jdddf}tj| |fddS)N.r7)r9r%r;)xx1x2rrr _rotate_halfszSinkCache._rotate_halfrcossincCs|||||}|Sr)r[)rrr\r]rotated_key_statesrrr_apply_key_rotary_pos_embsz#SinkCache._apply_key_rotary_pos_embc Cs|jd|jvrk|tj}|tj}||j|jdd}||j|jd }||j|jdd}||j|jd }||||}| |||} ||jd| |jdf|j|jd<|j|jdS)Nr6r)r9rUrEr%float32rTdtype unsqueeze) rrr\r] original_cos shifted_cos original_sin shifted_sinrerotation_cosrerotation_sinrrr_get_rerotation_cos_sins  z!SinkCache._get_rerotation_cos_sinrr cCr<r=r>rrrrrszSinkCache.get_seq_lengthcCs|jS)z9Returns the maximum sequence length of the cached states.)rSrrrrrszSinkCache.get_max_lengthr r cCs:|d}|d}|d}|duo|du}|dkr%|j|jd7_t|j|kr9|j||j|n|jd|||jkret j |j||gdd|j|<t j |j||gdd|j|<n|j|dddd|j |j |jddf} |r| ||d|j|d|j\} } |dur| dd|f| d|df} } | | | | } |durt j | | fd d} |j|ddddd|j f} t j | | |gdd|j|<|j|ddddd|j f}|j|dddd|j |j |jddf}t j |||gdd|j|<|j||j|fS) a; Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`Dict[str, Any]`, `optional`): Additional arguments for the cache subclass. The following arguments can be used in `SinkCache`: `sin`, `cos` and `partial_rotation_size`. These arguments are used with models using RoPE, to recompute the rotation as the tokens are shifted. Return: A tuple containing the updated key and value states. r]r\partial_rotation_sizeNrr6r7.rV)getr-r9r/r+r:r,rrSr%r;rTrir_)rrr r r r]r\rj using_rope keys_to_keeprgrh keys_pass sink_keys sink_valuesvalues_to_keeprrrrsD     (""(zSinkCache.updater?cCr@rArBrFrrrrG<rHzSinkCache.reorder_cacher r)r!r"r#r$r'r. staticmethodr[r%r&r_rrirrrrr(rrrOrGrrrrrRsL     OrR) typingrrrrrr%rr)rRrrrrs 3o