o h7@sdZddlmZddlmZddlmZmZddlm Z m Z m Z er-ddl Z ddl mZgdgd gd d d gdgd gd d d d Z    ddefddZddZddZddZddZddZdS)z;AWQ (Activation aware Weight Quantization) integration file)ACT2FN)PreTrainedModel)is_auto_awq_availableis_torch_available)AwqBackendPackingMethod AwqConfigAWQLinearVersionN)q_projk_projv_projo_proj) gate_projup_proj down_proj)input_layernormpost_attention_layernormnormF) attentionmlp layernorm use_alibi)mistralllamareturnc sL|durg}|j}tstd|tjkrddlm}m}n |tjkr)ddl m }|tjkr9|j t j kr6|n|} n|} |D]b\} } durIg| t| tjr| |vrtfdd|Ds| j} | j} | |j|j| | | jdu| jjd|j| <d }|j| d tt| dkrt | |||d \}}!d q?||fS) a Public method that recursively replaces the Linear layers of the given model with AWQ quantized layers. `accelerate` is needed to use this method. Returns the converted model and a boolean that indicates if the conversion has been successfull or not. During the module replacement, we also infer the backend to use through the `quantization_config` object. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. quantization_config (`AwqConfig`): The quantization config object that contains the quantization parameters. modules_to_not_convert (`list`, *optional*): A list of modules to not convert. If a module name is in the list (e.g. `lm_head`), it will not be converted. current_key_name (`list`, *optional*): A list that contains the current key name. This is used for recursion and should not be passed by the user. has_been_replaced (`bool`, *optional*): A boolean that indicates if the conversion has been successful or not. This is used for recursion and should not be passed by the user. NzAWQ (either `autoawq` or `llmawq`) is not available. Please install it with `pip install autoawq` or check out the installation guide in https://github.com/mit-han-lab/llm-awqr  WQLinear_GEMM WQLinear_GEMV)WQLinearc3s|] }|dvVqdS).N)join).0keycurrent_key_nameS/var/www/html/ai/venv/lib/python3.10/site-packages/transformers/integrations/awq.py asz*replace_with_awq_linear..)w_bit group_size in_features out_featuresbiasdevTF)modules_to_not_convertr$quantization_confighas_been_replaced)"backendr ValueErrorrAUTOAWQawq.modules.linearrrLLMAWQawq.quantize.qmodulerversionrGEMMnamed_childrenappend isinstancennLinearanyr*r+bitsr)r,weightdevice_modulesrequires_grad_lenlistchildrenreplace_with_awq_linearpop)modelr.r/r$r0r2rrr target_clsnamemoduler*r+_r%r#r&rH*sT       rHcCst|tstd|jj|jdur|j}|j|d<|S|jjt vrKt |jj}|jj }|jj }t |jd|}||d<||d<||d<|j|d<|Std)af Returns the fusing mapping given the quantization config and the model Args: model (`~PreTrainedModel`): The model to fuse - note this model should have been converted into AWQ format beforehand. quantization_config (`~transformers.quantization_config.AWQConfig`): The quantization configuration to use. z:The model should be an instance of `PreTrainedModel`, got N max_seq_lennum_key_value_heads hidden_sizenum_attention_headsaFusing mapping not found either on the quantization config or the supported `AWQ_FUSED_MAPPINGS`. Please pass a `fused_mapping` argument in the `quantization_config` or raise an issue on transformers https://github.com/huggingface/transformers to add its support.) r<rr3 __class____name__modules_to_fusefuse_max_seq_lenconfig model_typeAWQ_FUSED_MAPPINGSrQrRgetattr)rJr/current_fused_mappingrQrRrPr%r%r&get_modules_to_fuse~s&     r\c Cst|}|j}t||}|tjkr%ddlm}ddlm }ddl m }nt d| D]\}} t|d| |t|||d| |t|| |||q-|S)a8 Optionally fuse some modules in the model to speedup inference. Args: model (`~PreTrainedModel`): The model to fuse - note this model should have been converted into AWQ format beforehand. quantization_config (`dict`): The quantization configuration to use. r )QuantAttentionFused) QuantFusedMLP)FasterTransformerRMSNormz0Fusing is only supported for the AutoAWQ backendrr)r from_dictr2r\rr4awq.modules.fused.attnr]awq.modules.fused.mlpr^awq.modules.fused.normr_r3 named_modules_fuse_awq_layernorm _fuse_awq_mlp_fuse_awq_attention_layers) rJr/ awq_configr2rUr]r^r_rLrMr%r%r&fuse_awq_moduless     ricCsB|D]}t||rt||}||j|j|jj|j|<~qdS)a Fuse the LayerNorm layers into a target class using autoawq Args: fuse_module_names (`List[str]`): The list of module names to fuse module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse target_cls (`~autoawq.FasterTransformerRMSNorm`): The `FasterTransformerRMSNorm` class as it only supports that class for now. N)hasattrrZrAvariance_epsilontorBrC)fuse_module_namesrMrK module_name old_moduler%r%r&res   recCst|dkrdSt||drPt||d}t||d}t||d}|jj}t|jj} ||||| } |dd\} } | | } t | | | |~~~dSdS)a Fuse the MLP layers into a target class using autoawq Args: model (`~PreTrainedModel`): The input pretrained model current_module_name (`str`): The current submodule name fuse_module_names (`List[str]`): The list of module names to fuse. For the MLP layers it has to be an array of length 3 that consists of the 3 MLP layers in the order (gate (dense layer post-attention) / up / down layers) module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse target_cls (`~autoawq.QuantFusedMLP`): The `QuantFusedMLP` class as it only supports that class for now. r Nrr) rErjrZqweightrBrrW hidden_actrsplit get_submodulesetattrrl)rJcurrent_module_namermrMrKrrrprevious_device activation_fn new_module parent_name child_nameparentr%r%r&rfs    rfc Csddlm}m}t|ddkrdSt||ddrt||dd}|jj}t||r2|} d} nt||r<|} d} nt dt||dd} t||dd} t||dd} |j durmt j |j | j | j gdd nd}| |j |j|j|j| j| j|j dutt|j}t j |j| j| jg| d |_t j |j| j| jg| d |_t j |j| j| jg| d |_t||r|j|_||_ ||d |d |d || ||d |dd}d|_|dd\}}||}t||||~~ ~ ~ dSdS)a Fuse the Attention layers into a target class using autoawq Args: model (`~PreTrainedModel`): The input pretrained model module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse modules_to_fuse (`List[str]`): The module fusing mapping. The dictionary has to contain a field `attention` with attention module names in the correct order: q, k, v, o layer current_module_name (`str`): The current submodule name target_cls (`~autoawq.QuantAttentionFused`): The `QuantAttentionFused` class as it only supports that class for now. r rrNrpz'Unsupported q_proj type: {type(q_proj)}r)dimrQrRrPrOr)rTr)r5rrrErjrZrqrBr<r3r,torchcatr(r)r*r+nextiter state_dictvaluesqzerosscales split_k_itersis_hf_transformersrsrtrurl)rJrMrUrvrKrrr rwlinear_target_clscat_dimr r r r, qkv_layerfused_attention_layerrzr{r|r%r%r&rgs^  (     rg)NNNF)__doc__ activationsrmodeling_utilsrutilsrrutils.quantization_configrrrrtorch.nnr=rYboolrHr\rirerfrgr%r%r%r&s>    T&% %