o hR@sjddlmZmZmZmZmZmZddlZddlZddl m m Z ddlm Z mZmZm Z ddlZddlmZddlmZmZddlmZddlmZmZedd d ZGd d d ej jZGd ddej jZGdddej jZ Gddde j!Z"Gddde"Z#Gddde"Z$Gdddej jZ%ddZ&Gddde j!Z'Gddde j!Z(Gdd d e j!Z)dS)!)AnyDictOptionalTypeVarUnionoverloadN)Tensordevicedtypenn) QuantState) undo_layout get_tile_inds)GlobalOptimManager) OutlierTracerfind_outlier_dimsTztorch.nn.Module)boundcseZdZ        ddededeedeeded ed ed eed dffd d ZdddZ dddZ ded efddZ Z S)StableEmbeddingN@Fnum_embeddings embedding_dim padding_idxmax_norm norm_typescale_grad_by_freqsparse_weightreturnc sJt||||||||| | tjj|| d|_t|dddidSN)r weight optim_bits ) super__init__torchr LayerNormnormr get_instanceregister_module_override) selfrrrrrrrrr r  __class__M/var/www/html/ai/venv/lib/python3.10/site-packages/bitsandbytes/nn/modules.pyr$s    zStableEmbedding.__init__cCtjj|j|dSNr%r initxavier_uniform_r _fill_padding_idx_with_zeror*r-r-r.reset_parameters4 z StableEmbedding.reset_parameterscCN|jdur%t|j|jdWddS1swYdSdSNrrr%no_gradr fill_r5r-r-r.r4?  "z+StableEmbedding._fill_padding_idx_with_zeroinputc CsDt||j|j|j|j|j|j}|t }| ||jj Sr0) F embeddingr rrrrrtor%get_default_dtyper'r r*r>embr-r-r.forwardDs zStableEmbedding.forward)NNrFFNNNrN) __name__ __module__ __qualname__intrfloatboolrr$r6r4rE __classcell__r-r-r+r.rs@    rcseZdZ       ddededeedeeded ed ed eed eed dffdd Z dddZ dddZ ded efddZ Z S) EmbeddingNrFrrrrrrrrr rc s8tj||||||||| d t|dddidSr)r#r$rr(r)) r*rrrrrrrrr r+r-r.r$Vs  zEmbedding.__init__cCr/r0r1r5r-r-r.r6qr7zEmbedding.reset_parameterscCr8r9r:r5r-r-r.r4|r=z%Embedding._fill_padding_idx_with_zeror>c Cs&t||j|j|j|j|j|j}|Sr0)r?r@r rrrrrrCr-r-r.rEs zEmbedding.forward)NNrFFNNrF)rGrHrIrJrrKrLrr r$r6r4rErMr-r-r+r.rNUsB     rNc seZdZd deejdededed e d df d d Z e d!dejde e e fded dfddZddZed"dedeeeefdeeee fded ef ddZed#dedeee fded efddZed#dededed efddZfddZZS)$ Params4bitNT@fp4data quant_state blocksizecompress_statistics quant_typercCsD|dur td}tj|||}||_||_||_||_||_|Sr9) r%emptyr_make_subclassrTrUrVrSrR)clsrR requires_gradrSrTrUrVr*r-r-r.__new__s zParams4bit.__new__Fcudaquantized_statsrZcKsLtj|||}||_tj||d|_|jj|_|jj |_ |jj |_ |S)N)qs_dictr ) r%rrXrArZr from_dictrSrTnestedrUrV)rYrRr]rZr kwargsr*r-r-r.from_prequantizeds   zParams4bit.from_prequantizedcCsP|jdur|S|j|}tjj||j|j |j d\}}||_||_|S)N)rTrUrV) rSrR contiguoushalfr\bnb functional quantize_4bitrTrUrV)r*r ww_4bitrSr-r-r.r\s zParams4bit.cuda.r*r r non_blockingcCdSr0r-r*r r rjr-r-r.rAz Params4bit.tocCrkr0r-r*r rjr-r-r.rArmtensorcCrkr0r-r*rorjr-r-r.rArmcstjjj|i|\}}}}|dur#|jdkr#|jjjdkr#||S|jdur.|j |t t j |||d|j |j|j |j|jd}|S)Nr\cpur r rj)rZrSrTrUrV)r%_C_nn _parse_totyperRr r\rSrArOr#rZrTrUrVr*argsrar r rjconvert_to_format new_paramr+r-r.rAs    )NTNrPTrQ)Fr\....)rGrHrIrr%rr rJrLstrr[ classmethodrrrbr\rrrr r rArMr-r-r+r.rOs* (  :&rOcsBeZdZd fdd ZddZfdd Zd ejfd d ZZ S) Linear4bitTNrQcs8t||||t|jjd||d|_||_d|_dS)NF)rZrUrV)r#r$rOr rR compute_dtypecompute_type_is_set)r*input_featuresoutput_featuresbiasrrUrVr r+r-r.r$s zLinear4bit.__init__cCs|jtjtjfvr|j|_dS|jtjkrM|jtjkr0||jdkr0t dtj ddd|jtjkrO||jdkrQt dtj ddddSdSdSdS)NzInput type into Linear4bit is torch.float16, but bnb_4bit_compute_type=torch.float32 (default). This will lead to slow inference.ignorez .*inference.)messagezInput type into Linear4bit is torch.float16, but bnb_4bit_compute_type=torch.float32 (default). This will lead to slow inference or training speed.z.*inference or training) r r%float32bfloat16rfloat16numelshapewarningswarnfilterwarnings)r*xr-r-r.set_compute_types    zLinear4bit.set_compute_typecsdt|||t|jdddur.|jjjddD]\}}|r#|n|||d|<qdSdS)zc save weight and bias, then fill state_dict with components of quant_state rSNT)packedzweight.)r#_save_to_state_dictgetattrr rSas_dictitemsdetach)r* destinationprefix keep_varskvr+r-r.rs zLinear4bit._save_to_state_dictrcCs|jdur|jj|jkr|jj|j|j_t|jdddur#td|js.||d|_|j}|j dur<||j }|jdurCdn|j|j }t j ||j ||jj d}||}|S)NrSzhFP4 quantization state not initialized. Please call .cuda() or .to(device) on the LinearFP4 layer first.T)rrS)rr rRrArr printrrrre matmul_4bittrS)r*r inp_dtyperoutr-r-r.rEs    zLinear4bit.forward)TNTrQN) rGrHrIr$rrr%rrErMr-r-r+r.rs   rcseZdZdfdd ZZS) LinearFP4TNc t|||||d|dS)NrQr#r$r*rrrrrUr r+r-r.r$ zLinearFP4.__init__TNTN)rGrHrIr$rMr-r-r+r.r srcs"eZdZdZdfdd ZZS) LinearNF4a7 Implements the NF4 data type. Constructs a quantization data type where each bin has equal area under a standard normal distribution N(0, 1) that is normalized into the range [-1, 1]. For more information read the paper: QLoRA: Efficient Finetuning of Quantized LLMs (https://arxiv.org/abs/2305.14314) Implementation of the NF4 data type in bitsandbytes can be found in the `create_normal_map` function in the `functional.py` file: https://github.com/TimDettmers/bitsandbytes/blob/main/bitsandbytes/functional.py#L236. TNc r)Nnf4rrr+r-r.r$rzLinearNF4.__init__r)rGrHrI__doc__r$rMr-r-r+r.rs rc seZdZ     dddZfddZe   dd ed eee e fd eee e fd e d ef ddZedd ed ee e fd e d efddZedd eded e d efddZfddZZS) Int8ParamsNTFcCs4||_d|_d|_|durtd}tj|||Sr9)has_fp16_weightsCBSCBr%rWrrX)rYrRrZrrrr-r-r.r[s  zInt8Params.__new__csb|jr t|S|j|}tj|\}}}}}~~||_t |d|t |d||S)Nrr) rr#r\rRrcrdrerf double_quantsetattr)r*r BrCBtrSCBt coo_tensorBr+r-r.r\-s   zInt8Params.cuda.r*r r rjrcCrkr0r-rlr-r-r.rA=sz Int8Params.tocCrkr0r-rnr-r-r.rAFrmrocCrkr0r-rpr-r-r.rAJrmcsztjjj|i|\}}}}|dur#|jdkr#|jjjdkr#||Stt j |||d|j |j d}|j |_ |j|_|S)Nr\rqrr)rZr)r%rsrtrurvrRr r\rr#rArZrrrrwr+r-r.rANs&  )NTFNNr{r|)rGrHrIr[r\rrrrrJr r r}rLrArrMr-r-r+r.rs8  &rc Cs^||d}|durdS||dd}|dkr-t||j} t|| ||d<dSdS)Nr weight_formatrow)getpoprr r ) state_dictrlocal_metadatastrict missing_keysunexpected_keys error_msgsr r tile_indicesr-r-r.maybe_rearrange_weightgs rcsReZdZ  dfdd ZfddZfd d Zd d Zd ejfddZ Z S) Linear8bitLtTFNc s|t|||||rJdt|_||_||j_||j_||j_|dkr-|s-d|j_ t |j j ||d|_ | tdS)Nzvmemory_efficient_backward is no longer required and the argument is deprecated in 0.37.0 and will be removed in 0.39.0rTrrZ)r#r$re MatmulLtStatestateindex thresholdrmemory_efficient_backwarduse_poolrr rR"_register_load_state_dict_pre_hookr r*rrrrrrrr r+r-r.r$ts   zLinear8bitLt.__init__c st|||d}t|j|}t|j|}|jjdu}||}|d} |jjsi|dur=|r1|n|||<d|| <dS|durS|sS|rG|n|||<d|| <dS|durk|r[|n|||<|jj|| <dSdSdS)Nrrr) r#rrr rCxBrrformatB) r*rrrscb_nameparam_from_weightparam_from_statelayout_reorderedkey_name format_namer+r-r.rs&       z Linear8bitLt._save_to_state_dictc st|||||||t|}|D]4} | t|d} | dkrF|jjdur*td|| } |jj| |jjdurA|jj|j_| | qdS)NrzLoading a quantized checkpoint into non-quantized Linear8bitLt is not supported. Please call module.cuda() before module.load_state_dict()) r#_load_from_state_dictlistlenr r RuntimeErrorcopy_rremove) r*rrrrrrrunexpected_copykey input_name input_paramr+r-r.rs     z"Linear8bitLt._load_from_state_dictcC,|jj|j_|jj|j_d|j_d|j_dSr0r rrrr5r-r-r.init_8bit_state   zLinear8bitLt.init_8bit_statercCs|j|j_|jjdur||jdur%|jj|jkr%|jj |j|j_t j ||j|j|jd}|jj sJ|jjdurJ|jj durJ|j`|jj |j_|SN)rr)trainingr is_trainingr rrrr rRrArematmulrrr*rrr-r-r.rEs   zLinear8bitLt.forwardTTFrNN) rGrHrIr$rrrr%rrErMr-r-r+r.rss  rcs6eZdZd fdd ZddZddZd d ZZS) OutlierAwareLinearTNcs"t||||d|_d|_dS)NF)r#r$ outlier_dim is_quantized)r*rrrr r+r-r.r$s zOutlierAwareLinear.__init__cCtd)NzJPlease override the `forward_with_outliers(self, x, outlier_idx)` functionNotImplementedError)r*r outlier_idxr-r-r.forward_with_outliersz(OutlierAwareLinear.forward_with_outlierscCr)NzEPlease override the `quantize_weights(self, w, outlier_idx)` functionr)r*rhrr-r-r.quantize_weightrz"OutlierAwareLinear.quantize_weightcCsf|jdurt}|std||j}||_|js1||j|j}|jj |d|_dSdS)NzTPlease use OutlierTracer.initialize(model) before using the OutlierAwareLinear layerT) rrr(is_initializedr get_outliersr rrrRr)r*rtracerrrhr-r-r.rEs   zOutlierAwareLinear.forward)TN)rGrHrIr$rrrErMr-r-r+r.rs rcs:eZdZ      d fdd ZddZd d ZZS) SwitchBackLinearBnbTFrNc sft||||t|_||_||j_||j_||j_|dkr'|s'd|j_ t |j j ||d|_ dS)NrTr) r#r$rerrrrrrrrr rRrr+r-r.r$s     zSwitchBackLinearBnb.__init__cCrr0rr5r-r-r.rrz#SwitchBackLinearBnb.init_8bit_statecCsF|j|j_|jjdur|tj||jd|jd|j }dSr) rrrr rrre matmul_mixedrdrrr-r-r.rE s  (zSwitchBackLinearBnb.forwardr)rGrHrIr$rrErMr-r-r+r.rsr)*typingrrrrrrrr%torch.nn.functionalr rfr?rr r bitsandbytesrebitsandbytes.functionalr bitsandbytes.autograd._functionsr rbitsandbytes.optimrbitsandbytes.utilsrrrrNr ParameterrOLinearrrrrrrrrr-r-r-r.s,    @:?;J \