o h@sdddlZddlmZddlmZGdddeZGdddeZGdd d eZGd d d eZdS) N) Optimizer)Optimizer1Statecs0eZdZ         d fdd ZZS) LARSrF Nd{Gz?c s<|dkrtdtjd||||fd|||| | | dd dS)Nr'LARS without momentum is not supported!larsF max_unorm block_wiseNotImplementedErrorsuper__init__) selfparamslrmomentum dampening weight_decaynesterov optim_bitsargs min_8bit_sizepercentile_clippingr  __class__M/var/www/html/ai/venv/lib/python3.10/site-packages/bitsandbytes/optim/lars.pyr s$ z LARS.__init__) rrrFrNrrr__name__ __module__ __qualname__r __classcell__r r rr!r src.eZdZ        d fdd ZZS) LARS8bitrFNrrrc <|dkrtdtjd||||fd|d||| | dd dS)Nrr r r Fr r rrrrrrrrrrr rr r!r/$  zLARS8bit.__init__rrrFNrrrr"r r rr!r(.r(cr') LARS32bitrFNrrrc r))Nrr r r rFr rr+rr r!rQr,zLARS32bit.__init__r-r"r r rr!r/Pr.r/csHeZdZ      d fdd ZfddZed d d ZZS) PytorchLARS{Gz?rFrc s|dkr td||dkrtd||dkr!td|t||||||d}|r9|dks5|dkr9tdt||dS)Nr zInvalid learning rate: zInvalid momentum value: zInvalid weight_decay value: )rrrrrr rz8Nesterov momentum requires a momentum and zero dampening) ValueErrordictrr) rrrrrrrr defaultsrr r!rss* zPytorchLARS.__init__cs(t||jD]}|ddq dS)NrF)r __setstate__ param_groups setdefault)rstategrouprr r!r5s  zPytorchLARS.__setstate__NcCsd}|durt |}Wdn1swY|jD]}g}g}g}|d}|d}|d} |d} |d} |d} |dD]|} | jdurLqD|j| }| j}|d kr_|j| |d }|d kr|d d}|duryt|}||d <n | |j |d | d | r|||}n|}d }| dkr| j tj ksJt | }t |}|| |kr| ||}| j || |d qDq |S)zPerforms a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrr rrr)alphamomentum_bufferg?r )torch enable_gradr6gradr8addgetclonedetachmul_add_dtypefloat32norm)rclosurelossr9params_with_gradd_p_listmomentum_buffer_listrrrrr rpr8d_pbufupdate update_scalepnormunormr r r!stepsR          !zPytorchLARS.step)r1rrrFr)N) r#r$r%rr5r=no_gradrUr&r r rr!r0rs !r0) r= torch.optimrbitsandbytes.optim.optimizerrrr(r/r0r r r r!s  #""