o h'@sddlZddlZddlZddlZddlmZddlmZddlm Z GdddZ Gdd d Z Gd d d eZ Gd d d e Z GdddZGdddZdS)N)deepcopy)AcceleratedOptimizer)AcceleratedSchedulerc@sdeZdZdZddZddZddZdd d Zdd d ZddZ ddZ ddZ ddZ ddZ dS)HfDeepSpeedConfigaJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. c Cst|tr t|}nEtj|r.tj|ddd }t |}Wdn1s(wYn!zt | d}t |}WntttfyNtd|w||_|dS)Nrzutf-8)encodingzoExpected a string path to an existing deepspeed config, or a dictionary, or a base64 encoded string. Received: ) isinstancedictrospathexistsioopenjsonloadbase64urlsafe_b64decodedecodeloadsUnicodeDecodeErrorAttributeError ValueErrorconfigset_stage_and_offload)selfconfig_file_or_dictrfconfig_decodedrP/var/www/html/ai/venv/lib/python3.10/site-packages/accelerate/utils/deepspeed.py__init__*s"     zHfDeepSpeedConfig.__init__cCsj|dd|_d|_|s|r3tddg}t|d|dg}t||@dkr1d |_dSdSdS) Nzzero_optimization.stageFcpunvmez*zero_optimization.offload_optimizer.devicez&zero_optimization.offload_param.devicerT) get_value_stage_offloadis_zero2is_zero3setlen)roffload_devices_validoffload_devicesrrr r?s  z'HfDeepSpeedConfig.set_stage_and_offloadcCsH|j}|d}|}|D]}||}|durd|fSq||fS)N.)rsplitpopget)r ds_key_longrnodesds_keynoderrr find_config_nodeRs   z"HfDeepSpeedConfig.find_config_nodeNcCs&||\}}|dur |S|||S)zG Returns the set value or `default` if no value is set N)r6r1)rr2defaultrr4rrr r%_s zHfDeepSpeedConfig.get_valueFcCsj|j}|d}|D]}|}||}|dur'|r$td|d|jdSq |dur3||dSdS)z Deletes a sub-section of the config file if it's found. Unless `must_exist` is `True` the section doesn't have to exist. r.Nz Can't find z entry in the config: )rr/r1rr0)rr2 must_existrr3r5 parent_configrrr del_config_sub_treehs  z%HfDeepSpeedConfig.del_config_sub_treecCs||}|dur dSt|S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `True` (and it's not set to `False`` or isn't set). NFr%boolrr2valuerrr is_trues zHfDeepSpeedConfig.is_truecCs ||}|dur dSt| S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `False` (and it's not set to `True`` or isn't set). NFr;r=rrr is_falses zHfDeepSpeedConfig.is_falsecC |jdkS)Nrr&rrrr r( zHfDeepSpeedConfig.is_zero2cCrA)NrBrCrrr r)rDzHfDeepSpeedConfig.is_zero3cCs|jSN)r'rCrrr is_offloadszHfDeepSpeedConfig.is_offloadrF)F)__name__ __module__ __qualname____doc__r!rr6r%r:r?r@r(r)rGrrrr rs   rc@s eZdZdZddZddZdS)DeepSpeedEngineWrapperz Internal wrapper for deepspeed.runtime.engine.DeepSpeedEngine. This is used to follow conventional training loop. Args: engine (deepspeed.runtime.engine.DeepSpeedEngine): deepspeed engine to wrap cCs ||_dSrF)engine)rrMrrr r!rDzDeepSpeedEngineWrapper.__init__cKs"|jj|fi||jdSrF)rMbackwardstep)rlosskwargsrrr rNs zDeepSpeedEngineWrapper.backwardN)rHrIrJrKr!rNrrrr rLs rLcs>eZdZdZfddZd ddZddZed d ZZ S) DeepSpeedOptimizerWrapperz Internal wrapper around a deepspeed optimizer. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. cs$tj|dddt|jd|_dS)NF)device_placementscaleroverflow)superr!hasattr optimizer__has_overflow__)rrX __class__rr r!sz"DeepSpeedOptimizerWrapper.__init__NcCdSrFr)r set_to_nonerrr zero_gradz#DeepSpeedOptimizerWrapper.zero_gradcCr\rFrrCrrr rOr_zDeepSpeedOptimizerWrapper.stepcCs|jr|jjSdS)zTWhether or not the optimizer step was done, or skipped because of gradient overflow.F)rYrXrUrCrrr step_was_skippedsz*DeepSpeedOptimizerWrapper.step_was_skippedrF) rHrIrJrKr!r^rOpropertyr` __classcell__rrrZr rRs  rRcs(eZdZdZfddZddZZS)DeepSpeedSchedulerWrapperz Internal wrapper around a deepspeed scheduler. Args: scheduler (`torch.optim.lr_scheduler.LambdaLR`): The scheduler to wrap. optimizers (one or a list of `torch.optim.Optimizer`): cst||dSrF)rVr!)r scheduler optimizersrZrr r!sz"DeepSpeedSchedulerWrapper.__init__cCr\rFrrCrrr rOr_zDeepSpeedSchedulerWrapper.step)rHrIrJrKr!rOrbrrrZr rcs rcc@seZdZdZdddZdS) DummyOptima Dummy optimizer presents model parameters or param groups, this is primarily used to follow conventional training loop when optimizer config is specified in the deepspeed config file. Args: lr (float): Learning rate. params (iterable): iterable of parameters to optimize or dicts defining parameter groups weight_decay (float): Weight decay. **kwargs: Other arguments. MbP?rcKs||_||_||_||_dSrF)paramslr weight_decayrQ)rrhrirjrQrrr r!s zDummyOptim.__init__N)rgrrHrIrJrKr!rrrr rfsrfc@seZdZdZdddZdS)DummySchedulera Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training loop when scheduler config is specified in the deepspeed config file. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. total_num_steps (int, *optional*): Total number of steps. warmup_num_steps (int, *optional*): Number of steps for warmup. lr_scheduler_callable (callable, *optional*): A callable function that creates an LR Scheduler. It accepts only one argument `optimizer`. **kwargs: Other arguments. NrcKs"||_||_||_||_||_dSrF)rXtotal_num_stepswarmup_num_stepslr_scheduler_callablerQ)rrXrmrnrorQrrr r! s  zDummyScheduler.__init__)NrNrkrrrr rlsrl)rrrr copyrrXrrdrrrLrRrcrfrlrrrr s