o
hl, @ s0 d dl Z d dlmZ d dlmZ d dlZd dlZd dlm Z d dl
m Z ddl m
Z
mZmZmZmZmZmZmZmZmZmZmZmZ edd rPd dlm m Z dd
l m Z dd l m!Z! e e"Z# d d
e$dee% de&de&de&de'de de(de(fddZ) d ddZ*d!de'de(fdd Z+d"de'fd d Z,dS )# N)Path)List) load_file)
GradScaler )
MODEL_NAMEOPTIMIZER_NAMERNG_STATE_NAMESAFE_MODEL_NAMESAFE_WEIGHTS_NAME SAMPLER_NAME SCALER_NAMESCHEDULER_NAME WEIGHTS_NAMEget_pretty_nameis_tpu_availableis_xpu_availablesaveF) check_device)
get_logger) PartialStateT
output_dir model_states
optimizers
schedulers dataloaders
process_indexscalersave_on_each_nodesafe_serializationc C s t | } t|D ]-\} }
|stnt} | dkr | dd| d} | | } t|
| ||d td| qt|D ].\} }
|
}
| dkrKt
dnt
d| d}| |}t|
||dd td| q:t|D ].\} }| }
| dkr~t dnt d| d}| |}t|
||dd td | qmt|D ]D\} }| dkrt dnt d| d}| |}d
d l
m}m} t|j|r|jj}t||rt|||dd td | d
| q|dur| }
| t}t|
| td| i }t d| d}t |d< tj |d< t |d< t r*tj |d< ntj |d< t r;t |d< | |} t|| td| | S )a+
Saves the current states of the models, optimizers, scaler, and RNG generators to a given directory.
If `safe_serialization` is `True`, models will be saved with `safetensors` while the rest are saved using native
`pickle`.
Args:
output_dir (`str` or `os.PathLike`):
The name of the folder to save all relevant weights and states.
model_states (`List[torch.nn.Module]`):
A list of model states
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
dataloaders (`List[torch.utils.data.DataLoader]`):
A list of dataloader instances to save their sampler states
process_index (`int`):
The current process index in the Accelerator state
scaler (`torch.cuda.amp.GradScaler`, *optional*):
An optional gradient scaler instance to save
save_on_each_node (`bool`, *optional*):
Whether to save on every node, or only the main node.
safe_serialization (`bool`, *optional*, defaults to `True`):
Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
r ._)r r zModel weights saved in .binFzOptimizer state saved in zScheduler state saved in r IterableDatasetShardSeedableRandomSamplerz Sampler state for dataloader z
saved in Nz Gradient scaler state saved in .pkl random_statenumpy_random_seedtorch_manual_seedtorch_xpu_manual_seedtorch_cuda_manual_seedxm_seedzRandom states saved in )!r enumerater r replacejoinpathr loggerinfo
state_dictr r r data_loaderr$ r%
isinstancedatasetsamplerr
torchr randomgetstatenp get_state
get_rng_stater xpuget_rng_state_allcudar xm) r r r r r r r r r istate weights_nameoutput_model_fileoptoptimizer_nameoutput_optimizer_file schedulerscheduler_nameoutput_scheduler_file
dataloader sampler_nameoutput_sampler_filer$ r% r6 output_scaler_filestates states_nameoutput_states_file rR N/var/www/html/ai/venv/lib/python3.10/site-packages/accelerate/checkpointing.pysave_accelerator_state3 s` )
"
"
"
rT c K s |dvrt d|du rd}n|dkrt j}t| } t|D ]B\} }
| dkr,d| nd} | t | d } | rEt| t |d
}
n| t
| d } t j | |d }
|| j
|
fi | q td
t|D ](\} }| dkrxt d nt d| d }| |}t j ||d }||
| qktd t|D ]"\} }| dkrt d nt d| d }| |}|
t | qtd t|D ]:\} }| dkrt d nt d| d }| |}ddlm}m} t|j|r|jj}t||rt ||j_qtd |dur | t}|
t | td zLt | t d| d} t | d t j | d t | d t rRt j!"| d nt j#"| d t$ ret% | d td W dS t&y| td Y dS w ) a
Loads states of the models, optimizers, scaler, and RNG generators from a given directory.
Args:
input_dir (`str` or `os.PathLike`):
The name of the folder to load all relevant weights and states.
models (`List[torch.nn.Module]`):
A list of model instances
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
process_index (`int`):
The current process index in the Accelerator state
scaler (`torch.cuda.amp.GradScaler`, *optional*):
An optional *GradScaler* instance to load
map_location (`str`, *optional*):
What device to load the optimizer state onto. Should be one of either "cpu" or "on_device".
load_model_func_kwargs (`dict`, *optional*):
Additional arguments that can be passed to the model's `load_state_dict` method.
)Ncpu on_devicezaUnsupported optimizer map location passed, please choose one of `None`, `'cpu'`, or `'on_device'`NrU rV r r! z .safetensors)devicer" map_locationz%All model weights loaded successfullyz(All optimizer states loaded successfullyz(All scheduler states loaded successfullyr r# z1All dataloader sampler states loaded successfullyz$GradScaler state loaded successfullyr& r' r( r) r* r+ r, z%All random states loaded successfullyz Could not load random states)' TypeErrorr rX r r- r/ r
existsr strr r7 loadload_state_dictr0 r1 r r r r3 r$ r% r4 r5 r6 r
r r8 setstater: set_state
set_rng_stater r= set_rng_state_allr? r r@ Exception) input_dirmodelsr r r r r rZ load_model_func_kwargsrA modelendinginput_model_filer2 rE rF input_optimizer_fileoptimizer_staterH rI input_scheduler_filerK rL input_sampler_filer$ r% r6 input_scaler_filerO rR rR rS load_accelerator_state sr
"
"
"
rp indexc C sD t |d| d }tdt| d| t| ||d dS )zL
Saves the state of `obj` to `{path}/custom_checkpoint_{index}.pkl`
custom_checkpoint_r& zSaving the state of z to )r N)r r0 r1 r r r2 )objpathrq r
save_locationrR rR rS save_custom_state s rv c C sB | d| d}t dt| d| | tj|dd dS )zL
Loads the state of `obj` at `{path}/custom_checkpoint_{index}.pkl`
z/custom_checkpoint_r& zLoading the state of z from rU rY N)r0 r1 r r_ r7 r^ )rs rt rq
load_locationrR rR rS load_custom_state
s rx )NFT)NN)r F)r )-r8 pathlibr typingr numpyr: r7 safetensors.torchr torch.cuda.ampr utilsr r r r
r r r
r r r r r r torch_xla.core.xla_modelcore xla_modelr@ loggingr rB r __name__r0 r] dictlistintboolrT rp rv rx rR rR rR rS sP <
m
j