o h@sddlZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z dddZddZd d Zd eeejfd eee jffd dZGdddeZGdddeZd eee jfdeefddZdS)N)Mapping)DictListOptionalUnion) safe_opencCsd}t|jdkr|tj}d}|}tj ||d}|dur8|dur.t|j}|t |j d||<|j dkrA|d}t j||jd|j d}|dd|dd<||S)Nztorch.bfloat16bfloat16.dat)dtypeshaperzw+)r moder )strr viewtorchint16cpunumpyospathjoinlistr ndimnpmemmapflush)weight weight_nameoffload_folderindexr array tensor_file file_arrayr"N/var/www/html/ai/venv/lib/python3.10/site-packages/accelerate/utils/offload.pyoffload_weights     r$cCs|t|d}|dkr d}|d}|dkrd}tj|||dd}t|dd kr+|d }t|}|ddkr<|tj}|S) Nr r")r rrr)r r r r)tuplerrlenrtensorrr) weight_file weight_infor r rr"r"r#load_offloaded_weight.s    r,cCs|dus t|dkr dStj|d}tj|r6t|ddd }t|}Wdn1s0wYni}||t|ddd}tj ||ddWddS1sXwYdS) Nr index.jsonr&zutf-8)encodingw)indent) r(rrrisfileopenjsonloadupdatedump)rroffload_index_filef current_indexr"r"r#save_offload_indexDs   "r;save_dir state_dictcCsBtj|ddi}|D] \}}t||||d}q t||dS)a Offload a state dict in a given folder. Args: save_dir (`str` or `os.PathLike`): The directory in which to offload the state dict. state_dict (`Dict[str, torch.Tensor]`): The dictionary of tensors to offload. T)exist_ok)rN)rmakedirsitemsr$r;)r<r=rname parameterr"r"r#offload_state_dictUs  rCc@s:eZdZdZdedefddZddZdd Zd d Z d S) PrefixedDatasetz Will access keys in a given dataset by adding a prefix. Args: dataset (`Mapping`): Any map with string keys. prefix (`str`): A prefix to add when trying to access any element in the underlying dataset. datasetprefixcCs||_||_dSNrErF)selfrErFr"r"r#__init__qs zPrefixedDataset.__init__cCs|j|j|SrGrH)rIkeyr"r"r# __getitem__uszPrefixedDataset.__getitem__cstfddjDS)Ncsg|] }|jr|qSr") startswithrF.0rKrIr"r# ysz,PrefixedDataset.__iter__..)iterrErPr"rPr#__iter__xszPrefixedDataset.__iter__cC t|jSrG)r(rErPr"r"r#__len__{ zPrefixedDataset.__len__N) __name__ __module__ __qualname____doc__rr rJrLrSrUr"r"r"r#rDhs  rDc@sfeZdZdZ    ddeeejfdee ee j fde fddZ defd d Zd d Zd dZdS)OffloadedWeightsLoaderaC A collection that loads weights stored in a given state dict or memory-mapped on disk. Args: state_dict (`Dict[str, torch.Tensor]`, *optional*): A dictionary parameter name to tensor. save_folder (`str` or `os.PathLike`, *optional*): The directory in which the weights are stored (by `offload_state_dict` for instance). index (`Dict`, *optional*): A dictionary from weight name to their information (`dtype`/ `shape` or safetensors filename). Will default to the index saved in `save_folder`. Nr= save_folderrcs|dur|dur|durtd|durin|_|_|durB|durBttj|d }t|}Wdn1s=wY|durHin|_ t j _ j fddj D|_dS)NzWNeed either a `state_dict`, a `save_folder` or an `index` containing offloaded weights.r-csg|] }|jvr|qSr")all_keysrNrPr"r#rQsz3OffloadedWeightsLoader.__init__..) ValueErrorr=r\r3rrrr4r5rrkeysr]extenddevice)rIr=r\rrar9r"rPr#rJs  zOffloadedWeightsLoader.__init__rKcCs||jvr |j|S|j|}|ddurR|jdurdn|j}t|dd|d}||d|}Wdn1s=wYd|vrP|tt|dS|St j |j |d}t ||S)Nsafetensors_filerpt) frameworkrarr r )r=rgetrar get_tensortogetattrrrrrr\r,)rIrKr+rar9r)r*r"r"r#rLs    z"OffloadedWeightsLoader.__getitem__cCrTrG)rRr]rPr"r"r#rSrVzOffloadedWeightsLoader.__iter__cCrTrG)r(r]rPr"r"r#rUrVzOffloadedWeightsLoader.__len__)NNNN)rWrXrYrZrr rTensorrrrPathLikerrJrLrSrUr"r"r"r#r[s    r[submodule_namescs.i}|D]|fdd|Dq|S)a Extract the sub state-dict corresponding to a list of given submodules. Args: state_dict (`Dict[str, torch.Tensor]`): The state dict to extract from. submodule_names (`List[str]`): The list of submodule names we want to extract. cs,i|]\}}|ks|dr||qS).)rM)rOrKparam module_namer"r# s z1extract_submodules_state_dict..)r6r@)r=rkresultr"rnr#extract_submodules_state_dicts rrrG)r4rcollections.abcrtypingrrrrrrr safetensorsrr$r,r;r rjrirCrDr[rrr"r"r"r#s   &$;