o ha@s dZddlZddlZddlmZmZddlmZmZddl Z ddl m Z ddl m Z dd lmZmZdd lmZmZmZed d rJddlmmZerSdd lmZddZddZddZddZddZed dddZ dVddZ!ddZ"dd Z#d!d"Z$d#d$Z%d%d&Z&d'd(Z'd)d*Z(Gd+d,d,e)Z*d-d.Z+d/d0Z,e+d1d2Z-d3efd4d5Z.d3efd6d7Z/dWd8d9Z0dXd;d<Z1e+dWd=e2fd>d?Z3dWd=e2fd@dAZ4dYdBdCZ5dWdDdEZ6GdFdGdGe7Z8e,dZdHdIZ9e+d[dLdMZ:dNdOZ;GdPdQdQZdS)\zB A set of basic tensor ops compatible with tpu, gpu, and multigpu N)update_wrapperwraps)AnyMapping) PartialState)!TORCH_DISTRIBUTED_OPERATION_TYPES)DistributedTypeTensorInformation)is_torch_distributed_availableis_torch_versionis_tpu_availableF) check_device)ReduceOpcCs t|tjSN) isinstancetorchTensortensorrQ/var/www/html/ai/venv/lib/python3.10/site-packages/accelerate/utils/operations.pyis_torch_tensor(s rc Cs2t|tjjtjjtjjtjjtjjtjjtjj Sr) rrxpu FloatTensor ByteTensor IntTensor LongTensor HalfTensor DoubleTensorBFloat16Tensorrrrris_torch_xpu_tensor,sr"cCs t|tSr)rr  tensor_inforrris_tensor_information9 r%cCsVt|}|j}t|dks|dtkrdSt|dd}t|ts"dStdd|DS)z Checks if `x` is a `namedtuple` or not. Can have false positives, but only if a user is trying to mimic a `namedtuple` perfectly. rrF_fieldsNcss|]}t|tVqdSr)rstr).0memberrrr Isz is_namedtuple..)type __bases__lentuplegetattrrall)data data_typebasesfieldsrrr is_namedtuple=s  r6cCs$t|r t|t|St||S)zO Cast a generator to the same type as obj (list, tuple, or namedtuple) )r6r,list)obj generatorrrr honor_typeLs r: test_typeerror_on_other_typecst|ttfrt|fdd|DSt|tr/t|fdd|DS|r>|gRiSrRtdt|djdjd|S) a9 Recursively apply a function on a data structure that is a nested list/tuple/dictionary of a given base type. Args: func (`callable`): The function to recursively apply. data (nested list/tuple/dictionary of `main_type`): The data on which to apply `func` *args: Positional arguments that will be passed to `func` when applied on the unpacked data. main_type (`type`, *optional*, defaults to `torch.Tensor`): The base type of the objects to which apply `func`. error_on_other_type (`bool`, *optional*, defaults to `False`): Whether to return an error or not if after unpacking `data`, we get on an object that is not of type `main_type`. If `False`, the function will leave objects of types different than `main_type` unchanged. **kwargs: Keyword arguments that will be passed to `func` when applied on the unpacked data. Returns: The same data structure as `data` with `func` applied to every object of type `main_type`. c3s.|]}t|gRdVqdS)r;Nrecursively_apply)r)oargsr=funckwargsr<rrr+ps z$recursively_apply..cs0i|]\}}|t|gRdqS)r;r>r)kvrArr ysz%recursively_apply..zUnsupported types (z ) passed to `z?`. Only nested list/tuple/dicts of objects that are valid for `z` should be passed.) rr/r7r:rr,items TypeError__name__)rCr2r<r=rBrDrrArr?Ws, r?cst|ttfrt|fdd|DSt|tr:ttr#gndur)gt|fdd|DSt|drVz|j dWSt yU| YSw|S)a Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to send to a given device. device (`torch.device`): The device to send the data to. Returns: The same data structure as `tensor` with all tensors sent to the proper device. c3s |] }t|dVqdS) non_blocking skip_keysNsend_to_device)r)tdevicerMrNrrr+sz!send_to_device..Nc s.i|]\}}||vr |nt|dqS)rLrO)r)rFrQrRrrrHsz"send_to_device..to)rM) rr/r7r:rr(r,rIhasattrrTrJ)rrSrMrNrrRrrPs*     rPcCdd}t||S)aK Recursively gathers the information needed to rebuild a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with [`~utils.TensorInformation`] instead of tensors. cSst|j|jdS)N)shapedtype)r rWrXrrrr_get_data_structuresz/get_data_structure.._get_data_structurer>)r2rYrrrget_data_structure rZcCrV)a: Recursively gathers the shape of a nested list/tuple/dictionary of tensors as a list. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with lists of tensor shapes instead of tensors. cSs t|jSr)r7rWrrrr _get_shaper&zget_shape.._get_shaper>)r2r\rrr get_shaper[r]cCsdd}t||tdS)z Recursively initializes tensors from a nested list/tuple/dictionary of [`~utils.TensorInformation`]. Returns: The same data structure as `data` with tensors instead of [`~utils.TensorInformation`]. cSstj|jd|jiSNrX)remptyrWrXr#rrr_initialize_tensorz.initialize_tensors.._initialize_tensorr<)r?r%)data_structurer`rrrinitialize_tensorssrdcCst|tttfrt|dkrtdt|dt|ttfr%t|dSt|tr9|D] }t||Snt|t j sIt dt|d|j dS)a Recursively finds the batch size in a nested list/tuple/dictionary of lists of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. rz&Cannot find the batch size from empty .z0Can only find the batch size of tensors but got ) rr/r7rr. ValueErrorr,find_batch_sizekeysrrrJrW)r2rFrrrrgs      rgcCrV)aS Recursively finds tensors in a nested list/tuple/dictionary and converts them to a list of numbers. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to convert to regular numbers. Returns: The same data structure as `data` with lists of numbers instead of `torch.Tensor`. cSs,|}|jtjkr|tj}|Sr)detachcpurXrbfloat16rTfloat32tolistrrrr_convert_to_lists   z!listify.._convert_to_listr>)r2rnrrrlistifys rocCs"dd}t||dd}t|S)NcSs0|jdkr |d}|s|}t|S)Nr)ndimclone is_contiguous contiguousxm all_gatherrrrr_tpu_gather_ones   z$_tpu_gather.._tpu_gather_oneTr=)r?rt mark_step)rrvresrrr _tpu_gathers rzcs>ttddr tjjntjjfdd}t||ddS)Nz>=z1.13csjdkr dsjjdkr#jjdkr#tdjdurPjdkrPtj j j jd}||j dgddRSfdd tj D}tj|tj|dd S) NrcudazOne or more of the tensors passed to `gather` were not on the GPU while the `Accelerator` is configured for CUDA. Please move it to the GPU before calling `gather`.gloo)rXrSrcsg|]}tqSr)r empty_liker)_rrr Dsz8_gpu_gather.._gpu_gather_one..dim)rprqrrrsrSr, RuntimeErrorbackendrr_ num_processesnumelrXviewsizerange distributedrucat)routput_tensors gather_opstaterr_gpu_gather_one%s&    z$_gpu_gather.._gpu_gather_oneTrw)rr rrall_gather_into_tensor_all_gather_baser?)rrrrr _gpu_gathers   #rc@seZdZdZdS)DistributedOperationExceptionz An exception class for distributed operations. Raised if the operation cannot be performed due to the shape of the tensors. N)rK __module__ __qualname____doc__rrrrrKsrctfdd}|S)zv Verifies that `tensor` is the same shape across all processes. Only ran if `PartialState().debug` is `True`. cstjtjks tjs|i|Sjdj}d|vr$|d}n|d}t|}t|g}|ddurZ| |dt |k}|sZd ddt |D}t d|d||i|S) Nrerrz - cSs g|] \}}d|d|qS)zProcess z: r)r)irWrrrrgs z5verify_operation..wrapper..znCannot apply desired operation due to shape mismatches. All shapes across devices must be valid. Operation: `z` Input shapes: - )rdistributed_typer NOdebugrrKr] gather_objectcountr.join enumerater)rBrD operationrshapesoutputare_sameprocess_shape_strfunctionrrwrapperYs(   z!verify_operation..wrapperrrrrrrverify_operationTsrcr)z Checks that `verify_operation` failed and if so reports a more helpful error chaining the existing `DistributedOperationException`. c sNz|i|WSty&}zjdj}td|d|d}~ww)NrezError found while calling `z1`. Please see the earlier error for more details.)rrrK)rBrDerrrrrxs z"chained_operation..wrapperrrrrrchained_operationrs rcCs.tjtjkr t|Stjtvrt|S|S)a4 Recursively gather tensor in a nested list/tuple/dictionary of tensors from all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. Returns: The same data structure as `tensor` with all tensors sent to the proper device. )rrr TPUrzr rrrrrgathers   robjectcCs2ddttjD}tj||dd|DS)NcSsg|]}dqSrrrrrrrsz&_gpu_gather_object..cSsg|] }|D]}|qqSrr)r)yxrrrrs)rrrrrall_gather_object)routput_objectsrrr_gpu_gather_objectsrcCs.tjtjkr tdtjtvrt|S|S)a5 Recursively gather object in a nested list/tuple/dictionary of objects from all devices. Args: object (nested list/tuple/dictionary of picklable object): The data to gather. Returns: The same data structure as `object` with all the objects sent to every device. z&gather objects in TPU is not supported)rrr rNotImplementedErrorr r)rrrrrs   rcCsddd}t||d|dS)NrcSstjj||d|S)Nsrc)rr broadcast)rrrrr_gpu_broadcast_onesz*_gpu_broadcast.._gpu_broadcast_oneT)r=rrr>)r2rrrrr_gpu_broadcasts rbroadcast tensorcsht|ttfrt|fddt|DSt|tr)t|fdd|DSt |fddS)Nc3s*|]\}}t|d|dVqdS)rnameN_tpu_broadcast)r)rrQrrrr+(z!_tpu_broadcast..cs(i|]\}}|t|d|dqS)rrrrErrrrH(z"_tpu_broadcast..c|Srrrrrrz _tpu_broadcast..) rr7r/r:rrr,rIrt mesh_reduce)rrrr)rrrrs  r from_processcCs8tjtjkrt||ddStjtvrt||dS|S)a Recursively broadcast tensor in a nested list/tuple/dictionary of tensors to all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. from_process (`int`, *optional*, defaults to 0): The process from which to send the data Returns: The same data structure as `tensor` with all tensors broadcasted to the proper device. zaccelerate.utils.broadcast)rrr)rrr rrr r)rrrrrrs   rcs^tjtjkrt|D]\}}td|fdd||<q |Stjtvr-tj j |d|S)a Broadcast a list of picklable objects form one process to the others. Args: object_list (list of picklable objects): The list of objects to broadcast. This list will be modified inplace. from_process (`int`, *optional*, defaults to 0): The process from which to send the data. Returns: The same list containing the objects from process 0. z&accelerate.utils.broadcast_object_listcrrrrrrrrrz'broadcast_object_list..r) rrr rrrtrr rrbroadcast_object_list) object_listrrr8rrrrs  rcCsdd}t|||S)aN Recursively takes a slice in a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to slice. tensor_slice (`slice`): The slice to take. Returns: The same data structure as `data` with all the tensors slices. cSs||Srr)r tensor_slicerrr _slice_tensorz$slice_tensors.._slice_tensorr>)r2r process_indexrrrrr slice_tensorss rcstdttfrtdfddttdDStdtr9tdfdddDStdt j sLt dtdt j dS)a Recursively concatenate the tensors in a nested list/tuple/dictionary of lists of tensors with the same shape. Args: data (nested list/tuple/dictionary of lists of tensors `torch.Tensor`): The data to concatenate. dim (`int`, *optional*, defaults to 0): The dimension on which to concatenate. Returns: The same data structure as `data` with all the tensors concatenated. rc3s*|]tfddDdVqdS)cg|]}|qSrrr)drrrrz)concatenate...rN concatenater)r2rrrr+rzconcatenate..cs(i|]tfddDdqS)crrrrrFrrrrz*concatenate...rrrrrrrHrzconcatenate..z%Can only concatenate tensors but got r) rr/r7r:rr.rr,rhrrrJrrrrrrs *(rc@s eZdZdS)CannotPadNestedTensorWarningN)rKrrrrrrrsrcCsddd}t||d|||dS) a3 Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so they can safely be gathered. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. dim (`int`, *optional*, defaults to 0): The dimension on which to pad. pad_index (`int`, *optional*, defaults to 0): The value with which to pad. pad_first (`bool`, *optional*, defaults to `False`): Whether to pad at the beginning or the end. rFc st|ddrtdt|St|jkr|Stj|j|jdd}t | }t fdd|D|jkr<|S|jt }|<| t||}|rdtfddtt|D}ntfddtt|D}|||<|S) N is_nestedFzHCannot pad nested tensors without more information. Leaving unprocessed.)rSc3s|]}|VqdSrr)r)srrrr+;szFpad_across_processes.._pad_across_processes..c3s2|]}|krtntdVqdSrslicer)rrmax_sizeold_sizerrr+Ds" c3s.|]}|krtdntdVqdS)rNrr)rrrrr+Hs,)r0warningswarnrr.rWrrrSrrjmaxr7 new_zerosr/r) rr pad_index pad_firstrsizesnew_size new_tensorindicesrrr_pad_across_processes-s0    z3pad_across_processes.._pad_across_processesT)r=rrrNrrFr>)rrrrrrrrpad_across_processess  rmean?cCsddd}t||d||dS) aX Recursively reduce the tensors in a nested list/tuple/dictionary of lists of tensors across all processes by the mean of a given operation. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to reduce. reduction (`str`, *optional*, defaults to `"mean"`): A reduction method. Can be of "mean", "sum", or "none" scale (`float`, *optional*): A default scaling value to be applied after the reduce, only valied on XLA. Returns: The same data structure as `data` with all the tensors reduced. rrcSslt}|}|jtjkr|S|jtjkrtd||n|jjt vr+t j |t j |dkr4||j}|S)Nsumr)rrqrr rrrt all_reducevaluer rrrSUMr)r reductionscaler cloned_tensorrrr_reduce_across_processescs    z(reduce.._reduce_across_processesT)r=rrNrrr>)rrrrrrrreduceQs  rcCsdd}dd}t|||dS)av Recursively converts the elements nested list/tuple/dictionary of tensors in FP16/BF16 precision to FP32. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to convert from FP16/BF16 to FP32. Returns: The same data structure as `tensor` with all tensors that were in FP16/BF16 precision converted to FP32. cSs|Sr)floatrrrr_convert_to_fp32rz)convert_to_fp32.._convert_to_fp32cSst|do |jtjtjfvSr^)rUrXrfloat16rkrrrr_is_fp16_bf16_tensorsz-convert_to_fp32.._is_fp16_bf16_tensorrbr>)rrrrrrconvert_to_fp32us rc@s(eZdZdZddZddZddZdS) ConvertOutputsToFp32ad Decorator to apply to a function outputing tensors (like a model forward pass) that ensures the outputs in FP16 precision will be convert back to FP32. Args: model_forward (`Callable`): The function which outputs we want to treat. Returns: The same function as `model_forward` but with converted outputs. cCs||_t||dSr) model_forwardr)selfrrrr__init__szConvertOutputsToFp32.__init__cOst|j|i|Sr)rr)rrBrDrrr__call__razConvertOutputsToFp32.__call__cCs td)NzCannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.)pickle PicklingError)rrrr __getstate__sz!ConvertOutputsToFp32.__getstate__N)rKrrrrrr rrrrrs   rcstfdd}|_|S)Ncs|i|Srr)rBrDrrrforwardsz(convert_outputs_to_fp32..forward)r __wrapped__)rr rr rconvert_outputs_to_fp32s rcCs~t|tr|D]}t|}|dur|Sq dSt|ttfr4|D]}t|}|dur1|Sq#dSt|tjr=|jSdS)z Finds the device on which a nested dict/list/tuple of tensors lies (assuming they are all on the same device). Args: (nested list/tuple/dictionary of `torch.Tensor`): The data we want to know the device of. N) rrvalues find_devicer/r7rrrS)r2r8rSrrrrs"   r)FNr)rr)NNrr)?rr r functoolsrrtypingrrrrr constantsr dataclassesr r importsr r rtorch_xla.core.xla_modelcore xla_modelrttorch.distributedrrr"r%r6r:r?rPrZr]rdrgrorzr Exceptionrrrrrrrrintrrrr UserWarningrrrrrrrrrrrsf      3%-       4 #