o hn@sddlZddlZddlmZmZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddgZGd d d ZGd ddeZd d ZGdddZdS)N)abc defaultdict)Enum)AnycastDictListOptionalTuple)amp_definitely_not_availableOptState GradScalerc@s4eZdZdZdejddfddZdejfddZdS) _MultiDeviceReplicatorz_ Lazily serves copies of a tensor to requested devices. Copies are cached per-device. master_tensorreturnNcCs&|js |jjdks J||_i|_dS)Nxla)is_cudadevicetypemaster_per_device_tensors)selfrrP/var/www/html/ai/venv/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py__init__s z_MultiDeviceReplicator.__init__cCs6|j|d}|dur|jj|ddd}||j|<|S)NT)r non_blockingcopy)rgetrto)rrretvalrrrrs  z_MultiDeviceReplicator.get)__name__ __module__ __qualname____doc__torchTensorrrrrrrrsrc@seZdZdZdZdZdS)r rr N)r!r"r#READYUNSCALEDSTEPPEDrrrrr %scCs tjidS)N)stagefound_inf_per_device)r r(rrrr_refresh_per_optimizer_state+s r-c@s,eZdZUeejed<eejed<eeee e ffed<     d=d d Z d e ejejffd d Z ddZddZddZddZddZddZd>ddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd-d.Zd/d0Zd1d2Zd3d4Z d5d6Z!d7d8Z"d9d:Z#d;d<Z$dS)?r_scale_grows_tracker_per_optimizer_states@@?TcCs|rtrtdd|_n||_|jr@|dksJd|dks$Jd||_d|_||_||_||_d|_ d|_ t t |_ dSdS)NzLtorch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling.F?z The growth factor must be > 1.0.z!The backoff factor must be < 1.0.r)r warningswarn_enabled _init_scaler._growth_factor_backoff_factor_growth_interval_init_growth_tracker_growth_trackerrr-r0)r init_scale growth_factorbackoff_factorgrowth_intervalenabledrrrrts$ zGradScaler.__init__rcCsLd}|jdusJd|d||jdus Jd|d||j|jfS)NzaThis may indicate your script did not use scaler.scale(loss or outputs) earlier in the iteration.z Attempted z but _scale is None. z but _growth_tracker is None. )r.r>)rfuncnamefixrrr_check_scale_growth_trackers   z&GradScaler._check_scale_growth_trackercCsF|jdus Jdtjd|jtj|d|_tjd|jtj|d|_dS)Nz)_growth_tracker initialized before _scalerdtyper)r>r%fullr9float32r.r=int32)rdevrrr_lazy_init_scale_growth_trackers   z*GradScaler._lazy_init_scale_growth_trackercsjs|St|tjr3|js|jjdksJjdur!|jjdus(J|jj |jddSgfdd|S)a2 Multiplies ('scales') a tensor or list of tensors by the scale factor. Returns scaled outputs. If this instance of :class:`GradScaler` is not enabled, outputs are returned unmodified. Args: outputs (Tensor or iterable of Tensors): Outputs to scale. rNTrrcst|tjr;|js|jjdksJtdkr1jdur"|jjdus)J t j|d |jSt|t j rUt|}t|ttfrSt||S|Std)Nrrz2outputs must be a Tensor or an iterable of Tensors) isinstancer%r&rrrlenr.rMappendrrrIterablemaplisttuple ValueError)valiterable apply_scalerstashrrrZs       z%GradScaler.scale..apply_scale) r8rOr%r&rrrr.rMr)routputsrrYrscales    zGradScaler.scalec Cst|}t|}tdd}tl|jD]A}|dD]:} | jdur$q|s1| jjtjkr1td| jj rH| jjtjurB| j | _| j } n| j} || j | j | qq|D]\} } | D]} t| || || qdq\Wd|jS1swY|jS)NcSsttSN)rrTrrrrsz,GradScaler._unscale_grads_..paramsz%Attempting to unscale FP16 gradients.)rrr%no_grad param_groupsgradrHfloat16rV is_sparsecoalesce_valuesrrQitemsvalues*_amp_foreach_non_finite_check_and_unscale_rr)r optimizer inv_scale found_inf allow_fp16per_device_inv_scaleper_device_found_infper_device_and_dtype_gradsgroupparam to_unscalerper_dtype_gradsgradsrrr_unscale_grads_sB          zGradScaler._unscale_grads_cCs|jsdS|d|jt|}|dtjurtd|dtjur'td|jdus.J|j }t j ddt j|jjd}||||d |d <tj|d<dS) as Divides ("unscales") the optimizer's gradient tensors by the scale factor. :meth:`unscale_` is optional, serving cases where you need to :ref:`modify or inspect gradients` between the backward pass(es) and :meth:`step`. If :meth:`unscale_` is not called explicitly, gradients will be unscaled automatically during :meth:`step`. Simple example, using :meth:`unscale_` to enable clipping of unscaled gradients:: ... scaler.scale(loss).backward() scaler.unscale_(optimizer) torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm) scaler.step(optimizer) scaler.update() Args: optimizer (torch.optim.Optimizer): Optimizer that owns the gradients to be unscaled. .. note:: :meth:`unscale_` does not incur a CPU-GPU sync. .. warning:: :meth:`unscale_` should only be called once per optimizer per :meth:`step` call, and only after all gradients for that optimizer's assigned parameters have been accumulated. Calling :meth:`unscale_` twice for a given optimizer between each :meth:`step` triggers a RuntimeError. .. warning:: :meth:`unscale_` may unscale sparse gradients out of place, replacing the ``.grad`` attribute. Nunscale_r+zMunscale_() has already been called on this optimizer since the last update().z(unscale_() is being called after step().rrGFr,)r8rFr0idr r) RuntimeErrorr*r.double reciprocalfloatr%rIrJrrw)rrkoptimizer_staterlrmrrrrxs"  zGradScaler.unscale_cOs2d}tdd|dDs|j|i|}|S)Ncss|]}|VqdSr^)item).0vrrr :sz-GradScaler._maybe_opt_step..r,)sumristep)rrkrargskwargsr rrr_maybe_opt_step8szGradScaler._maybe_opt_stepc s|js |j|i|Sd|vrtd|d|jt|}|dtjur*tdd}t|dr|j r|}dt |jj v}|rOt d t|d|in2|dtjur[|||ttjtfd d |d D}|dtjkr{dn|_||_|j|i|}tj|d<|s|`|`|S|dtjur||t|d d ksJd|j||g|Ri|}tj|d<|S)a :meth:`step` carries out the following two operations: 1. Internally invokes ``unscale_(optimizer)`` (unless :meth:`unscale_` was explicitly called for ``optimizer`` earlier in the iteration). As part of the :meth:`unscale_`, gradients are checked for infs/NaNs. 2. If no inf/NaN gradients are found, invokes ``optimizer.step()`` using the unscaled gradients. Otherwise, ``optimizer.step()`` is skipped to avoid corrupting the params. ``*args`` and ``**kwargs`` are forwarded to ``optimizer.step()``. Returns the return value of ``optimizer.step(*args, **kwargs)``. Args: optimizer (torch.optim.Optimizer): Optimizer that applies the gradients. args: Any arguments. kwargs: Any keyword arguments. .. warning:: Closure use is not currently supported. closurez@Closure use is not currently supported if GradScaler is enabled.rr+z7step() has already been called since the last update().N_step_supports_amp_scaling grad_scalerzGradScaler is going to stop passing itself as a keyword argument to the passed optimizer. In the near future GradScaler registers `grad_scale: Tensor` and `found_inf: Tensor` to the passed optimizer and let the optimizer use them directly.csg|] }|jjddqS)T)r)rr)rtscalerrr sz#GradScaler.step..r,rz/No inf checks were recorded for this optimizer.)r8rr{rFr0rzr r*hasattrrinspect signature parametersr6r7 FutureWarningupdater(_check_inf_per_device_get_scale_asyncrr%r&rrir) grad_scalermrxrPr) rrkrrrr kwargs_has_grad_scaler_kwargrmrrrr>sl        zGradScaler.stepNcs|jsdS|d\}|durCt|tr|j|ndd}t|tjjs)J|| dks3J||j dus.rz,No inf checks were recorded prior to update.)r8rFrOr~r.fill_r%cuda FloatTensornumel requires_gradcopy_r0rirPrange_amp_update_scale_r:r;r<rr-)r new_scaler>reason found_infsfound_inf_combinedirrrrs8    zGradScaler.updatecC|jSr^rrrrrrszGradScaler._get_scale_asynccCs&|jr|jdur |jS|SdS)z Returns a Python float containing the current scale, or 1.0 if scaling is disabled. .. warning:: :meth:`get_scale` incurs a CPU-GPU sync. Nr5)r8r.r9rrrrrr get_scales  zGradScaler.get_scalecCr)zL Returns a Python float containing the scale growth factor. r:rrrrget_growth_factorzGradScaler.get_growth_factorcC ||_dS)zd Args: new_scale (float): Value to use as the new scale growth factor. Nrr new_factorrrrset_growth_factor zGradScaler.set_growth_factorcCr)zM Returns a Python float containing the scale backoff factor. r;rrrrget_backoff_factor rzGradScaler.get_backoff_factorcCr)ze Args: new_scale (float): Value to use as the new scale backoff factor. Nrrrrrset_backoff_factorrzGradScaler.set_backoff_factorcCr)zF Returns a Python int containing the growth interval. r<rrrrget_growth_intervalrzGradScaler.get_growth_intervalcCr)za Args: new_interval (int): Value to use as the new growth interval. Nr)r new_intervalrrrset_growth_intervalrzGradScaler.set_growth_intervalcCs$|jr|jdur |jS|jSdS)Nr)r8r>r=rrrrr_get_growth_tracker%s zGradScaler._get_growth_trackercCr)zM Returns a bool indicating whether this instance is enabled. )r8rrrr is_enabled/rzGradScaler.is_enabledcCs(|jr||j|j|j|dSiS)a Returns the state of the scaler as a :class:`dict`. It contains five entries: * ``"scale"`` - a Python float containing the current scale * ``"growth_factor"`` - a Python float containing the current growth factor * ``"backoff_factor"`` - a Python float containing the current backoff factor * ``"growth_interval"`` - a Python int containing the current growth interval * ``"_growth_tracker"`` - a Python int containing the number of recent consecutive unskipped steps. If this instance is not enabled, returns an empty dict. .. note:: If you wish to checkpoint the scaler's state after a particular iteration, :meth:`state_dict` should be called after :meth:`update`. )r]r@rArBr>)r8rr:r;r<rrrrr state_dict5s zGradScaler.state_dictcCs|jsdSt|dkrtd|d|_|jdur!|j|d|d|_|d|_|d|_|d|_ |j durD|j |ddSdS) z Loads the scaler state. If this instance is disabled, :meth:`load_state_dict` is a no-op. Args: state_dict(dict): scaler state. Should be an object returned from a call to :meth:`state_dict`. NrzeThe source state dict is empty, possibly because it was saved from a disabled instance of GradScaler.r]r@rArBr>) r8rPr{r9r.rr:r;r<r=r>)rrrrrload_state_dictQs         zGradScaler.load_state_dictcCsR|j}|jr't|jdksJd||d<||d<d|d<d|d<|S)NrzpA GradScaler instance may only be pickled at the beginning of an iteration, or at the end after scaler.update().r9r=r.r>)__dict__rr8rPr0rrrrrrr __getstate__ks   zGradScaler.__getstate__cCs|j|dSr^)rrrrrr __setstate__{szGradScaler.__setstate__cCsj|d\}}tjddtj|jd}tjddtj|jd}||||d|jt|d<|jt|dS)Nrrr5rGryTr,)rFr%rIrJrrwr0rz)rrkr._dummy_inv_scalermrrrr~s z GradScaler._check_inf_per_devicecCs|jt|dS)Nr,)r0rz)rrkrrr_found_inf_per_devicesz GradScaler._found_inf_per_device)r1r2r3r4Tr^)%r!r"r#r r%r&__annotations__rintstrrrr rFrMr]rwrxrrrrrrrrrrrrrrrrrrrrrrrr/sJ C   .,8 hE  )rr6 collectionsrrenumrtypingrrrrr r r%commonr __all__rr r-rrrrrs