o hG @sddlZddlmZmZmZmZmZmZmZddl Z ddl m Z m Z ddl m Z mZee j ee j fZgdZ  dded ed ed ed eed e j f ddZ  dded ed ed ed eed e j f ddZddeded eed dfddZdS)N)UnionIterableListDictTupleOptionalcast)Tensorinf)"_group_tensors_by_device_and_dtype_has_foreach_support)clip_grad_norm_clip_grad_normclip_grad_value_@F parametersmax_norm norm_typeerror_if_nonfiniteforeachreturncst|tjr |g}dd|D}t|}tt|dkr#tdS|djtdd|Dg}tkrRfdd|D}t|dkrI|dnt t |}nNg}| D]6\\} } \\}} |dusg|rwt || d rw| t|qX|rtd | jd | fd d|DqXtjt fd d|D}|rt||rtdd||d} tj| dd} | D];\\} } \\}} |dus|rt || d rt|| | q|rtd | jd | | } |D] }|| qq|S)aDClips gradient norm of an iterable of parameters. The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float): max norm of the gradients norm_type (float): type of the used p-norm. Can be ``'inf'`` for infinity norm. error_if_nonfinite (bool): if True, an error is thrown if the total norm of the gradients from :attr:`parameters` is ``nan``, ``inf``, or ``-inf``. Default: False (will switch to True in the future) foreach (bool): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: Total norm of the parameter gradients (viewed as a single vector). cSg|] }|jdur|jqSNgrad.0prN/var/www/html/ai/venv/lib/python3.10/site-packages/torch/nn/utils/clip_grad.py 'z#clip_grad_norm_..rgcSsg|]}|qSr)detachrgrrrr .scs$g|]}tj|tqSr)torchlinalg vector_normr"r tor# first_devicerrr 1s$Ndevice:foreach=True was passed, but can't use the foreach API on  tensorscsg|] }tj|qSr)r%r&r'r#)rrrr ;scsg|]}|qSr)r()rnormr)rrr =szThe total norm of order z for gradients from `parameters` is non-finite, so it cannot be clipped. To disable this error and scale the gradients by the non-finite norm anyway, set `error_if_nonfinite=False`gư>g?)max) isinstancer%r floatlentensorr-r r r1stackitemsr extend _foreach_norm RuntimeErrortyper&r' logical_orisnanisinfclamp _foreach_mul_r(r"mul_)rrrrrgrads grouped_gradsnorms total_normr-_ clip_coefclip_coef_clampedclip_coef_clamped_devicer$r)r*rrr sJ    &"   r cCstjdddt|||||S)zClips gradient norm of an iterable of parameters. .. warning:: This method is now deprecated in favor of :func:`torch.nn.utils.clip_grad_norm_`. z[torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.) stacklevel)warningswarnr )rrrrrrrrrWs r clip_valuec Cst|tjr |g}t|}dd|D}t|g}|D]a\\}}\\}}|dus,|rOtttt||drOt ttt|| t ttt||q|rZt d|j dt |D] }tt|j| |dqaWdn1sywYqdS)aClips gradient of an iterable of parameters at specified value. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized clip_value (float): maximum allowed value of the gradients. The gradients are clipped in the range :math:`\left[\text{-clip\_value}, \text{clip\_value}\right]` foreach (bool): use the faster foreach-based implementation If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` cSrrrrrrrr yr!z$clip_grad_value_..Nr,r.r/)minr1)r2r%r r3r r7r rr_foreach_clamp_min__foreach_clamp_max_r:r;no_gradclamp_)rrNrrBrCr-rFrrrrres$  " r)rFNr)rLtypingrrrrrrrr%r r torch.utils._foreach_utilsr r _tensor_or_tensors__all__r3boolr rrrrrrsL$ L $