o h@sUdZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z m Z mZmZmZmZgdZeeed <ee ZGd d d ZGd d d e jjZGddde jjZdS)zkAutograd functions for stream-aware CUDA copy. It is used to overlap copy and computation on the same GPU. )deque)DequeListOptionalTupleSequenceN)Tensor)AbstractStreamcurrent_stream get_device record_stream use_stream wait_stream)ContextCopyWait__all__c@seZdZUeed<eed<dS)r prev_stream next_streamN)__name__ __module__ __qualname__r __annotations__rrZ/var/www/html/ai/venv/lib/python3.10/site-packages/torch/distributed/pipeline/sync/copy.pyrs  rc @TeZdZdZededededefddZedede de e e d ffd d Z d S) rz#Copies tensors on specific streams.ctxrrreturnc Gs||_||_g}tt|}t|Qt|0|D]%}t|r:|jt|dd}||t ||t ||q||qWdn1sJwYWdt |SWdt |S1sfwYt |S)NT non_blocking) rrr r rtorch is_tensortoappendr tuple)rrrinputoutput output_streamxyrrrforward!s&      ( z Copy.forward grad_output.c Gs|j}|j}tt|d}tt|}t|;t|'t|D]}|jt|dd}| |t ||t ||q!Wdn1sFwYWdn1sUwYd}|t |S)N)maxlenTrNN) rrrlenr r rreversedr# appendleftr r%) rr,rr grad_input input_streamr)r* grad_streamsrrrbackward:s       z Copy.backwardN rrr__doc__ staticmethodrr Tensorsr+rrrr5rrrrrs (rc @r) rzSynchronizes a stream to another stream. Place it just before you want to start an operation on the next stream, provided that all operations on the previous stream are done. rrrrcGs(||_||_t||tdd|DS)Ncss&|]}t|r |n|VqdS)N)r!r"detach).0r)rrr as$zWait.forward..)rrrr%)rrrr&rrrr+Ys z Wait.forwardr2.cGs"|j}|j}t||d}||S)Nr.)rrr)rr2rrr4rrrr5cs  z Wait.backwardNr6rrrrrQs (r)r7 collectionsrtypingrrrrrr!rstreamr r r r rrrstrrr9rautogradFunctionrrrrrrs   3