o h@snddlZddlZddlmZmZmZddlZgdZdZ ddZ ddZ dd Z d d Z d eejeejfd dfddZde ddfddZdde ddfddd eejdeeejeejfdededeeejjdeeejd dfddZ d d eejded dfddZ  d!d eejdeejd dfddZe ddfd eejdeejded dfddZdS)"N)OptionalSequenceUnion) all_reducereduce broadcast all_gatherreduce_scattercCstttjds tddSt}|D]%}|jrdS|s!dS|js'dS| }||vr2dS| |qdS)N_nccl_all_reducez)PyTorch is not compiled with NCCL supportFT) hasattrtorch_Cwarningswarnset is_sparse is_contiguousis_cuda get_deviceadd)tensorsdevicestensordevicerE/var/www/html/ai/venv/lib/python3.10/site-packages/torch/cuda/nccl.py is_available s   rcCs0tj}|d?}|d?d@}|d@}|||fS)N i)r r _nccl_version)vermajorminorpatchrrrversion"s   r$cCs tjSN)r r _nccl_unique_idrrrr unique_id*s r'cCstj|||Sr%)r r _nccl_init_rank) num_ranksuidrankrrr init_rank.sr,inputsreturncCs&t|tjjr t|tjrtddS)Nz(Inputs should be a collection of tensors) isinstance collectionsabc Containerr Tensor TypeError)r-rrr_check_sequence_type2s r5cCs4t||dur |}t|tj|||||dSr%)r5r r r r-outputsopstreamscommsrrrr9s r)r7outputrootr8r9r7cCst||dur|durtdtd||}n!t|tjs1t|tjj r1td||}n |dur9||n|}tj ||||||dS)Nz'output' and 'outputs' can not be both specified. 'outputs' is deprecated in favor of 'output', taking in a single output tensor. The signature of reduce is: reduce(inputs, output=None, root=0, op=SUM, streams=None, comms=None).znccl.reduce with an output tensor list is deprecated. Please specify a single output tensor with argument 'output' instead instead.z\nccl.reduce with an output tensor list is deprecated. Please specify a single output tensor.) r5 ValueErrorrrr/r r3r0r1rr _nccl_reduce)r-r;r<r8r9r:r7_outputrrrrCs&   rcCst|tj||||dSr%)r5r r _nccl_broadcast)r-r<r9r:rrrrjsrcCs&t|t|tj||||dSr%)r5r r _nccl_all_gather)r-r7r9r:rrrrqsrcCs(t|t|tj|||||dSr%)r5r r _nccl_reduce_scatterr6rrrr |sr )rNN)NN)r0rtypingrrr torch.cudar __all__SUMrr$r'r,r3r5rintcudaStreamrrrr rrrrs~"   (