o h @sddlZddlZddlmZddlmZddlmZee Z dZ dZ dZ dZ  dd edededededef ddZddZddZdejfddZdS)N)closing) get_loggerzAddress already in usezSocket Timeoutz_tcp_store/num_membersz_tcp_store/last_memberXT is_server server_addr server_port world_sizetimeoutwait_for_workersc Cs|dkr|dkrtd|d||dkrtd||dkr"|nd} |dkr,|}nt}td|||||ztj||||tj|d|d } |rPt| |td | WSt y} z*t | t kr||krvt d ||||d7}n t d |d |d| WYd} ~ nd} ~ wwq%)NrrzCserver_port must be specified when world_size > 1, got server_port=z , world_size=z+sever_port: %s, specified, ignoring retriesTzVCreating c10d store on %s:%s world_size : %s is_server : %s timeout(sec): %s )seconds) host_nameportr is_masterr r zSuccessfully created c10d storez)port: %s already in use, attempt: [%s/%s]zon z, port: z already in use) ValueErrorloginfo get_free_portdistTCPStoredatetime timedelta_check_full_rank RuntimeErrorstr_ADDRESS_IN_USEwarning) rr r r r r retriesattemptrstoreer#a/var/www/html/ai/venv/lib/python3.10/site-packages/torch/distributed/elastic/utils/distributed.pycreate_c10d_storesZ         r%c Csj|td}||kr|tdz|tWdSty4}zt|tkr/td|d|d}~ww)Nrz ztimed out waiting for all z members to join) add_MEMBER_CHECKINset_LAST_MEMBER_CHECKINgetrr_SOCKET_TIMEOUT TimeoutError)r!r idxr"r#r#r$r\s    rcCs>t}t||dWdS1swYdS)Nr)get_socket_with_portr getsockname)sockr#r#r$rls  $rreturnc Cstjddtjtjd}|D]<}|\}}}}}t|||}z|d|d|WStyI}z|tj d|dWYd}~q d}~wwt d) a Returns a free port on localhost that is "reserved" by binding a temporary socket on it. Close the socket before passing the port to the entity that requires it. Usage example :: sock = _get_socket_with_port() with closing(sock): port = sock.getsockname()[1] sock.close() # there is still a race-condition that some other process # may grab this port before func() runs func(port) localhostN)hostrfamilytype)r2rrzSocket creation attempt failed.)exc_infozFailed to create a socket) socket getaddrinfo AF_UNSPEC SOCK_STREAMbindlistenOSErrorcloserrr)addrsaddrr4r5proto_sr"r#r#r$r.rs     r.)rrrTr)rr7 contextlibrtorch.distributed distributedr'torch.distributed.elastic.utils.loggingr__name__rrr+r'r)boolrintfloatr%rrr.r#r#r#r$s>    C