o he(@sddlZddlZddlmZmZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZmZmZmZe r=ddlZd d Zd d Zd dZdS)N) SAGEMAKER_PARALLEL_EC2_INSTANCESTORCH_DYNAMO_MODES)ComputeEnvironmentSageMakerDistributedType)is_boto3_available)SageMakerConfig)DYNAMO_BACKENDS _ask_field _ask_options_convert_dynamo_backend_convert_mixed_precision#_convert_sagemaker_distributed_mode_convert_yes_no_to_boolcCstd}ddddiddgd}z+|j|tj|d d d ddgd d dgd}|j||dtj|d d dWdS|jjyPtd|dYdSw)Niamz 2012-10-17AllowServicezsagemaker.amazonaws.comzsts:AssumeRole)Effect PrincipalAction)Version Statement)indent)RoleNameAssumeRolePolicyDocument)z sagemaker:*zecr:GetDownloadUrlForLayerzecr:BatchGetImagezecr:BatchCheckLayerAvailabilityzecr:GetAuthorizationTokenzcloudwatch:PutMetricDatazcloudwatch:GetMetricDatazcloudwatch:GetMetricStatisticszcloudwatch:ListMetricszlogs:CreateLogGroupzlogs:CreateLogStreamzlogs:DescribeLogStreamszlogs:PutLogEventszlogs:GetLogEventszs3:CreateBucketz s3:ListBucketzs3:GetBucketLocationz s3:GetObjectz s3:PutObject*)rrResource_policy_permission)r PolicyNamePolicyDocumentzrole z# already exists. Using existing one) boto3client create_rolejsondumpsput_role_policy exceptionsEntityAlreadyExistsExceptionprint) role_name iam_clientsagemaker_trust_policypolicy_documentr/Z/var/www/html/ai/venv/lib/python3.10/site-packages/accelerate/commands/config/sagemaker.py_create_iam_role_for_sagemaker&s0   r1cCstd}|j|dddS)Nr)rRoleArn)r"r#get_role)r+r,r/r/r0_get_iam_role_arn\s r5cCstdddgt}d}|dkrtddd}|tjd<ntd td }|tjd <td }|tjd <tddd}|tjd<tdddgt}|dkrLtd}nd}td|dt|tdtddd}d}|rmtddd}tdtddd} d} | rtd d!d} td"tddd} d} | rtd#d$d} td%d&d'gt} i}td(tddd}|rd)}td*d+d,t Dt d-d||d.<td/tddd}|rtd0t d1ddd||d2<td3tddd||d4<td5tddd||d6<d7}| t j krt|td8d}n |d97}t|d:dd;d}d}| t j krtdtd=d}td?gd@t}|r=|dAkr=tdBt|tj| d|||||||| | |dCS)DNzHow do you want to authorize?z AWS Profilez7Credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) rz'Enter your AWS Profile name: [default] default)r6 AWS_PROFILEzNote you will need to provide AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY when you launch you training script with,`accelerate launch --aws_access_key_id XXX --aws_secret_access_key YYY`zAWS Access Key ID: AWS_ACCESS_KEY_IDzAWS Secret Access Key: AWS_SECRET_ACCESS_KEYz"Enter your AWS Region: [us-east-1]z us-east-1AWS_DEFAULT_REGIONzMDo you already have an IAM Role for executing Amazon SageMaker Training Jobs?zProvide IAM Role namez%Create new IAM role using credentialszEnter your IAM role name: #accelerate_sagemaker_execution_rolez$Accelerate will create an iam role "z " using the provided credentialsz2Do you want to use custom Docker image? [yes/NO]: FzPlease enter yes or no.)r6 error_messagezEnter your Docker image: cS t|SNstrlowerxr/r/r0 z%get_sagemaker_input..zODo you want to provide SageMaker input channels with data locations? [yes/NO]: z\Enter the path to the SageMaker inputs TSV file with columns (channel_name, data_location): cSr=r>r?rBr/r/r0rDrEz3Do you want to enable SageMaker metrics? [yes/NO]: z[Enter the path to the SageMaker metrics TSV file with columns (metric_name, metric_regex): cSr=r>r?rBr/r/r0rDrEzWhat is the distributed mode?zNo distributed trainingzData parallelismz?Do you wish to optimize your script with torch dynamo?[yes/NO]:dynamo_z+Which dynamo backend would you like to use?cSsg|]}|qSr/)rA).0rCr/r/r0 sz'get_sagemaker_input..rbackendzGDo you want to customize the defaults sent to torch.compile? [yes/NO]: zWhich mode do you want to use?cS tt|Sr>)rintrBr/r/r0rDrEmodez\Do you want the fullgraph mode or it is ok to break model into several subgraphs? [yes/NO]: use_fullgraphz7Do you want to enable dynamic shape tracing? [yes/NO]: use_dynamicz:Which EC2 instance type you want to use for your training?cSrJr>)rrKrBr/r/r0rDrEz? [ml.p3.2xlarge]:cSr=r>r?rBr/r/r0rDrEz ml.p3.2xlargezShould distributed operations be checked while running for errors? This can avoid timeout issues but will be slower. [yes/NO]: rz(How many machines do you want use? [1]: z2Do you wish to use FP16 or BF16 (mixed precision)?)nofp16bf16fp8rOzTorch dynamo used without mixed precision requires TF32 to be efficient. Accelerate will enable it by default when launching your scripts.) image_uricompute_environmentdistributed_typeuse_cpu dynamo_configec2_instance_typeprofileregion iam_role_namemixed_precision num_machinessagemaker_inputs_filesagemaker_metrics_filedebug)r rKr osenvironr*r1rrr r rrNOr DATA_PARALLELMODEL_PARALLELrr rAMAZON_SAGEMAKER)credentials_configuration aws_profileaws_access_key_idaws_secret_access_key aws_regionrole_managementr[is_custom_docker_image docker_imageis_sagemaker_inputs_enabledr^is_sagemaker_metrics_enabledr_rUrW use_dynamoprefixuse_custom_optionsec2_instance_queryrXr`r]r\r/r/r0get_sagemaker_inputas2           ru)r%rautils.constantsrrutils.dataclassesrr utils.importsr config_argsr config_utilsr r r r rrrr"r1r5rur/r/r/r0s  $ 6