o h._find_pt_tf_differencesN)r*)setkeys ValueError) pt_outputs tf_outputs pt_out_attrs tf_out_attrsr#rLr$find_pt_tf_differencess   z$PTtoTFCommand.find_pt_tf_differencesrrrrrr r!r"c GsRtd|_||_|r |ntjd||_||_||_ ||_ ||_ ||_ ||_ dS)Nztransformers-cli/pt_to_tfz/tmp)r get_logger_logger _model_nameospathjoin _local_dir _max_error _new_weights_no_pr_push_extra_commit_description_override_model_class) selfrrrrrr r!r"rr#r#r$__init__s  zPTtoTFCommand.__init__cCs*dd}t|j}|tvr$t|j}|tvr#|jjdur#|jj |j_n2|t vr/t |j}n'|t vr:t |j}n|tvrNt|j}|jdurM|j |_ntd|dtt|jj}i}d|vrr|ddgd d d d |vrtd d ddddd} |d| id|vrt|jj} d| vr| dj} | dur| dur| } nd } nd } ||| dd|vr||d d|d!i|ddi} |d!i|ddi}|jst|drt|dsd|vrtjdgdggtd|jjpd }| dt !|i|dt"#|i| |fS)"zQ Returns the right inputs for the model, based on its signature. cSs@tdddd}|dtdddd}dd |D}|S) Nz)hf-internal-testing/librispeech_asr_dummyclean validationsplitidraudiocSsg|]}|dqS)arrayr#).0xr#r#r$ szFPTtoTFCommand.get_inputs.._get_audio_input..)rsortselectrange)dsspeech_samples raw_samplesr#r#r$_get_audio_inputs z2PTtoTFCommand.get_inputs.._get_audio_inputNz1Unknown data processing type (model config type: r7 input_idsz Hi there!z@I am a batch with more than one row and different input lengths.T)textpadding truncation pixel_valuescifar10 plain_texttestrgrimgimagesinput_featuresrxF)rjrx input_valuesreturn_tensorspttfencoderdecoderdecoder_input_idsr)dtyperr#)$r(configr rfrom_pretrainedr\r tokenizer pad_token eos_tokenrr rr rrPrNinspect signatureforward parametersrOupdaterfeature_extractorr+is_encoder_decoderhasattrr=asarrayintdecoder_start_token_idr;tensorrconvert_to_tensor)rcpt_modeltf_dummy_inputsrrumodel_config_class processormodel_forward_signatureprocessor_inputs sample_imagesfeature_extractor_signaturedefault_strategypadding_strategypt_inputtf_inputrr#r#r$ get_inputssj       "zPTtoTFCommand.get_inputsc! sttjtdkrtdddlm}m}ddlm}|j j d}t j }|j }jdurjdrBjddg}njg}z ttd |d}Wntyatd jd wz ttd d|d}Wnjtytd jd w|durttd d }ttd d}jdnAt|dkrtd|djd|dttd |d}z ttd d|d}Wntytd|ddw||} | j} ~ | j } | | | |\} } t| dBi| ddi}Wdn 1swY~ |j j dd} | dBi| ddd}||}dd|D}dd|D}t|dkrW|durWtd|d|r`t| nd }t| }|j!kst|j!krtd!d"j!d#d$"fd%d&|Dd'j!d#d$"fd(d&|Dt#j$"j t%}t#j$"j t&}t#j$'|st#j$'|rĈj(r| )j ~ | j }|dBi| ddi}||}d)d|D}d*d|D}t|dkr|durtd|d|rt| nd }t| }|j!ks%|j!krStd+d"j!d#d$"fd,d&|Dd'j!d#d$"fd-d&|Dj(rYd.nd/}j*r||j+dd0|,||j-dd1jd2j dSj.sjd3d4|d5d6|d5d7|d5d8|d5d9 }j!t/kr|d:j!d;7}j0r|d<j07}t#j$'|r|t&|d=g}t1j2j34j d>D]}||t#j$5||d=g7}qn|t%|d=g}|j |||d?dd@j6} jdA| dSdS)CNz0.9.0zrThe huggingface_hub version must be >= 0.9.0 to use this command. Please update your huggingface_hub installation.r) Repository create_commit)CommitOperationAdd)r clone_fromTFr transformersz Model class z not found in transformers.zTF model class TF AutoModel TFAutoModelz5No detected architecture, using AutoModel/TFAutoModelrzAMore than one architecture was found, aborting. (architectures = r7zDetected architecture: zThe TensorFlow equivalent of z doesn't exist in transformers.output_hidden_statesT)from_ptF)rtrainingcSi|] \}}d|vr||qShiddenr#rlkvr#r#r$ Nz%PTtoTFCommand.run..cSi|] \}}d|vr||qSrr#rr#r#r$rOrz;Something went wrong -- the config file has architectures (zF), but no model head output was found. All outputs start with 'hidden'gzOThe cross-loaded TensorFlow model has different outputs, something went wrong! z9 List of maximum output differences above the threshold (z):  c*g|]\}}|jkr|d|dqSz: .3er]rrcr#r$rn[*z%PTtoTFCommand.run..z@ List of maximum hidden layer differences above the threshold (crrrrrr#r$rn]rcSrrr#rr#r#r$rkrcSrrr#rr#r#r$rlrzLThe converted TensorFlow model has different outputs, something went wrong! crrrrrr#r$rnxrcrrrrrr#r$rnzrzUpdate TF weightszAdd TF weights)auto_lfs_track)blockingzTF weights pushed into z&Uploading the weights into a new PR...aModel converted by the [`transformers`' `pt_to_tf` CLI](https://github.com/huggingface/transformers/blob/main/src/transformers/commands/pt_to_tf.py). All converted model outputs and hidden layers were validated against its PyTorch counterpart. Maximum crossload output difference=rz,; Maximum crossload hidden layer difference=z'; Maximum conversion output difference=z-; Maximum conversion hidden layer difference=z; zB CAUTION: The maximum admissible error was manually increased to !z ) path_in_repopath_or_fileobjz/tf_model-*.h5model)repo_id operationscommit_messagecommit_description repo_type create_prz PR open in r#)7rparsehuggingface_hub __version__ ImportErrorrrhuggingface_hub._commit_apirr\rXr r architecturesrb startswithgetattrrAttributeErrorrPrWwarninglen from_config dummy_inputsevalrr;no_gradrUitemsr>valuesr]r[rYrZrrexistsr^save_pretrainedr`git_add git_commitgit_pushr_r3rariogfileglobbasenamepr_url)!rcrrrreporrpt_classtf_classtf_from_pt_modelrrrrrQtf_from_pt_outputscrossload_differencesoutput_differenceshidden_differencesmax_crossload_output_diffmax_crossload_hidden_difftf_weights_pathtf_weights_index_pathtf_modelrRconversion_differencesmax_conversion_output_diffmax_conversion_hidden_diffrcommit_descritionr shard_path hub_pr_urlr#rr$run s                 $               zPTtoTFCommand.runN) __name__ __module__ __qualname__ staticmethodrr6rUr1r2boolrdrrr#r#r#r$rKs0> %  Fr))rrYargparserr importlibrrr@r= packagingrr*rrr r r r r rrrrrutilsrrrr tensorflowrr experimental enable_tensor_float_32_executionr;datasetsrr3r%rr#r#r#r$s(  8