o ;ήc[@sPdZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZmZddlZddlZddlZddlmZddlmZmZmZddlmZmZmZmZmZmZm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&ej'(ej')ej'*e+d d dd l,m-Z-dd l.m/Z0ej'(ej')ej'*e+d dddlm1Z1ddl2m3Z4ddl5m6Z6m7Z7ddl8m9Z9e:dZ;GdddeZfddZ?dej>fddZ@dej>fddZAdTd e=d!eBfd"d#ZCd$e=d%eBde$fd&d'ZDd(ejd)efd*d+ZEd(ejd)efd,d-ZFd(ejd)efd.d/ZG 0 1dUd2ed3ed4e=d5eHfd6d7ZId8ed9efd:d;ZJ 1dVd(ed5eHde efdd>edAeee fdBejNdCejNdDeHdEeHdFe e eHde e=e ffdGdHZOdWdej>dJee e=dKeBfdLdMZPdSdej>dJee e=fdNdOZQdXdee e=dJee e=fdPdQZ/eRdRkre/dSdS)Ya This converts GPT2 or T5 model to onnx with beam search operator. Example 1: convert gpt2 model with beam search: python convert_generation.py -m gpt2 --output gpt2_beam_search.onnx Example 2: convert T5 model with beam search in two steps: cd ./models/t5 python convert_to_onnx.py -m t5-small cd ../.. python convert_generation.py -m t5-small --model_type t5 --decoder_onnx ./models/t5/onnx_models/t5-small_decoder.onnx --encoder_decoder_init_onnx ./models/t5/onnx_models/t5-small_encoder_decoder_init.onnx --output ./models/t5/onnx_models/t5_small_beam_search.onnx Example 3: convert T5 model with beam search. All in one step: python convert_generation.py -m t5-small --model_type t5 --output ./models/t5/onnx_models/t5_small_beam_search.onnx Example 4: convert MT5 model with external data file like mt5-base-beamsearch.onnx.data in below example. python convert_generation.py -m google/mt5-base --model_type mt5 --output mt5-base-beamsearch.onnx -e Example 5: convert gpt2 model with greedy search: python convert_generation.py -m gpt2 --output gpt2_greedy_search.onnx --num_beams 1 --num_return_sequences 1 N)Enum)Path)AnyDictListOptionalUnion) Precision) GraphProto ModelProto TensorProto) GPT2ConfigGPT2LMHeadModel GPT2Tokenizer MT5ConfigMT5ForConditionalGenerationT5ConfigT5ForConditionalGeneration T5Tokenizer)GraphOptimizationLevelInferenceSessionSessionOptionsget_available_providersmodelsgpt2)PRETRAINED_GPT2_MODELS)maint5) setup_logger)export_onnx_models)PRETRAINED_MT5_MODELSPRETRAINED_T5_MODELS) OnnxModelc@seZdZdZdZddZdS)GenerationType beam_search greedy_searchcCs|jSN)value)selfr*R/tmp/pip-target-vg8gfxp4/lib/python/onnxruntime/transformers/convert_generation.py__str__JszGenerationType.__str__N)__name__ __module__ __qualname__ BEAMSEARCH GREEDYSEARCHr,r*r*r*r+r$Fs r$argvreturnc Cslt}|d}|jdddtddtttd|jdd td gd d dgd d |jdd tt j dddd|jdd tddd|jdd tddd|jdd ddd|j d d|d}|jddtdd|jd d!d t t j t j t jgd"d |jd#d$d dd%d|j d d&|jd'd(d dd)d|j d d*|jd+d,d dd-d|j d d.|d/}|jd0d dd1d|j d d2|jd3d dd4d|j d d5|jd6d dd7|j d d8|jd9td d:d;d<|jd=d dd>d|j d d?|jd@d ddAd|j d dB|jdCd ddDd|j d dE|dF}|jdGtd dHdId<|jdJtd dKdLd<|jdMtd dNdOd<|jdPtd dHdQd<|jdRtd dHdSd<|jdTtd dHdUd<|jdVtd dWdXd<|dY}|jdZd dd[d|j d d\|jd]d dd^d|j d d_|jd`d ddad|j d db|jdcd tdHddd|jded ddfd|j d dg||}|S)hzParse arguments Args: argv (Optional[List[str]], optional): _description_. Defaults to None. Returns: argparse.Namespace: Parsed arguments. z Input optionsz-m--model_name_or_pathTzEPytorch model checkpoint path, or pretrained model name in the list: , )requiredtypehelpz --model_typeFr)rrmt5z*Model type (default is gpt2) in the list: )r6r7defaultchoicesr8z --cache_dir. cache_modelsz%Directory to cache pre-trained models)r6r7r:r8z--decoder_onnxr#zLPath of onnx model for decoder. Specify it when you have exported the model.z--encoder_decoder_init_onnxzgPath of ONNX model for encoder and decoder initialization. Specify it when you have exported the model.z --verbose store_truezPrint more information)r6actionr8)verbosezOutput options--outputz,Output path for onnx model with beam search.z-p --precisionzTPrecision of model to run. fp32 for full precision, fp16 for half or mixed precisionz-e--use_external_data_formatz!save external data for model > 2G)use_external_data_formatz-sz--run_shape_inferencezrun shape inference)run_shape_inferencez-iz--disable_shared_initializerszado not share initializers in encoder and decoder. It will increase memory usage of t5/mt5 models.)disable_shared_initializersz6Beam search parameters that stored in the output modelz--output_sequences_scoreszoutput sequences scores)output_sequences_scoresz--output_token_scoreszoutput token scores)output_token_scoresz--early_stopping)r6r?)early_stoppingz--no_repeat_ngram_sizerzNo repeat ngram size)r7r6r:r8z --vocab_maskz\Enable vocab_mask. This mask applies only to every generated token to filter some bad words.) vocab_maskz--prefix_vocab_maskzeEnable prefix_vocab_mask. This mask can be used to filter bad words in the first generated token only)prefix_vocab_maskz--custom_attention_maskz]Enable custom_attention_mask. This mask can be used to replace default encoder attention mask)custom_attention_maskzYBeam search parameters not stored in the output model, for testing parity and performancez --min_lengthzMin sequence lengthz --max_length2zMax sequence lengthz --num_beamsz Beam sizez--num_return_sequencesz&Number of return sequence <= num_beamsz--length_penaltyz1 to penalize and <1 to encourage short sentence.z--repetition_penaltyz-Positive. >1 to penalize and <1 to encourage.z --vocab_sizezIVocab_size of the underlying model used to decide the shape of vocab maskz0Other options for testing parity and performance --use_gpuz)use GPU for inference. Required for fp16.)use_gpuz--disable_parityzdo not run parity test)disable_parityz--torch_performanceztest PyTorch performance)torch_performancez --total_runsz4Number of times of inference for latency measurementz--save_test_dataz.save test data for onnxruntimer_perf_test tool)save_test_data)argparseArgumentParseradd_argument_group add_argumentstrjoinrr!r ospath set_defaultsr FLOAT32FLOAT16intfloat parse_args)r2parser input_group output_group model_groupbeam_parameters_group test_groupargsr*r*r+parse_argumentsNs                         rkrjc Cs|j}d|d|jdd|jtjkrdndddd d d d g }|jr#|d |jr+|d|jtjkr?|js8Jd| gd|j rJt d|t |ddS)zqConvert GPT-2 model to onnx Args: args (argparse.Namespace): arguments parsed from command line r4rAz--optimize_onnxrBfp32fp16z --test_runs1z --test_cases10z--use_int32_inputsz --overwriterQrCzEfp16 or mixed precision model cannot run in CPU. Please add --use_gpu)z--op_block_listAddLayerNormalizationFastGeluzarguments for convert_to_onnx:)r2N)model_name_or_path decoder_onnx precisionr r_rRappendrDr`extendr@loggerinfoconvert_gpt2_to_onnx)rj model_name argumentsr*r*r+ gpt2_to_onnxBs2   r}cCsxt|j|jt|jj|j|jd|jdddddd|j d}t d|dt d|d|d|_ |d|_ dS) znConvert T5 model to onnx Args: args (argparse.Namespace): arguments parsed from command line FT) rRrD optimize_onnxrur@use_decoder_start_tokenmerge_encoder_and_decoder_init overwritedisable_auto_mixed_precisionuse_int32_inputs model_typezonnx model for encoder: rzonnx model for decoder: rMN)export_t5_onnx_modelsrs cache_dirroutputparentrRrDrurrxdebugencoder_decoder_init_onnxrt)rjpathsr*r*r+ t5_to_onnxks(  rT onnx_pathrDcCsPddlm}tj|dd}|j|ddd}|r!tj|||dd Stdd S) zShape inference on an onnx file, which will be overwritten. Args: onnx_path (str): Path of onnx model use_external_data_format(bool): output tensors to external data or not. r)SymbolicShapeInferenceTload_external_dataF) auto_mergeguess_output_rank)save_as_external_dataz4Failed to run symbolic shape inference on the model.N) &onnxruntime.tools.symbolic_shape_inferronnx load_model infer_shapesr"saverxwarning)rrDrmodeloutr*r*r+shape_inferences r model_pathrRcCsRt}tj|_|r ddgndg}|r dtvrtdtdt|||d}|S)a,Create OnnxRuntime session. Args: model_path (str): onnx model path use_gpu (bool): use GPU or not Raises: RuntimeError: CUDAExecutionProvider is not available when --use_gpu is specified. Returns: onnxruntime.InferenceSession: The created session. CUDAExecutionProviderCPUExecutionProviderz5CUDAExecutionProvider is not available for --use_gpu!zuse CUDAExecutionProvider) providers) rrORT_DISABLE_ALLgraph_optimization_levelr RuntimeErrorrxryr)rrR sess_optionsexecution_providers ort_sessionr*r*r+create_ort_sessions   rgraphruc Cstj|k}t|j}|d}|dksJgdddt|D}t|jt|kr9tdt|dt|jt|D]E\}}|j|j|krZtd|d |d|j|jtj }|dkri|rftjntj }|j|j j j } | |krtd|d |d| q=td d gd dt|D} t|jt| krtdt| dt|jt| D]>\}} |j|j| krtd|d | d|j|j|rtjntj }|j|j j j } | |krtd|d |d| qtddS)aVerify GPT-2 subgraph Args: graph (onnx.GraphProto): onnx graph of GPT-2 precision (Precision): Precision (FLOAT16 or FLOAT32) of the model. Raises: ValueError: Number of inputs not expected. ValueError: Input name is not expected. ValueError: Input data type is not expected. ValueError: Number of outputs not expected. ValueError: Output name is not expected. ValueError: Output data type is not expected. rM) input_ids position_idsattention_maskcSg|]}d|qS)past_r*.0ir*r*r+ z(verify_gpt2_subgraph.. Number of inputs expected to be . Got Input  is expected to be $ is expected to have onnx data type z:Verifying GPT-2 graph inputs: name and data type are good.logitscSr)present_r*rr*r*r+rr!Number of outputs expected to be Output z;Verifying GPT-2 graph outputs: name and data type are good.N)r r`leninputrange ValueError enumeratenamer INT32FLOATr7 tensor_type elem_typerxryr) rru is_float16 input_count layer_countexpected_inputsrexpected_input expected_type input_typeexpected_outputsexpected_output output_typer*r*r+verify_gpt2_subgraphs>   " " rc Cs8tj|k}|r tjntj}t|j}|dd}|dksJgd}t|D]}|d||d|q&t|D]}|d||d|q=t|jt|krhtd t|d t|jt |D]?\}}|j|j |krtd |d |d |j|j |d krtj n|} |j|j j j} | | krtd |d| d | qldg} t|D]}| d|| d|qt|jt| krtdt| d t|jt | D]7\}} |j|j | krtd|d | d |j|j |j|j j j} | |krtd|d|d | qdS)Verify T5 decoder subgraph Args: graph (onnx.GraphProto): onnx graph of T5 decoder precision (Precision): Precision (FLOAT16 or FLOAT32) of the model. Raises: ValueError: Number of inputs not expected. ValueError: Input name is not expected. ValueError: Input data type is not expected. ValueError: Number of outputs not expected. ValueError: Output name is not expected. ValueError: Output data type is not expected. rrOrM)rencoder_attention_maskencoder_hidden_statespast_key_self_past_value_self_past_key_cross_past_value_cross_rrrrrrpresent_key_self_present_value_self_rrN)r r`r rrrrrvrrrrr7rrr)rrur float_typerrrrrrrrrrr*r*r+verify_t5_decoder_subgraphsH      " " rc Cstj|k}t|jdd}|dksJgd}t|jt|kr0tdt|dt|jt|D]9\}}|j|j|krQtd|d|d|j|jtj }|j|j j j }||krmtd|d |d|q4d d g} t |D]}| d || d |qvt |D]}| d|| d|qt|jt| krtdt| dt|jt| D]>\}} |j|j| krtd|d| d|j|j|rtjntj}|j|j j j } | |krtd|d |d| qtddS)rrrOrM)encoder_input_idsrdecoder_input_idsrrrrrrrrrpresent_key_cross_present_value_cross_rrzMT5 encoder graph verified: name and data type of inputs and outputs are good.N)r r`rrrrrrr rr7rrrrvrrxry) rrurrrrrrrrrrr*r*r+'verify_t5_encoder_decoder_init_subgraph9s@  "  "rshared_graph1graph2 shared_prefix min_elementsc Csi}i}g}g}g}|jD]L} | jrt| j|ksq |jD];} | jr)t| j|ks*qt| | rX|| j|| j<|| | j|vrV|| j} | || j<|| || nqq td||j D]} t t | j D]} | j | |vrt d| j | qnqe|j D]} t t | j D]} | j | |vrt d| j | qq|D]}|j|q|jD] }|j|vr||j|_q|j D]4} t t | j D]*} | j | |vr|| j | }td| jd| d| j | d||| j | <qq|D]}|j|q|jD]}|j|vr||j|_q|j D]7} t t | j D],} | j | |vrM|| j | }td| jd| d| j | d||| j | <q"q|D] }||j|_qS|D] }tj|j}tj|j|j|}|j||j|q_|S) aRemove initializers with same value from two graphs. Args: graph1 (GraphProto): the first graph to process graph2 (GraphProto): the second graph to process shared_prefix (str): add prefix to the shared initializers among two graphs min_elements (int, optional): minimal number of elements for initializers to be considered. Defaults to 1024. zshared initializers:zname is found in graph 1: zname is found in graph 2: zgraph 2 rename node z input z from z to zgraph 1 rename node ) initializerdimssumr"has_same_valuerrvrxrnoderrrrremove value_infor numpy_helperto_arrayshapehelpermake_tensor_value_info data_type)rrrrmapping_initializers_1mapping_initializers_2shared_initializers_1shared_initializers_2shared_initializers_names initializer1 initializer2 shared_namerjrrnew_namerr*r*r+remove_shared_initializers}s               *     *  r encoder_model decoder_modelcCsLt|}t|}|d|d||t|jj|jjd}|S)Ne_d_s_)r"add_prefix_to_namesremove_duplicated_initializerrrr)rrencoderdecoder initializersr*r*r+get_shared_initializerss  r cCsg}|jD]}|jrt|j|ksq||q|D]}|j|q|D]}tj|j}tj |j |j |}|j |q%|S)a^Remove initializers of a graph, when they have number of elements larger than a threshold. Args: graph (GraphProto): the graph. min_elements (int, optional): minimal number of elements for initializers to be considered. Defaults to 1024. Returns: List[TensorProto]: initializers that are removed from the graph. )rrrrvrrrrrrrrrr)rrmoved_initializerstensorrrrr*r*r+move_initializerss  rgeneration_typec" Cs<|jdk}|tjk}|r |std|jrtd|jr td|re|jr6tj |jr6t d|jnQ|jsRd |j tjkrCdnd}tt|jj||_t d |jd |jd t|n"|jry|jryt d |jd |jnt d|jdt||jrt d|jdt|j|j|rtj|j|jd}n|jdkrtj|j|jd}n tj|j|jd}|j rt d||j!}|r|j!n|j"}|j#}|j#dkr|j#}t$j%|jdd} |jd| j&_'|jdkrt(| j&|j nt)| j&|j |s gdngd} |j*r| +dn| +d|j,r(| +dn| +d|j-r7| +dn|s?| +ddg} |jrK| +d |jr\|jsWJd!| +d"|smt$j.j/d#| | d$|jd%n t$j.j/d&| | d'|jd%} d(| _0|st$j.1d)|t$j.1d*|t$j.1d+|j2t$j.1d,|j3rd-nd.t$j.1d/|jdkrd.nd-gn"t$j.1d)|t$j.1d*|t$j.1d/|jdkrd.nd-t$j.1d+|j2g} | j45| g}|jd0vrO|jrt d1|jdt|j|jt$j%|jdd}|jd2|j&_'t6|j&|j |j7s(t8|| }t t9|d3d4d5|Dd6| j45t$j.1d7|j&t$j.1d8| j&t$j.1d9t9|j&j:d:krI|j;ndgnt<| j&}t t9|d;| j4+t$j.1d8| j&t$j.=dj?d=d>g}t$j.=d?t>j?d-g}t$j.=d@t>j?d-g}t$j.=dAt>j?d-g}t$j.=dBt>j?d-g}t$j.=dCt>j@d-g}t$j.=dDt>j@d-g}|s|||||||gn||||g}|j*rt$j.=dt>j?|g}|+||j,rt$j.=dt>j?d=|g}|+||j-rt$j.=dt>j?d=d>g}|+||s t$j.=dt>j?gdEn t$j.=dt>j?d=d?g}t$j.=d t>j@d=dBg}t$j.=d"t>j@dFd=dA|g}|g}|jr;|+||jrD|+|t$j.A| g|sR|jdGn|jdH|||}t$j.jB|dI| jCdJ} |jrd.dKlDmE}!|!Ft$jG|!FdLkrt HdMtIjJ| |jdddNnt$J| |jt dO|jdPS)QzConvert model according to command line arguments. Args: args (argparse.Namespace): arguments parsed from command line rz3Currently only gpt2 with greedy search is supportedzCoutput_sequences_scores currently is not supported in greedy searchz?output_token_scores currently is not supported in greedy searchz)skip convert_to_onnx since path existed: zgpt2_past_{}.onnxrmrlzConvert GPT model z to onnx z ...z,skip convert_to_onnx since paths specified: z and zConvert model z to onnx ...z Run symbolic shape inference on z. The file will be overwritten.rrzConfig=rPTrz decoderr max_length min_length num_beamsnum_return_sequenceslength_penaltyrepetition_penaltyrrrrrJr#rKr sequencessequences_scoresz8--output_token_scores requires --output_sequences_scoresscores BeamSearch BeamSearch_)inputsoutputsr GreedySearch GreedySearch_z com.microsoft eos_token_id pad_token_idno_repeat_ngram_sizerIrMrrrr9zSymbolic shape inference on z encoder and decoder initz shared initializers (cSsg|]}|jqSr*)rrr*r*r+rz,convert_generation_model..z*) in subgraphs are moved to the main graphrr decoder_start_token_idrz: initializers from the decoder are moved to the main graphr batch_sizesequence_lengthrrrrrr)r(rrzmax_length - sequence_lengthz beam searchz greedy searchzonnxruntime.transformers) producer_name opset_imports)versionz1.12.0z0Require onnx >= 1.12 to save large (>2GB) model!)rall_tensors_to_one_filezmodel save to N)Krr$r1NotImplementedErrorrGrHrtr\r]existsrxryformatrur r`rrras_posixrsr}rrrErrDr from_pretrainedrrrr@r"r# vocab_sizerrrrrrrJrvrKrLr make_nodedomainmake_attributer$rI attributerwrrFr rrr'rrr rr make_graph make_model opset_import packagingr,parse __version__rr"r)"rjris_gpt2is_greedysearch onnx_filenameconfigr"r#r3rrrrattr_to_extendr rrrrrrrr graph_inputsrJrKrrrr graph_outputs new_graph new_modelr,r*r*r+convert_generation_model s                                            rGrrrr"r# bad_words_idsc Cs|jr tjs td|jtjkr|t |jrdnd}| |t d| |}| |}g}t |j D]/} t} |j|||j|j|j|j|j|||j|j|j|d|jp^|jd} |t| q;|jd} ddlm} | || S) aTest PyTorch performance of text generation. Args: args (argparse.Namespace): arguments parsed from command line model (Union[GPT2LMHeadModel, T5ForConditionalGeneration]): PyTorch model input_ids (torch.Tensor): input_ids attention_mask (torch.Tensor): Attention mask eos_token_id (int): EOS token ID pad_token_id (int): Padding token ID bad_words_ids (List[List[int]]): Words shall not be generated. Raises: RuntimeError: PyTorch with CUDA is not available for --use_gpu Returns: Dict[str, Any]: A dictionary with string with metric name, and value can be integer or string. z=Please install PyTorch with Cuda for testing gpu performance.zcuda:0cpuFTrrrrrrIr$r"r#rrrrHreturn_dict_in_generate output_scoresrget_latency_result)rRtorchcuda is_availablerrur r`halfdevicetoset_grad_enabledr total_runstimegeneraterrrrIr$rrrrGrHrvrbenchmark_helperrN) rjrrrr"r#rHrS torch_latency_startr(rNr*r*r+test_torch_performance4sB         r]F sentences is_greedyc+Cs|jdksJtj|j|jd}d|_|j|_tj|j|j|j d}|dur*gd}||ddd }|d }|d }d }|j |dd } dd| D} |j rSt d| ng} |j} | j } | j } | j} g}d}|jstdtd|j|||j|j|j|j|j| | |j|j|j| r| ndd|jp|jd}td |tdtd|j|jrtd|j|jrtd|jt |jD]\}}|j!|dd}|"|t|d|qtdtdt#|j$|j%}|r|&'(t)j*t)j+|jgt)j*dt)j+|jgt)j*dt)j+|jgt)j,dd}nB|&'(t)j*t)j+|jgt)j*dt)j+|jgt)j*dt)j+|jgt)j*dt)j+|jgt)j*dt)j+|jgt)j,dt)j+|jgt)j,dd}|j rnt)j-| t)j*d}|j rj| D]}d||<qb||d<|j.d}|j/rt 0d t)j-|| ft)j*d}||d!<t d"||1d|}|j2rt3|j$j45}t d#|dd$l6m7}|g}t |D]\}}t8j9:|d%t;|}|||qg}t<|j=D]}t>>}|1d|}|"t>>|qdd&l?m@} |j.d}| ||}!td'|d}"td|"|jrtd|d(|jrtd|d)|rG|"j.\}}#g}$t<|D]}|j!|"|dd}|$"|td*|d+|q*n5|"j.\}}%}#g}$t<|D](}t<|%D] }&|j!|"||&dd}|$"|td*|d,|&d|qYqS|r|jA||jd-}'tBC|"}(tdtd.t|'t|tdtd/t|(t|$td||$k})td0|)rd1nd2|)|!d3<|jDrtE||||| | | }*td4|*td5|!|!S)6a9Test GPT-2 model Args: args (argparse.Namespace): arguments parsed from command line sentences (Optional[List[str]], optional): input text. Defaults to None. Returns: Union[Dict[str, Any], None]: A dictionary with string with metric name, and value can be integer or string. rrleft)rr#N)zThe product is releasedzI enjoy walking in the parkzTest best way to investptTreturn_tensorspaddingrr walk in park)add_prefix_spacecSg|]}|gqSr*r*rword_idr*r*r+rr&z"test_gpt_model..rH2--------------------------------------------------CTest PyTorch model and beam search with huggingface transformers...rJ!huggingface transformers outputs:rrrskip_special_tokens: 'Testing beam search with onnxruntime...dtyperrrrJzYUse prefix vocab mask with all ones in ORT, but no corresponding setting for Torch model.rK ORT inputs test_data_diroutput_test_datatest_data_set_rM ORT outputs:rMrbatch z sequence: sequence rPTorch Sequences:ORT Sequences:Torch and ORT result is same differentparity Torch LatencyORT)Frrr2rsr padding_side eos_token pad_tokenrr"encoderJrxrrAr3rSprintrXrrrrIr$rrrrGrHrrrrdecodervrrrRrInumpyastypenpint32arrayfloat32onesrrKryrunrUrrr1bert_test_datarvr\r]r[rZrrVrWrYrNreshaperO LongTensorrTr])+rjr^r_ tokenizerrrrr bad_wordsrHrAr"r#r3torch_decoded_sequences beam_outputsrsequencedecoded_sequencerrJ bad_word_idr(rKresultrtrv all_inputsdirlatencyr[r\rNrrrort_decoded_sequences num_sequencesrtorch_sequences ort_sequencesis_sametorch_latency_outputr*r*r+test_gpt_modelvs4                              rc*Cs|jdvsJ|jrtddStj|j|jd}d|_|jdkr,t j|j|jd}n t j|j|jd}|dur=ddg}||d d d }|d }|d }d}| |dd}dd|D}|j rhtd|ng}|j } | j} | j} | j} td| d| d| g} |jstdtd|j|||j|j|j|j|j| | |j|j|j|r|ndd |jp|jd}td |tdtd|j|jrtd|j|jrtd|j t!|jD]\}}|j"|d d}| #|td$||qtdtdt%|j&|j'}t(j)| t(j*d }|j r|D]}d!||<q|+,-t(j*t(j.|jgt(j*d t(j.|jgt(j*d t(j.|jgt(j*d t(j.|jgt(j*d t(j.|jgt(j/d t(j.|jgt(j/d d"}|j rc||d#<|j0rt(j)|j1t(j*d }t2|j1d!D]*}d!}t2|j1d$D]}|||| kr|d!krd!|||<q|d$7}qqw||d <|j3rt4|j&j56}td%|d!d&l7m8}|g}t!|D]\}}t9j:;|d't<|}|||qtd(|g}t2|j=D]}t>>}|?d|}|#t>>|q|j1d!}d!d)l@mA} | ||}!td*|d!}"td|"|jr'td|d$|jr2td|d+|"j1\}}#}$g}%t2|D](}t2|#D] }|j"|"||d d}|%#|td,|d-|d.|qDq>|js|jB||jd}&tCD|"}'tdtd/t|&t| tdtd0t|'t|%td| |%k}(td1|(rd2nd3|(|!d4<|jErtF||||| | |})td5|)td6|!|!S)7a=Test T5 or MT5 model Args: args (argparse.Namespace): arguments parsed from command line sentences (Optional[List[str]], optional): input text. Defaults to None. Returns: Union[Dict[str, Any], None]: A dictionary with string with metric name, and value can be integer or string. r%zLSkipping parity test as prefix vocab mask is not implemented by Hugging FaceNrr`rz4translate English to French: The product is releasedzsummarize: research continues to show that pets bring real health benefits to their owners.Having a dog around can lead to lower levels of stress for both adults and kids.raTrbrrrerPcSrgr*r*rhr*r*r+rlr&z!test_t5_model..rHz eos_token_id:z, pad_token_id:z , vocab_size:rjrkrJrlrrrrmz{}: {}rprqrrrJrMrtrurwrsrMrxrryrzror{r|r}r~rrrr)GrrKrxrrr2rsrrrrrrJrAr"r#r3rSrrXrrrrIr$rrrrGrHrrrrrrvr0rrrRrrrrIrrrrrLrrrUrrr1rrvr\r]r[rZrVrWrrYrNrrOrrTr])*rjr^rrrrrrrHrAr"r#r3rrrrrrrJrabs_posrrtrvrrrr[r\rr(rNrrrrrrrrrr*r*r+ test_t5_model>s0                           rcCs<t|}t|j|jdvrC|jr tj|js td|j|j r2tj|j s2td|j |jr8|j r>|j rB|jsBtdn|j rJt d|j dkoS|j dk}|jdkrb|rbt|tjnt|td|jdvrwt||d }nt|||d }|r|jrtd |jd |jd |Std|j|S)a/Main entry function Args: argv (Optional[List[str]], optional): _description_. Defaults to None. sentences (Optional[List[str]], optional): input text. Defaults to None. Raises: ValueError: Path does not exist: --encoder_decoder_init_onnx ValueError: Path does not exist: --decoder_onnx ValueError: --decoder_onnx and --encoder_decoder_init_onnx are not used together for T5 Returns: Union[Dict[str, Any], None]: A dictionary with string with metric name, and value can be integer or string. r%z1Path does not exist: --encoder_decoder_init_onnx z$Path does not exist: --decoder_onnx zB--decoder_onnx shall use together with --encoder_decoder_init_onnxz>custom_attention_mask is only supported in t5 with beam searchrMrzstart testing model...)r^)r^r_zOutput files: r5z.dataz Output file: )rkrr@rrr\r]r/rrtrLr.rrrGr$r1rxryrrrDr)r2r^rjr_rr*r*r+rs<     r__main__r')T)rr)r)NF)NN)S__doc__rVloggingr\sysrWenumrpathlibrtypingrrrrrrrrrOrYr r r r transformersr rrrrrrr onnxruntimerrrrr]rvr[dirname__file__ gpt2_helperrmodels.gpt2.convert_to_onnxrrzrmodels.t5.convert_to_onnxrrmodels.t5.t5_helperr r! onnx_modelr" getLoggerrxr$rZ Namespacerkr}rboolrrrrrrarr rr0rGTensorr]rrr-r*r*r*r+s   (        u)8MG f  )    "BI$G 6