o ;ήc)A@sddlZddlZddlZddlmZmZddlZddlmZddl m Z m Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZeeZe d d fed d fed dfeddfed d fed dfed d fdZdddgfdededeedeedef ddZ  d.de dedededeef dd Z     d/d!ededededeedeeded"efd#d$Z!dedeeeffd%d&Z"d'd(Z#d)d*Z$d+d,Z%ed-kre%dSdS)0N)DictOptional) FusionOptions) ModelProto load_model) BartOnnxModel) BertOnnxModel)BertOnnxModelKeras)BertOnnxModelTF) Gpt2OnnxModel) TnlrOnnxModelpytorchtf2onnx keras2onnx)bartbertbert_tf bert_kerasgpt2gpt2_tftnlrFconnx_model_pathuse_gpuoptimized_model_path opt_levelreturnc Cs&|dvsJddl}|rd|vrtd|S|}|dkr'|jj|_n|dkr1|jj|_n|jj |_|durK|dd}d |||rHd nd }||_ i}|rV||d <|sf|j ||fd dgi|} n|j ||fd dgi|} d| vs{Jtj|rtj|sJtd ||S)a Use onnxruntime to optimize model. Args: onnx_model_path (str): the path of input onnx model. use_gpu (bool): whether the optimized model is targeted to run in GPU. optimized_model_path (str or None): the path of optimized model. opt_level (int): graph optimization level. disabled_optimizers (List[str]): a list of names of disabled optimizers Returns: optimized_model_path (str): the path of optimized model )rrrNCUDAExecutionProviderz3There is no gpu for onnxruntime to do optimization.rrz{}_o{}_{}.onnxgpucpudisabled_optimizers providersCPUExecutionProviderz)Save optimized model by onnxruntime to {}) onnxruntimeget_available_providersloggererrorSessionOptionsGraphOptimizationLevelORT_ENABLE_BASICgraph_optimization_levelORT_ENABLE_EXTENDEDORT_ENABLE_ALLformatoptimized_model_filepathInferenceSession get_providersospathexistsisfiledebug) rrrrr#r& sess_options path_prefixkwargssessionr=I/tmp/pip-target-vg8gfxp4/lib/python/onnxruntime/transformers/optimizer.pyoptimize_by_onnxruntime5sJ      r?rmodel model_type num_heads hidden_sizeoptimization_optionsc Cs|dkr|dks |dkrtdt|\}}}|jr-||jkr-td|d|jd|dur5t|}||||}|||d|j_dd lm } | |j_ |S) adOptimize Model by graph fusion logic. Note that ONNXRuntime graph optimizations (like constant folding) will not be applied. So it is better to enable constant folding during exporting ONNX model, or run optimize_by_onnxruntime on the model first like optimize_model. For BERT model, num_heads and hidden_size are optional. For other model types, you need specify these parameters. Args: model (ModelProto): model object model_type (str, optional): model type - like bert, bert_tf, bert_keras or gpt2. Defaults to 'bert'. num_heads (int, optional): number of attention heads. Defaults to 0. 0 allows detect the parameter from graph automatically (for model_type "bert" only). hidden_size (int, optional): hidden size. Defaults to 0. 0 allows detect the parameter from graph automatically (for model_type "bert" only). optimization_options (FusionOptions, optional): optimization options that turn on/off some fusions. Defaults to None. Returns: object of an optimizer class. rrTPlease specify parameters of num_heads and hidden_size when model_type is not 'bert'z&Model producer not matched: Expected "z", Got "z0".Please specify correct --model_type parameter.Nzonnxruntime.transformers) __version__) r(warning MODEL_TYPES producer_nameroptimizetopological_sortr@r&rFproducer_version) r@rArBrCrDoptimizer_classproducer_ optimizeronnxruntime_versionr=r=r>optimize_by_fusionps     rRinputonly_onnxruntimecCs|dus |dvs J|dkr|dks|dkrtdt|\}} } |dur(| }d} |dkr?|r2gngd} t|||| d} n |dkrJt|d dd } |rS| sStd t| pW|} |rb|| ||}nt| ||||}| ryt| td | |S) a Optimize Model by OnnxRuntime and/or python fusion logic. ONNX Runtime has graph optimizations (https://onnxruntime.ai/docs/resources/graph-optimizations.html). However, the coverage is limited. We also have graph fusions that implemented in Python to improve the coverage. They can combined: ONNX Runtime will run first when opt_level > 0, then graph fusions in Python will be applied. To use ONNX Runtime only and no Python fusion logic, use only_onnxruntime flag and a positive opt_level like optimize_model(input, opt_level=1, use_gpu=False, only_onnxruntime=True) When opt_level is None, we will choose default optimization level according to model type. When opt_level is 0 and only_onnxruntime is False, only python fusion logic is used and onnxruntime is disabled. When opt_level > 1, use_gpu shall set properly since the optimized graph might contain operators for GPU or CPU only. If your model is intended for GPU inference only (especially float16 or mixed precision model), it is recommended to set use_gpu to be True, otherwise the model is not optimized for GPU inference. For BERT model, num_heads and hidden_size are optional. For other model types, you need specify these parameters. Args: input (str): input model path. model_type (str, optional): model type - like bert, bert_tf, bert_keras or gpt2. Defaults to 'bert'. num_heads (int, optional): number of attention heads. Defaults to 0. 0 allows detect the parameter from graph automatically (for model_type "bert" only). hidden_size (int, optional): hidden size. Defaults to 0. 0 allows detect the parameter from graph automatically (for model_type "bert" only). optimization_options (FusionOptions, optional): optimization options that turn on/off some fusions. Defaults to None. opt_level (int, optional): onnxruntime graph optimization level (0, 1, 2 or 99) or None. Defaults to None. When the value is None, default value (1 for bert and gpt2, 0 for other model types) will be used. When the level > 0, onnxruntime will be used to optimize model first. use_gpu (bool, optional): use gpu or not for onnxruntime. Defaults to False. only_onnxruntime (bool, optional): only use onnxruntime to optimize model, and no python fusion. Defaults to False. Returns: object of an optimizer class. NrrrrrrrEr)MatMulScaleFusionMatMulAddFusionSimplifiedLayerNormFusionGemmActivationFusionBiasSoftmaxFusion)rrr#F)rrzKPlease specify a positive value for opt_level when only_onnxruntime is TruezRemove temporary model: {}) r(rGrHr?rrRr4remover8r0)rSrArBrCrDrrrTrM _producerdefault_opt_leveltemp_model_pathr#r@rPr=r=r>optimize_models<.     r_cCst|ddd}t|}|S)z Get counter of fused operators in optimized model. Args: optimized_model_path (str): the path of onnx model. Returns: A dictionary with operator type as key, and count as value NT)r0load_external_data)rrget_fused_operator_statistics)rr@rPr=r=r>get_fusion_statistics s rbc Csjtjdd}|jddtdd|jddtdd|jd d tjd ttd d td|jdd t ddd|jdd t ddd|jdd ddd|j d d|jdd ddd|j d dt ||jdd ddd|j d d|jd d dd!d|j d d"|jd#d dd$d|j d d%|jd&d t gd'dd(d)|jd*d dd+d|j d d,| }|S)-NzuGraph optimization tool for ONNX Runtime. It transforms ONNX graph to use optimized operators for Transformer models.) descriptionz--inputTzinput onnx model path)requiredtypehelpz--outputzoptimized onnx model pathz --model_typeFrz!Model type selected in the list: z, )rdredefaultchoicesrfz --num_headsrznumber of attention heads like 12 for bert-base and 16 for bert-large. Default is 0 to detect automatically for BERT. For other model type, this parameter need specify correctly.)rdrergrfz --hidden_sizezhidden size like 768 for bert-base and 1024 for bert-large. Default is 0 to detect automatically for BERT. For other model type, this parameter need specify correctly.z --input_int32 store_truezyUse int32 (instead of int64) inputs. It could avoid unnecessary data cast when EmbedLayerNormalization is fused for BERT.)rdactionrf) input_int32z --float16zConvert all weights and nodes in float32 to float16. It has potential loss in precision compared to mixed precision conversion (see convert_float_to_float16).)float16z --verbosezshow debug information.verbosez --use_gpuzZUse GPU for inference. Set this flag if your model is intended for GPU when opt_level > 1.)rz--only_onnxruntimez 1 is used, optimized model for GPU might not run in CPU. Level 2 and 99 are intended for --only_onnxruntime.)rdrerhrgrfz--use_external_data_formatz4use external data format to store large model (>2GB))use_external_data_format)argparseArgumentParser add_argumentstrlowerlistrHkeysjoinint set_defaultsr add_arguments parse_args)parserargsr=r=r>_parse_argumentss          r~cCs&|r tjddddStjdddS)NDEBUGz8[%(filename)s:%(lineno)s - %(funcName)20s()] %(message)s)levelfmtz%(funcName)20s: %(message)s)r) coloredlogsinstallrmr=r=r> _setup_loggerts  rc Cst}t|jtd|tj|jtj|j kr#t dt |}t |j|j|j|j|j||j|jd}|jrD|jdd|jrK|||j |j|r^tddStddS)Nz arguments:zYSpecified the same input and output path. Note that this may overwrite the original model)rrDrrTT) keep_io_typesz#The model has been fully optimized.zThe model has been optimized.)r~rrnr(r8r4r5realpathrSoutputrGrparser_rArBrCrrrTrlconvert_float_to_float16rkchange_graph_inputs_to_int32save_model_to_filerois_fully_optimizedinfo)r}rDrPr=r=r>main~s0    r__main__)rrrN)rrrNNFF)&rploggingr4typingrrrfusion_optionsronnxrronnx_model_bartronnx_model_bertronnx_model_bert_kerasr onnx_model_bert_tfr onnx_model_gpt2r onnx_model_tnlrr getLogger__name__r(rHrsboolrxr?rRr_rbr~rrr=r=r=r>s         = 8 c\ %