o ;ήcK@s<ddlmZddlmZmZddlmZmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlmZddlm Z ddl!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/ee0Z1GdddeZ2Gddde/Z3dS)) getLogger)ListOptional) AttentionMaskFusionAttention)FusionBiasGelu)FusionEmbedLayerNormalization)FusionFastGelu) FusionGelu)FusionGeluApproximation)FusionLayerNormalizationFusionLayerNormalizationTF) FusionOptions)FusionQOrderedAttention)FusionQOrderedGelu) FusionQOrderedLayerNormalization)FusionQOrderedMatMul) FusionReshape) FusionShape) FusionBiasSkipLayerNormalizationFusionSkipLayerNormalization) FusionUtils) GraphProto ModelProto TensorProtoValueInfoProtohelper) OnnxModelcs eZdZdZfddZZS)BertOptimizationOptionszThis class is deprecatedcstdt|dS)NzIBertOptimizationOptions is depreciated. Please use FusionOptions instead.)loggerwarningsuper__init__)self model_type __class__O/tmp/pip-target-vg8gfxp4/lib/python/onnxruntime/transformers/onnx_model_bert.pyr""s z BertOptimizationOptions.__init__)__name__ __module__ __qualname____doc__r" __classcell__r'r'r%r(rsrcs"eZdZdAdededeffdd ZddZd d Zd d Zd dZ ddZ ddZ ddZ ddZ ddZddZddZdedeedefd d!Zdefd"d#Zejfd$ed%ed&efd'd(Zd)d*ZdBd-d.Zd/d0Zd1d2Zd3d4Zd5d6Z dCd9e!e"d:efd;d<Z#d=d>Z$d?d@Z%Z&S)D BertOnnxModelrmodel num_heads hidden_sizecs|dkr|dks|dkr||dksJt|||_||_t||_t||j|j|j|_t||j|j|j|_ t ||_ dS)aGInitialize BERT ONNX Model. Args: model (ModelProto): the ONNX model num_heads (int, optional): number of attention heads. Defaults to 0 (detect the parameter automatically). hidden_size (int, optional): hidden dimension. Defaults to 0 (detect the parameter automatically). rN) r!r"r0r1rattention_maskrattention_fusionrqordered_attention_fusionrutils)r#r/r0r1r%r'r(r"(s(  zBertOnnxModel.__init__cCs|j|jdSN)r3applyr4r#r'r'r(fuse_attention=s zBertOnnxModel.fuse_attentioncC4t|}|t|}|t|}|dSr6)r r7r rr#fusionr'r'r( fuse_geluBs  zBertOnnxModel.fuse_gelucCst||}|dSr6)rr7)r# is_fastgelur<r'r'r(fuse_bias_geluKs  zBertOnnxModel.fuse_bias_gelucCt|}|dSr6)r r7r;r'r'r(gelu_approximationO z BertOnnxModel.gelu_approximationcCr@r6)rr7r;r'r'r(fuse_add_bias_skip_layer_normSrBz+BertOnnxModel.fuse_add_bias_skip_layer_normcCr@r6)rr7r;r'r'r( fuse_reshapeWrBzBertOnnxModel.fuse_reshapecCr@r6)rr7r;r'r'r( fuse_shape[rBzBertOnnxModel.fuse_shapecCr@r6)rr7r;r'r'r(fuse_embed_layer_rBzBertOnnxModel.fuse_embed_layercCr:r6)r r7r rr;r'r'r(fuse_layer_normcs  zBertOnnxModel.fuse_layer_normcCr@r6)rr7r;r'r'r(fuse_skip_layer_normnrBz"BertOnnxModel.fuse_skip_layer_normcCr@r6)rr7r;r'r'r(fuse_qordered_mamtulsrBz"BertOnnxModel.fuse_qordered_mamtulop_type input_indicescastedc sg}|}||}|D]>fdd|D}|D]0}||r)|s(||q||vrJ||} | jdkrJ|| jddurJ|rJ|| jdqq |S)z Get graph inputs that feed into node type (like EmbedLayerNormalization or Attention). Returns a list of the graph input names based on the filter whether it is casted or not. cs$g|]}|tjkrj|qSr')leninput).0inoder'r( s$zABertOnnxModel.get_graph_inputs_from_node_type..CastrN)output_name_to_nodeget_nodes_by_op_typefind_graph_inputappendrJrN) r#rJrKrL graph_inputsrUnodes bert_inputs bert_inputparentr'rQr(get_graph_inputs_from_node_typews$    z-BertOnnxModel.get_graph_inputs_from_node_typecCs*|dgd|}||ddg|7}|S)NEmbedLayerNormalization)r Attention)r^)r#rLinputsr'r'r(!get_graph_inputs_from_fused_nodessz/BertOnnxModel.get_graph_inputs_from_fused_nodesgraph graph_inputnew_typecCsxt|tsJt|tsJ||jsJ|jjjt|kr#dgfSd}g}| }|j|vr||j}dd|D}|r}| d} | d|j} |j } | || | _tjd|jg| gt|jjj| d}|j|g|D] } t| |j| qrdd|D} | D]#} t| dt|kr|| jd |j|| jd s|| q|r||t||jj_||fS) aChange graph input type, and add Cast node if needed. Args: graph (GraphProto): graph graph_input (TensorProto): input of the graph new_type (int, optional): new data type. Defaults to TensorProto.INT32. Returns: NodeProto: a new Cast node that added. None if Cast node is not added. List[NodeProto]: Cast nodes that have been removed. NcSsg|] }|jdkr|qSrTrJrOrRr'r'r(rSz9BertOnnxModel.change_graph_input_type..rT_)tonamecSsg|] }|jdkr|qSrirjrkr'r'r(rSrlrnr) isinstancerrrWrotype tensor_type elem_typeintinput_name_to_nodescreate_node_name value_infoaddCopyFromr make_noderRextendrreplace_node_inputget_node_attributereplace_input_of_all_nodesoutputfind_graph_outputrX remove_nodes)r#rfrgrh new_cast_nodenodes_to_removerurZnodes_not_cast node_name output_namenew_value_inforR nodes_castr'r'r(change_graph_input_typesL        z%BertOnnxModel.change_graph_input_typecCsd|}d}d}|jD]}|||tj\}}|r|d7}|t|7}q td|d|ddS)zPChange data type of all graph inputs to int32 type, and add Cast node if needed.rr`z)Graph inputs are changed to int32. Added z Cast nodes, and removed z Cast nodes.N)rfrNrrINT32rMrinfo)r#rfadd_cast_countremove_cast_countrgnew_node removed_nodesr'r'r(change_graph_inputs_to_int32s z*BertOnnxModel.change_graph_inputs_to_int32 batch_size max_seq_lencCs|jdd|jdd}i}|jjjD]!}|j|vr4|jjjjd}||_ |dur4|jjjjd}||_ q|jjj D] }|jjjjd}||_ q:dS)zD Update input and output shape to use dynamic axes. T)rLFrNr`) rer/rfrNrorqrrshapedim dim_paramr)r#dynamic_batch_dimdynamic_seq_lenbert_graph_inputsdynamic_batch_inputsrN dim_protorr'r'r(use_dynamic_axess$  zBertOnnxModel.use_dynamic_axescCs |dSr6)adjust_reshape_and_expandr8r'r'r( preprocessszBertOnnxModel.preprocessc Cs2g}|D]}}|jdkr||jd}|dur1|jdkr1||g||jd|jdq||gdgd| }|dur|d}||jd}|d}||jd}|d} |dur|durt |d krt |dkr|d|dkr| jd|jd<q|r| |t d t |dSdS) NReshaper`r)ExpandrrSlice)rrrrz"Removed Reshape and Expand count: )rZrJget_constant_valuerNsizer{r~rmatch_parent_pathrUrMrrr) r#rrR reshape_shape reshape_path expand_nodeexpand_shape_valuereshape_before_expand shape_value slice_noder'r'r(rsH      z'BertOnnxModel.adjust_reshape_and_expandc Csd|}g}|D]}dddd}|j|vrQ||j}||gd|dddddg|}|durQ|\}}} } } } | jd|jdjkrQ| jd|jd<|}|jdkr||gdgd|}|dur|d jd|jdjkrtj d|jdt |jd|j|jd d } d | _ | j td |jg|| || j||q ||dS)Nr`rrc)r_ ReduceSumrb)rTConstantOfShapeConcat UnsqueezeGatherShaperb)rrTrr)rcrrrr _remove_mask)rdoutputsroz com.microsoftr0)rUrZrJrrNrfrorrrzrMdomain attributer{make_attributer0add_nodeget_graph_by_noderXr)r#rUrrR op_input_idrP parent_nodescastconstantOfShapeconcat unsqueezegatherrattention_noder'r'r( clean_graph#sZ      zBertOnnxModel.clean_graphcCs||dSr6)r prune_graphr8r'r'r( postprocesserBzBertOnnxModel.postprocessNFoptionsadd_dynamic_axescCsl|dur |js ||j|j|dus|jr ||dus'|jr+|| | |dus:|j r>| |dusE|j rT|durP|j|j||dus[|jr_|||dusj|jrn||j||dus~|jr|jdd|jdd|dus|jr||dur|jr|||r| t!"d|#dS)NT)r>Fzopset version: )$enable_shape_inferencedisable_shape_inferencer5remove_identity_nodesremove_useless_cast_nodesenable_layer_normrG enable_gelur=rrDenable_skip_layer_normrHenable_attentionr2set_mask_formatattention_mask_formatr9enable_qordered_matmulrIrEenable_embed_layer_normrFremove_useless_reshape_nodesrenable_bias_gelur?enable_bias_skip_layer_normrCenable_gelu_approximationrAremove_unused_constantrrrget_opset_version)r#rrr'r'r(optimizeisD     zBertOnnxModel.optimizecCs@i}gd}|D] }||}t|||<qtd||S)z8 Returns node count of fused operators. ) r_rbQOrderedAttentionGelu QOrderedGeluFastGeluBiasGeluLayerNormalizationQOrderedLayerNormalizationSkipLayerNormalizationQOrderedMatMulzOptimized operators:)rVrMrr)r#op_countopsoprZr'r'r(get_fused_operator_statisticss z+BertOnnxModel.get_fused_operator_statisticscCs|}|d}|d|d}|d|d|d}|d|d}|d ko5|d ko5||ko5|d |k}|d kr?td |d krHtd |d krQtd |d krZtd|S)zA Returns True when the model is fully optimized. r_rbrrrrrrrrzLayer Normalization not fusedzGelu/FastGelu not fusedzEmbed Layer not fusedzAttention not fused)rrdebugr )r#rembed attentiongelu layer_norm is_perfectr'r'r(is_fully_optimizeds$    z BertOnnxModel.is_fully_optimized)rr)rr)NF)'r)r*r+rrtr"r9r=r?rArCrDrErFrGrHrIstrrboolr^rerrrrrrrrrrrrrrrrr-r'r'r%r(r.'s@    A )B?r.N)4loggingrtypingrrfusion_attentionrrfusion_biasgelurfusion_embedlayerrfusion_fastgelur fusion_gelur fusion_gelu_approximationr fusion_layernormr r fusion_optionsrfusion_qordered_attentionrfusion_qordered_gelurfusion_qordered_layernormrfusion_qordered_matmulrfusion_reshaper fusion_shaperfusion_skiplayernormrr fusion_utilsronnxrrrrr onnx_modelrr)rrr.r'r'r'r(s.