o ;ήcu@sddlmZddlmZmZmZmZddlmZddl m Z ddl m Z m Z mZddlmZeeZGdddeZGd d d eZd S) ) getLogger)DictListTupleUnion)Fusion) FusionUtils) NodeProto TensorProtohelper) OnnxModelc seZdZdZd2dedeffdd Zdeded e eefffd d Z d ed e ee efde de fddZddZddZddZddZddZdede eed efffddZ d3ded ed ed!ed"ed efd#ef d$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1ZZS)4FusionEmbedLayerNoMaskz Fuse embedding layer into one node (EmbedLayerNormalization). It supports the following model types: BERT, DistilBert, ALBert. no maskmodel descriptioncsBt|dddg|t||_|jjidd|_d|_d|_dS)NEmbedLayerNormalizationLayerNormalizationSkipLayerNormalizationT)update) super__init__rutilsrinfer_runtime_shapeshape_infer_helper attention embed_node)selfrr __class__Q/tmp/pip-target-vg8gfxp4/lib/python/onnxruntime/transformers/fusion_embedlayer.pyrs  zFusionEmbedLayerNoMask.__init__addreturnNcCsP|j|dgdg}|durdS|j|dgdg}|dur dS|d|dfS)NGatherr)rmatch_parent_path)rr! gather_0_path gather_1_pathrrr match_two_gather$sz'FusionEmbedLayerNoMask.match_two_gather layernorminput_name_to_nodesis_distil_bertc Csf|jj|d|dd|_|jdur|jd|vrdS||jd}t|dkrx|djdkrx|djd|vrx||djd}t|dkrw|djdkrw|djd|vrw||djd}|D] }|jdkrm||_d Sq`td d |D}n td d |D}|r|gd kr|gdkr|gdkrtddSd S|gdkr|gdkrtddSd S)aCheck that LayerNormalization has a child of Attention node or subgraph like Attention. Args: layernorm (NodeProto): LayerNormalization node input_name_to_nodes (Dict[str, List[NodeProto]]): map from input name to nodes is_distil_bert (bool): whether it is DistilBert or not Returns: bool: whether there is Attention node or subgraph like Attention AttentionF) recursiveNrr$MatMulAddTcSg|]}|jqSrop_type.0childrrr UzCFusionEmbedLayerNoMask.check_attention_subgraph..cSr0rr1r3rrr r6Wr7)r.r.r.Shaper)r/r.r.r.r8r8)r/r.r.r.r8z Equal(B=?) -- Not -- Cast(to=6) -- CumSum(axis=1) -- Mul -- Cast(to=7) -- Add(B=1) -- Cast(to=7)* --> Gather | ^ V | +------------------------------+ Roberta new pattern from transformers v4.9: (input_ids) --> Equal(B=?) -- Not -- Cast(to=6) -- CumSum(axis=1) -- Add(B=0) -- Mul -- Cast(to=7) -- Add(B=1) --> Gather | ^ V | +-------------------------------------------+ start_node = position_embedding_gather start_index = 1 # match optional Cast node. parent = self.model.get_parent(start_node, start_index, output_name_to_node) if parent is None: return if parent.op_type == "Cast": if OnnxModel.get_node_attribute(parent, "to") != 7: return start_node = parent start_index = 0 i, path, return_indices = self.model.match_parent_paths( start_node, [ (['Add', 'Cast', 'Mul', 'CumSum', 'Cast', 'Not', 'Equal'], [start_index, 0, 0, 0, 0, 0, 0]), (['Add', 'Cast', 'Mul', 'Add', 'CumSum', 'Cast', 'Not', 'Equal'], [start_index, 0, 0, 0, 0, 0, 0, 0])], output_name_to_node) if path is not None: # constant input of Add shall be 1. i, value = self.model.get_constant_input(path[0]) if value != 1: return False _, self.padding_word_id = self.model.get_constant_input(path[-1]) return input_ids == path[-1].input[0] FrrrQrRrSrrr match_position_embedding_robertas-z7FusionEmbedLayerNoMask.match_position_embedding_robertac Cs|j|ddgddg|}|durdS|\}}|j|jd}|durTt|jdkrT|jddkrT|j|ddgrT|j|ddgrTt|jd ksV|j|d dgsVdS|j}|d krjt |d dgsidSn |j|ddgsudS|j |d|} | durdS| j d kr|j| ddsdS|j | d|} n| } | dus| j d krdS|j| ddsdS|j | d|} | dus| j dkrdS|| jdkS)a Match position embedding path from input_ids to Gather for BERT. BERT Embedding Layer Pattern: (input_ids) / / Shape / | / Gather (indices=1) / | / Add (optional, B=0) / | Gather (segment_ids) Unsqueeze (axes=0) \ | | \ Gather Slice (data[1,512], starts=0, ends=*, axes=1, steps=1) \ / | Add Gather \ / Add | LayerNormalization SlicerJr$rHNFr axesr/r#r8) rr%get_constant_valuerNr;rVrrPget_opset_versionrcheck_node_attribute get_parentr2) rrQrRrSpathslice unsqueeze slice_weight opset_versionrBgatherrVrrr match_position_embedding_berts`     z4FusionEmbedLayerNoMask.match_position_embedding_bertcCs(||||r dS||||rdSdS)NTF)rnr\r]rrr match_position_embedding#s z/FusionEmbedLayerNoMask.match_position_embeddingc Cs|jd}|r |jdnd}|jd}|jdura|j|}|j|}|r(|s*Jt|dkr>t|dkr>|d|dksItd||dS|ra|j||satd||j|dS|j |jd} | dusut| j dkr|tddS|j |jd} | dust| j dks| j d| j dkrtd dS|r|j |jd} | dust| j dks| j d| j dkrtd dS| j d| j dkrt d |jdd | j dd |jdd | j d|rB| j d| j dkrt d |jdd | j dd|jdd | j d| j d| j dkrBt d|jdd | j dd|jdd | j ddS)zXSanity check of embedding weights, and match hidden_size of weights and shape of inputs.r$NrHzfCannot fuse EmbedLayerNormalization: input_ids and position_ids not matched in 2nd dimension: {} vs {}FzaCannot fuse EmbedLayerNormalization: input_ids and segment_ids does not have same shape: {} != {}rzICannot fuse EmbedLayerNormalization: word embedding table is not expectedzMCannot fuse EmbedLayerNormalization: position embedding table is not expectedzLCannot fuse EmbedLayerNormalization: segment embedding table is not expectedzword_embedding_table (z) size z <= position_embedding_table (z <= segment_embedding_table (zposition_embedding_table (T) rNrget_edge_shaper;r=infoformat compare_shaperrdrVwarning) rword_embedding_gathersegment_embedding_gatherrQrR segment_ids position_idsinput_ids_shapeposition_ids_shapeword_embedding_tableposition_embedding_tablesegment_embedding_tablerrr check_embedding1s              222z&FusionEmbedLayerNoMask.check_embedding input_namecCsdd}|j|}|dur&|jjjtjkr |j|\}}||fS|}||fS|j|\}}||fS)aCast a graph input or node input to int32. Args: input_name (str): name of graph input or node input Returns: A tuple of casted input name and the cast node. int32_output (str): If input is int32, it is the input name, Otherwise it is output name of Cast node. input_cast_node (Union[None, NodeProto]): Cast node. It could be None if input is int32. N) rfind_graph_inputtype tensor_type elem_typer INT32rcast_input_to_int32)rrinput_cast_node graph_input int32_outputrrr cast_to_int32|s z$FusionEmbedLayerNoMask.cast_to_int32FrRrurQrvrxcCsg}||\}} |jd} |jdkr|jd} |jd} n |jd} |jd} d} |durL||jd\}} |||jd|jd|jd| | g} n|d|jd|jdd| | g} |durp| d||\}} | || d | d g}|r|| d tjd| || d }d |_|j D]}|j dkr|j |gqt |j dkr|j t ddg|||D] }|j|j|j <q|j |||_|S)agCreate an EmbedLayerNormalization node. Note that segment embedding is optional. Args: input_ids (str): input_ids for word embeddings layernorm (NodeProto): LayerNormalization or SkipLayerNormalization node. word_embedding_gather (NodeProto): the Gather node for word embedding position_embedding_gather (NodeProto): the Gather node for position embedding segment_embedding_gather (Union[None, NodeProto]): the Gather node for segment embedding, or None. Returns: NodeProto: the EmbedLayerNormalization node created. rrr$rHr`Nr_output_dummy_mask_index_embedding_sum)outputsnamez com.microsoftepsilong-q=)rrcreate_node_namer2rNappendr make_nodedomain attributerextendr;make_attributethis_graph_namenode_name_to_graph_name nodes_to_addr)rrRr)rurQrvrxembedding_sum_outputrrW node_namegammabetaembed_node_inputsrwembed_node_outputsrattrBrrr create_fused_nodesj             z(FusionEmbedLayerNoMask.create_fused_nodecCs$|j|jd|jdd|_dS)NrT)rreplace_input_of_all_nodesr: prune_graph)rr)rrrr finish_fusions z$FusionEmbedLayerNoMask.finish_fusioncCs|j|}t|dkS)a@Check that Add before layer norm has an output to add before next layernorm Args: add_before_layer_norm (NodeProto): Add before any LayerNormalization node in topological order of graph Returns: bool: whether there is an extra output needed out of embed layer norm node r$)r get_childrenr;)radd_before_layer_normrArrr is_embedding_sum_neededs z.FusionEmbedLayerNoMask.is_embedding_sum_neededc Cs||}|dur dS|jd}|\}}|jd} |jd} |j||dds(dS||d|s1dSd} ||r:d} || |||d| | } |j|jd| jd| r^|j|| jddS)NFrr$r+TrH) r(r:rNrDr~rrrr) rr)add_before_layernormr*rS two_gather add_outputrurQrRrxoptional_embedding_sum_outputrrrr fuse_gpt2s6      z FusionEmbedLayerNoMask.fuse_gpt2c Cs||}|dur dS|\}}|jd}|j||ddsdS||||s'dS||d|s0dS|||||d} ||| dS)aFuse embedding layer for DistilBert Args: layernorm (NodeProto): node of LayerNormalization or SkipLayerNormalization add_before_layernorm (NodeProto): the Add node before LayerNormalization, or the SkipLayerNormalization itself input_name_to_nodes (Dict[str, List[NodeProto]]): map from input name to nodes output_name_to_node (Dict[str, List[NodeProto]]): map from output name to nodes NFr$Tr)r(rNrDror~rr) rr)rr*rSrrurQrRrrrr fuse_distilbert>s    z&FusionEmbedLayerNoMask.fuse_distilbertcCs|j|dgdg}|durdS||d}|durdS|\}}|jd} |j||dds0dS|j|dgdg} | dur@dS| d} || | |sZ||| |sTdS|} | }| } |||| scdS|| ||| |} ||| dS) aFuse embedding layer for Bert Args: layernorm (NodeProto): node of LayerNormalization or SkipLayerNormalization add_before_layernorm (NodeProto): the Add node before LayerNormalization, or the SkipLayerNormalization itself input_name_to_nodes (Dict[str, List[NodeProto]]): map from input name to nodes output_name_to_node (Dict[str, List[NodeProto]]): map from output name to nodes r/rNFr$rr#T) rr%r(rNrDror~rr)rr)rr*rS add_2_gatherrrurvrRposition_embedding_pathrQtemprrrr fuse_bertgs>   z FusionEmbedLayerNoMask.fuse_bertcCsx|jdkr|j|dgdg}|durdS|d}n|}|||||r&dS|||||r0dS|||||r:dSdS)Nrr/r)r2rr%rrr)rrBr*rSfirst_add_pathrrrr fuses  zFusionEmbedLayerNoMask.fuse)r)NF)__name__ __module__ __qualname____doc__r strrr rrr(rrboolrDr\r^rnror~rrrrrrrr __classcell__rrrr r sP"   ?>/H"K  `8)2r cs.eZdZdeffdd ZfddZZS)FusionEmbedLayerNormalizationrcst|ddS)Nz with mask)rr)rrrrr rsz&FusionEmbedLayerNormalization.__init__csd|_d|_t||||jrC|jrE|jjd}||vrG||}|jdkrI|j}|jd}|j|g|j|||j d<dSdSdSdSdS)Nr` ReduceSumrr$) rrrrrNr2nodes_to_removerrr:)rrBr*rS mask_indexrmask_input_namerrr rs      z"FusionEmbedLayerNormalization.fuse)rrrr rrrrrrr rsrN)loggingrtypingrrrr fusion_baser fusion_utilsronnxr r r onnx_modelr rr=r rrrrr s