o ;ήc(@sddlZddlZddlZddlmZddlmZmZmZddl Z ddl m Z m Z m Z ddlmZmZddlmZmZddlmZmZddlmZejejejed d dd lmZdd l m!Z!dd l"m#Z#e$e%Z&gd Z'gdZ(GdddZ)dS)N)Path)DictListUnion) T5DecoderT5DecoderHelper T5DecoderInit) T5EncoderT5EncoderHelper)T5EncoderDecoderInitT5EncoderDecoderInitHelper)MT5ForConditionalGenerationT5ForConditionalGeneration)InferenceSessionz..)float_to_float16_max_diff) OnnxModel)optimize_model)zt5-smallzt5-basezt5-largezt5-3bzt5-11b)zgoogle/mt5-smallzgoogle/mt5-basezgoogle/mt5-largez google/mt5-xlzgoogle/mt5-xxlc@s.eZdZe  d*dededededef dd Ze d+ded ed ejdedede eej j ff ddZ e  d,de eeeefd ejdededededefddZegdfdedeefddZe  d-deded ed!ed"eded#efd$d%Zede eeeefd&ed ejdefd'd(Zd)S).T5HelperF output_dirmodel_name_or_pathsuffix new_folderreturncCs^|}tj|rt|jd}n|dd||7}|r$tj||n|}tj||dS)aBuild onnx path Args: output_dir (str): output directory model_name_or_path (str): pretrained model name, or path to the model checkpoint suffix (str, optional): suffix like "_encoder" or "_decoder_fp16" will be appended to file name. Defaults to None. new_folder (bool, optional): create a new directory for the model. Defaults to False. Returns: str: path of onnx model /z.onnx)ospathisdirrpartssplitjoin)rrrr model_name directoryr$S/tmp/pip-target-vg8gfxp4/lib/python/onnxruntime/transformers/models/t5/t5_helper.py get_onnx_path!s zT5Helper.get_onnx_pathTt5 cache_dirdevicemerge_encoder_and_decoder_init model_typec Cs|dkr tj||d}n|dkrtj||d}ntdt|j|j|j}| ||r@t |j |j|j|jdd}||dSt |j |j}| |t |j|j|j} |  |||| dS) abLoad model given a pretrained name or path, then build models for ONNX conversion. Args: model_name_or_path (str): pretrained model name or path cache_dir (str): cache directory device (torch.device): device to run the model merge_encoder_and_decoder_init (bool, optional): Whether merge encoder and decoder initialization into one ONNX model. Defaults to True. is_mt5 (bool, optional): whether the model is MT5 instead of T5 Returns: Dict[str, torch.nn.Module]: mapping from name to modules for ONNX conversion. r')r(mt5z only support mode_type=t5 or mt5N)decoder_start_token_id)encoder_decoder_initdecoder)encoderr/ decoder_init)rfrom_pretrainedr ValueErrorrr/lm_headconfigevaltor r0r r) rr(r)r*r+modelr/r.r0r1r$r$r% load_model>s0 zT5Helper.load_modelr8onnx_model_pathverboseuse_external_data_formatuse_decoder_input_idsuse_int32_inputsc Cs^t|trt||||||dSt|tr#t|||||||dSt||||||dS)N) isinstancer r export_onnxr r r)r8r)r:r;r<r=r>r$r$r%r@os6   zT5Helper.export_onnx)Pow ReduceMeanAddSqrtDivMulSoftmaxRelu onnx_model op_block_listcCsTtdd|D}t|}||}td|d||jdj}d}|}||vs3J||}d} |j dkrq|} td |jd} |j D] } | | } | dur[nqNt | } t d |jd | | d k}n td |j d|jg} g}|s| dur|g} | jg}| |||d}td||jdddi||S)aConvert model to mixed precision. It detects whether original model has fp16 precision weights, and set parameters for float16 conversion automatically. Args: onnx_model (OnnxModel): optimized ONNX model op_block_list (List[str], optional): . Defaults to ["Pow", "ReduceMean", "Add", "Sqrt", "Div", "Mul", "Softmax", "Relu"] Returns: parameters(dict): a dictionary of parameters used in float16 conversion cSsg|]}|jqSr$)op_type).0noder$r$r% sz1T5Helper.auto_mixed_precision..z fp32 op: z fp16 op: rFNMatMulz#Found last MatMul node for logits: z3max diff of converting weights in last MatMul node z: gư>z-Failed to find MatMul node for logits. Found z of node ) keep_io_typesrJnode_block_listforce_fp16_initializersz!auto_mixed_precision parameters: use_symbolic_shape_inferTr$)setnodes differenceloggerinfographoutputnameoutput_name_to_noderKinputget_initializerrdebugwarningconvert_float_to_float16)rIrJ op_full_set fp32_op_set fp16_op_setlogits_output_nameis_weight_fp16_precisionr\rMlast_matmul_node initializerr]max_diffrPrQ parametersr$r$r%auto_mixed_precisionsH       zT5Helper.auto_mixed_precisionoptimized_model_path is_float16num_attention_heads hidden_sizerkc CsJt|d||dddd}|r|rt|n|jdd|j||dddS) zHOptimize ONNX model with an option to convert it to use mixed precision.bertrNF)r+ num_headsro opt_leveloptimization_optionsuse_gpu)cast_input_outputT)all_tensors_to_one_file)rrrk convert_model_float32_to_float16save_model_to_file)r:rlrmrnror<rkmr$r$r% optimize_onnxs    zT5Helper.optimize_onnx ort_sessioncCsDt|tr t||||St|trt||||St||||S)zQCompare the result from PyTorch and OnnxRuntime to verify the ONNX model is good.)r?r r verify_onnxr r r)r8r{r)r>r$r$r%r|s  zT5Helper.verify_onnxN)rF)Tr')TFTF)FT)__name__ __module__ __qualname__ staticmethodstrboolr&torchr)rnnModuler9rr rrr r@rrrkintrzrr|r$r$r$r%r s  0 & G r)*loggingrsyspathlibrtypingrrrr t5_decoderrrr t5_encoderr r t5_encoder_decoder_initr r transformersr r onnxruntimerrappendr!dirname__file__float16rrIr optimizerr getLoggerr}rWPRETRAINED_T5_MODELSPRETRAINED_MT5_MODELSrr$r$r$r%s&