o ;ήcDK@sdddlZddlZddlmZddlmZddlZddlZddlmZddlm Z ddl m Z m Z mZdZdZd Zd Zd Zd Zd ZdZdZddddddddddddddddZGd d!d!eZGd"d#d#eZGd$d%d%eZGd&d'd'eZe jjed(e jj ed)iZ!dtd*d+Z"dud-d.Z#dud/d0Z$dvd1d2Z%dvd3d4Z&Gd5d6d6Z'Gd7d8d8Z(Gd9d:d:Z)d;d<Z*d=d>Z+d?d@Z,dAdBZ-dCedDe.dEefdFdGZ/dHdIZ0dJdKZ1dwdMdNZ2dOefdPdQZ3dOedRefdSdTZ4dUdVZ5dWdXZ6dYdZZ7d[d\Z8dOefd]d^Z9dOed_e:fd`daZ;dbdcZdhdiZ?djdkZ@dldmZAdndoZBdpdqZCdrdsZDdS)xN)Enum)Path)external_data_helper)onnx_pb)GraphOptimizationLevelInferenceSessionSessionOptionsz onnx.quantizez0.1.0zai.onnxz com.microsoftQuantizeLinear_QuantizeLinear_InputDequantizeLinear_DequantizeLinear_Output _quantizedFLOATUINT8INT8UINT16INT16INT32INT64STRINGBOOLFLOAT16DOUBLEUINT32UINT64 COMPLEX64 COMPLEX128) c@(eZdZdZdZddZeddZdS)QuantizationModerrcC|jSNnameselfr4K/tmp/pip-target-vg8gfxp4/lib/python/onnxruntime/quantization/quant_utils.py__str__3zQuantizationMode.__str__cC zt|WStytwr/)r-KeyError ValueError)moder4r4r5 from_string6   zQuantizationMode.from_stringN)__name__ __module__ __qualname__ IntegerOps QLinearOpsr6 staticmethodr<r4r4r4r5r-/ r-c@r,)QuantizedValueTyperrcCr.r/r0r2r4r4r5r6Br7zQuantizedValueType.__str__cCr8r/)rEr9r:)vr4r4r5r<Er=zQuantizedValueType.from_stringN)r>r?r@Input Initializerr6rCr<r4r4r4r5rE>rDrEc@r,) QuantTyperrcCr.r/r0r2r4r4r5r6Qr7zQuantType.__str__cCr8r/)rIr9r:)tr4r4r5r<Tr=zQuantType.from_stringN)r>r?r@QInt8QUInt8r6rCr<r4r4r4r5rIMrDrIc@r,) QuantFormatrrcCr.r/r0r2r4r4r5r6`r7zQuantFormat.__str__cCr8r/)rMr9r:)formatr4r4r5r<cr=zQuantFormat.from_stringN)r>r?r@ QOperatorQDQr6rCr<r4r4r4r5rM\rDrMint8uint8c Cs|tvs Jd|t|}t|tjkrdnd|durdn|}t|tjkr)dnd|dur0dn|}t|tj| |} tj | ||| d| |S)NEUnexpected data type {} requested. Only INT8 and UINT8 are supported.r)out) ONNX_TYPE_TO_NP_TYPErNmaxnumpyrRminasarrayastypefloat32roundclip) qTypearrscale zero_pointlowhighdtypecliplowcliphigharr_fp32r4r4r5quantize_nparrayqs$$ rkFcCs|dks|dkrtd|d|t|d}t|d}|r-tt|t|}| }| }||t||}|ttjjkrHd}d}||gSt |||}||gS)aCalculate the scale s and zero point z for the quantization relation r = s(q-z), where r are the original values and q are the corresponding quantized values. r and z are calculated such that every value within [rmin,rmax] has an approximate representation within [qmin,qmax]. In addition, qmin <= z <= qmax is enforced. If the symmetric flag is set to True, the interval [rmin,rmax] is symmetrized to [-absmax, +absmax], where absmax = max(abs(rmin), abs(rmax)). :parameter rmin: minimum value of r :parameter rmax: maximum value of r :parameter qmin: minimum value representable by the target quantization data type :parameter qmax: maximum value representable by the target quantization data type :return: zero and scale [z, s] rzBqmin and qmax must meet requirement: qmin <= 0 <= qmax while qmin:z, qmmax:?) r:r[rYabsfloatrZfinfor^tinyr_)rminrmaxqminqmax symmetricabsmaxrcrdr4r4r5compute_scale_zp}s  rwc Cspd}d}d}d}t|r't|}t|}t|||d\}} t|||| |\}}t|t|||} ||||| fS)a :param data: data to quantize :param qType: data type to quantize to. Supported types UINT8 and INT8 :param symmetric: whether symmetric quantization is used or not. This is applied to INT8. :return: minimum, maximum, zero point, scale, and quantized weights To pack weights, we compute a linear transformation - when data `type == uint8` mode, from `[rmin, rmax]` -> :math:`[0, 2^{b-1}]` and - when data `type == int8`, from `[-m , m]` -> :math:`[-(2^{b-1}-1), 2^{b-1}-1]` where `m = max(abs(rmin), abs(rmax))` and add necessary intermediate nodes to trasnform quantized weight to full weight using the equation :math:`r = S(q-z)`, where - *r*: real original value - *q*: quantized value - *S*: scale - *z*: zero point rrlru)lenr[rYget_qmin_qmax_for_qTyperwrkrZr\) datararu reduce_rangerqrrrdrcrsrtquantized_datar4r4r5 quantize_datasr~cCsr|tjjkr|r dnd\}}||fS|tjjkr2|r&|rdnd\}}||fS|r*dnd\}}||fStd|)z Return qmin and qmax, the minimum and maximum value representable by the given qType :parameter qType: onnx.onnx_pb.TensorProto.UINT8 or onnx.onnx_pb.TensorProto.UINT8 :return: qmin, qmax )rrV)rrU)i@)rTrV)irVrS) onnx_proto TensorProtorrr:rNrar|rursrtr4r4r5rzs  rzcCst|||d\}}||S)z Helper function to get the quantization range for a type. parameter qType: quantization type. return: quantization range. rx)rzrr4r4r5get_qrange_for_qTypesrc@s eZdZdZggdfddZdS)QuantizedInitializerzJ Represents a linearly quantized weight input from ONNX operators Nc Cs:||_||_||_||_||_||_||_||_| |_dSr/) r1 initializerrminsrmaxs zero_pointsscalesr{r}axis) r3r1rrrrrr{r}rr4r4r5__init__s  zQuantizedInitializer.__init__r>r?r@__doc__rr4r4r4r5rs  rc@seZdZdZ dddZdS)QuantizedValuezI Represents a linearly quantized value (input\output\intializer) NcCs(||_||_||_||_||_||_dSr/) original_nameq_name scale_namezp_name value_typer)r3r1new_quantized_namerzero_point_namequantized_value_typerr4r4r5rs   zQuantizedValue.__init__r/rr4r4r4r5r s rc@seZdZdZddZdS)BiasToQuantizez+ Represents a bias to be quantized cCs||_||_||_dSr/) bias_name input_name weight_name)r3rrrr4r4r5r%s zBiasToQuantize.__init__Nrr4r4r4r5r s rcCs|jdkr td|j|jdkr|j}n[|jdkr|j}nR|jdkr(|j}nI|jdkr1|j}n@|jdkr:|j}n7|jdkrC|j }n.|jd krL|j }n%|jd krU|j }n|jd kr^|j }n|jd krg|j }n td |j|j|j|iS)z Convert attribute to kwarg format for use with onnx.helper.make_node. :parameter attribute: attribute in AttributeProto format. :return: attribute in {key: value} format. rz*attribute {} does not have type specified.rrrr r!r"r#r$r%r&z%attribute {} has unsupported type {}.)typer:rNr1fisrJgfloatsintsstringstensorsgraphs) attributevaluer4r4r5attribute_to_kwarg+s0            rcs*fdd|D}t|dkr|dSdS)z Helper function to find item by name in a list. parameter item_name: name of the item. parameter item_list: list of items. return: item if found. None otherwise. csg|] }|jkr|qSr4r0).0item item_namer4r5 Wsz find_by_name..rN)ry)r item_listitemsr4rr5 find_by_namePsrcCs,d}tdt|D] }|||kr|}q |S)zC Helper function to return index of an item in a node list r)rangery) elem_name elem_listelem_idxrr4r4r5get_elem_index[s  rcCstjd||g|S)z Helper function to create a Mul node. parameter inputs: list of input names. parameter output: output name. parameter name: name of the node. return: Mul node in NodeProto format. Mul)onnxhelper make_node)inputsoutputr1r4r4r5 get_mul_nodefsrfilename identifierreturncCs|j|j||jS)zp Helper function to generate a identifiable filepath by concatenating the given identifier as a suffix. )parentjoinpathstemsuffix)rrr4r4r5generate_identified_filenameqsrcCsddl}ddlm}ddl}|j|jdtdt|tdt||j||dd|d| d| d | dS) Nr) thresholdz Histogram:zHistogram Edges:T)fillz Tensor valueCountszTensor value V.S. Counts) sysmatplotlib.pyplotpyplotrZset_printoptionsmaxsizeprintstairsxlabelylabeltitleshow)hist hist_edgesrpltrZr4r4r5 apply_plotxs     rc Cs4ddl}ddl}ddlmmm}ddlmmm}t d |t dd}| | |Wdn1s@wY|d}g}t|D]:}||} ttt| dt| d} ||} || } ||||| ||| ||} || qR||t||D]} || q|}|||||| |}|!||"}t dd  }| |Wdn1swY t d d3}t|D]#}||} |d ttt| dt| d}| || d qWddS1swYdS)z> Helper function to write calibration table to files. rNzcalibration cache: {}zcalibration.jsonwirzcalibration.flatbufferswbFzcalibration.cache  ))json flatbuffers5onnxruntime.quantization.CalTableFlatBuffers.KeyValue quantizationCalTableFlatBuffersKeyValue5onnxruntime.quantization.CalTableFlatBuffers.TrtTableTrtTablelogginginforNopenwritedumpsBuildersortedkeysstrrYrm CreateString KeyValueStartKeyValueAddKeyKeyValueAddValue KeyValueEndappendTrtTableStartDictVectorryPrependUOffsetTRelative EndVector TrtTableStartTrtTableAddDict TrtTableEndFinishOutputGetRootAsTrtTable DictLengthrDictKeyValue)calibration_cacherrrrfilebuilderkey_value_listkeyvaluesrflat_key flat_value key_value main_dict cal_tablebufdict_lenrrr4r4r5write_calibration_tablesR                 &  $r -C6?c Csddl}|dk|j}|dk|j}|}|j|}|s!dS|t|t|}|dks8Jd|||f||j}|||| |7}|dkdksSJ|S)a~Given a discrete distribution (may have not been normalized to 1), smooth it by replacing zeros with eps multiplied by a scaling factor and taking the corresponding amount off the non-zero values. Ref: http://web.engr.illinois.edu/~hanj/cs412/bk3/KL-divergence.pdf https://github.com//apache/incubator-mxnet/blob/master/python/mxnet/contrib/quantization.py rNrrlz"n_zeros=%d, n_nonzeros=%d, eps1=%f)rZr]r^sumsizern) pepsnpis_zeros is_nonzerosn_zeros n_nonzeroseps1rr4r4r5smooth_distributions"   r model_pathcCs4tj|dd}|jjD] }t|rdSq dS)NF)load_external_dataT)rloadas_posixgraphrruses_external_data)rmodel intializerr4r4r5model_has_external_datas   r!opt_model_pathcCs0t}||_tj|_t||dgd}dS)z Generate model that applies graph optimization (constant folding, etc.) parameter model_path: path to the original onnx model parameter opt_model_path: path to the optimized onnx model :return: optimized onnx model CPUExecutionProvider) providersN)rroptimized_model_filepathrORT_ENABLE_BASICgraph_optimization_levelr)rr" sess_option_r4r4r5optimize_models r*cCs>ddi}|jr|jD] }||j|jiq tj||dS)z>Tag the model that it went through quantization pre-processingonnx.quant.pre_processonnxruntime.quantNmetadata_propsupdaterrrrset_model_props)rr.propr4r4r5add_pre_process_metadatas  r2cC0|jr|jD]}|jdkr|jdkrdSqdS)zCCheck the model whether it went through quantization pre-processingr+r,TFr.rr)rr1r4r4r5model_has_pre_process_metadata s  r5cCs>ddi}|jr|jD] }||j|jiq tj||dS)N onnx.inferr,r-)rr.rr4r4r5add_infer_metadatas  r7cCr3)Nr6r,TFr4)rrr4r4r5model_has_infer_metadatas  r8cCs:t|d}tjt|t|t|}||S)Nz -inferred)rrshape_inferenceinfer_shapes_pathrrrunlink)rinferred_model_pathrr4r4r5load_model_with_shape_infer#s r= need_optimizecCsltjdd&}|rt|st|d}t|||}t|}t||WdS1s/wYdS)N ort.quant.prefix model.onnx)tempfileTemporaryDirectoryr!rrr*r=r7)rr> quant_tmp_dirr"rr4r4r5 load_model+s  $rFcCshtjdd$}t|d}tjj|ddt||t |dWdS1s-wYdS)Nr?r@rBT)all_tensors_to_one_fileF) rCrDrrrrconvert_model_to_external_data save_modelrrF)rrErr4r4r5save_and_reload_model7s $rJcCs*t|rt}|||St|}|Sr/)r8r ModelProtoCopyFromrJ)r cloned_modelr4r4r5clone_model_with_shape_infer?s  rNcCs6|jtjjkr tj|Std|jdt |j)Nz&Only float type is supported. Weights z is ) data_typerrrr numpy_helperto_arrayr:r1 type_to_name)rr4r4r5tensor_proto_to_arrayHs  rScC|dS)N_QuantizeLinearr4 tensor_namer4r4r5add_quant_suffixQrXcC|tSr/)QUANT_INPUT_SUFFIXrVr4r4r5add_quant_input_suffixUrYr\cCrT)N_QuantizeLinear_Outputr4rVr4r4r5add_quant_output_suffixYrYr^cCrT)N_DequantizeLinearr4rVr4r4r5add_dequant_suffix]rYr`cCrT)N_DequantizeLinear_Inputr4rVr4r4r5add_dequant_input_suffixarYrbcCrZr/)DEQUANT_OUTPUT_SUFFIXrVr4r4r5add_dequant_output_suffixerYrd)NN)F)FF)r )ErrCenumrpathlibrrZrrrr onnxruntimerrr __producer__ __version__ onnx_domain ms_domain QUANT_OP_NAMEr[DEQUANT_OP_NAMErcTENSOR_NAME_QUANT_SUFFIXrRr-rErIrMrrrgrrXrkrwr~rzrrrrrrrrrrrr rr!r*r2r5r7r8r=boolrFrJrNrSrXr\r^r`rbrdr4r4r4r5s      + '   %    C