onnx-community-BERT-tiny-RAID-int8-uint8-onnx

ONNX Community BERT Tiny RAID Text Detector INT8/UINT8 ONNX

INT8/UINT8 ONNX version of ONNX Community BERT Tiny RAID Text Detector for text classification.

• Quantization: Dynamic 8-bit quantization with INT8 weights and UINT8 dynamic activations using DynamicQuantizeLinear and MatMulInteger

Model Files

Parameter Summary

Stored initializer elements includes weights, biases, quantization scales, zero-points, and other constant tensors. It is not a trainable-parameter count.

Original Model Inference

The original artifact is already ONNX, so the ONNX example below applies.

The upstream artifact is already ONNX. The model.onnx file above is the source-compatible downloadable model for this variant.

Converted ONNX Inference

pip install huggingface_hub numpy onnxruntime

import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download

repo_id = "ketiswp/onnx-community-BERT-tiny-RAID-int8-uint8-onnx"
model_path = hf_hub_download(repo_id=repo_id, filename="model.onnx")

options = ort.SessionOptions()
options.intra_op_num_threads = 1
options.inter_op_num_threads = 1
options.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
session = ort.InferenceSession(
    model_path,
    sess_options=options,
    providers=["CPUExecutionProvider"],
)

dtype_by_ort_type = {
    "tensor(float)": np.float32,
    "tensor(double)": np.float64,
    "tensor(float16)": np.float16,
    "tensor(int64)": np.int64,
    "tensor(int32)": np.int32,
    "tensor(int16)": np.int16,
    "tensor(int8)": np.int8,
    "tensor(uint8)": np.uint8,
    "tensor(bool)": np.bool_,
}
feeds = {}
for item in session.get_inputs():
    shape = [dim if isinstance(dim, int) and dim > 0 else 1 for dim in item.shape]
    feeds[item.name] = np.zeros(shape, dtype=dtype_by_ort_type[item.type])

outputs = session.run(None, feeds)
print([(item.name, value.shape, str(value.dtype))
       for item, value in zip(session.get_outputs(), outputs)])

Paired Model

FP32 version

Source

• Original artifact
• Source project

Project Validation

FP32/quantized comparison, conversion results, and reproduction code

This repository also includes the Netron graph, ONNX Dialect MLIR, and static MLIR dependency graph for this model variant.