joytag-onnx-Q8_0-quantized

joytag-onnx-Q8_0-quantized

Q8_0 Quantized JoyTag Model, working well with merely 88MiB in size (74.9% smaller), using 200+ MiB less RAM and achieving ~10% performance improvement.

It's fully compatible with your existing ONNXRuntime code. Replace the model file and it will work.

[!WARNING] Precision Loss is expected. You may get unwanted outcome when using this model.

Deploying as a Server

You can refer to joytag-node, a simple Node.js implementation for serving JoyTag by me.

It will use approximately 750MiB of RAM (when using this quantized model), which is relatively a small amount. And it's fairly easy to run.

Dynamic Quantizing Used

Pre-process the original model for quantizing:

python3 -m onnxruntime.quantization.preprocess --input original-model.onnx --output input.onnx

Quantize the pre-processed model using Q8_0:

import onnx
from onnxruntime.quantization import quantize_dynamic, QuantType

quantize_dynamic(
        model_input="input.onnx",
        model_output="output.onnx",
        weight_type=QuantType.QInt8
)

License

This project is licensed under Apache-2.0 License, as the same as the original model.