QuantizedSarcBERT-static-v2

QuantizedSarcBERT-static-v2

INT8 statically quantized ONNX build of Vandita/Bert-finetuned-Sarc, for CPU inference.

Padding invariance — read this before serving

Activation scales are frozen at calibration time, so they cannot change with input shape. Measured on this build:

pad-sensitivity = max change in P(sarcastic) for the same text alone vs padded, at batch size 1. mixed-batch = same text alone vs inside a mixed-length batch.

Dynamic PTQ recomputes activation scales from each tensor's observed range at inference time. Padded positions enter that range, so they change the scale, so they change how the real tokens are quantized — the attention mask stops padded positions contributing to attention, but not to the quantization range. Static quantization removes the dependence entirely.

Results

SarcOjiTest1

SarcOjiTest2

SarcOjiTest2 is ~75/25 negative-skewed; MCC and ROC-AUC are the meaningful columns there.

Performance (x86 CPU, single thread, batch-of-1)

Not an on-device ARM measurement.

Quantization recipe

• Static PTQ, QDQ format, per-channel weights
• Calibration: percentile on 128 in-domain SarcOji samples, padded as deployed
• Ops quantized: MatMul and Gemm only (where BERT's compute is; leaving Add/LayerNorm/Softmax in float preserves accuracy at negligible speed cost)
• Exported with eager attention, dynamic batch and sequence axes, opset 17

Usage

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Vandita/QuantizedSarcBERT-static-v2")
model = ORTModelForSequenceClassification.from_pretrained(
    "Vandita/QuantizedSarcBERT-static-v2", file_name="model_int8_static.onnx")

inputs = tok(["Oh great, another meeting."], return_tensors="pt", padding=True)
logits = model(**inputs).logits   # batching is safe on this build