Downloads · 30 days
5
8% of all-time downloads
TonyRaju/pszemraj-grammar-synthesis-small-onnx-int8
pszemraj-grammar-synthesis-small-onnx-int8 is a machine learning model from TonyRaju. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc-by-nc-sa-4.0.
Critical Warning: INT8 quantization catastrophically degrades this model. BLEU drops from 57 to 3. All 50 test sentences produce incoherent output. The FP32 ONNX version works correctly. Do not use the INT8 version fo…
Downloads · 30 days
5
8% of all-time downloads
All-time downloads
62
Public
Repo size
150 MB
Likes
0
Public
Click a slice to open those files.
.onnx150 MB · 98%
From the Hugging Face model README
Critical Warning: INT8 quantization catastrophically degrades this model. BLEU drops from 57 to 3. All 50 test sentences produce incoherent output. The FP32 ONNX version works correctly. Do not use the INT8 version for any production application.
This is an ONNX INT8 dynamically quantized version of pszemraj/grammar-synthesis-small for grammatical error correction in English.
This is a derivative work. All credit for the original model goes to Peter Szemraj (@pszemraj). This repository only provides a quantized ONNX conversion for easier deployment. We do not claim ownership of the model architecture or weights.
| Original Repository | pszemraj/grammar-synthesis-small |
| Original Author | Peter Szemraj (@pszemraj) |
| Architecture | T5-small-lm-adapt (encoder-decoder) |
| Parameters | 77.0M |
| Training Data | Extended JFLEG |
| License | CC BY-NC-SA 4.0 / Apache 2.0 (dual-licensed in original) |
encoder_model.onnx — Encoder graph (INT8 quantized)decoder_model.onnx — Decoder graph (INT8 quantized)decoder_with_past_model.onnx — Decoder with KV-cache (INT8 quantized)config.json, generation_config.json — Model configurationtokenizer.json, tokenizer_config.json, special_tokens_map.json — Tokenizer files| Step | Detail |
|---|---|
| Export | PyTorch → ONNX FP32 via HuggingFace Optimum (ORTModelForSeq2SeqLM) |
| Quantization | onnxruntime.quantization.quantize_dynamic with QuantType.QInt8 |
| Validation | Verified ONNX FP32 output matches PyTorch FP32 exactly (0 delta) before quantizing |
Benchmarked on a 50-sentence English grammar correction test set (2-thread CPU, Google Colab).
| Format | BLEU | chrF++ | Exact Match | Δ BLEU | Δ chrF++ |
|---|---|---|---|---|---|
| PyTorch FP32 (baseline) | 57.11 | 77.13 | 28% | — | — |
| ONNX INT8 (this repo) | 3.15 | 18.72 | 0% | -53.96 | -58.42 |
| Metric | PyTorch FP32 | ONNX INT8 (this repo) |
|---|---|---|
| P50 Latency | 363 ms | 138 ms |
| P95 Latency | 629 ms | 199 ms |
| Disk Size | 294 MB | 153 MB |
| Throughput | — | 13.70 sentences/sec |
| Metric | Value |
|---|---|
| Sentences with changed output (INT8 vs FP32) | 50/50 sentences changed |
| Deployment Scorecard Verdict | NO-GO |
Provided for completeness and research purposes only.
This model was benchmarked alongside 6 other grammar correction models. All models were evaluated on the same 50-sentence test set under identical conditions.
| Model | Params | BLEU | chrF++ | P50 (ms) | Size (MB) | Exact Match |
|---|---|---|---|---|---|---|
| prithivida | 222.9M | 76.88 | 86.40 | 602 | 850 | 58% |
| coedit-small | 60.5M | 77.42 | 86.84 | 226 | 231 | 60% |
| vennify-t5-base | 222.9M | 78.57 | 87.87 | 539 | 850 | 58% |
| aventiq-t5-small | 60.5M | 34.43 | 45.88 | 213 | 231 | 16% |
| visheratin-mini | 31.2M | 66.84 | 81.77 | 133 | 119 | 46% |
| visheratin-tiny | 15.6M | 66.08 | 82.69 | 85 | 59 | 48% |
| pszemraj-small ← | 77.0M | 57.11 | 77.13 | 363 | 294 | 28% |
| Model | Params | BLEU (FP32) | BLEU (INT8) | chrF++ (INT8) | P50 (ms) | Disk (MB) | Exact Match | Scorecard |
|---|---|---|---|---|---|---|---|---|
| prithivida | 222.9M | 76.88 | 74.44 | 84.79 | 222 | 426 | 54% | NO-GO |
| coedit-small | 60.5M | 77.42 | 77.60 | 86.69 | 117 | 152 | 60% | GO |
| vennify-t5-base | 222.9M | 78.57 | 77.24 | 86.74 | 224 | 426 | 56% | GO |
| aventiq-t5-small | 60.5M | 34.43 | 40.66 | 51.95 | 120 | 152 | 22% | GO* |
| visheratin-mini | 31.2M | 66.84 | 66.65 | 82.67 | 78 | 93 | 46% | GO |
| visheratin-tiny | 15.6M | 66.08 | 64.51 | 82.28 | 60 | 55 | 44% | GO |
| pszemraj-small ← this model | 77.0M | 57.11 | 3.15 | 18.72 | 138 | 153 | 0% | NO-GO |
Key cross-model comparisons (INT8 variants):
| Model A | Model B | chrF++ A | chrF++ B | p-value | Significant? |
|---|---|---|---|---|---|
| coedit-small | vennify-t5-base | 86.69 | 86.74 | 1.000 | No (identical) |
| coedit-small | visheratin-mini | 86.69 | 82.67 | 0.038 | Yes |
| coedit-small | visheratin-tiny | 86.69 | 82.28 | 0.010 | Yes |
| coedit-small | prithivida | 86.69 | 84.79 | 0.324 | No |
| visheratin-mini | visheratin-tiny | 82.67 | 82.28 | 0.824 | No (identical) |
| prithivida | vennify-t5-base | 84.79 | 86.74 | 0.324 | No |
| Use Case | Recommended Model | Why |
|---|---|---|
| Best overall (desktop) | coedit-small INT8 (152 MB) | Highest INT8 quality (chrF++ 86.69), 117ms latency, only 1 regression |
| Smallest (mobile/edge) | visheratin-tiny INT8 (55 MB) | 60ms latency, 55 MB disk, acceptable quality (chrF++ 82.28) |
| Highest baseline quality | vennify-t5-base INT8 (426 MB) | chrF++ 86.74, but 3x larger than coedit-small for no significant quality gain |
from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer
model_id = "YOUR_USERNAME/pszemraj-grammar-synthesis-small-onnx-int8"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = ORTModelForSeq2SeqLM.from_pretrained(model_id)
text = "(none — empty string)She go to school yesterday"
inputs = tokenizer([text], return_tensors="pt", max_length=128, truncation=True)
outputs = model.generate(**inputs, max_new_tokens=128, num_beams=1, repetition_penalty=1.3)
corrected = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(corrected)
Input prefix: (none — empty string)
All credit for the original model goes to Peter Szemraj (@pszemraj). This repository only provides an ONNX INT8 quantized conversion to make the model easier to deploy in production environments (mobile, edge, browser).
Quantization and benchmarking performed as part of the Smart Desktop Keyboard Grammar Engine project.
If you use this model, please cite the original authors:
@misc{pszemraj_small_onnx_int8,
title = {pszemraj/grammar-synthesis-small — ONNX INT8 Quantized},
note = {Quantized version of pszemraj/grammar-synthesis-small},
url = {https://huggingface.co/pszemraj/grammar-synthesis-small},
}