Gliner-small-edge-wo-int8-emb4

GLiNER small v2.5 Edge — weight-only INT8 (ONNX)

A smaller ONNX build of GLiNER small v2.5, a zero-shot named-entity recognition model: give it text plus any list of labels you want ("person", "api key", "ticket id", whatever), and it finds matching spans without being trained on those labels ahead of time.

This build is 131.9 MB — 58% smaller than the standard fp16 ONNX export (317 MB) — with no accuracy loss measured on our test set. It scores identically to fp16 (19/22 recall) and misses the exact same three entities. This is not the same trick as the existing patronus-studio/gliner_small-v2.5-edge int4/int8 build, which is a similar size (127 MB) but loses more than half its recall (10/22) because of how it was quantized — see Why this exists below.

Quick start

pip install gliner onnxruntime huggingface_hub

from gliner import GLiNER

model = GLiNER.from_pretrained(
    "Tazner/Gliner-small-edge-wo-int8-emb4",
    load_onnx_model=True,
    load_tokenizer=True,
    onnx_model_file="model.onnx",
)

text = "Contact Jane Doe at [email protected] or +1 415 555 0100."
labels = ["person", "email", "phone number"]

entities = model.predict_entities(text, labels, threshold=0.4)
for e in entities:
    print(e["text"], "->", e["label"], round(e["score"], 2))

That's it — from_pretrained downloads and caches this repo the same way it would any other model on the Hub.

Why this exists

The standard way to shrink an ONNX model is dynamic quantization (onnxruntime.quantization.quantize_dynamic), which compresses both the model's stored weights and the numbers flowing through it at runtime (activations) down to INT8.

That second part is the problem here. This model's backbone, microsoft/deberta-v3-small, uses an attention mechanism ("disentangled attention") that produces occasional very large activation values. Squeezing those into INT8's 256 buckets means the range has to stretch to cover the outliers, which crushes all the ordinary values into a couple of adjacent buckets — the model effectively goes half-blind. That's measurably what happens to the existing edge build: recall drops from 19/22 to 10/22.

This build instead uses weight-only quantization: only the numbers stored on disk are compressed. Activations stay in full precision the entire time the model is running, so none of that outlier damage happens. You get most of the size reduction, none of the accuracy loss — at the cost of somewhat higher latency, because the runtime has to decompress each weight back to a float before using it (see Trade-offs).

Benchmark

Measured with the 22-entity multilingual hard test in benchmark.py (multi-paragraph text with a JWT, an API key, an internal IP, a URL, and names in five different naming conventions), 8 timed runs after 2 warmup runs, CPU-only (onnxruntime CPUExecutionProvider, Intel64 Windows 11):

This model and the fp16 source miss the exact same three gold entities (Deutsche Bahn AG, INFRA-4821, +65 6789 0123) — the match isn't a coincidental tie, it behaves like the same model.

Trade-offs

• Slower than fp16 and slower than the int4/int8 edge build (512 ms vs 375 ms vs 236 ms mean latency in the benchmark above). Weight-only quantization saves disk space, not compute — every quantized weight has to be unpacked back to a float before each matmul, which costs CPU time. If your deployment is latency-bound rather than disk-bound, the fp16 source or the standard edge build may suit you better.
• Benchmarked on one CPU, one short multilingual text, 22 gold entities. It's a real test with real multilingual and structured-data coverage (PII patterns, five naming conventions), but it's not a large-scale NER benchmark. Validate on your own data before relying on this in production.
• Embeddings are quantized to 4-bit, not 8-bit — onnxruntime's weight-only quantizer only supports 4-bit for Gather ops (the embedding lookup), 8-bit isn't an available option there. This turned out not to cost any measured recall in our test, but it's worth knowing the embedding table has more aggressive compression than the attention/FFN weights do.

How it was built

Starting from patronus-studio/gliner_small-v2.5-edge's onnx/fp16/model_fp16.onnx:

1. MatMul weights → INT8, block-wise (block size 128, symmetric), via onnxruntime.quantization.matmul_nbits_quantizer.MatMulNBitsQuantizer. Activations are untouched — this only rewrites what's stored on disk.
2. Embedding table (Gather) → INT4, block-wise, same tool, separate pass (the quantizer requires 4-bit specifically for Gather ops).
3. Everything not touched by those two passes (biases, layer norms, the small classification heads) stays at the source's original fp16.

The full script (quantize.py) and the benchmark used to validate it (benchmark.py) are part of the working project this model was built in; ask the uploader if you'd like them.

Files

Attribution

The upstream Apache-2.0 license and its notices are retained. The MIT terms continue to apply to the portions originating from the backbone model.

Changes made to the source work

In accordance with Section 4(b) of the Apache License 2.0, the modifications relative to patronus-studio/gliner_small-v2.5-edge's onnx/fp16/model_fp16.onnx are:

• Applied weight-only block quantization: INT8 for MatMul weights, INT4 for the token-embedding table, both block-wise with block size 128. Runtime activations are left in float precision (this is the key difference from the source repo's own quantized variant, which uses dynamic quantization that also compresses activations).
• No retraining, fine-tuning, or change to model architecture or learned behavior beyond the numerical effects of quantization.

License

This work is distributed under the Apache License 2.0, the same license as the upstream model. A full copy is included as LICENSE in this repository.

Copyright the original GLiNER authors and contributors.
Modifications (fp16 ONNX export) Copyright Casdo Labs GmbH.
Further modifications (weight-only INT8/INT4 quantization) for this repository.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.