berteus-onnx

BERTeus ONNX (int8 + int4 quantized)

⚠️ This is an unofficial community conversion. This is not the original BERTeus repository. The original model was created by Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, and Eneko Agirre at the IXA NLP Group (University of the Basque Country, UPV/EHU).

🔗 Original model: ixa-ehu/berteus-base-cased

📄 Paper: Give your Text Representation Models some Love: the Case for Basque (LREC 2020)

---

What is this?

This repository provides ONNX-quantized versions of BERTeus, a monolingual BERT model pre-trained on Basque (Euskara). Two quantization levels are available:

• int8 (model_quantized.onnx, 119 MB) — dynamic int8 quantization
• int4 (model_q4.onnx, 85 MB) — weight-only int4 (MatMul) + int8 (embeddings)

Both enable efficient inference in:

• JavaScript / TypeScript via @huggingface/transformers (Transformers.js) — runs in the browser via WebAssembly
• Python via optimum with ONNX Runtime
• Any runtime that supports the ONNX format

Why use this instead of the original?

The original PyTorch model remains the best choice for training, fine-tuning, or GPU inference. Use this ONNX version when you need browser deployment, small download size, or fast CPU inference.

int4 vs int8 — which should I use?

Recommendation: Use int4 (dtype: 'q4') for browser deployment — it is smaller, faster to load, and produces equivalent or better quality on the re-ranking benchmark. Use int8 if you need maximum backward compatibility with older Transformers.js versions.

---

Model details

Note: The original BERTeus checkpoint was saved as BertModel (encoder only) — it does not include the masked LM (MLM) prediction head. This means BertForMaskedLM will produce random MLM outputs. For masked LM tasks, use the static word embeddings approach described below, or fine-tune an MLM head on the original model.

Quantization details

int4 (model_q4.onnx): Two-step quantization using ONNX Runtime's MatMulNBitsQuantizer:

1. Encoder MatMul ops → int4 (weight-only, block_size=128, asymmetric) via MatMulNBits operator
2. Embedding Gather ops → int8 (QDQ: DequantizeLinear) via dynamic quantization

This approach keeps the embedding table in int8 (not int4) to avoid GatherBlockQuantized, which is not implemented on the WASM backend. The MatMulNBits operator IS supported on WASM in Transformers.js v3+.

int8 (model_quantized.onnx): Standard dynamic int8 quantization using MatMulInteger + DynamicQuantizeLinear operators.

---

Files

Why a separate embedding file?

The ONNX model outputs contextual hidden states, but the static word embedding matrix (the input embedding layer) is useful for tasks like lexical similarity and candidate scoring. We extract it as a separate float16 binary file (74 MB) to avoid loading the full model when only embeddings are needed.

---

Usage

Python (ONNX Runtime via Optimum)

from optimum.onnxruntime import ORTModelForFeatureExtraction
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("itzune/berteus-onnx")
model = ORTModelForFeatureExtraction.from_pretrained("itzune/berteus-onnx")

inputs = tokenizer("Euskara munduko hizkuntzarik ederrena da.", return_tensors="pt")
outputs = model(**inputs)

# Contextual embeddings: [1, seq_len, 768]
hidden_states = outputs.last_hidden_state

JavaScript / Browser (Transformers.js)

int4 (recommended — smaller, faster):

import { AutoModel, AutoTokenizer, env } from '@huggingface/transformers';

const model = await AutoModel.from_pretrained('itzune/berteus-onnx', {
  dtype: 'q4',    // int4 quantization — 85 MB
  device: 'wasm',
});

const tokenizer = await AutoTokenizer.from_pretrained('itzune/berteus-onnx');

const inputs = tokenizer('Euskara mundiko hizkuntzarik ederrena da.', {
  truncation: true,
  max_length: 512,
  padding: false,
});

const outputs = await model(inputs);
// outputs.last_hidden_state.data → Float32Array of shape [1, seq_len, 768]

int8 (alternative — slightly larger):

const model = await AutoModel.from_pretrained('itzune/berteus-onnx', {
  dtype: 'q8',    // int8 quantization — 119 MB
  device: 'wasm',
});

Using the embedding matrix (lexical similarity)

import numpy as np
import struct

# Load float16 embeddings
with open("word_embeddings_f16.bin", "rb") as f:
    raw = np.frombuffer(f.read(), dtype=np.float16)

embeddings = raw.reshape(50099, 768).astype(np.float32)

# Get embedding for a token ID
token_id = tokenizer.convert_tokens_to_ids("euskara")
word_embedding = embeddings[token_id]  # 768-dim vector

// Browser: load and convert float16 → float32
const response = await fetch('word_embeddings_f16.bin');
const buffer = await response.arrayBuffer();
const uint16 = new Uint16Array(buffer);
const float32 = new Float32Array(uint16.length);
// Convert each float16 → float32 (bit manipulation, see source)

---

Use cases

✅ What this model is good for

❌ What this model cannot do

Masked embedding similarity (no MLM head needed)

Since the checkpoint lacks an MLM head, the recommended approach for "which word fits here?" tasks is:

1. Replace the target position with [MASK] (token ID 4)
2. Run the BERT encoder on the full sentence (bidirectional context)
3. Extract the [MASK] position's hidden state (768-dim)
4. For each candidate word, compute its static embedding (mean of subword piece embeddings)
5. Score by cosine similarity between the [MASK] hidden state and the candidate embedding

This is based on the lexical substitution approach from Paetzold & Specia (2017) and works well for spell-checker candidate re-ranking.

---

Performance

int4 vs int8 vs fp32 (933-case spell-correction re-ranking benchmark)

Both quantized variants slightly outperform fp32 PyTorch on this benchmark — quantization noise occasionally helps break ties in the re-ranking task. The int4 model is the best overall: smallest, fastest, and highest accuracy.

Browser validation (30-case subset, Transformers.js WASM)

The float16 embedding matrix has negligible precision loss compared to float32.

---

Citation

If you use this model, please cite the original BERTeus paper:

@inproceedings{agerri2020give,
  title     = {Give your Text Representation Models some Love: the Case for Basque},
  author    = {Rodrigo Agerri and Iñaki San Vicente and Jon Ander Campos and Ander Barrena and Xabier Saralegi and Aitor Soroa and Eneko Agirre},
  booktitle = {Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC)},
  year      = {2020}
}

---

License

The original BERTeus model is released under the Apache 2.0 license. This ONNX conversion is provided under the same license.

Acknowledgements

All credit for the BERTeus model goes to the IXA NLP Group at the University of the Basque Country (UPV/EHU). This repository is merely a format conversion for deployment convenience.