gector-eus-onnx

GECToR-eus ONNX (int4) — Unified v3

ONNX int4 export of the GECToR-eus unified v3 model — a GECToR grammar correction model fine-tuned on RoBERTa-eus-base for Basque grammatical error correction. This is the unified model, trained on the Elhuyar GEC corpus plus augmented data covering spelling, capitalization/punctuation, proper nouns, and real-word confusions. Trained by Itzune as part of the gector-eus project.

For the PyTorch checkpoint (fine-tuning/research), see itzune/gector-eus.

Overview

This model corrects real-word grammar errors in Basque — cases where every word is a valid dictionary word but the inflection is wrong in context (e.g. dio → zaio, zaidalaren → zaidalako, etortzen → etorriko). It also provides per-word error detection via a dedicated detect head (P(INCORRECT) per token).

Performance

Elhuyar benchmark (F0.5 via M² scorer)

Evaluated on the Elhuyar Dem_single (221 errorful sentences) and Dem_none (250 clean sentences) benchmarks. The unified v3 model achieves F0.5=33.8 on eval_gector.py (token-level exact-match scoring), comparable to the v1 grammar-only model (F0.5=34.1). The M² scorer gives higher absolute numbers due to a different alignment method.

Unified benchmark (220-case, 10 categories)

The v3 unified model was evaluated on a 220-case benchmark spanning 10 error categories (spelling, cap-punct, grammar, real-word, multi-error, paragraph, markdown, mixed, clean):

The hybrid configuration — unified v3 for grammar, MarianMT for cap-punct, Hunspell+BERTeus for spelling — is the best-performing setup. Key v3 improvements over v1: real-word error detection (+50%), grammar correction (+12.5%).

For comparison, GECToR-2024 (English, RoBERTa-large 300M) scores F0.5=72.9 on BEA-dev.

Files

Usage

JavaScript (Transformers.js / ONNX Runtime Web)

import { InferenceSession, Tensor } from 'onnxruntime-web';

// Load model
const session = await InferenceSession.create('/models/gector/onnx/model_q4.onnx', {
  executionProviders: ['wasm'],
});

// Load vocab
const vocab = await fetch('/models/gector/gector_vocab.json').then(r => r.json());

// Run inference
const inputIds = new Tensor('int64', BigInt64Array.from(ids.map(BigInt)), [1, len]);
const attentionMask = new Tensor('int64', BigInt64Array.from(mask.map(BigInt)), [1, len]);
const outputs = await session.run({ input_ids: inputIds, attention_mask: attentionMask });

// outputs.logits_labels  → [1, seq_len, 5000]  (label predictions)
// outputs.logits_d       → [1, seq_len, 2]     (detection: CORRECT vs INCORRECT)

Python (ONNX Runtime)

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession('onnx/model_q4.onnx')
outputs = session.run(None, {
    'input_ids': np.array([input_ids], dtype=np.int64),
    'attention_mask': np.array([attention_mask], dtype=np.int64),
})
logits_labels = outputs[0]  # [1, seq_len, 5000]
logits_d = outputs[1]       # [1, seq_len, 2]

How it works

GECToR uses an edit-based approach — instead of generating text, it predicts a label per token:

• $KEEP — leave this word unchanged
• $DELETE — remove this word
• $REPLACE_xxx — replace this word with xxx
• $APPEND_xxx — append xxx after this word

The model runs iteratively (up to 5 passes): each pass applies the predicted edits, then re-runs on the corrected text until no more changes are needed.

The detect head provides an independent P(INCORRECT) per token — a confidence score for whether each word is wrong. This is used as a threshold (min_error_prob) to control the precision/recall tradeoff, and also powers error detection visualizations.

int4 Quantization

The int4 model uses a two-step quantization process:

1. int4 MatMul weights — MatMulNBitsQuantizer (bits=4, block_size=128, asymmetric) quantizes all 96 MatMul operations in the RoBERTa encoder to 4-bit. This reduces the encoder weight matrix from 147 MB (fp32) to ~20 MB.
2. int8 embeddings — quantize_dynamic quantizes the 3 Gather operations (token/position embeddings) to int8. This avoids GatherBlockQuantized which is not supported on the WASM backend.

Result: 83 MB (int4) vs 122 MB (int8) vs 487 MB (fp32) — 83% smaller than fp32, 32% smaller than int8, with no accuracy loss (99.8% token label match vs PyTorch).

Training data

Elhuyar/Orai GEC corpus (1M pairs)

The primary training data is the Elhuyar GEC corpus, created by the Elhuyar Foundation (now Orai NLP) and distributed under CC-BY-NC-SA.

• Source: Correct sentences were extracted from a Basque news corpus compiled from Berria.eus, Argia.eus, and the Tokikom.eus proximity-media network (500,015 news items, 4,927,748 sentences, ~66M words).
• Error generation: Synthetic grammar errors (R1–R4: tense, verb agreement/argument, case/agreement, suffix) were introduced by rule application.
• Scale: 9.3M sentence pairs total (4.71M with errors, 5.29M clean). We trained on 1M pairs.
• Distribution: Orai.eus resources page / HiTZ corpus page

Augmented data (234k pairs)

To create a unified model that handles multiple error types, we augmented the Elhuyar corpus with 234k synthetic pairs generated from the ZelaiHandi news corpus (professionally edited Basque texts):

The augmented data introduces new labels (e.g. $APPEND_!, $APPEND_,, $TRANSFORM_CASE_CAPITAL) that the v1 grammar-only model did not have.

ℹ️ This is distinct from the Elhuyar web corpus (186M tokens, Leturia 2014), a separate general web-crawl resource on Orai's resources page that is not the source of the GEC data.

Training

• Base model: ixa-ehu/roberta-eus-euscrawl-base-cased
• Training data: 1.23M pairs (1M Elhuyar GEC + 234k augmented, see above)
• Epochs: 5 trained (best at epoch 3, early stopping on valid_loss)
• Cold epochs: 2 (lr=1e-3)
• Fine-tune lr: 1e-5, constant with warmup
• Batch size: 32
• Max length: 128 tokens
• GECToR implementation: gotutiyan/gector
• Training code: itzune/gector-eus

Related

• itzune/gector-eus — PyTorch checkpoint (487 MB, for fine-tuning/research)
• itzune/gector-eus — Training code and scripts
• txukun — Browser-based Basque text corrector using this model

Credits

• RoBERTa-eus-base was pre-trained by the IXA NLP Group (UPV/EHU) — see Artetxe et al. (2022)
• GECToR was introduced by Omelianchuk et al. (2020) (Grammarly)
• Elhuyar GEC corpus was created by the Elhuyar Foundation (now Orai NLP) and HiTZ Zentroa (UPV/EHU)
• GECToR implementation: gotutiyan/gector (MIT)

License

CC-BY-NC-SA 4.0

This model's weights are a derivative work of the Elhuyar GEC corpus, which is licensed under CC-BY-NC-SA. Under the ShareAlike clause, the trained model weights are distributed under the same license:

• NC (NonCommercial) — You may not use this model for commercial purposes.
• SA (ShareAlike) — If you remix, transform, or build upon this model, you must distribute your contributions under the same license.

The base encoder (RoBERTa-eus-base, Apache 2.0) and the GECToR architecture (gotutiyan/gector, MIT) are separately licensed, but the trained weights inherit the CC-BY-NC-SA restriction from the training data.

The ONNX export code and inference pipeline (itzune/gector-eus) are MIT licensed.

Citation

If you use this model, please cite the original works:

@inproceedings{beloki2020gec,
  title     = {Grammatical Error Correction for Basque through a seq2seq neural architecture and synthetic examples},
  author    = {Beloki, Zuhaitz and Saralegi, Xabier and Ceberio, Klara and Corral, Ander},
  booktitle = {Proceedings of the 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020)},
  address   = {Granada, Spain},
  year      = {2020}
}

@article{omelianchuk2020gector,
  title={GECToR--Grammatical Error Correction: Tag, Not Rewrite},
  author={Omelianchuk, Kostiantyn and Atrasevych, Vitaly and Chernodub, Artem and Skurzhanskyi, Oleksandr},
  journal={arXiv preprint arXiv:2005.12592},
  year={2020}
}

@article{artetxe2022roberteus,
  title={Roberteus: a monolingual basque language model},
  author={Artetxe, Mikel and Adebisi, Iyanuoluwa and Azkarate, Mikel and Ceberio, Itziar and Campos, Jon Ander and Esnal, Gorka and Fernandez de Landa, Oscar and Goikoetxea, Joseba and Gutierrez, Aitor and Igondi, Maite and others},
  journal={Procesamiento del Lenguaje Natural},
  volume={69},
  pages={27--34},
  year={2022}
}