plgec-herbert-large-v2

plgec-herbert-large-v2 — Polish grammar error correction

plgec-herbert-large-v2 is the second versioned release of a GECToR-style token-edit classifier over HerBERT-large. It reads a Polish sentence and proposes minimal, conservative edits (replace, append, delete), each tagged with an error class (orth, infl, agree, punct, conf, spell) and a raw confidence. It is an advisory WARN-level gate for document pipelines: flag suspected errors for a human, never rewrite on its own.

The v2 checkpoint is a continued fine-tune of the v1 model, trained to stop flagging the domain it kept misfiring on (the numbers are in the domain-adaptation section below). The decode got more permissive and the per-class filter moved to schema version 3; both changes are disclosed in full, next to the calibration and ablation evidence that shaped them. What did not change is the design goal: a low spurious-edit rate you can trust as a linter, per-edit classes you can route, and honest per-class numbers — including the weak classes, which stay weak and are documented as such.

This card is generated from committed measurement manifests; no number on it was typed by hand. The one deliberate gap is the re-benchmark verdict slot in its own section: that number arrives from a once-only measurement on real documents and will be recorded whatever it shows.

Model identity

What changed from v1

The decode pair moved from 0.30/0.30 to 0.25/0.20 on a fresh 135-combination sweep over this checkpoint, selected on validation F0.5 under the inherited spurious gate; the pairing is more permissive, and the per-class bars in thresholds.yaml still filter consumer-side after decoding.

Consumers pin the v2.0.0 tag, not the default branch. The v1 repository stays online and unchanged; its corrigendum points here.

Results

One instrument: the frozen 180-sentence curated test set, token-level detection, attribution-derived per-class precision. The v1 card reported detection and char-span correction side by side; v2 reports the detection instrument with Wilson 95% intervals and refers the correction story to the domain-adaptation and conf sections, where the v2 work actually moved the numbers.

The measurement decode for this table is the v1-published stack (bias 0.30, min error probability 0.30, v1 temperature), so every v2-versus-v1 delta below compares models, not decode settings. The shipped decode differs and is disclosed in the changes section. Per-class detection precision is attribution-derived: an edit counts toward a class only when the gold edit at that position is of the same class.

Movement against v1, same instrument

Positive deltas are improvements. Worst absolute per-class F0.5 drop accepted by the release gate: 0.23 points (bar: 3 points per class, F0.5 and recall separately).

orth is the headline: the v1 card's accepted flagship regression is gone, replaced by the largest per-class gain on the card. infl gives back a fraction of a point of F0.5 (within the release bar of 3 points per class); no other class moved against v1 by more than a rounding step in the wrong direction. The gate that judged this table ran lexicographically over both F0.5 and recall drops per class, accepted the worst absolute drop of 0.23 points, and fired no retry.

A caveat carried over from v1: the curated test set has been read end-to-end several times across the project (selection, re-gate, this eval). The v2 numbers were not selected on it, but the v1 baseline they sit beside was. Read the point estimates with that history in mind; the intervals do not undo it.

Confidence calibration

v2 re-fit the confidence temperature on this checkpoint's own logits and shipped identity, T = 1.0. The ship gate has two legs and the fit passed only one:

• Wilson leg — passed. On the 4,000-sentence clean carve-out (80,371 candidate positions), 3,870 sentences had lower weighted |confidence − accuracy| after the fitted T: improvement proportion 0.968 (95% CI [0.962, 0.973], n=4,000) — the interval excludes 0.5.
• ECE leg — failed. Carve-out expected calibration error rose from 0.00175 to 0.00195 instead of decreasing.

A temperature that sharpens sentence-level improvement while making the calibration error worse is not a calibration; the gate returned NO-SHIP. The fitted value (T = 0.8360, L-BFGS, 57 iterations) is recorded here and in the evidence manifest, and is not applied. Nothing was inflated to convert the verdict. Because the shipped constant is identity, the confidences on this card and in the decoder output are raw softmax probabilities.

The conf class: a marginal verdict, shipped disabled

The paired gate re-measured the confusable class at the shipped configuration and returned a marginal verdict (CONF_GATE_MARGINAL). The two gated axes:

• char-span precision 0.44 (95% CI [0.19, 0.73], n=9) against the pre-registered bar 0.30 — the point estimate clears the bar, the interval's lower bound does not. At this sample size the measurement cannot certify the bar; that is the verdict, not a near-miss to be rounded up.
• token recall 0.17 (95% CI [0.07, 0.34], n=30), point-identical to its v1 baseline of 0.17 — recall did not move.

The cost side: 16 of 20 clean-text spurious edits were conf-class (80% of the clean false-positive budget).

So conf ships disabled by default — the same consumer contract as v1, now with a schema-v3 floor key so an operator who opts in can demand extra confidence on top of the class bar. The documented path to revisiting the class is a dedicated real-text confusable mining pass for a future release; a threshold tweak is not that pass and was not attempted.

Domain adaptation: what v2 actually fixed

v1's worst live behavior was register mismatch: on a frozen 2,000-sentence carve of in-domain production Polish it flagged 114 sentences (0.057). The v2 checkpoint is a continued fine-tune of v1 over mined false-positive sites from that same domain. Two instruments measure what it bought, and they are not equal witnesses:

• Held-out carve (the honest number): flag rate fell to 0.007 (14 of 2,000 sentences) — an 87.7% relative reduction (114 → 14). The carve was frozen before Stage-IV training began, and the training never saw it.
• Mined false-positive sites (an upper bound by construction): 10,061 sites flagged by v1 fell to 387 under v2 — 96.15% suppressed. These sites are the training source, so this number says the lesson was learned, not generalized.

Both are domain-side instruments on one production domain; the re-benchmark section below is the in-the-wild verdict on documents the model has no relationship with.

RoBERTa encoder ablation

Before freezing v2, a Polish RoBERTa-large encoder was trained through the same pipeline as an ablation, to answer the v1 regression question directly. The answer is mixed and the candidate is not shipped:

• orth improves under RoBERTa at both comparability surfaces: F0.5 +0.80 pp and recall +5.26 pp (stage-3 leg); F0.5 +0.80 pp and recall +5.26 pp (stage-4 leg).
• infl regresses at both surfaces: F0.5 -6.61 pp and recall -6.25 pp (stage-3 leg); F0.5 -7.16 pp and recall -9.38 pp (stage-4 leg).
• The per-class gate verdicts were cliffs at both legs (agree and punct fell beyond the 8-point bound), and strict detection precision fell 18.91 pp (stage-3 leg) and 22.32 pp (stage-4 leg) against the shipping model.

The recommendation recorded in the ablation evidence was to keep HerBERT-large and not extend the re-benchmark to the RoBERTa candidate; that reading stands. The full four-column comparison lives in the project's ablation evidence manifest, which is not redistributed with this model.

INT8 position

No INT8 quantization of this checkpoint exists. The v1 repository documents a gate-rejected INT8 attempt for the v1 graph — nothing quantized was published there either, and that evidence stays with v1. This release ships the fp32 ONNX graph and the schema-v3 thresholds file; there is no quantized sibling to download and none is claimed.

Re-benchmark on real documents

Measured once, read-only, on real production documents (3 docs, 1,962 sentences after extraction): 35 flagged sentences (0.0178 overall).

Spot check (class-balanced, n=32 flags): strict precision 0.250 (95% CI [0.133, 0.421]); precision counting debatables 0.375. Under-powered classes are listed inside the spot-check artifact, never padded.

Parity: the in-container fp32 ONNX decode agreed with the live service on 100.0% of sentences (0 disagreements recorded). Latency at report scale: 102.5 ms per sentence amortized.

No URL false-edit occurrences were observed.

The punctuation blocklist

Carried verbatim from v1 and still enabled: an exact-token rule that drops a ; → , replacement (class punct, reason punct_style_swap) after decoding, in the same filter stage as the per-class bars — it never re-decodes. A context guard keeps the rule from firing near a clause start (min_tokens_before: 3), so short enumerations keep their gold semicolons. Literary-register consumers who want the swaps back flip one line of yaml (enabled: false).

Limitations

• The confusable class is disabled by default on a marginal verdict; the full Wilson disclosure is above. Enabling it opts into the class that produced 80% of the clean-text false positives measured at the shipped configuration.
• The curated evaluation set was machine-verified against the generator and LanguageTool but never reviewed by a human linguist, and it has been read end-to-end several times across the project's history (the v1 baseline numbers it is compared against were themselves selected on it).
• The per-class table is the token-detection instrument; v2 did not re-measure char-span correction on the curated set. The domain-adaptation section is the correction-side evidence for v2.
• Domain adaptation tuned against one production domain's false positives; the re-benchmark verdict above covers three real documents, and registers beyond them remain unmeasured.
• The decode is deliberately single-pass (max_iters 1): multi-error sentences get at most one edit per pass.
• Polish only; the decoder emits nothing useful for other languages.

Intended use

A WARN-level advisory gate for Polish prose in document pipelines: each sentence gets zero or more flagged edits with class, span, and confidence; a human accepts or dismisses. Conservative by construction — the decoder only emits an edit when its raw probability clears a high bar. Not suitable for unattended rewriting, for literary editing, or for languages other than Polish. Self-hosted LanguageTool remains the deterministic first gate; this model is the neural second opinion on the error classes rules structurally miss.

License

The model weights are published under CC BY-NC 4.0 (non-commercial, attribution). The base encoder, allegro/herbert-large-cased, is CC BY 4.0. Training included a licensed corpus that is not redistributed here; no datasets ship with this model.

Citation

@software{plgec_herbert_large_v2_2026,
  title  = {plgec-herbert-large-v2: a Polish GECToR grammar-error
            corrector over HerBERT-large (second release)},
  author = {Krystian Lewandowski},
  year   = {2026},
  url    = {https://huggingface.co/citan/plgec-herbert-large-v2},
  note   = {Per-class detection results with 95% CIs on a 180-sentence
            curated Polish set; conservative decode, per-class
            thresholds, domain-adapted continued fine-tune}
}

Usage

Minimal standalone inference — public dependencies only (onnxruntime, tokenizers; no deep-learning framework, no serving stack):

pip install onnxruntime tokenizers
hf download citan/plgec-herbert-large-v2 --revision v2.0.0 --local-dir plgec-herbert-large-v2
cd plgec-herbert-large-v2
python inference_example.py "Mieszkam w Warszawa od pięciu lat."

inference_example.py ships in this repository, next to this card. It loads the ONNX model, tokenizer, label vocabulary and thresholds.yaml from its own directory and prints one JSON object per sentence — per edit: token index, character span on the source text, original → replacement, tag, class, and confidence (the chosen tag's probability under the published decode settings and identity temperature).

thresholds.yaml is the consumer-side filter, applied after decoding. Schema version 3 carries: per-class min_error_probability and enabled keys (conf off by default), an optional per-class confidence floor inside the conf block, a calibration block with the shipped temperature, and a blocklist section with the ; to , rule. Edit the shipped file (or pass --thresholds with your own, or --thresholds none) — the model decode itself never changes:

schema_version: 3
default:
  min_error_probability: 0.30
  enabled: true
conf:
  min_error_probability: 0.30
  enabled: false   # opt in to confusable-class edits
  floor: 0.30      # additive second bar (schema v3)
# calibration + blocklist sections follow in the shipped file

Runtimes that read this file must support schema version 3; a v1-era runtime refuses the file rather than misreading it. Consumers coming from v1: pin this repository's tag, load the thresholds.yaml that ships with it, and expect the conf class to stay off unless you switch it on.

Note the corrected-text field joins edit-applied tokens with single spaces; consumers that care about typography should apply the per-edit char_spans to the source text instead.

Training lineage

Fourteen runs, in order: seven that built v1, seven that produced and judged this checkpoint. IDs are internal experiment-registry run identifiers; each row was verified finished at release time. The released checkpoint is the Stage-IV epoch-1 model, continued from the v1 (Stage-III) checkpoint, and exported to ONNX with 100% argmax agreement.

Two early smoke attempts failed before the successful Stage-IV train run; they are recorded as exceptions in the evidence chain, not hidden.

Data and training

Training pairs (~1.2M) were generated by injecting synthetic errors into clean Polish sentences (CC100-pl and Polish Wikipedia sources, plus a licensed corpus staged internally; no dataset is redistributed with this model). Errors come from a Morfeusz2-driven generator: inflection and agreement corruptions derived from morphological analysis, curated confusable-pair swaps, single-character orthography swaps (ż/rz, ó/u, ch/h, ą/ę), punctuation, and spelling. The evaluation set is the frozen 180-sentence curated set inherited from v1, machine-verified against the generator and LanguageTool but never reviewed by a human linguist. On top of the v1 mixture, v2 adds a continued fine-tune over mined false-positive sites from in-domain text — the suppression evidence in the domain-adaptation section is what that pass was trained to produce.

The GECToR label vocabulary grew from 6,680 to 6,757 tags — an append-only extension that keeps every v1 id in place, so the two vocabularies load side by side without aliasing (a v2 checkpoint must not load the v1 vocabulary) — and covers 0.9002 of distinct real-error edit forms over its training pool; the ceiling is measured, not assumed.

v3 milestone close (2026-09-21)

The v3 program attempted two enrichment legs beyond this release, and both were dropped by their pre-registered fail-closed gates: the LLM-corruption generator was non-viable at its entry probe (it would not corrupt grammatical Polish on demand), and the mined wiki revision corpus measured 0.47 label purity (n=200; Wilson 95% CI [0.402, 0.539]) under the pre-registered 0.75 drop floor, so the leg was dropped rather than shipped diluted. Nothing about the published artifact changed as a result: no weights, decode parameters, or thresholds were touched, and release tag v2.0.0 remains the canonical pin. No data from either dropped leg (no mined wiki-revision pairs, no LLM-generated rows) entered any published artifact; the license dispositions for both legs were evaluated, and the legs then dropped (evaluated-not-consumed), so the license posture is unchanged (CC BY-NC 4.0).

v4 milestone close (2026-09-27)

The v4 eval-flywheel program ran one pre-registered encoder challenger against this release; it was killed by its pre-registered stage-I rule (detection recall 0.199 against the 0.476 bar, measured before a spot-check existed), the gated contingency candidate never ran, and the mechanical ship decision produced no certified beat of v2. The program closed honest-negative: nothing about the published artifact changed, no weights, decode parameters, or thresholds were touched, release tag v2.0.0 remains the canonical pin, and the license posture is unchanged (CC BY-NC 4.0).