t5-efficient-base-grammar-correction

t5-efficient-base-grammar-correction

Full fine-tune (all parameters, no adapters) of google/t5-efficient-base for English sentence-level grammatical error correction on agentlans/grammar-correction.

Model

Training data

• Token length (train, input with prefix): mean 38.23, p99 129; truncated at 256/256: 0.10% sources, 0.03% targets.
• Duplicate pairs: 0 (train), 0 (validation); validation inputs present in train: 0.
• Pairs with input == output: 0.
• Targets containing characters outside the SentencePiece vocabulary (<unk>): 1.31%.
• Text is used as-is (no normalization of case, punctuation or contractions).

Training procedure

Evaluation

Checkpoint selected by eval_sari on 2,000 validation examples: epoch 5.00, step 15,625, validation loss 0.6534 (perplexity 1.922).

Validation loss 0.6504, perplexity 1.916. Decoding: beam search, 4 beams.

Over-correction check: 1,000 grammatical validation references fed as inputs; 24.20% were modified (chrF vs. input 98.81).

Metric notes: BLEU and chrF measure overlap with the reference, so copying the input already scores high (see baseline). SARI (Xu et al., 2016) scores kept/added/deleted n-grams relative to the source on lowercased 13a tokens, so casing edits are not reflected. Exact match is case- and punctuation-sensitive. ERRANT F0.5 was not computed.

Generation config

{
  "max_new_tokens": 256,
  "early_stopping": true,
  "do_sample": false,
  "num_beams": 4,
  "pad_token_id": 0,
  "eos_token_id": 1,
  "decoder_start_token_id": 0
}

Usage

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "thealper2/t5-efficient-base-grammar-correction"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id).eval()

text = "She go to school yesterday."
inputs = tokenizer("grammar correction: " + text, return_tensors="pt", truncation=True, max_length=256)
with torch.inference_mode():
    output_ids = model.generate(**inputs)  # uses the bundled generation_config.json
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Limitations

• Training pairs are synthetic (C4_200M-derived corruptions filtered by a grammar classifier); error distribution differs from learner or native-speaker text.
• English only, sentence/short-paragraph level; inputs longer than 256 tokens are truncated.
• The training set contains no already-correct (input == output) pairs, so leaving correct text unchanged is not directly supervised; check the over-correction numbers above.
• The T5 SentencePiece vocabulary cannot represent some characters (e.g. {, }, <, ~, many accented letters); they map to <unk> and are dropped from generated text.
• Not evaluated on standard GEC benchmarks (CoNLL-2014, BEA-2019).