qwen2.5-3b-gec-v2

qwen2.5-3b-gec-v2

LoRA fine-tune of Qwen/Qwen2.5-3B-Instruct for English grammatical error correction (GEC). v2 improvements over qwen2.5-3b-gec-bea2019:

• +0.038 F0.5 (0.495 → 0.533) on held-out BEA-dev.
• Mixed training corpus: BEA-2019 W&I+LOCNESS (22,668 pairs) + Grammarly Coedit GEC subset (19,823 pairs) = 42,491 minimal-edit pairs.
• All 36 transformer layers LoRA-adapted (vs 16 in v1) → 14.97 M trainable params (vs 6.65 M).
• 5,000 iters (vs 3,000) on the larger corpus.
• LoRA weights are fused into the base, so this repo is a drop-in replacement for the base model.

Results

Scored with ERRANT on a 100-sample held-out BEA-dev split (ABCN.dev.gold.bea19.m2, identity rows skipped). Post-processing strips a small set of BEA-2019 tokenization artifacts (trailing " ." → ".", collapsed double spaces, source-aligned leading-cap).

Validation loss trajectory (held-out 500-pair mix of BEA + Coedit):

Trained 5,000 iters on an Apple M2 Max in ~36 minutes (peak memory 10.5 GB).

Use

from mlx_lm import generate, load

model, tokenizer = load("amiya/qwen2.5-3b-gec-v2")
prompt = tokenizer.apply_chat_template(
    [
        {"role": "system", "content": "Correct the grammar of the user text. Preserve meaning."},
        {"role": "user", "content": "He go to school"},
    ],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=80))
# -> "He goes to school"

PyTorch users: transformers.AutoModelForCausalLM.from_pretrained("amiya/qwen2.5-3b-gec-v2") works the same.

System prompt

Correct the grammar of the user text. Preserve meaning.

Use that exact prompt for in-distribution behavior. Chat template is Qwen's ChatML.

Recommended decoding

• Greedy (temperature 0).
• max_new_tokens ~ 1.5× the prompt length.
• Single-pass: ~80 tokens / sentence on M2 Max greedy.
• Iterative re-feed: same F0.5 on this dataset (trades precision for recall); add it if you need higher recall.
• Post-processor (~20 LOC, in the source repo) trims " .", collapses runs of whitespace, and aligns leading capitalisation to the input. Worth +0.01–0.02 F0.5.

Training

• Base: Qwen/Qwen2.5-3B-Instruct (3.09 B params)
• LoRA: rank 16, alpha 32, all 36 layers (vs 16 in v1), 14.97 M trainable (0.485 %)
• Optimizer: AdamW, lr 5e-5
• Batch size: 2, max sequence 256
• Iters: 5,000 (best snapshot at iter 3,500 was promoted)
• Hardware: Apple M2 Max, 32 GB unified RAM, mlx-lm v0.31.3
• Data:
  • BEA-2019 W&I+LOCNESS ABC.train.gold.bea19.m2 → 22,668 unique src→tgt pairs after dropping identity rows.
  • Grammarly Coedit train.jsonl, filtered to task == "gec" → 19,823 pairs after stripping instruction prefixes.
  • 500 held-out for val.

Limitations

• ERRANT F0.5 = 0.533 is below the GECToR paper's 0.65+ benchmark. Closing the gap further requires NUCLE + FCE + Lang-8 + the multi-stage curriculum (synthetic pretrain → BEA mid-train → W&I+L finetune).
• Greedy decoding is recommended. Temperature > 0 produces fluent but less-faithful corrections.
• English only.
• Trained on ≤ 256-token sequences; long inputs may degrade.

Comparison to v1

The v1 model is BEA-only and uses a smaller LoRA (16 layers). v2 doubles LoRA capacity and adds Coedit data; the gains come about evenly from both. Use v2 unless you specifically want a smaller-footprint adapter.

Citation

@misc{qwen25_3b_gec_v2,
  title  = {qwen2.5-3b-gec-v2: LoRA-tuned Qwen2.5-3B for English GEC},
  author = {amiya},
  year   = {2026},
  url    = {https://huggingface.co/amiya/qwen2.5-3b-gec-v2}
}

Datasets: Bryant et al. The BEA-2019 Shared Task on GEC. Raheja et al. CoEdIT: Text Editing by Task-Specific Instruction Tuning.