qwen2.5-7b-gec-v3

qwen2.5-7b-gec-v3

LoRA fine-tune of Qwen/Qwen2.5-7B-Instruct for English grammatical error correction (GEC). LoRA was trained against the 4-bit MLX quantization of the base, then fused and dequantized to bf16 for distribution. +0.018 F0.5 over the 3B v2 model on the same held-out BEA-dev split.

Results

ERRANT scores on a 100-sample held-out BEA-dev split (ABCN.dev.gold.bea19.m2, identity rows skipped, post-process: trailing-space-punct + space-collapse + leading-cap):

Best snapshot: iter 4000 (val 0.252). Training also tested iter 1000 (0.504), iter 2500 (0.532), iter 5000 (0.535).

Use

from mlx_lm import generate, load

model, tokenizer = load("amiya/qwen2.5-7b-gec-v3")
prompt = tokenizer.apply_chat_template(
    [
        {"role": "system", "content": "Correct the grammar of the user text. Preserve meaning."},
        {"role": "user", "content": "He go to school"},
    ],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=80))
# -> "He goes to school"

PyTorch users: transformers.AutoModelForCausalLM.from_pretrained("amiya/qwen2.5-7b-gec-v3") works the same.

System prompt

Correct the grammar of the user text. Preserve meaning.

Use that exact prompt. Chat template is Qwen ChatML.

Recommended decoding

• Greedy (temperature 0)
• max_new_tokens ~ 1.5× the prompt length
• Single-pass; iterative re-feed did not help on the v2 ablation
• Post-processor (scripts/eval_mlx_gec.py --post-process in the source repo) is worth +0.01–0.02 F0.5

Training

• Base: mlx-community/Qwen2.5-7B-Instruct-4bit (4-bit quantised) — fused weights here are dequantised to bf16
• LoRA: rank 16, alpha 32, dropout 0.05, all 28 layers, 40.37 M trainable (0.53 %)
• Optimizer: AdamW, lr 1e-5 (5× lower than v2 to keep Q4 base stable — 5e-5 diverged)
• Batch size: 2, max sequence 256
• Iters: 5,000 (best at iter 4000)
• Hardware: Apple M2 Max, 32 GB unified RAM, mlx-lm v0.31.3
• Wall-clock: ~60 minutes, peak memory 12 GB

Data

Identity rows dropped; sentences over 50 words dropped from sarayusapa to keep distribution focused.

Smoke samples

Limitations

• F0.5 = 0.551 is still below GECToR's 0.65+ paper benchmark. Closing the gap further needs NUCLE + FCE + Lang-8 (license-gated) plus the multi-stage curriculum (synthetic pretrain → BEA → W&I+L finetune).
• The fused model here is dequantised to bf16 (~14 GB). For inference-only deployment, re-quantise with mlx_lm.convert -q --q-bits 4.
• English only.
• Trained on ≤ 256-token sequences; long inputs may degrade.

Citation

@misc{qwen25_7b_gec_v3,
  title  = {qwen2.5-7b-gec-v3: LoRA-tuned Qwen2.5-7B for English GEC},
  author = {amiya},
  year   = {2026},
  url    = {https://huggingface.co/amiya/qwen2.5-7b-gec-v3}
}

Datasets: Bryant et al. BEA-2019 Shared Task. Raheja et al. CoEdIT. Owishiboo, sarayusapa public HF datasets.