qwen2.5-3b-gec-bea2019

qwen2.5-3b-gec-bea2019

LoRA fine-tune of Qwen/Qwen2.5-3B-Instruct for English grammatical error correction (GEC). Trained on the W&I+LOCNESS subset of BEA-2019 (33,432 minimal-edit pairs). LoRA weights are fused into the base, so this repo is a drop-in replacement for the base model.

Results

ERRANT scores on a 100-sample held-out BEA-dev split (sampled from ABCN.dev.gold.bea19.m2):

Validation loss converged from 2.875 (baseline) to 0.281 over 3,000 LoRA iters on an Apple M2 Max in ~17 minutes.

Use

from mlx_lm import generate, load

model, tokenizer = load("amiya/qwen2.5-3b-gec-bea2019")
prompt = tokenizer.apply_chat_template(
    [
        {"role": "system", "content": "Correct the grammar of the user text. Preserve meaning."},
        {"role": "user", "content": "He go to school"},
    ],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=80))
# -> "He goes to school"

PyTorch users: transformers.AutoModelForCausalLM.from_pretrained("amiya/qwen2.5-3b-gec-bea2019") works the same way.

System prompt

The model was trained with the system prompt:

Correct the grammar of the user text. Preserve meaning.

Use that exact prompt for in-distribution behavior. The chat template is Qwen's ChatML.

Smoke samples

Trailing-space + period is a BEA-2019 tokenization artifact; a post-processor can strip it.

Training

• Base: Qwen/Qwen2.5-3B-Instruct (3.09 B params)
• LoRA: rank 16, alpha 32, 16 layers, 6.65 M trainable params (0.216 %)
• Optimizer: AdamW, lr 5e-5
• Batch size: 2, max sequence 256
• Iters: 3,000
• Hardware: Apple M2 Max, 32 GB unified RAM, mlx-lm v0.31.3
• Data: BEA-2019 W&I+LOCNESS ABC.train.gold.bea19.m2, converted to ChatML SFT by the data-pipeline repo. 33,432 minimal-edit (src, tgt) pairs after dropping identity rows.
• Wall-clock: 17 minutes solo (3.2 it/s).

Limitations

• F0.5 = 0.4950 is below the GECToR paper's 0.65+ benchmark. That gap is expected: the published number comes from a multi-stage curriculum (C4_200M pretrain → BEA-2019 → W&I+L fine-tune) on a token-classification head. This run is a single LoRA pass with seq2seq generation. Closing the gap needs more data (NUCLE, FCE, Lang-8) and the multi-stage recipe.
• Greedy decoding is recommended. Temperature > 0 produces fluent but less-faithful corrections.
• English only.
• Trained on ≤ 256-token sequences; long inputs may degrade.

Citation

@misc{qwen25_3b_gec_bea2019,
  title  = {qwen2.5-3b-gec-bea2019: LoRA-tuned Qwen2.5-3B for English GEC},
  author = {amiya},
  year   = {2026},
  url    = {https://huggingface.co/amiya/qwen2.5-3b-gec-bea2019}
}

Underlying corpus: Bryant, Felice, Andersen, Briscoe. The BEA-2019 Shared Task on Grammatical Error Correction, BEA 2019.