Llama-3-8B-cefr-qlora

Llama-3-8B-cefr-qlora

Built with Meta Llama 3.

QLoRA adapter (r=16, alpha=32, all attention and MLP projections) that rates the CEFR difficulty of a single English sentence on the 1 to 6 scale (A1 to C2). The completion is one bare digit token, so classification is a single forward pass with an argmax over six token logits: no generation, no output parsing.

Prompt format:

Rate the CEFR difficulty of this English sentence on a scale of 1 (A1, easiest) to 6 (C2, hardest).
Sentence: {sentence}
Level:

Benchmark vs GPT-4o-mini few-shot

Identical held-out test set (CEFR-SP public portion, n=1460, split checksum e45e1389a015), single evaluation pass for both models.

Checkpoint 1347 (epoch 3 of 3) selected by validation macro F1 (0.5132). McNemar exact test on paired correctness: p = 2.3e-33.

Ablation adapter: os500/

The os500/ folder holds an adapter trained identically except A1 and C2 training rows were oversampled to 500 each. Test accuracy 64.11% and macro F1 0.4489, but A1 F1 stayed exactly 0.00 and C2 dropped to 0.18 (McNemar vs the base recipe: p = 0.09, not significant). Duplicating 47 unique sentences adds signal strength, not information. The root adapter remains the final model, selected by validation macro F1 (0.5132 vs 0.5023). Raw prediction files for both runs are under benchmark/.

Known limitations

Training data contains only 47 A1 and 91 C2 sentences. The adapter under-predicts both extremes (test F1: A1 0.00, C2 0.20) while beating the baseline by wide margins on A2, B1, B2, and C1. Judgments follow the stricter of the two expert annotators in CEFR-SP by design.

Trained on the public portion of CEFR-SP (Wiki-Auto CC BY-SA 3.0, SCoRE CC BY-NC-SA 4.0): treat this adapter as research and non-commercial use.

Full data pipeline, training notebook, and evaluation harness: https://github.com/akallam04/cefr-qlora-benchmark