Downloads · 30 days
7
15% of all-time downloads
dougalldeepmind/2026-08-03-qwen36-lora-500k-numina-only
2026-08-03-qwen36-lora-500k-numina-only is a image-text-to-text model from dougalldeepmind. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
LoRA adapter trained on 497,968 tokens of NuminaMath-CoT with loss on assistant tokens only, for 1 epoch at lr 4e-5.
Downloads · 30 days
7
15% of all-time downloads
All-time downloads
47
Public
Repo size
658 MB
Likes
0
Public
Click a slice to open those files.
.safetensors638 MB · 97%
From the Hugging Face model README
LoRA adapter trained on 497,968 tokens of NuminaMath-CoT with loss on assistant tokens only, for 1 epoch at lr 4e-5.
Training data: qwen3.6-27b-mixture-500k-numina-only.
| Examples / tokens | 934 / 497,968 |
| Supervised | 409,975 / 497,968 = 82.3% |
| Epochs / steps | 1 / 59 |
| lr / schedule | 4e-5, cosine, 3% warmup |
| Runtime | 36 min, 1x H100 80GB |
| r / alpha / dropout | 32 / 64 / 0.05 |
| batch x grad-accum | 1 x 16 |
| max seq len / packing | 3072 / off |
| Final loss | 0.538 |
| Token accuracy | 0.878 |
0.538 is roughly half the loss of every mixed-corpus arm in this family (0.76-0.95), and token accuracy is ~8 points higher. That is the narrowness of the target, not model quality: NuminaMath solutions follow a highly regular mathematical structure, so next-token prediction is far easier than on open-ended instruction data. Loss converged by about step 15 of 59.
| Run | Data | Loss | Token acc |
|---|---|---|---|
| this | NuminaMath only | 0.538 | 0.878 |
| 500k maths-weighted | 67% maths + mixed | 0.878 | 0.793 |
| 500k da20 + maths | 20% difficult-advice | 0.946 | 0.776 |
Everything outside an assistant turn is -100. A supervised span starts immediately after the
<|im_start|>assistant\n header and ends after the closing <|im_end|>. Verified before
training: zero user or system tokens inside any supervised span.
loss_type: nll is set because TRL's default chunked-CE path patches the LM head and reads
forward.__func__, which fails on this checkpoint's functools.partial forward.
Not yet evaluated.
from peft import PeftModel
from transformers import AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "LASR-Callum/2026-08-03-qwen36-lora-500k-numina-only")
model = model.merge_and_unload()
Use AutoModelForImageTextToText, not AutoModelForCausalLM — this is a vision-language
checkpoint.