Downloads · 30 days
4
10% of all-time downloads
dougalldeepmind/2026-08-02-qwen36-lora-500k-numina-heavy
2026-08-02-qwen36-lora-500k-numina-heavy is a image-text-to-text model from dougalldeepmind. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
LoRA adapter trained on 498,035 tokens weighted toward maths (NuminaMath-CoT ~67%, TULU3 and No Robots ~16.5% each), with loss on assistant tokens only, for 1 epoch.
Downloads · 30 days
4
10% of all-time downloads
All-time downloads
39
Public
Repo size
658 MB
Likes
0
Public
Click a slice to open those files.
.safetensors638 MB · 97%
From the Hugging Face model README
LoRA adapter trained on 498,035 tokens weighted toward maths (NuminaMath-CoT ~67%, TULU3 and No Robots ~16.5% each), with loss on assistant tokens only, for 1 epoch.
Training data: qwen3.6-27b-mixture-500k-numina-heavy.
| Source | Examples | Tokens | Share |
|---|---|---|---|
| no_robots | 271 | 82,239 | 16.51% |
| numinamath_cot | 611 | 333,351 | 66.93% |
| tulu3 | 119 | 82,445 | 16.55% |
| Supervised | 390,921 / 498,035 = 78.5% |
| Epochs / steps | 1 / 63 |
| Runtime | 38 min, 1x H100 80GB |
| r / alpha / dropout | 32 / 64 / 0.05 |
| batch x grad-accum | 1 x 16 |
| lr / schedule | 1e-4, cosine, 3% warmup |
| max seq len / packing | 3072 / off |
| Final loss | 0.897 |
| Token accuracy | 0.798 |
Final loss 0.897 sits below the ~1M-token difficult-advice arms (0.88-0.93) at a comparable step count, and far below the 100k/14-step run (1.547). That reflects the data, not model quality: NuminaMath solutions are highly structured mathematical prose and an easier next-token target than open-ended advice. Do not read it as a better model.
Everything outside an assistant turn is -100. A supervised span starts immediately after
the <|im_start|>assistant\n header and ends after the closing <|im_end|>, which the model
must emit in order to stop. Verified: zero user or system tokens inside any supervised span.
TRL's own assistant_only_loss cannot do this on Qwen3.6 — its chat template has no
{% generation %} markers. Spans are derived from the rendered text via the fast
tokenizer's offset mapping instead, and TRL receives finished labels.
loss_type: nll is set because TRL's default chunked-CE path patches the LM head and reads
forward.__func__, which fails on this checkpoint's functools.partial forward. The loss is
mathematically the same.
from peft import PeftModel
from transformers import AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "LASR-Callum/2026-08-02-qwen36-lora-500k-numina-heavy")
model = model.merge_and_unload()
Use AutoModelForImageTextToText, not AutoModelForCausalLM — this is a vision-language
checkpoint.