Downloads · 30 days
11
15% of all-time downloads
dougalldeepmind/2026-08-02-qwen36-lora-100k-tulu-numina-norobots
2026-08-02-qwen36-lora-100k-tulu-numina-norobots is a image-text-to-text model from dougalldeepmind. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
LoRA adapter trained on 99,794 tokens in equal thirds from three instruction corpora, with loss on assistant tokens only, for 1 epoch.
Downloads · 30 days
11
15% of all-time downloads
All-time downloads
71
Public
Repo size
658 MB
Likes
0
Public
Click a slice to open those files.
.safetensors638 MB · 97%
From the Hugging Face model README
LoRA adapter trained on 99,794 tokens in equal thirds from three instruction corpora, with loss on assistant tokens only, for 1 epoch.
Training data: qwen3.6-27b-mixture-100k-tulu-numina-norobots.
| Source | Examples | Tokens | Share |
|---|---|---|---|
| no_robots | 98 | 33,254 | 33.32% |
| numinamath_cot | 64 | 33,261 | 33.33% |
| tulu3 | 49 | 33,279 | 33.35% |
| Supervised | 72,200 / 99,794 = 72.3% |
| Epochs / steps | 1 / 14 |
| Runtime | 8 min, 1x H100 80GB |
| r / alpha / dropout | 32 / 64 / 0.05 |
| batch x grad-accum | 1 x 16 |
| lr / schedule | 1e-4, cosine, 3% warmup |
| max seq len / packing | 3072 / off |
| Final loss | 1.547 |
| Token accuracy | 0.736 |
100k tokens is 14 optimizer steps. With 3% warmup and a cosine decay, most of the schedule is spent warming up and annealing, so the adapter is lightly trained and its final loss (1.547) sits well above the ~1M-token arms (0.88-0.93), which ran 86-98 steps. Treat it as a small-data datapoint rather than a converged model.
Everything outside an assistant turn is -100. A supervised span starts immediately after
the <|im_start|>assistant\n header and ends after the closing <|im_end|>, which the model
must emit in order to stop. Verified: zero user or system tokens inside any supervised span.
TRL's own assistant_only_loss cannot do this on Qwen3.6 — its chat template has no
{% generation %} markers. Spans are derived from the rendered text via the fast
tokenizer's offset mapping instead, and TRL receives finished labels.
loss_type: nll is set because TRL's default chunked-CE path patches the LM head and reads
forward.__func__, which fails on this checkpoint's functools.partial forward. The loss is
mathematically the same.
from peft import PeftModel
from transformers import AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.6-27B", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "LASR-Callum/2026-08-02-qwen36-lora-100k-tulu-numina-norobots")
model = model.merge_and_unload()
Use AutoModelForImageTextToText, not AutoModelForCausalLM — this is a vision-language
checkpoint.