Muse-Glimmer-30B-Fable-Distill-LoRA

Muse-Glimmer-30B-Fable-Distill-LoRA

The QLoRA adapter (PEFT, r=32) from a fine-tune of meta-models/Muse-Glimmer-30B on agentic coding traces and chat distilled from Claude Fable 5, with a multilingual slice. Text-only: the adapter touches the language model's projections (self_attn.{q,k,v,o,gate}_proj, mlp.{gate,up,down}_proj, 416 modules, 209.6M parameters); the vision tower, projector, embeddings and lm_head are untouched, so the model's image input keeps working exactly as in the base.

Trained against the NF4 base unsloth/Muse-Glimmer-30B-unsloth-bnb-4bit; load it on that same 4-bit base (or the bf16 original) with PEFT.

Chat format and serving

Muse Glimmer uses its own harmony-style "Onyx ATEM" format — reasoning on a <|start|>assistant to=self<|message|>…<|eom|> channel, tool calls as <atem:function_calls> XML, tool results as <tool_output> blocks, <|eot|> end-of-turn. The chat template drives all of it (reasoning_strength: low/medium/high/xhigh, default high; tool-call arguments must be a dict, not a JSON string).

No stock vLLM (≤ 0.28) reasoning or tool parser matches this format — do not enable --reasoning-parser / --tool-call-parser with a stock name, they will mis-parse. Serve without parsers (the channel markers arrive in content) or register custom parser plugins.

Training data

651 examples / 3.62M tokens after rendering at a 24,576-token context (trim_followups; median 1.1k tokens, p90 20.4k — most of the longest Cursor traces exceed 24k and were dropped). Prepared with teich 0.3.6 plus a format patch (teich did not know this chat format; the patch teaches its span expansion the <|start|>/<|eot|>/<|eom|> scaffolding and <atem:function_calls> blocks, and parses string tool arguments to dicts before rendering). Reasoning kept and supervised. Rows without any assistant reasoning were rendered at reasoning_strength: low, rows with reasoning at high, so the strength flag in the system prompt actually predicts whether a reasoning channel follows (a first run rendered everything at high and learned to skip thinking). Harness text recorded as assistant turns in the raw sessions (API Error: …, Please run /login …, No response requested.) was removed before rendering — the first run memorised it verbatim. Supervision is scoped so the base's think-first routing survives: on rows without reasoning the initial routing header ( to=user<|message|> / to=<fn><|message|>) is left unsupervised and only the answer or <atem:function_calls> block is trained; on rows with reasoning the headers after <|eom|> are supervised, and the <|start|>assistant continuation that follows every <|eom|> is supervised so thinking → answer is learnable. (A run that supervised every routing header stopped opening the reasoning channel even at xhigh and called tools 1/4 vs the base's 4/4.)

Validated before training over every row: final answers supervised as to=user<|message|>…<|eot|>, reasoning as to=self<|message|>…<|eom|>, tool calls as the recipient header + full <atem:function_calls> block + terminator; user/system/tool-output text masked; 40% of tokens supervised.

Training

QLoRA (NF4 base), LoRA r=32 / α=32 / dropout 0, 24,576-token context, LR 8e-5 linear, warmup 5, 2 epochs (164 steps), batch 1 × grad-accum 8, paged_adamw_8bit, max_grad_norm 0.3, gradient checkpointing, single 64 GB GPU. Plain transformers (git main — the muse_glimmer architecture needs ≥ 5.15) + PEFT + TRL with a fused linear cross-entropy that reproduces the model's output_multiplier pre-scale and tanh logit soft-cap (T·tanh(logits·m/T), T=20) — verified equal to the model's own loss before training. Base-model loss on this data at step 1: 1.17.

Limitations

Non-English is ~10% of tokens. Not benchmarked against an answers-only variant. Training used the NF4-quantized base; any bf16 merge is therefore not bit-identical to what training saw.