glimmer-b200-final

Gemma 4 E4B — B200 production-scale instruct-subspace-masked mix CPT

This is the B200 production-scale validation of the instruct-subspace gradient-mask technique on google/gemma-4-E4B-it. The model was trained on a deliberately wild 7-corpus mix at 16k context, with a high-rank mask (rank 256) protecting the directions in weight space that carry instruction-following behavior.

The goal was to see whether the mask could absorb a much larger and more aggressive corpus than any of the small-scale single-source experiments (T2 INSTRUCT, T2 CPT-marvin, SD CPT — each ~6-34M tokens) while still preserving IFEval. Answer: largely yes, with one caveat (long-form repetition).

Training recipe

Data mix (~62.6M tokens, seed=42 deterministic concat)

Total chunked at 32k tokens with 256-token overlap (Gemma 4 tokenizer); shuffled with seed=42; trained at 16k effective ctx (chunks split when longer).

Evaluation

Same harness as the rest of the E4B masking experiment lineage (IFEval strict 541 prompts + 10-prompt style gen + grpo_v12d metrics including V3 humanness classifier output). All numbers from the bf16 GGUF.

What to read from this

• The mask did its job at scale. -5.36pp IFEval is worse than the SD r=256 result (-2.03pp on 11M tokens) but better than the r=64 single-source runs (-6.8 to -8.5pp). For 5-6× the training data on a much more aggressive mix, the mask absorbed the brunt.
• Best style movement in the cohort. h_delta Δ +3.85 edges out the TIES merge (+3.64) and is essentially tied with T2 INSTRUCT (+3.91). Confirms the larger/richer corpus is moving the model further.
• Lowest AI-slop of any variant including TIES. Suggests the mix actively pushes the model away from generic LLM tics.
• ⚠️ rep3g/1k 50.30 is the open problem. 7× base rate, much worse than any other masked variant. Diagnostic: long-form descriptive passages in the smut/worm/adventure corpora teach the model to echo phrases. Same failure mode observed on the Stage 2/3 Marvin (Qwen) runs. Fix: antirep DPO pass. This model is not deployment-ready as-is; it is a successful training-technique demonstration.

Caveats

• No matched-recipe unmasked baseline was trained. The mask's effect at this scale is inferred from cross-recipe comparison with smaller-scale masked vs unmasked pairs (T2 INSTRUCT/CPT/SD CPT each had matched pairs showing 41-83% IFEval recovery from the mask). A 1:1 direct A/B at B200 scale would cost another ~$40.
• The V3 humanness classifier was trained against Gemma 4 31B-it outputs, not E4B-it. E4B is gemma-family-similar so V3 is still measuring the right axis, but distance-to-marvin is not literally comparable across model sizes.
• q_prob (ModernBERT trained on Qwen-Marvin DPO chosen/rejected pairs) is a weak quality oracle for a gemma derivative. The 0.182 jump from 0.098 is real but the absolute value shouldn't be over-read.
• rep3g/1k of 50.3 means real generations will exhibit visible phrase echoes at long context. Useful for technique validation; not useful for end-user RP/chat without an antirep cleanup pass.

Files

A bf16 GGUF + Q6_K quant are published at ToastyPigeon/Gemma4-Test-GGUFs/Gemma4-E4B-b200-mix-masked-r256.{bf16,Q6_K}.gguf for llama.cpp use.

Provenance

• Trained 2026-05-16 → 2026-05-17 by ToastyPigeon (training) + autumn (server + Discord automation) + twistedshadows (B200 compute)
• Pod: RunPod B200 secure tier (~7.5h pod lifetime including bootstrap)
• Datasets: see ToastyPigeon's private dataset collection (sources listed above, NSFW corpora not redistributed)
• Method: instruct-subspace gradient mask, rank 256, on top of LoRA r=64 α=128
• Sibling experiments: small-scale variants live in ToastyPigeon/Gemma4-Test-GGUFs (T2 INSTRUCT/CPT masked, SD CPT masked at r=64 and r=256, TIES merge)

Recommended next step

Apply antirep DPO (the Stage 4 approach that worked on Qwen Marvin: ~250-500 pairs, chosen=clean, rejected=repetitive, mask_thinking=true) to clean up the rep3g/1k score, then re-eval. Target: bring rep3g/1k below 15 while preserving the h_delta/slop wins.

Sampling

Gemma 4 defaults: temperature=1.0, no top_p / top_k / rep_penalty.