teutonic-II-110B-A7B-router08-fulldepth

teutonic-II-110B-A7B-router08-fulldepth

A 110B MoE checkpoint for Bittensor subnet 3 (Teutonic), derived from the reign-14 king teutonic-II-110B-A7B-5DALwjE4-v44 by a router recalibration plus full-depth training in which all 45 layers adapt simultaneously.

What this is, and what it is not

It does not clear the subnet's acceptance bar. Teutonic accepts a challenger when the lower confidence bound of paired per-sequence cross-entropy improvement exceeds delta_threshold = 0.025. This model measures LCB@4000 ≈ +0.0013, roughly 19× short. It is published as a reproducible measurement, not a competitive checkpoint.

Measured result

Paired against the reign-14 king on 1,200 held-out sequences (panel fingerprint 4f50f8cc798ddbe8), drawn at the chain's own mixture (dclm-baseline-1.0 0.52, automathtext-v2 0.26, finewebedu 0.22, seq_len=2048). Every row below was produced by the same scorer, against the same baseline, on the same panel — see "Why one code path" below.

* projection to the validator's n=4000 by normal approximation, not a validator result.

Positive on all three sources, and better than the previous best on each:

The finding worth reporting

Layers trained alone are nearly worthless; the same layers trained together are not.

A per-block scan, each block trained in isolation and scored against a matched baseline, found essentially nothing:

A single block has to push its change through 30+ frozen downstream blocks. Training all 45 depths at once lets them co-adapt, and the same parameter budget then yields +0.000885 (7.3σ) over the identical starting point — more than 11× the best isolated block.

The budget is the key to why this fits at all. Full-parameter training of 36 layers needs ~1.4 TB of optimiser state. But the routed experts are what make this model 110B — attention, shared experts and layernorms across all 45 layers total only 3.008 B parameters, fewer than a single block-44 arm (2.95 B). Freezing the 107.3 B of routed experts leaves everything that mixes and routes information free to move at every depth.

trainable   45 x self_attn.qkv_proj + o_proj          attention, every layer
            44 x mlp.shared_experts.{gate,up,down}    shared experts
            45 x input_layernorm + post_attention     norms, plus model.norm
             3 x layer-0 dense mlp
            = 316 tensors, 3.008 B parameters;  107.3 B frozen

Improvement was still rising when the run was stopped — this is not a converged number:

Recipe

1. Router recalibration. b' = 0.8*(b - mean(b)) on e_score_correction_bias in all 44 MoE layers. Centring is a no-op for top-8 selection (n_group=1), so the effect is the scaling. The bias enters only through torch.topk selection, never the gating weights, so it has an identically zero gradient and must be set by rule, not learned.
2. Full-depth training. The 316 tensors above, AdamW-8bit lr=5e-6, 4.0M prediction tokens of the eval mixture, microbatch 1 × accum 2, cosine-free constant LR with 20-step warmup, sequence order shuffled.

Training runs in model.eval() throughout: the pinned modeling_mimo_v2.py raises inside the router when self.training is true, but that guard tests mode, not requires_grad, so gradients still reach every trained tensor. attention_dropout is 0.0, so eval mode is numerically identical to train mode.

Learning rate is the whole game

The king is converged enough that the usable LR window is narrow. At lr=1.5e-5 — a value an earlier holdout grid called optimal — block training is catastrophic:

Tripling the LR flips the sign and multiplies the magnitude by six. The damage is fully present at 5M tokens and never recovers, so it is the learning rate, not overtraining.

Why one code path

Every number here was re-measured with a single scorer against a single baseline, because mixing measurement paths produced errors of the same size as the effects:

• Prefix caches taken at different split points are not interchangeable. Scoring the identical model through a K=39 and a K=42 cache gives means of 1.981205 and 1.981249 — they round the cached hidden states at different layer boundaries. That 4.4e-5 gap is the same order as a single-block effect. It cancels only when both arms share a cache.
• Held-out-split readings ran optimistic repeatedly. One arm read +0.001906 on a 244-sequence holdout and −0.000923 on the dev panel.
• Re-measured on one path, the previous best is +0.001402, not the +0.001456 that had been quoted from the split-cache path.

Panels are stored grouped by source tag (dev: automathtext 312 | dclm 624 | finewebedu 264; train: 1904 | 3809 | 1611). Two consequences, both of which silently produce wrong results: a trainer that walks the panel in order and stops early trains on one domain, and a first-N subset for cheap ranking contains zero finewebedu. Training order is shuffled and subset scoring is stratified per tag.

Reproducing the evaluation

Loss is mean per-token cross-entropy over the 2,047 next-token positions per sequence, matching teutonic/evaluator/engine.py::compute_per_sequence_loss. Verdicts use the subnet's own teutonic.evaluation.policy.paired_bootstrap_verdict. results/ contains raw per-sequence losses for every arm above, so each number can be recomputed without rerunning the model.

Architecture

Unmodified from the base: MiMo-v2 MoE, 45 layers, hidden 3072, 256 routed experts top-8 plus 1 shared, hybrid sliding-window/full attention (sliding_window=128, 9 full-attention layers), vocab 152,576. All contract files (config.json, modeling_mimo_v2.py, tokenizer) are byte-identical to the king, so the subnet's architecture and contract-hash gates pass unchanged.