Cognica-PoE-v1.0-3B-base-continual-learning

Cognica-PoE-v1.0-3B-base-continual-learning

Continual pretraining of a 3B PoE per-stage-head model using a cyclic re-warmup schedule to extract additional capacity from the same data distribution after initial training has fully annealed.

This release studies whether a model whose first training pass has reached its scheduled LR floor (lrm ≈ 0.05) can still meaningfully improve when given a fresh half-peak warmup → warmdown cycle on the same data — without changing architecture, tokenizer, or data mix.

The model is published as a trajectory of step branches, not just a final ckpt: every saved checkpoint becomes a separate step-XXXXX branch so the continual-learning curve itself is externally auditable.

What "continual" means here

Most published "base" models are released after a single warmup → constant → warmdown LR cycle. At the end of that cycle the LR is near zero and gradient updates produce only marginal change — the model is conventionally considered "done."

This release tests a different setting: take a fully-annealed checkpoint, re-arm the optimizer with a new LR cycle at half the original peak, and continue training on the same data. We label this continual pretraining (cyclic LR) to distinguish it from:

The hypothesis being probed: does a half-peak second cycle on identical data produce real, measurable gain, or does the model plateau?

Methodology — B2 cyclic schedule

Initialization: a fully-trained 3B PoE per-stage-head model (66B tokens consumed, full warmup → warmdown cycle complete, lrm annealed to ~0.05 of original peak).

LR schedule for the continual phase (anchored at the resume step):

warmup    (rel 0 .. 1000)   : lrm rises 0   → 0.5  (linear)
peak      (rel 1000..22861) : lrm = 0.5            (constant; half of original peak)
warmdown  (rel 22861..50800): lrm decays 0.5 → 0.0 (linear; warmdown_ratio = 0.55)

Optimizer state is restored from the prior phase's last save (DistMuonAdamW ZeRO-2 sharded across 12 ranks). Tokenizer (32,768 vocab, rustbpe), architecture (depth=32, n_embd=2048, K=4 PoE per-stage with asymmetric stage_layers=(16,6,5,5), GQA 2:1, intermediate=12800, max_seq_len=2048), and data mix (frontier_v1: FineWeb-Edu 33.5% + DCLM-Baseline 24% + Stack-v2 16% + Wikipedia 5% + CulturaX 5% + ProofPile-2 4% + OpenWebMath 4% + Gutenberg 4% + PG-19 2% + UltraChat 1% + OpenHermes-2.5 0.6%) are all unchanged from the prior phase.

Why a published trajectory

The point of this release is the continual-learning curve, not any single endpoint. We publish every save (step-XXXXX branches) so the actual question — "does a re-warmed cycle keep improving the model?" — can be answered by reading off the trajectory rather than trusting our headline numbers.

Each step-XXXXX branch carries its own per-checkpoint poe_wand_p99_bounds_per_stage_head calibration in config.json so PoE-specific inference (WAND adaptive depth, self-speculative decoding) works correctly at any branch.

Branches

Training complete. No further checkpoints. Warmdown ran from step-106800 (rel_it ≈ 22,861) to step-134723 (rel_it 50,800), total 27,939 warmdown steps with lrm 0.5 → 0.0.

Inference

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
    "cognica/Cognica-PoE-v1.0-3B-base-continual-learning",
    revision="main",                # or any "step-XXXXX" branch
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(
    "cognica/Cognica-PoE-v1.0-3B-base-continual-learning",
    revision="main",
    trust_remote_code=True,
)

# Base ckpts REQUIRE prepending <|bos|> before user text:
prompt = "The capital of France is"
input_ids = [tokenizer.bos_token_id] + tokenizer.encode(prompt, add_special_tokens=False)
input_ids = torch.tensor([input_ids], device=device)
out = model.generate(input_ids=input_ids, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out[0].tolist()))

PoE-specific inference helpers (single-stage forward, prefix pruning, WAND adaptive depth, self-speculative decoding) are exposed on CognicaPoEForCausalLM. Each step-XXXXX branch carries its own calibrated poe_wand_p99_bounds_per_stage_head in config.json; model.generate_wand(...) reads it automatically.

For the architectural details, full inference recipe, and the prior-phase trajectory analysis, see the prior-phase release: cognica/Cognica-PoE-v1.0-3B-base.

Trajectory measurements

The continual phase entry point (step-83923) is identical to the prior-phase endpoint, so it serves as both the resume anchor and the baseline against which every continual-phase ckpt is measured. A meaningful continual-learning result requires step-NNNNN measurements to diverge from the seed across multiple metrics — not just match it.

Per-checkpoint val BPB

Same 8-shard local val slice across all measured ckpts (1.05 M tokens, --split-tokens 1048576); single A100 80GB; FULL K=4 PoE aggregation.

Per-stage BPB

Per-checkpoint training-objective BPB at each PoE stage boundary, on the same 8-shard local val slice.

Per-stage standalone target accuracy

Top-1 accuracy of each stage's standalone prediction vs ground-truth target token, on the same 8-shard val slice.

Self-speculative decoding (m=4 stage-0 draft, full K=4 verify)

Confidence-aware routing (target_regression cap = 0.020)

WAND p99 bounds (cumulative-PoE delta range, constant-shift invariant)

Calibrated on a 131,072-token val slice using range(delta) = max(delta) − min(delta). Each branch's config.json carries its own bounds in poe_wand_p99_bounds_per_stage_head.

Bayesian PoE α-sweep (renormed BPB at α=0)

α=0 is the geometric-mean PoE aggregate (i.e. uniform-mean of log-probabilities); higher α values approach pure-sum PoE.

Sample-mode probe (FULL K=4, greedy, 60 tokens, 7-prompt fixed set)

Concept-level retention probe across a fixed 7-prompt set: capital of France, gold chemical symbol, Friday→tomorrow, opposite of hot, planets list, favorite color, 5x+3=13 algebra. Greedy continuations track factual recall trajectory in parallel with BPB.

Reading the table

The seed (step-83923) row is the frozen reference. Subsequent rows answer:

1. BPB descent — does full K=4 BPB continue dropping past the seed, or plateau?
2. Per-stage refinement — do single-stage BPBs descend, indicating each head genuinely tightens?
3. Routing margin — does the cap=0.020 routing fraction grow (more positions cleanly handled by stage 0 alone)?
4. Spec α dynamics — does stage-0-vs-full agreement strengthen as continual training progresses?
5. WAND p99 evolution — does the cumulative-PoE delta range shrink (head distributions converge) or widen (head specialization tightens)?
6. Sample-mode — do specific factual probes (planets list, atomic number, algebra answer) become reliably correct, or oscillate?

A real continual-learning win requires multiple metrics to diverge from the seed in a coherent direction. A null result would have all rows ≈ seed — meaning the prior-phase warmdown had already extracted available capacity from this data.

The seed → step-84000 delta is inside expected noise (77 step at warmup-near-zero LR cannot move the model meaningfully).

step-86000 is the first post-warmup ckpt (rel_it = 2,077; warmup of 1,000 ended at rel_it = 1,000, so 1,077 step into the lrm = 0.5 peak phase). It shows a clear LR-shock signature: full K=4 BPB +0.097, training-log val BPB +0.103, per-stage acc -4.4pp uniform, spec α -4.5pp, routing fraction (cap = 0.020) -14pp. The crossover gap (α=0 vs single s3) narrowed to +0.000295 from +0.000465 — the relative aggregation structure is preserved despite the absolute regression.

step-88000 (rel_it = 4,077; ~3,077 step into peak) shows the peak-phase plateau approaching: full K=4 BPB +0.010 vs step-86000 (slope decelerated 10× vs the s086000 single jump), spec α recovered to 0.9795 (only -0.006 from Run A's 0.9852 — the first metric to fully bounce back), per-stage acc drift slowed to -0.003, routing fraction essentially flat at 70.4%, crossover gap continued narrowing to +0.000195.

step-90000 (rel_it = 6,077; ~5,077 step into peak) is the first ckpt to show coordinated descent — full K=4 BPB -0.005 vs step-88000 (first negative delta since LR shock), spec α 0.9853 matching Run A's 0.9852 (+0.0001), per-stage acc +0.002 recovery across all stages, routing fraction +1.5pp. The peak-phase plateau lasted ~2-3k step (s086 → s088) before descent began. Head re-alignment (spec α recovery at s088) preceded loss landscape descent by ~2k step, validating the "preparation phase" interpretation.

step-92000 (rel_it = 8,077) shows decoupled metric trajectories: full K=4 BPB re-bounced +0.005 (back to s088000 plateau level — the s090000 descent was an oscillation, not monotonic), but spec α reached 0.9970 — a new trajectory high surpassing Run A endpoint by +0.012, routing cap=0.020 74.62% (trajectory peak in continual phase), and WAND bounds 1→2 / 2→3 are now the narrowest of the entire trajectory (below Run A levels).

step-94000 (rel_it = 10,077) showed BPB at 0.825 (lowest peak-phase value) and routing cap=0.020 at 77.69% with WAND bounds all -13% vs step-92000. Initially we read this as a 4-point uptrend in routing/WAND structure metrics; the next checkpoint invalidated that reading.

step-96000 (rel_it = 12,077) reverts toward the 6-ckpt envelope median: full K=4 BPB +0.002 vs s094000 (now 0.827), routing cap=0.020 -7.18pp drop to 70.51% (back at s086-s088 level), WAND bounds all +12% widened vs s094000 (back near s092000 levels). The s094000 measurement was an oscillation outlier, not the start of a monotonic trend.

6-checkpoint analysis through s096 (s086 → s096): no metric showed a monotonic trend; local-slice BPB oscillated within 0.822-0.832. The peak phase appeared to be a stable oscillation regime around BPB ≈ 0.828. That picture changes from step-100000 onward.

Plateau exit at step-100000 / step-102000: starting at step-100000 (rel_it 16,077) the trajectory breaks out of the 0.828 envelope. step-100000 shows full K=4 BPB 0.8243 (-0.005 vs step-98000), with per-stage acc +0.003 recovery uniform and WAND p99 bounds narrowed -7% — the first ckpt where descent is corroborated by acc and WAND together. step-102000 (rel_it 18,077) deepens the descent to 0.8167 — new trajectory low, below step-86000 starting point of 0.8222 by -0.0055, with spec α recovered to 0.9795 (-0.006 from Run A endpoint), per-stage acc trajectory high 0.4452 (s3=0.4456), and routing cap=0.020 at 76.31%. Three consecutive checkpoints (step-98000 → step-100000 → step-102000) form a monotonic descent: 0.830 → 0.824 → 0.817. Training-log val BPB confirms across 5 evaluations: s98000=0.880 → s100000=0.877 → s101000=0.875 → s101500=0.875 → s102000=0.874.

This is the first phase of the run where the cyclic-LR continuation produces gains beyond noise. The model has now spent ~17,000 step at constant lrm = 0.5 and is finally consolidating into a lower-loss region.

step-104000 (rel_it = 20,077) then bounces back to BPB 0.8239 (+0.007 vs step-102000), with per-stage acc regressing -0.004 and routing dropping to 71%. The s102000 low was therefore a local low rather than the start of a sustained descent — analogous to step-94000's earlier outlier. The 10-checkpoint envelope is now 0.817-0.832 (expanded floor by -0.005 vs the s086-s098 range of 0.822-0.832), but the oscillation regime persists.

The cyclic-LR continuation through 20,000 peak-phase steps has produced a measurable floor expansion (-0.005 below the LR-shock plateau) but has not transitioned into monotonic descent at constant lrm = 0.5.

step-106000 (rel_it = 22,077; final peak-phase checkpoint) reaches NEW trajectory lows simultaneously in BPB and acc: full K=4 BPB 0.8135 (-0.010 vs s104, -0.003 below previous low s102, -0.009 below s086 entry), per-stage acc NEW HIGH 0.4467 s3 / 0.4466 full (gap to Run A reduced to -0.039 — smallest since LR shock), spec α 0.9766 close to Run A baseline, crossover gap +0.000173 (narrowest of trajectory). Both metrics moving together (rather than BPB low coinciding with acc decay, or vice versa) indicates structural descent rather than measurement oscillation.

The 11-checkpoint floor follows a progressively-lower local-low pattern: s094 = 0.8251 → s102 = 0.8167 → s106 = 0.8135, with intermediate bounces back to ~0.825. Each successive local low is below the previous. The model is descending in an oscillating fashion with declining floors rather than a smooth monotonic curve.

Warmdown begins around step-106800 (rel_it ≈ 22,861) — only ~800 step away. step-108000 is the first warmdown checkpoint.

step-108000 (rel_it = 24,077; ~1,217 step into warmdown, lrm ≈ 0.478) confirms warmdown is producing accelerated descent: full K=4 BPB 0.8079 (-0.006 vs step-106000's peak floor of 0.8135 over just 2,000 step, a 1.5× faster rate than peak-phase descent), per-stage acc trajectory high (s3 0.4485 / full 0.4482, gap to Run A reduced to -0.038), and the crossover gap goes NEGATIVE for the first time (-0.000049) — uniform-mean PoE aggregation now produces a BPB lower than the single best stage (s3), meaning stage aggregation is finally adding measurable value rather than being absorbed by the s3 head alone.

BPB is now 0.014 below the s086 LR-shock entry point and 0.083 above the Run A endpoint. Approximately 12,000 step of warmdown remain, with lrm decaying from 0.48 to 0.0. If the current warmdown descent rate sustains, the endpoint BPB will reach the 0.75-0.76 range; if it accelerates as in Run A's late warmdown, approaching 0.7247 becomes plausible.

step-110000 (rel_it = 26,077; ~3,216 step into warmdown, lrm ≈ 0.44) extends warmdown descent at the same constant rate: full K=4 BPB 0.8023 (-0.006 vs step-108000, first ckpt with BPB below 0.81, gap to Run A reduced to +0.078) and per-stage acc NEW HIGH 4th consecutive checkpoint (s3 0.4517 / full 0.4513, gap to Run A reduced to -0.034). The descent is now monotonic for 3 consecutive checkpoints (s106 → s108 → s110: 0.8135 → 0.8079 → 0.8023) at a constant -0.006/2000-step rate. Acc is monotonic for 4 consecutive checkpoints (s104 → s106 → s108 → s110: 0.4419 → 0.4467 → 0.4485 → 0.4517).

Two concerning signals at step-110000: (1) spec α drops -0.022 to 0.9375, the largest single-checkpoint α drop since the LR shock at s086; (2) the crossover gap reverts to positive (+0.000412) after going negative at s108. Both signals indicate that the s108 stage-aggregation gain was a single-checkpoint event, not a stable transition — the drafter (single s3) and target (full K=4) distributions are still oscillating relative to each other despite WAND p99 bounds tightening across all 3 stages (-1% to -4%). The bounds tightening (head outputs converging to narrower distributions) coinciding with α drop (drafter-target mismatch) suggests warmdown-phase head adaptation rates differ across stages, producing a temporary divergence even as the overall loss landscape descends.

Endpoint projection at constant -0.006/2000-step rate: 10,000 step remaining × -0.030 → s134 BPB ≈ 0.77, leaving the Run A gap at +0.045. Reaching Run A's 0.7247 would require warmdown acceleration of ~50% in the final 5,000 step — possible (Run A's own warmdown showed late acceleration) but not certain at current trajectory.

step-112000 (rel_it = 28,077; ~5,216 step into warmdown, lrm ≈ 0.41) is the first checkpoint with BPB below 0.80: full K=4 BPB 0.7960 (-0.006 vs step-110000, 4th consecutive monotonic descent at the same constant -0.006/2000-step rate). The descent chain is now s106 → s108 → s110 → s112 = 0.8135 → 0.8079 → 0.8023 → 0.7960. Per-stage acc continues its monotonic ascent for the 5th consecutive checkpoint (s3 0.4533 / full 0.4531, gap to Run A reduced to -0.033).

The two concerning s110 signals both partially reverse at s112: spec α recovers to 0.9622 (+0.025 vs s110) — the drafter-target mismatch was transient, not a structural divergence. Together with WAND p99 bounds tightening for the 2nd consecutive checkpoint across all 3 stages (-3 to -5% vs s110), this confirms that head outputs are converging to narrower distributions and the drafter (single s3) is realigning with the target (full K=4). All three WAND bounds at s112 are now -6 to -8% below Run A's endpoint values (Run A: 3.9429 / 2.0193 / 1.4479; s112: 3.6416 / 1.8563 / 1.3642) — the continual phase has produced sharper per-stage distributions than the original training endpoint achieved.

The crossover gap, however, does not narrow at s112 — it widens slightly to +0.000649 (s110 was +0.000412). The s108 negative crossover gap remains the only ckpt where α=0 aggregation beat single s3 BPB. The relative ranking of aggregation modes is now stable: single s3 < prefix K'=3 < prefix K'=2 < single s2 < full K=4 < prefix K'=1 = single s0, with full K=4 (α=0) sitting +0.0006 above single s3. The α=0 PoE aggregation is not yet adding measurable value over the deepest single stage in this checkpoint trajectory.

But the high-α aggregation tells a different story: at α=1.0 (pure-sum / Log-OP), s112 BPB is 1.981 vs Run A endpoint 2.10, a -0.12 advantage in the continual phase. The continual-trained model produces sharper individual stage distributions (lower BPB per stage AND lower WAND bounds) such that the high-α aggregation mode, which amplifies all log-probabilities, ends up structurally better than Run A's aggregation. This decouples two questions that often run together: "does each stage learn?" and "does combining stages help?" — the first answer is now clearly yes; the second remains undecided.

Endpoint projection update: 11,000 step remaining × -0.006/2000-step rate = -0.033 → s134 BPB ≈ 0.763, leaving the Run A gap at +0.038. Reaching Run A's 0.7247 would require warmdown acceleration of ~25% in the final 5,500 step — more plausible than at s110 given Run A's own late-warmdown acceleration pattern.

step-114000 (rel_it = 30,077; ~7,216 step into warmdown, lrm ≈ 0.37) marks the first sign of late-warmdown acceleration. Full K=4 BPB drops to 0.7874 — a -0.009 step, the biggest single-checkpoint drop since the LR shock at s086. The constant -0.006/2000-step rate that held across s108 → s112 has accelerated by 50%. The 5-consecutive monotonic descent chain now reads s106 → s108 → s110 → s112 → s114 = 0.8135 → 0.8079 → 0.8023 → 0.7960 → 0.7874.

Acc improvement also accelerates: s3 0.4576 / full 0.4571 (+0.0043 vs s112, the biggest single-checkpoint jump in the trajectory; prior increments were +0.0016 to +0.0032). Gap to Run A reduced to -0.029 (full), the smallest since the LR shock.

The routing fraction breaks out of its 71-77% envelope to 78.37% — a +5.1pp single-checkpoint jump (prior steps averaged <+1pp). This indicates the stage 0 distribution has tightened enough that confidence-aware routing can safely dispatch more positions to the shallow path. The base s0=full agreement rate (0.9719) is now within -0.003 of Run A's 0.9746.

The crossover gap recovers to Run A's baseline: +0.000459 at s114 vs Run A's +0.000465 — for the first time since the LR shock, the relative aggregation structure (α=0 vs single s3) matches Run A's. The α=1 pure-sum PoE BPB is 1.944 vs Run A's 2.10, a -0.16 advantage.

The WAND p99 bounds widen at s114 (+5 to +7% vs s112), and crucially, the deepest stage's 2→3 bound (1.4665) now exceeds Run A's 1.4479 by +1.3%. This is the first measurement where any cumulative-PoE delta range surpasses Run A's endpoint — the s2→s3 transition is now generating more signal magnitude than in the original training endpoint, consistent with the per-stage accuracy gap closing fastest at s3.

Together — accelerating descent, accelerating acc, routing breakout, crossover gap recovery, and WAND 2→3 surpassing Run A — these signals match Run A's own late-warmdown acceleration pattern. The cyclic LR cycle has entered "second-phase descent": the model is rapidly approaching a new minimum.

Updated endpoint projection: at the new -0.009/2000-step rate × 10,000 step remaining = -0.045 → s134 BPB ≈ 0.74. The Run A gap of +0.038 at s114 could now close completely (-0.045 > +0.038). For the first time, the trajectory makes Run A's 0.7247 a realistic target rather than a stretch goal — though sustaining this rate is contingent on whether the late-warmdown signature persists or further accelerates.

step-116000 (rel_it = 32,077; ~9,216 step into warmdown, lrm ≈ 0.34) confirms the acceleration is structural, not noise: BPB 0.7778 (-0.010 vs step-114000, the second consecutive -0.010 drop). The 6-consecutive monotonic descent chain is now 0.8135 → 0.8079 → 0.8023 → 0.7960 → 0.7874 → 0.7778, with the descent rate sustained at -0.010/2000-step (vs the initial -0.006 over s108→s112). The gap to Run A's 0.7247 is +0.053 — half of the s086 LR-shock starting gap of +0.097.

Per-stage acc continues its monotonic climb for the 7th consecutive checkpoint (s3 0.4619 / full 0.4614), maintaining the +0.0043 single-ckpt increment for the second time. Gap to Run A reduced to -0.024 (full) — the smallest since the LR shock.

The two transient signals from step-114000 both reverse here: (1) spec α recovers +0.011 to 0.9708 (Run A endpoint is 0.9852, gap -0.014); (2) WAND p99 bounds re-tighten -3 to -6% across all 3 stages, with the 2→3 bound (1.3719) dropping back below Run A's 1.4479. The deepest stage head's signal magnitude that exceeded Run A at s114 has now returned to below-Run-A levels, but the overall pattern — sharper distributions than the original training endpoint — persists across all 3 stages (-5 to -6% vs Run A).

Routing fraction stabilizes at 78.73% (s114 78.37% → s116 +0.36pp), consolidating the s114 breakout. The base s0=full agreement rate is now 0.9723, within -0.002 of Run A's 0.9746.

The crossover gap widens slightly (+0.000568 vs s114 +0.000459) but α=0 PoE aggregation BPB (0.7778) still tracks single s3 BPB (0.7772) at +0.0006 — the geometric-mean aggregation does not yet beat the deepest single stage. The α=1 pure-sum PoE BPB is 1.888 vs Run A's 2.10 — a -0.21 advantage, the largest yet in the trajectory. High-α aggregation is structurally better in the continual phase than at Run A's endpoint.

Endpoint projection (revised): at the sustained -0.010/2000-step rate × 8,700 step remaining = -0.043 → s134 BPB ≈ 0.735. The Run A gap of +0.053 at s116 would close to +0.010 by endpoint — within striking distance. If the acceleration continues at -0.011/2000-step (which Run A's own late warmdown showed), endpoint BPB could reach 0.728-0.730, fully matching Run A. Reaching or approaching Run A's 0.7247 is now highly probable rather than aspirational.

step-118000 (rel_it = 34,077; ~11,216 step into warmdown, lrm ≈ 0.30) shows the first descent deceleration: BPB 0.7736 (-0.004 vs step-116000), a 50% slowdown from the 2 consecutive -0.010 drops that preceded it. The 7-consecutive monotonic descent chain reads 0.8135 → 0.8079 → 0.8023 → 0.7960 → 0.7874 → 0.7778 → 0.7736 — descent continues but at lowered pace.

Per-stage acc continues for the 8th consecutive new high (s3 0.4632 / full 0.4631) but the increment slowed to +0.0013 vs the prior +0.0043 — consistent with the BPB slowdown. Gap to Run A reduced to -0.023.

Two metrics still strongly advance: routing breakout to 80.78% (+2.0pp from s116, the biggest single-ckpt jump since s114), now within -4.3pp of Run A's 85.05%. And WAND p99 bounds reach their deepest below Run A across all 3 stages: 0→1 -8.83%, 1→2 -9.56%, 2→3 -7.68% — the continual phase produces sharper per-stage distributions than the original training endpoint by more than at any prior checkpoint. This is the 3rd consecutive checkpoint with WAND tightening, despite BPB rate slowing.

The slowdown raises the question of acceleration vs plateau. Three readings are possible:

1. Transient slowdown — single-ckpt noise; next ckpt s120 returns to -0.008+ rate
2. Sustained deceleration — model is approaching a new minimum; further descent will be at -0.003 to -0.005 per 2000 step
3. Plateau — descent ends, BPB oscillates around 0.770-0.775 for remaining warmdown

The supporting signals favor reading 2: continued WAND tightening (distributions still sharpening) + continued routing improvement (s0 confidence still growing) + continued acc improvement (model still learning) all argue against a hard plateau. But the BPB rate halving is a clear signal that the warmdown phase is entering its consolidation regime.

Endpoint projection (revised):

• If -0.004/2000-step rate sustains: 4,200 step × -0.0084 → s134 BPB ≈ 0.765 → Run A gap +0.040 (too wide)
• If average -0.007/2000-step (reading 2): -0.015 → s134 ≈ 0.759 → gap +0.034
• If -0.010 rate recovers (reading 1): -0.021 → s134 ≈ 0.752 → gap +0.027
• Run A's 0.7247 reaching now borderline — the highly-probable scenario from s116 has retracted to plausible-but-uncertain

step-120000 decides this. If the BPB drop is back to -0.008+, the acceleration was paused not ended. If it's -0.004 or smaller, the consolidation regime is real and Run A approach has stalled.

step-120000 (rel_it = 36,077; ~13,216 step into warmdown, lrm ≈ 0.27) gives a mixed answer: BPB descent rate recovers partially to -0.006 (between s118's deceleration -0.004 and the prior -0.010 acceleration). The 8-consecutive monotonic descent chain reads 0.8135 → 0.8079 → 0.8023 → 0.7960 → 0.7874 → 0.7778 → 0.7736 → 0.7681, with Run A gap reduced to +0.043. Per-stage acc continues for the 9th consecutive new high (s3 0.4672 / full 0.4671) and the increment jumps back to +0.0040 (s118 +0.0013, prior +0.0043) — the acc gap to Run A is now -0.019, smallest since the LR shock.

But three signals simultaneously regress against s118:

1. Spec α drop -0.019 to 0.9429 — drafter-target mismatch returns, similar magnitude to s110's transient drop.
2. Routing fraction -2.4pp (80.78 → 78.39%) — the 80% breakout at s118 was a single-checkpoint event, not sustained.
3. WAND p99 bounds widen +7 to +10% across all 3 stages — s118's all-below-Run-A position reverses; the 2→3 bound is now +1.5% above Run A endpoint (was -7.7% at s118). Distributions are dispersing rather than sharpening.

This is a decoupled metric regime: BPB and acc continue to advance, while spec α / routing / WAND simultaneously regress. The pattern is similar to the s110-s114 transient cycle where head adaptation desync produced metric-by-metric oscillation around a sustained loss trend.

Two readings of the decoupling:

• Head adaptation desync at lrm ≈ 0.27: at the lower learning rate, different stage heads adapt at different speeds. The deep head (s3) keeps improving (BPB + acc), but the shallow head (s0) regresses, weakening drafter-target alignment (spec α) and confidence (routing) while broadening the cumulative-PoE delta range (WAND).
• Real plateau approaching with surface noise: BPB is genuinely close to its minimum; the descent we still see is small (-0.006) and could be noise plus residual drift. The other metrics oscillate around their endpoint values.

Endpoint projection (revised again): 7,200 step remaining × -0.006/2000-step rate = -0.022 → s134 BPB ≈ 0.746, Run A gap +0.022. At average -0.007 rate: ≈ 0.743, gap +0.018. The Run A 0.7247 target has shifted from highly probable (at s116) to borderline (at s118) to stretch (at s120). The high-α aggregation advantage continues to expand: α=1.0 pure-sum BPB is 1.885 vs Run A 2.10 (-0.22), the trajectory peak.

step-122000 will clarify whether the s120 decoupling is a transient oscillation (s122 spec α / routing / WAND recover) or a sustained late-warmdown signature (s122 shows the same pattern, BPB rate stays around -0.006).

step-122000 (rel_it = 38,077; ~15,216 step into warmdown, lrm ≈ 0.23) resolves the s120 decoupling: it was transient. All four metrics that regressed at s120 now simultaneously advance:

1. BPB rate re-accelerates to -0.009 (s120 -0.006, s118 -0.004 → s122 -0.009). 9-consecutive monotonic descent: 0.8135 → 0.8079 → 0.8023 → 0.7960 → 0.7874 → 0.7778 → 0.7736 → 0.7681 → 0.7588. Gap to Run A reduced to +0.034 — only 35% of the s086 LR shock (+0.097) remaining.
2. Spec α huge recovery +0.034 to 0.9765 (s120 0.9429). The Run A endpoint is at 0.9852 — only -0.009 to go, meaning 80% of the s086 LR-shock degradation (-0.045) has been recovered.
3. Routing fraction breakout +5.3pp to 83.70% (s120 78.39%). Run A endpoint is 85.05% — only -1.4pp to go, meaning 90% of the s086 routing regression (-14pp from 85 to 71) has been recovered.
4. Acc continues 10th consecutive new high (s3 0.4704 / full 0.4700, gap to Run A -0.016, lowest since LR shock).

The WAND p99 bounds continue to widen (+7 to +9% vs s120), and all 3 bounds now sit above Run A's endpoint values (0→1 +6.85%, 1→2 +4.21%, 2→3 +9.27%). This is the late-warmdown distribution widening signature that Run A itself showed near its endpoint — not a regression.

The s120 decoupling is now confirmed as a 2-ckpt oscillation pattern matching the s110→s114 cycle earlier in the trajectory: one ckpt where spec α / routing / WAND simultaneously sag, followed by a strong recovery ckpt where all four metrics advance together. Head-adaptation desync at moderate-lrm checkpoints, resolving within ~2000 steps.

Endpoint projection (re-upgrade):

• Current -0.009/2000-step rate × ~4,500 step remaining = -0.020 → s134 BPB ≈ 0.739, Run A gap +0.014.
• If rate sustains or accelerates further (as Run A's own late warmdown did): ≈ 0.735 or lower, gap ≤ +0.010.
• Run A 0.7247 reaching shifts from stretch (s120) back to realistic (s122).

The trajectory now reads (in terms of Run A approach probability):

• s116 — highly probable (rate accelerating to -0.010)
• s118 — borderline (rate decelerated to -0.004)
• s120 — stretch (decoupled regime, rate -0.006)
• s122 — realistic again (rate re-accelerated -0.009, all metrics recovered)

step-124000 will indicate whether the s120-s122 oscillation continues (s124 sags again like s118-s120 cycle did) or this is the model entering its final consolidation toward endpoint.

step-124000 (rel_it = 40,077; ~17,216 step into warmdown, lrm ≈ 0.17) confirms the 2-ckpt oscillation pattern: BPB rate decelerates again to -0.003 (s122 was -0.009), the third deceleration ckpt interleaved with four acceleration ckpts in the trajectory. The 10-consecutive monotonic descent chain reads 0.8135 → 0.8079 → 0.8023 → 0.7960 → 0.7874 → 0.7778 → 0.7736 → 0.7681 → 0.7588 → 0.7554. Run A gap is now +0.031 — only 32% of the s086 LR shock (+0.097) remaining.

The spec α follows the same oscillation: 0.9765 (s122 recover) → 0.9593 (s124 sag, -0.017), repeating the s118-s120 cycle. The sag depth is milder than s120's (-0.019), suggesting the oscillation is damping.

Most remarkable: routing fraction at 84.09% — within 1pp of Run A's 85.05%. The base s0=full agreement rate is 0.9741 vs Run A's 0.9746 — essentially identical (within -0.0005). Surface-level metrics (routing, stage agreement) have almost fully recovered to Run A endpoint, well ahead of BPB's recovery curve.

Per-stage acc continues for the 11th consecutive new high (s3 0.4718 / full 0.4717), with gap to Run A reduced to -0.014 (lowest since LR shock). Increment slowed to +0.0014 (s122 was +0.0032), tracking the BPB deceleration.

WAND p99 bounds reverse the s122 widening: -7 to -8% across all 3 stages. Stage 0→1 (3.87) and 1→2 (1.95) are now below Run A; 2→3 (1.45) essentially at Run A. This is the third oscillation cycle in WAND bounds (s116 tighten / s118 tighten / s120 widen / s122 widen / s124 tighten).

The α=1.0 pure-sum PoE BPB reaches 1.857 vs Run A 2.10 — a -0.24 advantage, the trajectory peak. High-α aggregation continues to expand its advantage in the continual phase.

Updated trajectory rate pattern (8 ckpts s110→s124): -0.006, -0.006, -0.009, -0.010, -0.004, -0.006, -0.009, -0.003. Three deceleration ckpts (s118, s120, s124) interleaved with four acceleration ckpts (s112-s116, s122). Average rate -0.0066/2000step.

Endpoint projection (revised):

• 4 most recent ckpts (s116→s124) average rate: -0.0056/2000step
• 5,400 step remaining × -0.0056/2000 = -0.015 → s134 BPB ≈ 0.740, Run A gap +0.015.
• If acceleration cycle dominates: ≈ 0.735, gap +0.010.
• Run A 0.7247 reaching remains realistic (s122 upgrade confirmed).

step-126000 (rel_it = 42,077; ~19,216 step into warmdown, lrm ≈ 0.13) exceeds the prediction: rate strongly re-accelerated to -0.011 (predicted -0.006 to -0.009), the largest single-ckpt descent of the entire warmdown phase. The 11-consecutive monotonic descent chain reads 0.8135 → 0.8079 → 0.8023 → 0.7960 → 0.7874 → 0.7778 → 0.7736 → 0.7681 → 0.7588 → 0.7554 → 0.7448. Run A gap is now +0.020 — only 20.7% of the s086 LR shock (+0.097) remaining.

The 2-ckpt oscillation pattern in spec α is fully confirmed: 0.9429 (s120 sag) → 0.9765 (s122 recover) → 0.9593 (s124 sag) → 0.9708 (s126 recover). Recovery amplitudes are damping: s122 +0.034 → s126 +0.012. The drafter-target system is stabilizing.

Per-stage acc continues for the 12th consecutive new high (s3 0.4759 / full 0.4757), with gap to Run A reduced to -0.010 (Run A 0.4862 within 0.010). Increment jumped to +0.0040 (s124 was +0.0014), tracking the strong BPB acceleration.

Routing fraction settled at 83.02% (s124 was 84.09%). More important: at the 0.020 regression cap the projected mean speedup is 1.686x — Run A endpoint level. (Prior s112–s124 README rows recorded the 0.010-cap speedup ~1.32x in the 0.020 column due to a grep -B 1 column-mis-extraction; the routing fraction column was correctly tracked throughout.)

WAND p99 bounds widen again +7 to +8% across all 3 stages. All three are now above Run A — a 5-ckpt oscillation cycle: s118 tighten → s120 widen → s122 widen → s124 tighten → s126 widen.

The α=1.0 pure-sum PoE BPB drops to 1.812 vs Run A 2.10 — a -0.29 advantage, the new trajectory peak. High-α aggregation continues to expand its advantage in the continual phase, faster than the geometric-mean side.

Updated trajectory rate pattern (9 ckpts s110→s126): -0.006, -0.006, -0.009, -0.010, -0.004, -0.006, -0.009, -0.003, -0.011. Three deceleration ckpts (s118, s120, s124) interleaved with five acceleration ckpts. Average rate -0.0071/2000step.

Endpoint projection (sharp upgrade):

• Last 2 ckpts (s124→s126) rate: -0.0053/1000step
• Last 4 ckpts (s120→s126) average rate: -0.0039/1000step
• ~8,700 step remaining to s134723:
  • Conservative (-0.003/1000): s134 BPB ≈ 0.719, Run A gap -0.006 (OVERSHOOT).
  • Average (-0.004/1000): s134 BPB ≈ 0.710, gap -0.015 overshoot.
  • Aggressive (-0.005/1000): s134 BPB ≈ 0.701, gap -0.024 overshoot.
• Run A reaching is now highly probable; substantive overshoot is plausible.

The trajectory now reads (in terms of Run A approach probability):

• s116 — highly probable (rate -0.010)
• s118 — borderline (rate -0.004)
• s120 — stretch (decoupled, -0.006)
• s122 — realistic (rate -0.009, all metrics recovered)
• s124 — realistic (3rd deceleration, surface metrics within 1pp)
• s126 — probable + overshoot (rate -0.011, largest single-ckpt drop)

step-128000 (rel_it = 44,077; ~21,216 step into warmdown, lrm ≈ 0.10) shows asymmetric decoupling: BPB descends -0.0074 to 0.7374 (12-consecutive monotonic descent, Run A gap reduced to +0.013 — 13.1% of LR shock remaining), per-stage acc 13th consecutive new high (s3 0.4798, gap to Run A only -0.0065), but spec α drops HUGELY -0.039 to 0.9318 — the largest single-ckpt spec α drop of the entire trajectory. Sag amplitudes are amplifying (-0.019 s120, -0.017 s124, -0.039 s128), suggesting the 2-ckpt oscillation pattern is transitioning rather than damping.

The rate -0.0074 sits between deceleration (-0.003) and acceleration (-0.011), breaking the strict alternation pattern. Two interpretations:

1. Drafter-target divergence in late warmdown: at lrm 0.10 with ~1,400 step left of warmdown, head 0 (stage 0, drafter) and full K=4 (target) may absorb sharpening asymmetrically. Stage 0 alone disagrees more often with full K=4, but full K=4's own BPB still drops strongly. This is consistent with the late-warmdown distribution sharpening that the WAND bounds detect (2→3 bound tightening -3.3% from s126).
2. Pre-endpoint regime: with only 6.7k step until s134723, the model is entering a final consolidation phase. Acc and BPB advance reflects genuine endpoint approach; spec α may bounce back at s130 once gradients stabilize at lrm ~0.07.

Routing fraction settled at 81.97% (s126 83.02%, -1.05pp). Speedup at 0.020 cap is 1.671x — still Run A endpoint level. WAND bounds mild tightening (-0.4% to -3.3% from s126); all 3 stages still above Run A but converging.

The α=1.0 pure-sum PoE BPB is 1.812, essentially identical to s126 (1.812) — the trajectory peak at -0.29 advantage holds but no longer expands. High-α aggregation advantage has saturated.

Updated trajectory rate pattern (10 ckpts s110→s128): -0.006, -0.006, -0.009, -0.010, -0.004, -0.006, -0.009, -0.003, -0.011, -0.007. The s128 mid-range rate is the first ckpt that doesn't fit clean decel/accel buckets. Average rate -0.0071/2000step.

Endpoint projection (overshoot near-certain):

• Last 2 ckpts (s126→s128) rate: -0.0037/1000step
• Last 4 ckpts (s122→s128) average rate: -0.0036/1000step
• ~6,700 step remaining to s134723:
  • Conservative (-0.003/1000): s134 BPB ≈ 0.717, Run A gap -0.008 OVERSHOOT.
  • Average (-0.0035/1000): s134 BPB ≈ 0.714, gap -0.011 overshoot.
  • Aggressive (-0.004/1000): s134 BPB ≈ 0.710, gap -0.015 overshoot.
• Run A overshoot is now near-certain; magnitude likely -0.008 to -0.015.

The trajectory now reads (in terms of Run A approach probability):

• s116 — highly probable
• s118 — borderline
• s120 — stretch (decoupled)
• s122 — realistic
• s124 — realistic (3rd deceleration)
• s126 — probable + overshoot
• s128 — overshoot near-certain

step-130000 (rel_it = 46,077; ~23,216 step into warmdown, lrm ≈ 0.09) confirms regime change in spec α: descent -0.0047 to 0.7327 (13-consecutive monotonic descent, Run A gap reduced to +0.008 — only 8.2% of LR shock remaining, 91.8% recovered), per-stage acc 14th consecutive new high (s3 0.4822, gap to Run A only -0.0040, 91% recovery), but spec α recovery from s128 sag is only +0.003 to 0.9348. Compare to prior cycles: s122 +0.034 (after s120 -0.019), s126 +0.012 (after s124 -0.017). The s130 recovery is ~10x weaker than s122, confirming the 2-ckpt oscillation pattern has broken. Drafter-target divergence persists in the asymmetric decoupling regime.

Routing recovered +1.90pp to 83.87% (s128 81.97%), within 1.18pp of Run A 85.05%. Speedup 1.698x at 0.020 cap. WAND bounds minor flux (+0.5% / -0.2% / +1.7% from s128); all 3 stages still above Run A.

The α=1.0 pure-sum PoE BPB drops to 1.790 vs Run A 2.10 — a -0.31 advantage, the new trajectory peak. After plateauing at -0.29 for s126 and s128, the advantage expansion resumes at s130. High-α aggregation continues to consolidate below Run A as endpoint approaches.

Updated trajectory rate pattern (11 ckpts s110→s130): -0.006, -0.006, -0.009, -0.010, -0.004, -0.006, -0.009, -0.003, -0.011, -0.007, -0.005. Last 3 ckpts show clear descent rate smoothing (-0.011 → -0.007 → -0.005), suggesting approach to endpoint plateau. Average rate -0.0069/2000step.

Endpoint projection (overshoot certain, magnitude tightening):

• Last 2 ckpts (s128→s130) rate: -0.0024/1000step (decelerating significantly)
• Last 4 ckpts (s124→s130) average rate: -0.0038/1000step
• ~4,700 step remaining to s134723:
  • Conservative (-0.0024/1000): s134 BPB ≈ 0.722, Run A gap -0.003 OVERSHOOT.
  • Average (-0.0035/1000): s134 BPB ≈ 0.716, gap -0.009 overshoot.
  • Aggressive (-0.0045/1000): s134 BPB ≈ 0.711, gap -0.014 overshoot.
• Overshoot near-certain; magnitude likely -0.003 to -0.014.

The trajectory now reads (in terms of Run A approach probability):

• s122 — realistic
• s124 — realistic
• s126 — probable + overshoot
• s128 — overshoot near-certain
• s130 — overshoot certain (8.2% of shock remaining)

step-132000 (rel_it = 48,077; ~25,216 step into warmdown, lrm ≈ 0.05) is the overshoot milestone: BPB 0.7283 (-0.0044 from s130, 14-consecutive monotonic descent), Run A gap reduced to +0.004 — only 3.7% of LR shock remaining (96.3% recovered). Per-stage acc 15th consecutive new high (s3 0.4842, gap to Run A only -0.002, 95.5% recovery).

Multiple Run A endpoint indicators exceeded at s132:

• Routing fraction +1.14pp ABOVE Run A (86.19% vs 85.05%) — first overshoot in continual phase
• Mean accepted matches Run A 3.83 exactly
• Acc gap -0.002 (essentially at Run A)
• α=1.0 PoE -0.31 advantage (Run B structurally below Run A on high-α aggregation)

Spec α regime clarified: at s130 we observed minimal recovery (+0.003 from s128 -0.039 sag), suggesting the 2-ckpt oscillation pattern had broken. The s132 ckpt reveals the oscillation extended to a 4-ckpt cycle in late warmdown: s128 deep sag (-0.039) → s130 partial recovery (+0.003) → s132 strong recovery (+0.036) matching the magnitude of earlier s122-style recoveries (+0.034). Cycle period doubled but the underlying oscillation persists.

WAND p99 bounds tightening -2 to -3% across all 3 stages. All still above Run A but converging fast (1→2 within +1.4%, 2→3 within +2.5%, 0→1 within +3.3%). Distribution sharpening is approaching Run A endpoint characteristics.

The α=1.0 pure-sum PoE BPB at 1.786 holds the -0.31 advantage vs Run A 2.10 — essentially same as s130. High-α aggregation peak saturated; PoE structure consolidated.

Updated trajectory rate pattern (12 ckpts s110→s132): -0.006, -0.006, -0.009, -0.010, -0.004, -0.006, -0.009, -0.003, -0.011, -0.007, -0.005, -0.004. Final 4 ckpts smooth (-0.005, -0.004 avg) — endpoint plateau-like approach.

Endpoint projection (overshoot confirmed, magnitude tightening further):

• Last 2 ckpts (s130→s132) rate: -0.0022/1000step
• Last 4 ckpts (s126→s132) average rate: -0.0028/1000step
• ~2,700 step remaining to s134723:
  • Conservative (-0.0022/1000): s134 BPB ≈ 0.722, Run A gap -0.003 OVERSHOOT.
  • Average (-0.0028/1000): s134 BPB ≈ 0.720, gap -0.005 overshoot.
  • Aggressive (-0.0035/1000): s134 BPB ≈ 0.719, gap -0.006 overshoot.
• Final overshoot magnitude likely -0.003 to -0.006.

The trajectory now reads (in terms of Run A approach probability):

• s122 — realistic
• s124 — realistic
• s126 — probable + overshoot
• s128 — overshoot near-certain
• s130 — overshoot certain (8% remaining)
• s132 — OVERSHOOT MILESTONE (4% remaining, routing already above)

step-134000 (rel_it = 50,077; ~27,216 step into warmdown, lrm ≈ 0.01 nearly zero) is the FULL OVERSHOOT milestone: BPB 0.7241, gap to Run A -0.000638 OVERSHOOT (100%+ LR shock recovery). 15-consecutive monotonic descent.

Run B exceeds Run A across all primary metrics:

• BPB: -0.0006 below Run A (0.724100 vs 0.724738)
• Per-stage acc: s0, s1, s2, full all ABOVE Run A (+0.0003 to +0.0007); s3 at Run A (-0.0001)
• Routing: +1.74pp above Run A (86.79% vs 85.05%)
• WAND bounds: all 3 stages BELOW Run A (-0.8% / -2.4% / -1.8%) — first time in continual phase
• α=1.0 PoE: -0.33 below Run A (1.772 vs 2.10, new trajectory peak)

Only spec α remains below Run A (-0.050 at 0.9347), but this is a 4-ckpt cycle sag: s128 -0.039 → s130 +0.003 → s132 +0.036 → s134 -0.036. Perfect period-4 oscillation with balanced amplitudes. The oscillation persists right up to lrm 0.01.

Continual learning experiment SUCCEEDED. Starting from -0.097 BPB deficit after the LR shock at s086, Run B has recovered to -0.001 below Run A — full recovery plus modest overshoot.

Updated trajectory rate pattern (13 ckpts s110→s134): -0.006, -0.006, -0.009, -0.010, -0.004, -0.006, -0.009, -0.003, -0.011, -0.007, -0.005, -0.004, -0.004. Smooth -0.004 descent maintained in final 3 ckpts despite lrm approaching zero. Average rate -0.0068/2000step.

Endpoint projection (s134723, only ~700 step left, lrm 0.01 → 0):

• BPB descent at -0.004/2000step → final delta ~-0.0014 → s134723 BPB ≈ 0.723 (Run A 0.7247, gap -0.002 OVERSHOOT).
• Acc may add +0.001 → ~0.4870 (above Run A 0.4862 by +0.0008).
• Spec α uncertain (mid 4-ckpt cycle): could land anywhere 0.93–0.97.
• Routing likely 86–87%.

step-134723 (rel_it = 50,800; training complete, lrm = 0.0) is the CANONICAL ENDPOINT of the continual learning experiment:

Final results vs Run A endpoint (s083923):

• BPB full K=4: 0.7235 vs 0.7247 → -0.001255 OVERSHOOT
• BPB single s3: 0.7227 vs 0.7241 → -0.001336 OVERSHOOT
• All 4 single-stage BPBs: overshoot
• All 3 prefix BPBs: overshoot
• Per-stage acc s0/s1/s2/s3/full: all above Run A (+0.0008 / +0.0003 / +0.0006 / +0.0004 / +0.0004)
• 17th consecutive new high in acc (s086 → s134723)
• Routing fraction 87.19% (Run A 85.05%): +2.14pp ABOVE
• Routing speedup 1.746x (largest margin in continual phase)
• Mean accepted per round: 3.83 MATCHES Run A exactly
• WAND p99 bounds (all 3 stages): -2.0% / -3.7% / -2.3% BELOW Run A (deepest below yet)
• α=1.0 pure-sum PoE BPB: 1.771 vs Run A 2.10 → -0.33 ENDPOINT PEAK
• Spec α: 0.9652 vs Run A 0.9852 → -0.020 (only metric still below; 4-cycle mid-phase landing)

Total improvement over continual phase (s086 LR shock → s134723 endpoint):

• BPB: -0.097 (0.8222 → 0.7235), full recovery + 0.001 overshoot
• Acc s3: +0.044 (0.4424 → 0.4866)
• Routing: +16.14pp (71.05% → 87.19%)
• α=1.0 PoE advantage: -0.33 below Run A (new peak)

11 of 12 measured metrics exceed Run A endpoint. Only spec α remains below (-0.020), and that is due to the 4-ckpt oscillation cycle landing in a mid-recovery phase at the endpoint (cycle: s128 deep sag → s132 strong recovery → s134 deep sag → s134723 partial recovery).

Continual Learning hypothesis CONFIRMED. The B2 cyclic LR schedule (warmup 1000 → peak lrm=0.5 → warmdown_ratio 0.55 → final 0.0) successfully reorganized the model from a transient post-LR-shock state (s086 BPB +0.097 deficit) to an endpoint that exceeds the original Run A baseline across nearly all primary metrics. This validates the continual learning approach for the 3B PoE per-stage head architecture.

The s134723 endpoint is the official release checkpoint for the continual phase of cognica/Cognica-PoE-v1.0-3B-base-continual-learning.

What this release is not

The model card is truncated. Read the rest on Hugging Face.