geolip-aleph-qwen

geolip-aleph-qwen

The AlephLM generation stake: composing the certified components of the GeoLIP aleph program into a utilizable generation structure on a real pretrained substrate. Successor to geolip-aleph-differentiation (exp011–exp021, now on hold), whose measurement program produced the design rules this repo spends. Experiment numbering restarts at 1.

The architecture (every component certified upstream)

Design rules inherited from the closed line: pure Adam wd=0 on geometric paths; decoders read M̂ never M; slot-parallel reconstructive consumption (the collapse cure); CV is a readout; drift-check before any freeze; ≥2 seeds before any claim; every package ships standalone with a self-asserting results builder.

Experiment line

• exp001_relay — the relay retrofit at 0.5B ✅: frozen 17.798 → relay 14.09 / matched-MLP 13.93 (−21% ppl at <0.6% trainable, 2 seeds, replication <0.02). The GPT-2 ppl ordering inverts (substrate-scoped) while the gate mechanism replicates (relay gates grow 0.047→0.081, MLP gates shrink) — the gate-vs-ppl dissociation, certified at two substrates.

• exp002_refine — refining the relay ✅: the aleph-addressed patchwork consumer wins (13.997 mean, 2 seeds, first relay under 14); width/strobe saturated; coverage beats concentration (−0.48); the wide MLP diverges 1/2 seeds vs 0% across all relay variants; ordering budget-stable at 6k steps (2 seeds).

• exp003_instruct — instruct tasks + register differentiation ✅: hallucination reduction quantified (specialist grounding 0.979/0.958 vs frozen 0.542 — >10× fewer failures); register capture found (a single-task specialist answers every register in its own; perfect JSON, 0.0 task-validity); multi-task resolves it (all registers ≥0.875); the register-differentiation hypothesis confirmed on the sign-code surface (~2× deep-layer separation vs a wikitext-trained stack, 2 seeds) — the gauge only the aleph relay exposes.

• exp004_composite — composite construction gate ✅ (two acts, both ledgered): combination-scene targets from the fused datasets (qwen-deepfashion-fused, qwen-synth-characters-fused). Act 1: packed-window training teaches the ~3k-token composite register flawlessly (json_error 0.0 vs frozen 0.833) but never teaches stopping (truncated 1.000; still 0.938 with the cap lifted — true rows average 4.3k tokens, so 1024-token windows almost never contain an ending). Act 2: row-aligned SFT fixes it completely — structural_ok 1.000 (16/16), every failure bucket 0.0. Termination is a property of sample semantics, not the adapter stack. Honest caveats: entity over-emission 1.67×, grounding-lite 0.062 — construction solved, faithfulness is the next target. Ships the long-sequence memory rider (gradient checkpointing + chunked CE; 66 GB spill → 8.8 GB peak).

• exp005_taskmoe — multi-anchor task-MoE ✅ (with its own capacity control): the routed composite of three frozen register specialists + one trainable anchor serves all registers (0.875/1.0/1.0) under dense signed dispatch — but the control decides attribution: random frozen stacks match (0.979/0.979/1.0), so at register distance the specialists were passengers (the monolith-capacity trainable anchor carries; specialists may mildly interfere). No routing collapse (usage_ppl 3.7–3.9 of 4, 4/4 paths).

• exp006_story — story anchors, 4 request methods ✅: continuation/instruct/JSON registers all 1.0 (frozen 0.67/0.83/0.83), but the checkable keyword constraint stays at 0.083 — unmoved from frozen: format learning ≠ constraint compliance (teacher forcing makes ignorable instructions unlearned). Strong register separation (0.46 @L8).

• exp007_math — math anchors, 4 ask-methods ✅: format locks in 4/4 (frozen nl format 0.312), no correctness tax, and held-out f(x) evaluation improves 0.688 → 0.938. With exp006 this yields the predictability principle: the LM loss teaches exactly what target-token predictability demands — answers force using the question; story keywords are ignorable.

• exp008_tokendiff — CIFAR-10 discrete token diffusion through a gated adapter (feasibility gate): the frozen trunk + 5.5M adapter learns a genuine class-conditional discrete denoiser — beats the identity baseline at every corruption level (0.527 vs 0.252 at t=0.75; 0.378 vs 0.002 at pure noise). Greedy iterative decoding mode-collapses to the modal (all-dark) image — the ledgered v1 sampler lesson; stochastic-sampling amendment included.

• exp009_family — THE FAMILY ✅ (the flagship): six anchors — five frozen specialists (3 registers, story, math) + one trainable — under one dense signed dispatch serve 11 sub-competences at ~specialist level through a single routed model. Domain-level dispatch specialization without a selector (story prompts: 0.82 usage on the story anchor) and surgical modular decoupling both directions (story-off kills story, math stays perfect; math-off degrades math, story untouched). With exp005's control: the two-regime dispatch law — anchors specialize-and-carry at domain distance, blend into redundancy at register distance, and the sign-code register probe's separation predicts the regime in advance.

• exp010 — seed-2 ablations (folded into the packages above as amendments): the two-regime dispatch law certified on two seeds (story usage 0.821 vs 0.820; surgical decoupling both directions, both seeds), with the register-level control's verdict amended honestly: the seed-0 MATCH was seed-dependent — at seed 1 the random-stack control falls (0.917/0.708/0.563) while the real-specialist composite holds (0.938/1.0/1.0). Specialists confer seed-robustness. The preregistered OOD-band cell confirmed: the tokendiff anchor draws 0.918 usage (predicted >0.9 before the run), and the routed family preserves the standalone denoiser to the third decimal.

• exp011_newanchors — masked-infill + sentence similarity ✅: the infill anchor beats a real frozen-LM floor by +18/+14/+10 points across corruption rates; the similarity anchor reaches 0.494 recall@1 over 465 distractors (frozen 0.054, lexical 0.073) — and the aleph-native headline: sign-code Hamming retrieval at 0.359 on the trained stack (0.023 before training) — contrastive pressure reorganized the discrete surface itself. The absolute-objective arm (recon_target) works at half strength: a scoped boundary marker for the absolute-vs-comparative doctrine.

• exp012_bridges — foreign encoders as anchors ✅: the CLIP bridge is the program's first demonstrated cross-modal conditioning — captions generated under a frozen CLIP image tower score 0.273 against their true image vs 0.170 under shuffled images. The BERT bridge is an honest negative twice over (boilerplate targets, then content-bearing targets against a strong generic prior) — the predictability principle's third datum, and the roster decision it forced is ledgered.

• exp013_collective — the grand collective ✅: ten anchors under one dense signed dispatch. Three preregistrations, three confirmations (infill usage 0.549 in the predicted 0.5–0.8 band; simesh a passenger as its 0.22 separation predicted; tokendiff 0.886) — the register probe now calls dispatch regimes before mixtures are built. The full N×N matrix (12 configs × 5 domains vs a frozen reference row) keeps decoupling surgical where anchors carry, and returns two honest boundaries: the hardest extraction register starves at this domain count and budget (a capacity/schedule limit found by pushing limits; remedy arm recorded), and the sub-capacity attribution pair is inconclusive under that same confound. Mixed-batch test: per-row usage tracks each row's domain inside one padded batch — dispatch is per-position, no cross-batch coupling.

• exp014_toggles — the toggle registry ✅: span gating via harness-controlled control tokens (band 33000+, stripped before the trunk ever sees them — the moe_toggle tool). The mask path is bit-exact (all-off ≡ freshly loaded frozen trunk, max|Δlogit| = 0; strip ≡ identity). Per-segment activation beats all-on (1.0 vs 0.875 on the chained math segment — toggling removes mixed-context interference), the inverted-gating null shows directional damage (0.667) where the test discriminates, and the three-segment demo (denoise → describe → compute) ships with its two characterizations: greedy collapse in the demo path, and denominator damping of solo-toggled anchors (~1/10 amplitude at ten anchors) — the doctrinal trade-off, measured.

Each experiment lands in expNNN_*/ with code, README, self-asserting results builder, ledger, and checkpoints, reproducible from inside its folder.

Relation to prior work

The upstream repo holds the laws this build rests on: the consumption law, the address-bottleneck's scope, the projective-codebook law, the KD regime map, heredity-as-robustness, the memorization/generalization inverse law, and the frozen-keyer requirement — each with ledgers, controls, and replication seeds. This repo is where they compound.

License

MIT. AbstractPhil + Claude Fable 5.

Why the numbering jumps exp009 -> exp011

exp010 was the seed-ablation program (staged A/B/C): re-runs of earlier experiments' cells at a second seed plus preregistered composition extensions. It had no bed of its own, so its rows shipped as amendments inside the packages they test (exp005_taskmoe, exp008_tokendiff, ...). The exp010_seeds/ folder is an index: the A1/A1b verdicts (two-regime dispatch law certified at 2 seeds; the preregistered OOD-band usage prediction confirmed at 0.918 vs >0.9) and pointers to every amendment row.