dabl-dual-ssm

DABL — Dual-SSM Attention-Bridge Learning (research checkpoints)

Trained Mamba-2 / SSD student checkpoints from a study of whether softmax attention's normalizer (Σexp numerator / Σexp denominator) can be transplanted into an SSM student distilled from a Transformer teacher.

Answer: no. On fair held-out data (train on C4, evaluate on WikiText-2 test), every softmax-faithful normalizer variant ties or loses to a plain normalizer-free single-SSM. This is a negative-result artifact release.

• Code / full experimental record: https://github.com/tlstngud/mangochange
• Teachers: microsoft/phi-1_5 (MIT), Qwen/Qwen3-0.6B (Apache-2.0), TinyLlama (Apache-2.0)
• Distillation: 4-stage MOHAWK-style (matrix orientation → hidden-state / bridge → end-to-end KD).

Headline (WikiText-2 test PPL; C4 training, seq1024, 300M tokens, Phi-1.5 teacher)

single, sig_gate, and sieber cluster within ~1 PPL — whether the normalizer stream is absent, fixed, or learned (the Sieber et al. NeurIPS 2024 upgrade), it adds nothing on fair data.

Checkpoints

Each folder under checkpoints/ holds a run's student.pt (raw PyTorch state dict) and its results.json (config + eval metrics). Folder names encode the setting, e.g. sieber_c4_1024_300M, single_hd256_bs1_c4_300M, dabl_phi_300M_c4. Load with the model definition in the GitHub repo (dual_mamba/dabl_v4.py); the matching --denom-mode / dims are in each results.json.

Intended use

Research reproducibility and analysis of Transformer→SSM distillation and attention-normalizer transplantation. Not tuned for downstream deployment.