Spikewhale-SNN-Brain2Qwerty

SpikeWhale-SNN × Brain2Qwerty decoder

A brain2qwerty-style decoder (Conv encoder -> causal Transformer -> LM head) as a readout for Quazim0t0/SpikeWhale-SNN-216M, plus a jointly fine-tuned SNN+decoder checkpoint that beats the SNN's native linear readout on held-out FineWeb-Edu.

Brain2Qwerty decodes multichannel neural time-series (MEG/EEG) into text. SpikeWhale's all-layer pre-reset membrane trunk (state["hidden"], shape [B, T, num_layers*hidden] = [B, T, 5952]) is a multichannel neural signal, so I treat it as one and decode it.

No FAIR code or weights are used - the decoder is a clean re-implementation of the brain2qwerty pattern, so it carries no upstream license.

What was tested

All numbers are held-out bits/token on FineWeb-Edu (lower is better).

1. Frozen readout (SNN frozen, only the decoder trains)

The SNN trunk was extracted once and cached; a full sweep trained decoders on it: capacity 4.4M-27.8M params, dropout 0.1-0.4, weight decay 0.01-0.2, data 1024→4096 windows, training 8→24 epochs, with early stopping.

Every configuration lost to the linear head. Train loss could be driven to ~0.03 bits (the decoder memorizes the train windows) but held-out never crossed the baseline. Varying every axis and seeing no improvement showed the ceiling is the fixed trunk's information content, not decoder capacity, data, or regularization: a frozen trunk exposes no extra generalizable next-token signal.

2. Joint fine-tune (unfreeze the SNN, train end-to-end)

Unfreezing the 216M SNN and training it with the decoder via backprop-through-time (SNN lr 5e-5, decoder lr 1e-4, decoder warm-started from the frozen-readout best, 3000 steps on streamed FineWeb-Edu), then a continued fine-tune at a gentler LR (SNN lr 3e-5, decoder lr 4e-5, ~900 more steps):

Initial descent (monotonic, still falling at the end of the 3000-step run): 5.473 → 5.396 → 5.369 → 5.354 → 5.332 → 5.307 → 5.283 → 5.280 → 5.239 → 5.221.

Segmented fine-tune chain (joint_chain.py): repeatedly resume from the current best and run a fresh 5k-step segment with a NEW seed (a different SGD path over the same streamed pool), early-stopping each segment on patience and promoting only when held-out improves beyond noise. Each segment first drifts up, then finds a lower minimum around step ~1200-4800. It kept improving for four segments, then stalled:

5.187 → 5.107 → 5.094 → 5.057 → 5.020 (seg5 gained <0.004 → chain stopped).

joint_best.pt is the 5.020 checkpoint.

Takeaway. A heavier readout on a frozen trunk cannot beat the linear head. Once the SNN's recurrent weights are allowed to move, it learns to expose signal the brain2qwerty decoder can use - and repeated re-seeded continued fine-tuning compounds into a 0.21-bit win over the linear readout before it saturates. The gains still don't cross into qualitatively coherent text (the undertrained base SNN and the one-timestep-per-token axis cap that), but the direction is unambiguous: unfreezing the SNN and chaining fresh-seed fine-tune segments is what makes the decoder win, and each segment's different SGD path finds further headroom until it saturates.

Generation samples

⚠️ The base SpikeWhale SNN is explicitly undertrained research code, so both readouts produce locally-grammatical but globally-incoherent text. These samples are representative, not cherry-picked. Prompt in italics.

Jointly fine-tuned SNN + decoder (joint_best.pt, val 5.020, temp 0.7, top-k 20):

• In the beginning - X-ray regional structure in the United States. The United Nations General Agency is not an interpretant in the nation.
• Scientists have discovered - "No saturated it." The research results were based on the real University, but the eighth-grade ... Center for Environmental European Union.
• The history of - Johnson's "It is not to be the 'People'." They also makes an overcome it only. They were to be understood and not ...
• She opened the door and - King King Karl Park. As a Keep on July 5, 2007. The first week of the One of Kings.
• The most important thing about - "pretination" is something, not as the same as the amount of Vietnam. At the same word as a "sign" ...

Frozen-SNN decoder (for contrast):

• According to the report, - "It's important to the Atlantic Law." ... the reception of the United States
• During the war, - ... the town of the city in the northern. There are times of the atmospheric area ... the greenhouse gas

The joint model drifts toward an "encyclopedic" register (dates, institutions, place names) - a fingerprint of the FineWeb-Edu fine-tune - but the gain is not visible as a qualitative jump; the base SNN caps both.

Distilled variant (joint_distill.pt)

A sequence-level distillation experiment: continue-fine-tune the joint model on permissive assistant data so it imitates that style. (True logit-KD is impossible - the teachers use a different tokenizer than SpikeWhale's 16.5k byte vocab - so this is sequence-level KD: train on the teacher text.) Lineage, all resumed in order:

1. SmolLM2-135M-Instruct-generated corpus (educational explainer style)
2. SmolTalk2 SFT/OpenHermes_2.5_no_think → chat register, learns "Ah, …" openers (6.646 → 4.773)
3. SmolTalk2 SFT/smol_magpie_ultra_no_think → technical/coding-instruction register (4.597 → 3.841)

joint_distill.pt is stage 3. It clearly imitates an instruction/coding-assistant register - "Here's a…", "help you…", markdown and python def blocks - but the undertrained 216M SNN still can't produce coherent content, so output is style-accurate word-salad. Samples (ChatML prompt → assistant turn, temp 0.7):

• What is a lion? → "Here's a Python program that involves a list of integers that lists the list…"
• Once upon a time → "…-driven approach to help you create a great time complexity to your story… revise your own characters…"

This variant fits assistant chat far better (3.841 vs the 5.020 model's ~6.6 on the same data) but is worse on FineWeb by design - it is a style-transfer artifact, not a better general model. Teachers (SmolLM2, SmolTalk2) are Apache-2.0; MobileLLM-R1 was deliberately not used (noncommercial license - can't publish a derivative).

Reproduce stage 3:

PYTHONPATH=src python joint_finetune.py --resume joint_best.pt --out joint_distill.pt \
  --stream --dataset HuggingFaceTB/SmolTalk2 --config SFT \
  --split smoltalk_smollm3_smol_magpie_ultra_no_think \
  --seq-len 128 --batch-size 4 --grad-accum 2 --snn-lr 3e-5 --dec-lr 5e-5 \
  --max-steps 6000 --eval-every 300 --patience 6

Files

Usage

Requires the base repo's snn package on the path (clone SpikeWhale-SNN-216M and put its src/ on PYTHONPATH, and its snn_stream_program.pth alongside).

# sample from the jointly fine-tuned model
PYTHONPATH=src python generate_joint.py --n 20

Limitations

• Inherits the undertrained base SNN; not a usable general LM.
• The joint checkpoint bundles the full 216M SNN, so it needs the base repo's snn code to load.
• Held-out gain over the linear head reaches ~0.21 bits after the segmented chain, but this is a next-token bits/token metric - it does not translate into qualitatively coherent generation. The undertrained base SNN and the one-timestep-per-token axis cap output quality regardless of the readout.
• Copies of joint_finetune.py / joint_chain.py reference these locally-bundled files; the base repo's published snn package is the only external dependency.