dna-diskchat-2b-peer-v1

DNA-DiskChat-2B-PEER-v1 (v20 baseline - superseded)

Ternary (BitNet b1.58) controller with a PEER product-key expert FFN in all 16 blocks, each block owning a private pool of 30,976 rank-1 experts (507.5M expert params total), on top of the DNA codon memory table (512 MB, SSD-resident).

This is the v20 baseline. It is superseded by dna-diskchat-2b-peer-v21, which is 1.44x faster in training AND reaches lower CE per token by reorganising the same 507.5M expert parameters into ONE shared pool read by 3 layers (per Memory Layers at Scale, arXiv:2412.09764).

Measured (RTX 5060 Ti 16 GB, streamed FineWeb-Edu + Cosmopedia)

Architecture

16 recurrent ternary blocks (chunk-parallel gated linear recurrence, BitNet b1.58 STE)
  every block: PEER FFN -> its OWN pool of 176^2 = 30,976 rank-1 experts
codon memory table: 2048 x 2048 = 4.19M rows, 128 B/row (512 MB on SSD, 256 B/token)
routing: product-key, h*k = 8 active experts per layer

Why it was superseded

Placing a private expert pool in every block costs 16 expert gathers per token to buy capacity that 3 gathers can buy. Memory layers are memory-bandwidth bound, not FLOP bound, so the gathers dominate. See the v21 analysis for the full measured breakdown.

Files

checkpoints/ckpt_*.pt - {'model': state_dict, 'config': ...}. The architecture code is scripts/model_dna_peer.py + scripts/model_dna.py; tokenizer is tokenizer.json.

References

• He, Mixture of A Million Experts (PEER) - arXiv:2407.04153
• Berges et al., Memory Layers at Scale - arXiv:2412.09764
• Ma et al., BitNet b1.58 - arXiv:2402.17764