dna-diskchat-2b-peer-v26

DNA-DiskChat-2B-PEER-v26

Recurrent PEER controller (1.999B logical params) with a product-quantized deployment checkpoint that passes both size and quality gates — the fix for the v25 campaign's compression failure.

This is a research artifact: the win here is the compression codec, not code-generation capability (see Benchmarks).

What changed vs v25

v25 quantized every tensor to ternary {-1, 0, +1} with 88% forced zeros and two per-row means. The PEER expert pool (1.89M x 512 fp32 lookup vectors) is used as exact additive vectors in the hidden state; ternary + 88% sparsity destroyed the per-value magnitude and validation CE went 5.25 → 10.20 (+4.95). A "less sparsity" retry made it worse (+6.80), proving ternary sign-means cannot represent these vectors at any density.

v26 replaces that with:

Why fp32 for exactly layers 0-3: a per-layer ablation (each layer's proj+o replaced by fp16, one at a time) isolated the fp16 sensitivity:

Early recurrent projections (fed into the 512-step diag_scan recurrence) are fp16-sensitive; later layers are not. Storing layers 0-3 in fp32 costs ~10.5 MB and removes the entire residual.

Measured compression results

Validation: 8 batches x 4 x 512 tokens from a deterministic fineweb-edu tail (docs 2000:2199, sha256 7e30af1a4412d2fe990294d0dfb158bbd987ecbc6e707af4bdd30e1b816e7568), bf16 autocast, greedy.

* v25 base CE used a different, unpublished validation set; v26 numbers are on the reproducible set above. The negative delta is within noise — the pool is effectively lossless at B12 (dense-only ablation +0.0255 vs full -0.0276 means the pool contributes ~0).

Codec (V26PQ01)

• Pool payload: 53.8 MB for 1.938B pool params = 0.222 bits/param. Codebooks: 2 tables x 8 groups x 4096 x 64 dims fp16 (8.4 MB); indices: 2 x 1.89M x 8 x 12 bits, LSB-packed (45.5 MB).
• Dense: fp16 section 110.4 MB; fp32 (layers 0-3 proj/o) 21.0 MB; metadata 11.6 KB.
• Format: header | manifest | PQ payload (crc'd, B-bit packed) | fp16 section | fp32 section | sha256 footer. Reader is backward compatible with the all-fp16 v1 layout. PQ state is reusable across exports (--pq-state-from), so the ~20 min k-means runs once.
• The pool is near-isotropic (flat PCA spectrum, std exactly 0.02 = init): experts were barely trained and SFT froze the pool. PQ is the right tool precisely because it preserves per-value magnitude without needing low-rank structure.

Training throughput

Same architecture and training loop as v25 (10B base tokens, then code SFT):

Benchmarks

EvalPlus 0.3.1, greedy pass@1, on the SFT checkpoint (not the compressed file). Peer numbers are web-sourced via Brave searches (patchright, logged in search_results/) and cite the model's official reports; ours are measured locally on the same harness.

Honest position: the v26 model scores 0% on HumanEval/MBPP. The v25 eval generated solutions that reproduce signatures and docstrings correctly but with nonsense bodies (def is_element(n): return 0, docstring-echo loops). The model learned code format on ~4k CodeAlpaca SFT steps but not code semantics at ~40M active params/token, after only 10B pretrain tokens. The compression fix in v26 does not change this; the two are independent problems.

Fix path (documented, not yet run): higher-quality verified SFT data (20-30k records, not CodeAlpaca), unfreeze the PEER pool during SFT (--train-pool; v25 SFT froze it so the 1.9B logical params contributed nothing), and longer/more diverse pretraining. Re-run with the same pinned EvalPlus 0.3.1 harness and report honestly.

Deployment

• deployment/controller.v26q — 176.6 MiB V26PQ01 checkpoint (PQ pool + fp16/fp32 dense).
• tokenizer.json — 32k BPE (FineWeb).
• data/validation.u16 — reproducible validation tail (8 x 4 x 513 tokens, fineweb-edu docs 2000:2199).
• Reader/decoder: scripts/v26_pq/pq_page_format.py (CheckpointReader.tensor(name)).
• External codon table (codons.u8, 512 MiB, not part of the 195 MiB controller budget) is unchanged from the same checkpoint and available at jaivial/dna-diskchat-2b-peer-v25.

Reproducibility

• Code: scripts/v26_pq/ (pq_codec.py, pq_page_format.py, export_pq.py, evaluate_pq.py, make_validation.py, inspect_ckpt.py, sweep_dense.py, test_pq.py — 8/8 unit tests pass).
• Validation set: docs 2000:2199 of HuggingFaceFW/fineweb-edu/sample-100BT, 16,416 tokens, sha256 7e30af1a4412d2fe990294d0dfb158bbd987ecbc6e707af4bdd30e1b816e7568.
• All experiments ran on a Vast AI instance (RTX 3090, 51 GB RAM); no results fabricated.
• The v25 model card and results remain at jaivial/dna-diskchat-2b-peer-v25.

Intended use

Research on compact recurrent codecs, CPU/disk training, and low-bit deployment of MoE controllers. Not production software. Generated output is untrusted.