dna-diskchat-2b-peer-v27

DNA-DiskChat-2B-PEER-v27

Recurrent PEER code model — 2.0B logical parameters, 40M active per token, 185 MB deployable checkpoint.

v27 is a code-focused continuation of the v25/v26 DNA-DiskChat-2B-PEER controller. It trains the same architecture on 7.5B additional tokens (~50% code) and keeps the PEER expert pool unfrozen during SFT (a change from v25, where the pool was frozen and contributed nothing to instruction tuning).

Model Details

Pretraining

Code-focused continuation on 7.5B tokens (github-code-clean + fineweb-edu, ~50% code). The run survived several instance restarts and resumed from checkpoint each time.

Supervised Fine-Tuning

v27 unfreezes the PEER pool during SFT (sparse index_add_ SGD for the pool, AdamW for the dense controller), so the 1.9B logical pool parameters actually participate in instruction tuning.

Code Benchmarks

EvalPlus 0.3.1, greedy pass@1, evaluated on the SFT checkpoint (not the compressed file). Reported honestly.

Honest note: pass@1 is 0.0 on both, matching the v25 family behavior at 40M active parameters. This model is a research artifact for compact recurrent routing and low-bit MoE deployment, not a production code model.

Compression

The SFT checkpoint is product-quantized (8 groups, 12 bits) to a 185 MB controller.v27q file that fits comfortably under the 195 MiB deployment cap, with a negligible CE penalty.

Deployment

• deployment/controller.v27q — PQ checkpoint (product-quantized pool + fp16/fp32 dense).
• checkpoints/sft.pt — full SFT checkpoint (8.84 GB).
• checkpoints/v27_base.pt — final 7.5B pretraining checkpoint.
• tokenizer.json — 32k BPE (FineWeb).
• Reader: scripts/v26_pq/pq_page_format.py (O_DIRECT NVMe page lookups).

# load the PQ controller (see pq_page_format.py for the full reader)
python -c "
from pq_page_format import load_controller
ctrl = load_controller('controller.v27q')
print(ctrl.summary())
"

Reproducibility

• Validation set: fineweb-edu docs 2000:2199 (16,416 tokens, sha256 7e30af1a4412d2fe990294d0dfb158bbd987ecbc6e707af4bdd30e1b816e7568).
• SFT data: sahil2801/code_instructions_120k, contamination-filtered vs HumanEval/MBPP, 20,337 train / 408 validation records, 7.15M assistant tokens.
• All experiments on a Vast AI instance (RTX 3090).
• Training config: batch 24, seq 512, nk 1376, pool-lr 0.05 (pretrain) / 0.02 (SFT), AdamW lr 2e-5.

Intended Use

Research on compact recurrent codecs and low-bit MoE deployment. Generated code is untrusted; review it and execute only in a sandbox.

Limitations

• pass@1 is 0.0 on HumanEval/MBPP; the model does not produce working code solutions at this scale.
• Benchmark scores do not establish security, correctness, or production readiness.
• Missing metrics were not measured and are not estimated from plans or earlier model versions.
• EvalPlus execution runs untrusted model-generated Python and requires isolation.