memsplit-checkpoints

memsplit-checkpoints

Research checkpoints from the FCUO / post-training capacity-expansion project (Qwen3), backed up before B200 server release. Weights-only + sparse learned-memory (optimizer state and regenerable calib caches excluded).

License

Weights derive from Qwen3 (Apache-2.0) → Apache-2.0. Exception: ckd_* (agentic) were trained partly on APIGen-MT-5k (CC-BY-NC-4.0) → those subdirs are non-commercial / research-only. Training data: OpenR1-Math-220k (Apache-2.0), openwebmath (ODC-By), xlam-function-calling-60k (CC-BY-4.0), ToolACE (Apache-2.0), APIGen-MT-5k (CC-BY-NC-4.0), PopQA. Third-party datasets are NOT redistributed here.

Headline result

pk_p2_frozen — a FROZEN Qwen3-4B-Base backbone + a large content-addressed product-key memory (dq512 / 65536 keys / 1.6B params) reaches 52.5% direct fact recall (full 14k set; 77.5 on the first-3000 subset) vs small-PK 24.3 and bare-frozen 0 — the project's strongest near-zero-inference capacity positive (direct-probe only; generalization untested).

Format

• Weights: standard HF *.safetensors + config/tokenizer (+ ctx_gate.pt for grafts, pk_memory.pt for PK).
• sparse_memory/layer_*.pt (learned memory grafts only): nonzero rows only. Reconstruct dense:

  import torch
  d = torch.load("sparse_memory/layer_00.pt")
  dense = torch.zeros(d["V"], d["I"], dtype=d["mean_rows"].dtype)
  dense[d["rows"].long()] = d["mean_rows"]
  

• Graft checkpoints need their calib table to run; calib dirs are regenerable via fcuo/calibrate_selfgen_math.py (see code repo). Frozen-graft trained_memory (= calib copy) is omitted.

Code: github.com/hyunseoklee-ai/memory_split. Full result docs: private backup repo.