LiaLeen-Contracts-1

LiaLeen Contracts 1 (Archived – Nov 18, 2025)

Status: Archived artifact. The released RL checkpoint (nanochat d32: 32 layers, d_model=2048, 16 heads) can collapse into #### responses when it loses confidence. A fresh recovery run (rerunning base → contract mid → CUAD SFT) was attempted on Nov 18 2025 but halted after repeated checkpoint corruption and escalating Lambda GPU costs. The model stays online to document the journey—not as a production-ready system.

Why keep it online?

• Full postmortem and code live here: https://github.com/samerGMTM22/My-nanochat
• Shows exactly how a nanochat specialization can succeed (and fail).
• Serves as a persistence trophy and reference for anyone attempting domain-specific LLMs.

What still works

• Training scripts: tokenizer, base, contract mid, CUAD SFT, RL, eval, chat UI.
• Data prep: Atticus contract shards (120) + CUAD conversations can be recreated with the repo.
• Inference code still demonstrates the architecture; just expect the #### regression at runtime.

What failed

1. RL checkpoint behavior – For some prompts the model emits nothing but ####.
2. Recovery sprint (Nov 18) – Rebuilt data + tokenizer, but the new base checkpoint (model_002000.pt) was corrupted (PytorchStreamReader failed reading zip archive). Midtraining crashed when it tried to load that file. After burning another ~$100 on 8×H100 time, the run was terminated.

Lessons learned / next steps

1. Enable --save_every so partial checkpoints survive crashes.
2. Immediately validate every model_*.pt via python - <<'PY' ... torch.load(...) before advancing.
3. Archive {base,mid,sft}_checkpoints off the box before shutting down a cloud instance.
4. Budget for the full pipeline (base + mid + SFT) when attempting a “quick fix.”
5. Share failures as openly as successes.

Original pipeline (for reference)

• Model size: ~2B parameters (32 Transformer blocks × 2048-dim, 16 heads)
• Training stages: FineWeb-Edu base → Pile-of-Law (atticus_contracts) mid → CUAD SFT → RL (step 466)
• Hardware: 8× H100 80 GB (Lambda Labs)
• Total training time: ~41 hours (original run, Nov 2025)

Evaluation snapshot (RL checkpoint)

These scores show the model was always narrow: good for contract-style prompts, baseline elsewhere. After RL it picked up the #### habit.

If you still want to experiment

from nanochat.checkpoint_manager import load_model
from nanochat.engine import Engine

model, tokenizer, _ = load_model("rl", device="cuda", phase="eval")
engine = Engine(model, tokenizer)
# Expect to steer with system prompts or get hashes.

## Creator

- Samer Haddad – built on Andrej Karpathy’s nanochat (https://github.com/karpathy/nanochat)
- Website: http://www.givemethemicofficial.com
- CV: http://www.samerkhaddad.com

## License

MIT – everything here remains free to study, fork, and reuse.

———

Model status: Archived. Use as a reference for nanochat experiments, not as a production legal assistant. The next iteration will restart base → mid → SFT from scratch with strict
checkpoint hygiene.