crisp3b-v8-ckpts

crisp3b-v8 — 2.67B decoder-only LM, pretrained from scratch on 1× B200

Base language model (no instruction tuning). 2.67B parameters, Llama-style architecture, custom 49k byte-level BPE tokenizer with single-digit numbers, trained on 6.67B tokens of crisp-v8 — a corpus built for this run from Nemotron-CC-v2.1 (organic web + synthetic rephrasings + diverse-QA), FineWeb-Edu, FineMath-4plus, StarCoderData and peS2o — in 38.4 GPU-hours on a single NVIDIA B200 (≈$193 of GPU time).

Which file to use: the final model is 2666M_L34_E2560_KV20_b24_t2048_lr1e-03_muon1_3153_final_step45675.pt (end of the warmdown). The other 20 .pt files are intermediate checkpoints of the two training runs, kept for the training curves and for resuming — see Checkpoint files.

It is deliberately undertrained (≈2.5 tokens/parameter, ≈8× below compute-optimal): the run exercises the full stack — data pipeline, tokenizer, recipe search, scaling, checkpointing, evaluation — on one GPU, and is not a competitive model. Everything here is reproducible: the model code and tokenizer are in this repo (code/, tokenizer/), so the loading snippet below runs as-is.

Part of the collection LLM training from scratch: 90M → 2.7B on one GPU.

Architecture

Training

The recipe (optimizer, schedule, attention pattern, value residual, ReLU², FP8) was selected in a 167-run ablation sweep at 160M–500M scale (llm-speedrun) and checked at 280M–3B (llm-scaling-probe). It is identical to v7; the only changes in v8 are the tokenizer and the data.

Data

crisp-v8 was built from scratch for this run (it is not a re-tokenization of Crisp-165B). Two phases with different mixes, both nested-sampled so any prefix preserves the mixture:

So the stable phase is ≈42% real web + 20% synthetic web + 18% math + 13% code + 7% papers, and the anneal phase shifts to math/QA/code (≈40% synthetic). Pipeline: download → cleaning → MinHash near-dedup (316k documents removed) → 13-gram decontamination against WikiText-103, HellaSwag, GSM8K, ARC-Challenge and OpenBookQA → tokenization → nested interleaving → verification. The tokenizer was trained on a 14.6 GB sample drawn from all seven sources. About 44% of the stable tokens and 53% of the anneal tokens come from Nemotron-CC-v2.1; because of NVIDIA's redistribution terms the dataset repo is public but gated (access granted manually). The tokenizer itself is not data and is included here.

Evaluation

Final checkpoint. Validation loss 1.745 (ppl 5.49), 0.738 bits/byte on the anneal validation split — note this split is math/code/QA-heavy and more predictable than general web text, so do not compare this bpb to numbers reported on WikiText or similar. Loss is in v8-tokenizer units and is not comparable to GPT-2-tokenizer models; bits/byte on the same text would be.

Benchmarks below are 0-shot, scored on the full test/validation sets with a forward-only harness (code/eval/post_eval_v8.py): multiple choice by continuation log-likelihood (acc = raw, acc_norm = byte-length-normalized), LAMBADA by greedy exact match of the last word, MMLU macro-averaged over subjects. The same harness reproduces published GPT-2 numbers. (The in-training curves in W&B use a 1,000-example subset per task, which is why they differ by a point or two.)

• vs v7 (same architecture and recipe, GPT-2 tokenizer, ≈2.8B tokens): v8 is ahead or equal on every task except LAMBADA (ARC-challenge and OpenBookQA are within noise). LAMBADA is scored by exact greedy match of a whole word, which penalizes a tokenizer that splits more words (and every number) into several tokens; treat that gap as mostly a scoring artifact.
• vs same-size models trained on far more data: models of this size trained on ≈300B tokens (e.g. Pythia-2.8B, ≈45× the tokens) score well above this on the data-hungry tasks (HellaSwag, SciQ, PIQA, LAMBADA) and closer on ARC-easy and BoolQ. Their published numbers are not reproduced here because they were not run through this harness; the gap is tokens, not recipe. MMLU is at chance for any model at this scale.

Sample

Prompt Once upon a time, temperature 0.9, top-k 40:

Once upon a time there was a young man who was born that same day; and he was very fond of the flower, the rose, so much that he put a little flower in his pocket, and it stood a little over the window. Then he asked the old lady what was the rose, and she replied, "The flower I have seen, but it is very different; it is not the same flower, but a different one."

Fluent and grammatical, weak on facts: a 2.7B model that has seen 6.7B tokens.

Notes

• Warmdown is cheap loss. In v7, a 4-hour linear decay to zero moved validation loss from 2.555 to 2.374 after 12 hours at constant LR had plateaued. In v8 the constant-LR phase plateaued around 2.4B tokens and the warmdown is where the benchmark gains landed (BoolQ 0.45 → 0.62, SciQ 0.63 → 0.69).
• The recipe transfers. Nothing was tuned at 3B; everything came from the 160M–500M sweep and held.
• Tokenizer and data are the only changes from v7. GPT-2 BPE merges digit runs inconsistently; single-digit tokenization is the standard fix for arithmetic.
• Undertraining is deliberate. At this budget the loss-optimal size is ≈1.4B; a 500M model on the same 6.7B tokens would sit near compute-optimal and be competitive today. 2.7B was chosen for the single-GPU engineering exercise (memory, MFU, checkpointing, resume) and for its higher ceiling under continued pretraining.
• A plateaued constant-LR loss is not a finished model. The decay cashes out progress the constant-LR curve does not show: stop the stable phase when the loss is flat, then anneal.

Files and loading

Checkpoint files

All 21 .pt files are raw PyTorch state_dicts written by the training script: {"model", "optimizers", "step", ...} (≈10.9 GB each: bf16 weights ≈5.3 GB + optimizer states). The prefix 2666M_L34_E2560_KV20_b24_t2048_lr1e-03_muon1 encodes the architecture and recipe; the 4-digit suffix is the run id.

This repo also contains:

• code/ — the training script (training/pretrain_500m.py, which defines TransformerLM), the tokenizer adapter (training/v8_tokenizer.py), the in-training eval harness, the launch scripts, the corpus/tokenizer pipeline and the post-hoc eval harness, with a README describing the exact recipe and pipeline order.
• tokenizer/ — crisp3b_v8.json (HF tokenizers format), crisp3b_v8_meta.json (special ids), special_tokens_v8.py.

Requirements: torch>=2.5, tiktoken, tokenizers, huggingface_hub. The script picks CUDA if available, else MPS, else CPU (loading on CPU needs ≈11 GB of RAM).

import os, sys, torch
from huggingface_hub import snapshot_download, hf_hub_download

REPO = "juliannunezb/crisp3b-v8-ckpts"
local = snapshot_download(REPO, allow_patterns=["code/*", "tokenizer/*"])
sys.path.insert(0, f"{local}/code/training")

# The training script reads its architecture from env at import time.
os.environ.update(
    N_EMBD="2560", N_LAYERS="34", N_HEADS="40", N_KV_HEADS="20", INTERMEDIATE_SIZE="10752",
    BLOCK_SIZE="2048", USE_RELU2="1", USE_VALUE_RESIDUAL="1",
    VOCAB_SIZE="49152", EOT_ID="48896",
    V8_TOKENIZER=f"{local}/tokenizer/crisp3b_v8.json",
    SKIP_DATA_LOAD="1", USE_WANDB="0",   # architecture + tokenizer only; no training shards needed
)
import pretrain_500m as M          # defines TransformerLM and loads the tokenizer adapter

model = M.TransformerLM().to(M.device, M.torch_dtype).eval()
ckpt = torch.load(
    hf_hub_download(REPO, "2666M_L34_E2560_KV20_b24_t2048_lr1e-03_muon1_3153_final_step45675.pt"),
    map_location="cpu", weights_only=False,
)
model.load_state_dict({k.replace("_orig_mod.", ""): v for k, v in ckpt["model"].items()}, strict=True)

tok = M.tokenizer                   # .encode(str) -> list[int], .decode(list[int]) -> str
ids = torch.tensor([tok.encode("Once upon a time")], device=M.device)
with torch.no_grad():
    logits, _ = model(ids, targets=None)   # (1, T, 49152); attention is full-causal at inference

code/eval/post_eval_v8.py contains a sampling loop (sample_rollouts) if you want generation; it recomputes the prefix every step (no KV cache). An MLX implementation with a KV cache (≈33 tok/s on M1 Max) exists for the v7 tokenizer and is being ported.

Limitations

English only. Base model: not instruction-tuned, not aligned, not filtered — it will complete any text. Small token budget: factual recall is weak and MMLU is at chance. The tokenizer's 240 reserved ids and 15 of the 16 special tokens never appeared in pretraining and have random embeddings until post-training.

Citation

@misc{nunez_crisp3b_2026,
  author = {Julián Núñez Barrero},
  title  = {crisp3b-v8: a 2.67B language model pretrained from scratch on a single B200},
  year   = {2026},
  url    = {https://huggingface.co/juliannunezb/crisp3b-v8-ckpts}
}