Downloads · 30 days
0
crisp3b-v7-ckpts
juliannunezb/crisp3b-v7-ckpts
crisp3b-v7-ckpts is a text generation model from juliannunezb. Use it when you need the model to write or continue text. It is set up for pytorch. The card lists the license as mit.
Base language model. Same architecture and training recipe as crisp3b-v8 (34 layers, dmodel 2560, GQA 40/20, FlexAttention sliding window + global layers, ReLU², value residual, Muon + AdamW, FP8 forward), with the GP…
Access
Public
Updated Sep 16, 2026
Repo size
27.2 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt27.2 GB · 100%
At a glance
- Task
- Text Generation
- Library
- pytorch
- License
- mit
- Access
- Public
- Created
- Jun 17, 2026
- Updated
- Sep 16, 2026
- SHA
- 82a2744d
Try a prompt
Datasets
Model card
From the Hugging Face model README
crisp3b-v7 — 2.67B decoder-only LM, GPT-2 tokenizer
Base language model. Same architecture and training recipe as crisp3b-v8 (34 layers, d_model 2560, GQA 40/20, FlexAttention sliding window + global layers, ReLU², value residual, Muon + AdamW, FP8 forward), with the GPT-2 BPE tokenizer (vocab 50,257, padded to 50,304) instead of the v8 tokenizer. Trained on Crisp-165B with loader-side source reweighting (target mix: FineWeb-Edu 32%, DCLM 20%, code 13%, FineMath 9%, Dolmino 8%, peS2o 8%, Cosmopedia 4%, PG19 3%, Wikipedia 3%).
Part of the collection LLM training from scratch: 90M → 2.7B on one GPU.
Runs
| Run | W&B | Steps | Notes |
|---|---|---|---|
| Stable phase | …_0955 | 43,500 | Constant LR after 50-step warmup. Stopped when validation loss plateaued (best 2.535 at step 37k; band 2.54–2.60 for ≈6k steps). ≈2.14B tokens (43,500 steps × 49,152), ≈12h. |
| Warmdown | …_9225 | 13,561 | Resumed model + optimizer from step 40,000 (val 2.555). Linear LR decay to 0 over ≈4h. Final val loss 2.374. ≈0.67B tokens (13,561 steps × 49,152). |
Hardware: 1× NVIDIA B200, Vast.ai (≈$3.15/h). Batch 24 × 2048, ≈50k tokens/s. ≈2.8B tokens and ≈16 GPU-hours (≈$50) across both runs; no token was seen twice.
Evaluation (final warmdown checkpoint)
| Benchmark | Score |
|---|---|
| ARC-easy | 55.8 |
| SciQ | 66.4 |
| PIQA | ≈66 |
| HellaSwag (acc_norm) | ≈37 |
| LAMBADA | ≈34 |
| MMLU | 25.4 |
Validation loss 2.374 (GPT-2 tokenizer units).
Downstream checkpoints
| Stage | Repo | W&B | Data |
|---|---|---|---|
| SFT (ChatML, 3 epochs, batch 16, doc-masked packing) | crisp3b-sft-ckpts | crisp3b_sft_v1 | crisp-sft-chat — 329,722 conversations, 218M tokens |
Reasoning cold-start (<think>…</think>) | crisp3b-reason-ckpts | v1, v2 | crisp-reason-cold — 34,821 short verified math/science traces + 5k no-think |
| GRPO on arithmetic | — | crisp3b-grpo (4 runs, all crashed) | — |
Chat special tokens (<|im_start|>, <|im_end|>, tool-call and think tags) occupy ids 50257+ in the padded vocab; see chat_format.py in the training repo.
Files
| File | Contents |
|---|---|
2669M_..._9225_final_step13561.pt | Final warmdown checkpoint: model + optimizer states + step |
2669M_..._8830_final_step651.pt | Earlier short warmdown test |
w_final_weights.pt | Weights only, bf16, ≈5.3 GB; input of the SFT stage and the MLX playground |
Loading is the same as for crisp3b-v8, with tiktoken.get_encoding("gpt2") as tokenizer and logits for ids ≥ 50257 masked at inference.
Limitations
English only. Base model, not instruction-tuned, not aligned or filtered. ≈3 tokens/parameter; MMLU at chance.
- Task
- Text Generation
- Library
- pytorch
- License
- mit
- Languages
- en
- Created
- Jun 17, 2026
- Updated
- Sep 16, 2026