Downloads · 30 days
0
race-nanochat-d24
sahilj2701/race-nanochat-d24
race-nanochat-d24 is a machine learning model from sahilj2701. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for nanochat. The card lists the license as mit.
A full-RACE-attention GPT (depth-24, ~1.384B params) pretrained with nanochat's base pipeline, with every attention layer replaced by RACE attention (a hash-bucket linear-attention mechanism).
Access
Public
Updated Jul 2, 2026
Repo size
4.2 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt4.2 GB · 100%
At a glance
- Library
- nanochat
- License
- mit
- Access
- Public
- Created
- Jul 2, 2026
- Updated
- Jul 2, 2026
- SHA
- a40968db
Model card
From the Hugging Face model README
RACE-nanochat d24 (base)
A full-RACE-attention GPT (depth-24, ~1.384B params) pretrained with nanochat's base pipeline, with every attention layer replaced by RACE attention (a hash-bucket linear-attention mechanism).
- Params: 1,384,122,122 | layers: 24 | n_embd: 1536 | heads: 12
- Context length: 2048 | vocab: 32768
- Training: 5.84B tokens (ClimbMix), bf16, global-causal (window_pattern=L), no fp8
- Step: 5568 (final)
Usage
This is a base (not chat-tuned) model — it continues text. Run it with one command from the companion repo branch:
git clone -b race-chat https://github.com/sahiljoshi515/RACE_Attention.git
cd RACE_Attention/chat && bash run.sh
The launcher downloads these weights, JIT-builds the RACE CUDA kernel (needs an NVIDIA GPU + CUDA 12.x), and drops into an interactive prompt.
Files are laid out to match nanochat's NANOCHAT_BASE_DIR:
base_checkpoints/d24/model_005568.pt, .../meta_005568.json,
tokenizer/tokenizer.pkl, tokenizer/token_bytes.pt.
- Library
- nanochat
- License
- mit
- Created
- Jul 2, 2026
- Updated
- Jul 2, 2026