nanochat-d18-sft

nanochat-d18-sft

A small conversational language model trained with nanochat. This is the supervised fine-tuned (SFT) checkpoint — a base model pretrained from scratch, then fine-tuned on conversational data to follow chat turns, answer multiple-choice questions, do basic math/tool use, and spell. The pretrained base checkpoint it was fine-tuned from is available at nanochat-d18-base.

Model details

The SFT mixture covers general conversation (SmolTalk), MMLU, GSM8K, and spelling tasks, plus a small set of synthetic identity conversations.

Training infrastructure

This model was trained on a single consumer GPU (no multi-GPU node):

Because of the single-GPU / 16 GB memory budget, training used device_batch_size=1 with gradient accumulation, and pretraining was run as a resumable job. Pretraining used FP8 matmuls; SFT ran in BF16.

Important: format

This is not a 🤗 transformers model — you cannot load it with AutoModelForCausalLM.from_pretrained. It is a raw nanochat checkpoint and must be run through the nanochat codebase.

Repository contents

chatsft_checkpoints/d18/
  model_000478.pt     # fine-tuned weights
  meta_000478.json    # model config + training args
tokenizer/
  tokenizer.pkl       # tokenizer
  token_bytes.pt      # token byte lengths (used for bpb eval)

The optimizer state is intentionally omitted (inference does not need it).

Usage

# 1. Clone nanochat and set up the environment
git clone https://github.com/karpathy/nanochat
cd nanochat
uv venv && source .venv/bin/activate
uv sync --extra gpu

# 2. Download this repo into the nanochat cache directory
export NANOCHAT_BASE_DIR="$HOME/.cache/nanochat"
hf download nitishpandey04/nanochat-d18-sft --local-dir "$NANOCHAT_BASE_DIR"

# 3. Chat with the model
python -m scripts.chat_cli                       # interactive
python -m scripts.chat_cli -p "Why is the sky blue?"   # one-shot
python -m scripts.chat_web                        # web UI

chat_cli loads the SFT checkpoint (-i sft) by default.

License

MIT, following the nanochat project.