nanochat-d34-sft

NanoChat D34 SFT

A fine-tuned version of karpathy/nanochat-d34 with mid-training and supervised fine-tuning (SFT) for chat capabilities.

Model Details

• Base Model: karpathy/nanochat-d34 (2.2B parameters)
• Training Pipeline: Base → Mid-Training → SFT
• Hardware: Lambda Labs H100 80GB
• Training Time: ~6 hours total (~5.5h mid-training + ~30min SFT)

Training Details

Mid-Training

• Steps: 813
• Final validation BPB: 0.3282
• Batch size: 4 (reduced from default 32 to fit single H100)

SFT (Supervised Fine-Tuning)

• Steps: 700
• MMLU Accuracy: 42.6%
• ARC-Easy Accuracy: 72.0%

Files

Tokenizer

Checkpoints

Checkpoint Selection Guide:

• General use: model_000700.pt - best MMLU, solid reasoning
• Domain reasoning: full_custom_model_000055.pt - knows domain vocabulary, codebase patterns
• Note: For SQL syntax, use MCP (Context7) rather than relying on fine-tuning (see Recommendations below)

Training Data

The custom training data contains domain-specific examples for analytics queries, particularly ClickHouse SQL for trading/blockchain data analysis.

Usage

Setup

git clone https://github.com/karpathy/nanochat.git
cd nanochat
uv sync

# Download this model
huggingface-cli download victoremnm/nanochat-d34-sft --local-dir ~/nanochat-d34-sft

# Setup directories
mkdir -p ~/.cache/nanochat/tokenizer
mkdir -p ~/.cache/nanochat/chatsft_checkpoints/d34

cp ~/nanochat-d34-sft/tokenizer/* ~/.cache/nanochat/tokenizer/
cp ~/nanochat-d34-sft/model_000719.pt ~/.cache/nanochat/chatsft_checkpoints/d34/
cp ~/nanochat-d34-sft/meta_000719.json ~/.cache/nanochat/chatsft_checkpoints/d34/

Run Chat

# Web interface
uv run python -m scripts.chat_web --source=sft --model-tag=d34 --step=719 --temperature=0.6

# CLI interface
uv run python -m scripts.chat_cli --source=sft --model-tag=d34 --step=719 --temperature=0.6

Training Data Format

The training data uses the standard chat format (JSONL with role/content pairs):

[
  {"role": "user", "content": "How do I find the top traders by volume?"},
  {"role": "assistant", "content": "Here's the SQL query:\n\n```sql\nSELECT trader, SUM(amount) FROM trades GROUP BY trader ORDER BY 2 DESC LIMIT 10;\n```"}
]

Performance Comparison

Domain Evaluation Results

We evaluated the models on domain-specific tasks:

Key Finding: Custom training maintained codebase/reasoning but degraded SQL syntax. The model outputs SQL Server syntax (TOP 10) instead of Clickhouse (LIMIT 10) due to pre-training bias - even though training data was correct.

Recommendations: Fine-tuning + MCP Hybrid

Based on our evaluation, we recommend a hybrid approach:

What the fine-tuned model provides:

• Domain vocabulary (BonkBot, swap_events, materializer, etc.)
• Business context (trading analytics, DEX patterns, holder tracking)
• Codebase awareness (repository structures, API patterns)

What to defer to MCP:

• Current table schemas (DDLs change frequently)
• SQL dialect syntax (Clickhouse functions, BigQuery patterns)
• External documentation (always up-to-date via Context7)

Context7 has 12,916 Clickhouse code snippets available via /clickhouse/clickhouse-docs.

Limitations

• 2.2B parameter model - smaller than production chat models
• Trained on limited data compared to commercial models
• SQL dialect knowledge limited by pre-training bias (use MCP for syntax)
• Best used for learning/experimentation, not production

Training Resources

• NanoChat Repository
• Setup Guide

License

MIT (same as base nanochat)