NanoMind

NanoMind · 152M

A 152M parameter GPT-2 style language model trained from scratch on GPT-4 quality instruction data, with a hand-written C++ inference engine featuring AVX2 SIMD, OpenMP parallelism, and persistent KV-cache.

---

Model Details

---

Training Data

~220M tokens from GPT-4 quality sources:

All data formatted as:

System: You are a helpful, thoughtful, and articulate AI assistant.
User: <instruction>
Assistant: <response>

---

Inference Engine

This model ships with a custom C++ daemon (inference.cpp) — not transformers, not llama.cpp.

Features

• AVX2 + FMA matrix-vector multiply (8 floats/cycle)
• AVX2 attention dot products and weighted V accumulation
• OpenMP parallelism across attention heads and matmul rows
• Persistent KV-cache per session — no recomputation on follow-up turns
• LRU eviction — up to 20 concurrent sessions, oldest evicted automatically
• Streaming protocol over stdin/stdout — FastAPI wraps as SSE

Performance (HF Space T4)

Compile

g++ -O3 -march=native -fopenmp -ffast-math -std=c++17 \
    -o inference inference.cpp -lm

---

Files

model.bin format

Header: [n_layer, n_head, n_embd, block_size, vocab_size]  (5 × int32)
wte:    [vocab_size × n_embd]  float32
wpe:    [block_size × n_embd]  float32
Per layer (×16):
  ln1_w, ln1_b, c_attn_w, c_attn_b,
  c_proj_w, c_proj_b, ln2_w, ln2_b,
  mlp_fc_w, mlp_fc_b, mlp_proj_w, mlp_proj_b
ln_f_w, ln_f_b, lm_head_w

---

API

# Chat (streaming SSE)
curl -X POST http://localhost:7860/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "What is machine learning?", "session_id": "abc123"}'

# Health
curl http://localhost:7860/health

# Metrics
curl http://localhost:7860/metrics

# Reset session
curl -X POST http://localhost:7860/chat/reset \
  -d '{"session_id": "abc123"}'

---

Known Limitations

• Reasoning: 152M parameters cannot chain multi-step logic. Expect factual recall and pattern matching, not reasoning.
• Hallucination: No RLHF/DPO — model will confidently say wrong things.
• Context: Hard limit of 1024 tokens (~750 words).
• Evaluation: Trained on loss minimization only. No MMLU, HellaSwag, or held-out eval set — a proper eval harness is the next planned addition.
• serialize() note: shape_hint param is only used when t=None (bias=True config). Would refactor signature in v2.
• vocab_size=50304: GPT-2's actual vocab is 50,257. Padded to 50,304 (nearest multiple of 64) for memory alignment — standard trick, undocumented in v1.

---

Citation

@misc{nanomind2025,
  author       = {NOT-OMEGA},
  title        = {NanoMind: A 152M GPT-2 Model with Custom C++ Inference},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/NOT-OMEGA/NanoMind}},
}

---

Trained from scratch · Custom C++ engine · No frameworks at inference time