synapse_v2

---

language:

• en license: mit tags:
• transformer
• decoder-only
• from-scratch
• bpe-tokenizer
• instruction-tuning
• educational
• nlp
• pytorch
• small-language-model
• custom-architecture pipeline_tag: text-generation

---

Synapse v2 — Decoder-Only Transformer Built from First Principles

Built by Abhinav Tyagi
📄 GitHub • 💼 LinkedIn • 📜 Research Paper (PDF)

---

Overview

Synapse v2 is a 3.67M parameter decoder-only Transformer built entirely from scratch — no HuggingFace Trainer, no pre-built architecture. Every component is implemented manually: the attention mechanism, BPE tokenizer, positional embeddings, training loop, and inference pipeline.

This is the second generation of the Synapse series, representing a 4.6× parameter scale-up and 3× depth increase from Synapse v1 — with a core focus on transitioning from memorization to generalization.

"This work prioritizes understanding over performance. Building from first principles reveals what production models abstract away." — Abhinav Tyagi

---

Evolution: v1 → v2

The core lesson: systematic scaling + regularization + quality data = generalization.

---

Architecture

Raw Text
  ↓
Turbo BPE Tokenization (GPT-2 Regex Pattern)
  ↓
Token Embeddings (1037 → 128)
  +
Positional Embeddings (64 → 128)
  ↓
12× Transformer Blocks
  [LayerNorm → Multi-Head Attention (4 heads) → Residual]
  [LayerNorm → Feed-Forward (128→512→128) → Residual]
  ↓
Final LayerNorm
  ↓
LM Head (128 → 1037)
  ↓
Next Token

Model Specs

Parameter Breakdown

---

Tokenizer: Turbo BPE

Built from scratch — no dependency on HuggingFace or tiktoken.

Key innovation: GPT-2 regex pre-tokenization pattern (same pattern used by GPT-2, GPT-3, GPT-4, Llama):

's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+

This ensures correct handling of contractions, numbers, punctuation, and whitespace — preventing cross-boundary merges that degrade tokenization quality.

Optimizations implemented:

• Doubly-linked list for O(1) merge updates (vs O(n) list rebuilding)
• Hash map for O(1) pair lookup
• Position sets to track all occurrences efficiently
• Rank-based encoding (deterministic, matches GPT-2/GPT-4 behavior)

Compression achieved: 3–4× sequence length reduction.

Example:

"don't" → Basic BPE: ['d','o','n',"'",'t']  # 5 tokens
"don't" → Turbo BPE: ['don', "'t"]           # 2 tokens ✅

---

Training

Dataset

Hybrid instruction-tuned corpus (~18–20K pairs, ~200–500K tokens):

Training Configuration

Training Results

Starting from random initialization (loss 8.54 ≈ log(1037)), the model converged to 2.04 — demonstrating genuine learning, not memorization. The train/val gap (2.04 vs 3.26) confirms effective regularization via dropout.

---

Key Design Decisions Explained

Why 12 layers? Enables hierarchical learning: syntax (lower layers) → semantics (middle) → reasoning (upper). Empirically validated as the sweet spot for ~4M parameter models.

Why pre-norm (LayerNorm before attention)? More stable training — gradients don't explode or vanish as easily. Standard in modern architectures (GPT-3, Claude). Allows higher learning rates.

Why dropout 0.1? The v1 model had a perplexity of 1.05 — pure memorization, useless for generalization. Dropout forced the model to learn robust patterns. The validation gap (3.26 vs 2.04) proves it worked.

Why 64-token context (smaller than v1's 128)? Most conversational turns fit in 50–80 tokens. Reduces O(T²) attention cost by 4× — faster training, same capability for dialogue tasks.

---

Usage

import torch
from model import TinyGPT
from tokenizer import TurboBPE

# Load model
model = TinyGPT(
    vocab_size=1037,
    n_embd=128,
    n_head=4,
    n_layer=12,
    block_size=64,
    dropout=0.0  # Disable dropout at inference
)
model.load_state_dict(torch.load("synapse_v2.pt", map_location="cpu"))
model.eval()

# Load tokenizer
tokenizer = TurboBPE.load("tokenizer_state.json")

# Generate
prompt = "Instruction: What is a Transformer model?\nResponse:"
tokens = tokenizer.encode(prompt)
input_ids = torch.tensor([tokens])

with torch.no_grad():
    for _ in range(100):
        logits, _ = model(input_ids)
        next_token = torch.argmax(logits[:, -1, :], dim=-1)
        input_ids = torch.cat([input_ids, next_token.unsqueeze(0)], dim=1)

print(tokenizer.decode(input_ids[0].tolist()))

---

Lessons Learned

Building Synapse v2 from scratch produced insights that using pre-built frameworks hides:

1. Tokenization quality is foundational — Bad tokenization forces the model to waste capacity learning token boundaries instead of meaning
2. Dropout is not optional at small scale — Without it, small models memorize training data completely (perplexity 1.05 in v1)
3. Pre-norm vs post-norm matters — Pre-norm enabled stable training at 12 layers; post-norm would have required careful LR tuning
4. Validation loss is the only truth — Training loss is meaningless without a validation signal
5. Instruction formatting teaches task structure — The model learns what a "question" and "answer" look like before it learns the content

---

Synapse Series

---

About the Author

Abhinav Tyagi is an LLM Engineer who builds AI systems from the ground up — from custom tokenizers and transformer architectures to production RAG pipelines and agentic systems.

Other work:

• Synapse-124M — 124M parameter transformer from scratch: GQA, MoE, Sliding Window, NTK-RoPE, SwiGLU, custom BPE
• Synapse Wingman — Agentic AI desktop assistant (Telegram → PC control, vision, WhatsApp automation)
• Smart RAG Chatbot — Hybrid RAG with Chain of Verification (CoVe), multi-query generation, FAISS
• Psywarp — Published research on multimodal cognitive AI framework (DOI: 10.5281/zenodo.18182199)

📧 [email protected]
🐙 GitHub
💼 LinkedIn

---

Citation

@misc{tyagi2026synapsev2,
  author = {Tyagi, Abhinav},
  title  = {Synapse v2: A Decoder-Only Transformer Language Model Built from First Principles},
  year   = {2026},
  url    = {https://huggingface.co/Abhinav-Tyagi/synapse_v2}
}

---

License

MIT — free to use, modify, and distribute with attribution.

---

"Understanding requires building. Building requires breaking things. Breaking things requires documentation."
— Abhinav Tyagi