Raven-1

<div align="center">

🐦‍⬛ RAVEN-1

A 29.5M parameter language model, built entirely from scratch.

No frameworks. No shortcuts. Just PyTorch, math, and stubbornness.

---

</div>

---

What is RAVEN-1?

RAVEN-1 is a GPT-style decoder-only transformer language model with 29.51 million parameters. Every single component — tokenizer, model architecture, training loop, data pipeline, inference engine, and web UI — is written from scratch in Python and PyTorch.

No HuggingFace transformers. No pre-trained weights borrowed. No abstractions hiding the work.

Why Build From Scratch?

• Full understanding — Every matrix multiply, every attention mask, every gradient step is explicit and visible.
• Colab-friendly — Designed to train end-to-end on Google Colab's free T4 GPU (15GB VRAM, 12GB RAM).
• Two-phase training — Pre-trained on a 389MB text corpus for language modeling, then post-trained on 977MB of curated conversational data for personality.
• Production-grade tooling — Resume-safe checkpoints, mixed precision, Google Drive sync, streaming data pipelines, and a Gradio web UI for deployment.

Personality

RAVEN-1 was post-trained on conversational data that gives it a distinct personality — dry, sarcastic, concise. It's not trying to be helpful. It's trying to be honest. Sometimes annoyingly so.

User: I'm bored
Raven: Good. Boredom is where ideas start.

User: good morning
Raven: Morning. Let's keep it simple.

User: thanks!
Raven: Thanks. I'll file that away.

---

🚀 Quick Start (Download Weights from Hugging Face)

[!IMPORTANT] Because model checkpoints (.pt files) are large, they are not tracked in this GitHub repository. Instead, the full model weights and trained BPE tokenizer are hosted on Hugging Face.

To set up and run RAVEN-1 locally, follow these simple steps to download the repository and fetch the weights directly from Hugging Face:

1. Clone & Set Up Environment

# Clone the repository
git clone https://github.com/itslokeshx/Raven-1.git
cd Raven-1

# Create the checkpoints directory
mkdir -p checkpoints

# Install required packages
pip install -r requirements.txt

2. Download Tokenizer & Weights from Hugging Face

Download the files directly into the checkpoints/ directory:

# 1. Download BPE Tokenizer Config
wget -P checkpoints/ https://huggingface.co/itslokeshx/Raven-1/resolve/main/checkpoints/raven1_tokenizer.json

# 2. Download Post-Trained Best Weights (Highly Recommended - Chat & Logic)
wget -P checkpoints/ https://huggingface.co/itslokeshx/Raven-1/resolve/main/checkpoints/raven1_posttrain_best.pt

# 3. Download Pre-Trained Base Weights (Optional)
wget -P checkpoints/ https://huggingface.co/itslokeshx/Raven-1/resolve/main/checkpoints/raven1_best.pt

3. Run the Gradio Web Chat Interface

Once the weights are in checkpoints/, start the premium local web chatbot UI:

python app.py

Open your browser and navigate to http://localhost:7860 to chat with RAVEN-1!

---

Architecture

Input Tokens (sequence of integers)
     │
     ▼
┌──────────────────────────────────────┐
│    Token Embedding (8,192 → 512)     │  Weight-tied with LM Head
│  + Learned Position Embedding (256)  │
│  + Dropout (0.1)                     │
└──────────────────┬───────────────────┘
                   │
                   ▼
        ┌─────────────────────┐
        │                     │
        │   8× Transformer    │  ◄── Pre-LayerNorm design
        │       Block         │
        │                     │
        │  ┌───────────────┐  │
        │  │ LayerNorm     │  │
        │  │ ↓             │  │
        │  │ Multi-Head    │  │  8 heads × 64 dim = 512
        │  │ Causal Attn   │  │  FlashAttention (auto-dispatched)
        │  │ + Residual    │  │  Combined Q/K/V projection
        │  │               │  │
        │  │ LayerNorm     │  │
        │  │ ↓             │  │
        │  │ Feed-Forward  │  │  512 → 2048 → 512
        │  │ (GELU)        │  │  No bias terms
        │  │ + Residual    │  │
        │  └───────────────┘  │
        │                     │
        └──────────┬──────────┘
                   │
                   ▼
┌──────────────────────────────────────┐
│         Final LayerNorm              │
│         LM Head (512 → 8,192)       │  Weight-tied with Token Embedding
└──────────────────┬───────────────────┘
                   │
                   ▼
            Logits (8,192)

Model Specifications

Special Tokens

---

Project Structure

Raven-1/
│
├── config.py                  # Central configuration — ALL hyperparameters
├── model.py                   # Transformer architecture + generation
├── train.py                   # Pre-training loop
├── posttrain.py               # Post-training / fine-tuning loop
├── tokenizer_train.py         # Train byte-level BPE tokenizer
├── data_prep.py               # Tokenize text → binary memmap files
├── posttrain_data_prep.py     # Streaming post-train data pipeline
├── inference.py               # Terminal chat with slash commands
├── app.py                     # Gradio web UI (HF Spaces compatible)
│
├── eval/                      # Evaluation & benchmarks
│   ├── run_300_test.py        # 300-prompt benchmark suite
│   ├── stress_test.py         # Stress test with long inputs
│   ├── pretrain_results.txt   # Pre-train benchmark results
│   └── posttrain_results.txt  # Post-train benchmark results
│
├── notebooks/                 # Colab training notebooks
│   ├── colab_train.ipynb      # One-click pre-training notebook
│   └── colab_posttrain.ipynb  # One-click post-training notebook
│
├── data/                      # Training corpora (not in git)
│   ├── corpus.txt             # Pre-training corpus (389 MB)
│   └── posttrain.txt          # Post-training data (977 MB, ~13M lines)
│
├── checkpoints/               # Generated at runtime (not in git)
│   ├── raven1_tokenizer.json  # Trained BPE tokenizer (557 KB)
│   ├── raven1_best.pt         # Best pre-trained weights (338 MB)
│   ├── raven1_posttrain_best.pt  # Best post-trained weights (338 MB)
│   ├── train.bin / val.bin    # Pre-training binary data
│   └── posttrain_train.bin / posttrain_val.bin
│
├── requirements.txt
├── README.md
└── .gitignore

Total hand-written code: ~2,480 lines across 11 Python files.

---

Quick Start

1. Install Dependencies

pip install -r requirements.txt

2. Run Inference (Terminal Chat)

python inference.py

The model automatically loads the best available weights:

1. Post-train weights (raven1_posttrain_best.pt) — tried first
2. Pre-train weights (raven1_best.pt) — fallback
3. Random weights — if no checkpoints found

3. Run Web UI

python app.py
# → Opens at http://localhost:7860

---

Training Pipeline

The full pipeline has 5 stages, each with a dedicated script. Every stage is independently runnable, resume-safe, and Colab-compatible.

Stage 1 — Train Tokenizer

python tokenizer_train.py

Trains a byte-level BPE tokenizer on data/corpus.txt using HuggingFace's tokenizers library (Rust backend for speed).

Stage 2 — Prepare Data

# Pre-training data
python data_prep.py

# Post-training data (streaming, handles 1GB+ without RAM issues)
python posttrain_data_prep.py

Tokenizes text corpora into binary uint16 memmap files for zero-copy random access during training.

Key features:

• Chunked processing — reads 10MB (data_prep) or 8MB (posttrain_data_prep) at a time to stay within Colab's 12GB RAM
• Streaming parser — reconstructs <|eos|>-delimited samples across chunk boundaries
• Batch tokenization — uses encode_batch() with Rust multi-threading (4,096 samples per batch)
• Metadata cache — skips rebuild if source file and tokenizer haven't changed
• Malformed sample detection — warns if >1% of samples lack required special tokens

Stage 3 — Pre-train

python train.py

Full pre-training loop with production-grade features:

Pre-train results:

• Best checkpoint: step 9,000
• Best val loss: 0.5906

Stage 4 — Post-train

python posttrain.py              # Full post-training
python posttrain.py --resume     # Resume from checkpoint
python posttrain.py --dry-run    # Test 20 steps only

Fine-tunes the pre-trained model on curated conversational data. Uses the same training infrastructure with lower learning rates and generates before/after sample comparisons.

Post-train results:

• Best checkpoint: step 2,000
• Best val loss: 0.5950

Stage 5 — Test & Evaluate

python eval/run_300_test.py     # 300-prompt benchmark across 6 categories
python eval/stress_test.py      # Stress test with very long inputs
python inference.py             # Interactive terminal chat

---

Inference

Terminal Chat (inference.py)

python inference.py

Interactive chat with conversation history (last 3 turns), multiple sampling modes, and slash commands.

══════════════════════════════
  RAVEN-1  |  29.5M parameters
  Loaded: posttrain
  Device: cpu
══════════════════════════════

You: hey there
Raven: You found me. Now what.

You: I'm tired of everything
Raven: Everything is temporary. Including this conversation, hopefully.

You: /mode chaos
  Mode → chaos

You: /exit
Bye.

Slash commands:

Web UI (app.py)

python app.py
# → http://localhost:7860

Premium dark monochrome Gradio interface. Same generation logic as the terminal chat. Features mode selection dropdown and conversation history. Deployable directly to Hugging Face Spaces.

UI tech stack: Gradio with custom dark theme, Inter font (Google Fonts), #0a0a0a background, 12px border-radius, 700px max-width container.

Generation Pipeline

Both inference methods use the same pipeline:

1. Build prompt with conversation context (last 3 turns)
2. Encode with BPE tokenizer
3. Crop to block_size (256) if needed (keeps rightmost tokens)
4. Generate with autoregressive sampling:
   • Temperature scaling
   • Top-K filtering
   • Top-P (nucleus) filtering
   • Repetition penalty on previously generated tokens
5. Decode only new tokens
6. Clean leaked special tokens
7. Fallback to witty canned response if output is empty

---

Training Results

Pre-training

Post-training

Loss Trajectory

The model converges smoothly with cosine decay:

Pre-train:   random init → ~7.0 loss → 0.5906 val loss (9,000 steps)
Post-train:  0.5906 pretrain → 0.5950 posttrain val loss (2,000 steps)

---

300-Prompt Benchmark

Both the pre-trained and post-trained models were evaluated on 300 diverse prompts across 6 categories using identical sampling parameters (default mode).

Benchmark Comparison

Category Distribution

Sample Post-train Outputs

Casual / Conversational (strong improvement):

Personality-driven responses:

Note: The post-train corpus included some competitive programming data, which occasionally causes reasoning-style leakage (e.g., "To solve this problem...") on certain prompts. This is a data quality issue, not a model architecture issue.

---

Sampling Modes

All modes generate up to 60 new tokens per response and stop on <|eos|>.

---

Configuration Reference

All hyperparameters live in config.py — zero hardcoded values anywhere else in the codebase. Every script imports from this single source of truth.

# ── Model Architecture ──
vocab_size   = 8192        # BPE vocabulary size
block_size   = 256         # Maximum context length (tokens)
n_embd       = 512         # Embedding dimension
n_head       = 8           # Number of attention heads
n_layer      = 8           # Number of transformer blocks
dropout      = 0.1         # Dropout rate during training

# ── Pre-training ──
batch_size   = 32          # Micro-batch size per step
gradient_accumulation_steps = 2   # Effective batch = 64
max_iters    = 10000       # Total pre-training steps
learning_rate = 3e-4       # Peak learning rate
min_lr       = 3e-5        # Cosine floor
warmup_iters = 300         # Linear warmup steps
weight_decay = 0.1         # AdamW weight decay
grad_clip    = 1.0         # Gradient norm clipping
eval_interval = 500        # Evaluate every N steps
eval_iters   = 100         # Batches per evaluation
use_amp      = True        # Mixed precision training

# ── Post-training ──
posttrain_max_iters       = 2000
posttrain_learning_rate   = 5e-5
posttrain_min_lr          = 5e-6
posttrain_warmup_iters    = 100

# ── Special Tokens ──
user_token   = "<|user|>"
raven_token  = "<|raven|>"
eos_token    = "<|eos|>"
pad_token    = "<|pad|>"

# ── Device ──
device = "cuda" if torch.cuda.is_available() else "cpu"

---

Design Decisions

Architecture Choices

Training Choices

Data Pipeline Choices

Inference Choices

---

Google Colab Guide

RAVEN-1 is designed to train end-to-end on Google Colab's free tier (T4 GPU, 15GB VRAM, 12GB RAM). Two notebooks are provided:

Pre-training (notebooks/colab_train.ipynb)

1. Mount Drive & check GPU → verify T4
2. Install dependencies → tokenizers, numpy
3. Create directories → upload project files
4. Train tokenizer → python tokenizer_train.py
5. Prepare data → python data_prep.py (chunked, RAM-safe)
6. Pre-train → python train.py (auto-resumes, syncs to Drive)
7. Post-train → python posttrain.py
8. Test → python inference.py
9. Download weights → .pt + tokenizer .json

Post-training (notebooks/colab_posttrain.ipynb)

1. Mount Drive & check GPU
2. Install dependencies
3. Upload code files (6 files)
4. Copy posttrain.txt from Drive
5. Copy pre-trained weights from Drive
6. Pre-flight check (validates all files)
7. Tokenize data → python posttrain_data_prep.py (streaming, ~1 min for 1GB)
8. Post-train → python posttrain.py --resume (resume-safe)
9. Test the model
10. Download weights

💡 Tip: If the Colab session disconnects, just re-run from the training cell — it auto-resumes from the latest checkpoint synced to Google Drive. Checkpoints include optimizer state, scaler state, step count, and best val loss for exact continuation.

Colab Resource Usage

---

Requirements

torch>=2.1.0
tokenizers>=0.15.0
numpy>=1.24.0
gradio>=4.0.0

pip install -r requirements.txt

Compatibility

• Python: 3.10+
• PyTorch: 2.1+ (for F.scaled_dot_product_attention / FlashAttention)
• CUDA: Optional. Runs on CPU (slower) or any CUDA-capable GPU
• OS: Linux, macOS, Windows
• Hardware: Colab T4 (minimum), any modern GPU, or CPU

---

File Reference

---

Checkpoint Format

Each .pt checkpoint contains:

{
    "step": int,                 # Training step number
    "model_state": OrderedDict,  # Model weights (69 tensors)
    "optimizer_state": dict,     # AdamW state for exact resume
    "scaler_state": dict,        # AMP GradScaler state
    "best_val_loss": float,      # Best validation loss seen
    "config": {                  # Architecture config snapshot
        "vocab_size": 8192,
        "block_size": 256,
        "n_embd": 512,
        "n_head": 8,
        "n_layer": 8,
        "dropout": 0.1,
    }
}

Checkpoint size: ~338 MB (includes optimizer momentum buffers).

---

License

MIT

---