snip-0.4m-reward

SNIP-0.4M

SNIP means Small Neural Intelligence Prototype. SNIP-0.4M is a deliberately tiny causal transformer designed to make the complete language-model lifecycle inspectable on ordinary hardware.

Architecture

• approximately 400,000 parameters;
• 3 transformer layers;
• 96 hidden dimensions;
• 3 attention heads;
• 128-token context;
• custom 512-token byte-level BPE tokenizer;
• GPT-2-compatible architecture saved with Hugging Face Transformers.

Training

The base model is initialized from scratch and pretrained on a bounded stream from roneneldan/TinyStories. It does not inherit pretrained weights.

uv run python projects/snip-0.4m/prepare_data.py
uv run python projects/snip-0.4m/train.py
uv run python projects/snip-0.4m/evaluate.py

Intended use

SNIP is an educational and experimental base model. Its small capacity makes it useful for testing tokenizers, adapters, curricula, evaluation pipelines, compression, and RL methods quickly. It is not intended as a factual assistant or production chatbot.

Results

Measured parameter count, held-out loss, perplexity, throughput, and generation samples are written to artifacts/snip-0.4m-base/evaluation.json after training.

Base v1

• Parameters: 397,152
• Pretraining tokens: 1,638,400
• Held-out loss: 3.3656
• Held-out perplexity: 28.9508

These are local CPU results over 200 held-out 128-token blocks.

SNIP-0.09M compression variant

The compressed student has 87,360 parameters: two layers, 48 hidden dimensions, three attention heads, and the same 512-token vocabulary and 128-token context. It was trained for 500 steps over 1,024,000 tokens.

The distilled student is 4.55 times smaller than its teacher and improves perplexity by 4.82% over the architecture-matched scratch run. It still trails the larger teacher, making the compression-quality tradeoff explicit.

uv run python projects/snip-0.4m/train_distilled.py --steps 500
uv run python projects/snip-0.4m/train_distilled.py --steps 500 `
  --distillation-weight 0 --output-name snip-0.09m-scratch

Story LoRA

The first post-trained variant attaches a rank-4 LoRA adapter to the attention and feed-forward projections. It learns an instruction-shaped microfiction format while leaving the 397K base weights frozen.

uv run python projects/snip-0.4m/train_lora.py
uv run python projects/snip-0.4m/evaluate_variants.py

The run saves both a lightweight adapter and a standalone merged variant.

Pairwise reward model

SNIP Reward reuses the pretrained 397K transformer and replaces the language-model head with a scalar ranking head. It learns from 810 structured preference pairs and is evaluated on 90 held-out pairs.

uv run python projects/snip-0.4m/train_reward.py --epochs 4 --batch-size 128

The 397,248-parameter model reached 100% pairwise accuracy on both wrong-goal and wrong-lesson corruptions, with a mean chosen-minus-rejected margin of 3.3656. This is a narrow synthetic consistency test, not evidence of general human-preference alignment.

Verifiable-reward language RL

SNIP-GRPO applies an 18,432-parameter LoRA policy to parity, comparison, and threshold prompts with exact binary rewards. Each update samples 16 actions per prompt, centers and normalizes reward inside each group, clips the policy ratio, and penalizes KL divergence from the frozen base policy.

uv run python projects/snip-0.4m/train_grpo.py --steps 300

Across 176 held-out prompts, greedy accuracy improved from 52.27% to 69.32%. Less-than accuracy rose from 47.50% to 77.50%, and sum-threshold accuracy rose from 57.89% to 65.79%. Parity stayed at 50%, an explicit failure mode rather than a hidden average. The method is GRPO-style group-relative policy optimization over a constrained two-token action space; it is not a full open-ended reasoning model.

Logic SFT control

An architecture- and data-matched supervised LoRA control trained for 300 updates on the same 684 examples and evaluated on the same 176 held-out prompts:

SFT and GRPO tied in aggregate but did not learn the same behavior. SFT was stronger on comparison and thresholds while catastrophically degrading parity; GRPO preserved base parity performance. The merged SFT artifact was independently reloaded and re-evaluated after training.