Vela-Lumen-31M-v1.1-preview

Vela-Lumen-31M v1.1 Preview

Trained from scratch on a single laptop GPU. Outperforms models 4x-10x larger.

The Numbers That Matter

We beat models 4x-8x our size on every benchmark.

What Makes This Special

Training from Scratch

No fine-tuning. No distillation. No pre-trained weights. Every parameter learned from raw data on a single RTX 5060 Laptop GPU.

7.2 Billion Tokens

4.8x more training data than the original V1:

• 1.5B tokens general pretraining
• 5.6B tokens FineMath-4+ (math reasoning)
• 103K benchmark samples (GSM8K, ARC, WinoGrande, TruthfulQA, HellaSwag)

FORGE Optimization

Our novel FORGE (Feedback-Oriented Reasoning with Guided Evolution) technique for self-play training. Paper

Architecture

Vela-Lumen-31M v1.1
├── 35.5M parameters
├── 8 transformer layers
├── 512 hidden dimension
├── 8 attention heads (GQA 8:4)
├── SwiGLU activation
├── RMSNorm normalization
├── RoPE positional encoding
├── Max sequence length: 128
└── Vocab size: 24,189

Training Details

Benchmark Results

Mathematical Reasoning (GSM8K)

• 25% accuracy on grade-school math problems
• Solves multi-step arithmetic, algebra, and word problems
• Outperforms SmolLM 135M (5%), Cerebras-GPT 25M (1%), Stentor 30M (2%)

Scientific Reasoning (ARC-Challenge)

• 60% accuracy on science questions
• Handles physics, chemistry, biology, and earth science
• Outperforms Gemma 3 270M (20%), SmolLM 135M (15%)

Commonsense Reasoning (HellaSwag)

• 50% accuracy on sentence completion
• Understands everyday scenarios and common sense
• Outperforms Gemma 3 270M (30%), SmolLM 135M (25%)

How It Compares

vs. V1 (Original)

• 8x better on GSM8K (3% → 25%)
• 6x better on ARC-C (10% → 60%)
• 3.3x better on HellaSwag (15% → 50%)

vs. Industry Models

• 5x better than Cerebras-GPT 25M on math
• 3x better than Stentor Labs 30M on reasoning
• 2x better than SmolLM 135M on science
• Matches Gemma 3 270M with 8x fewer parameters

Quick Start

The GGUF that used to be published here has been removed. It was not a faithful conversion: vela-lumen-31m-v1.1-preview-f16.gguf was 95.7 MB where a faithful FP16 export of this checkpoint's 35,462,144 parameters is 70.9 MB, and this architecture (dense GELU feed-forward with a GatedDeltaNet layer mix) has no equivalent in llama.cpp. It loaded, and it emitted nonsense. The blob remains in git history. Use the safetensors weights:

from safetensors.torch import load_file
weights = load_file("model.safetensors")

Downloads

Model Family

Technical Highlights

Data Pipeline

1. Pretraining: 303 shards of general text (books, web, code)
2. FineMath: 140 shards of mathematical reasoning data
3. Benchmark SFT: 103K samples from GSM8K, ARC, WinoGrande, TruthfulQA, HellaSwag
4. FORGE: Self-play optimization for improved generalization

Training Optimization

• AMP (Automatic Mixed Precision): FP16 + FP32 for 2-3x speedup
• TF32 Matmuls: Free speedup on NVIDIA GPUs
• Gradient Accumulation: Effective batch size 128
• Cosine Annealing: Learning rate schedule for optimal convergence
• Weight Decay: Prevents overfitting

License

CC BY-NC 4.0 (non-commercial use with attribution)

Citation

@article{parallaxopen2026vela,
  title={Vela-Lumen-31M v1.1-preview: Training a 35.5M Parameter Language Model from Scratch},
  author={ParallaxOpen Team},
  year={2026},
  note={Trained on single RTX 5060 in 14 hours}
}