10k-financial-slm

10-K Financial SLM (11.5M params)

A tiny GPT language model trained exclusively on SEC 10-K filings from financial companies. ~20 experiments at 5 minutes each (~2 hours total GPU time) on a MacBook Air using Apple Silicon (MPS).

Model Details

Performance

Compression Quality (bits-per-byte)

23.3% better compression on financial text compared to the same architecture trained on general data.

Inference Speed (MacBook Air, MPS)

Cost Comparison (processing 80K filings)

Prices as of March 2026. Input tokens only.

Training Details

Built using Karpathy's autoresearch framework, which enables autonomous hyperparameter experimentation. An AI agent (Claude) iteratively modified the training configuration, ran 5-minute training sessions, and kept improvements.

Key hyperparameters (after optimization)

• Learning rates: 1.5x default (Embedding: 0.9, Matrix/Muon: 0.06)
• Warmdown ratio: 0.05 (LR stays at peak for 95% of training)
• Optimizer: MuonAdamW (Muon for matrix params, AdamW for embeddings)
• Batch size: 65,536 tokens per step

Data pipeline

1. Downloaded 10-K filing index from SEC EDGAR (2015-2025)
2. Filtered to financial companies (SIC codes 6000-6411): banks, insurance, investment firms
3. Sampled 1,500 filings, downloaded full text from EDGAR
4. Cleaned HTML/XBRL markup, removed filings that were too short or too numeric
5. Chunked into 2,048-token sequences, split 90/10 train/val
6. Trained a BPE tokenizer (8,192 vocab) on the financial text

Intended Use

This model is a research artifact demonstrating domain-specific SLM training. Potential applications:

• Document embeddings: Fast similarity search over financial filings
• Anomaly detection: Flag filings with unusual language patterns
• Pre-filtering: Cheap triage before sending documents to expensive API models
• Privacy-preserving analysis: All processing stays on-device
• Foundation for fine-tuning: Starting point for downstream financial NLP tasks

Limitations

• Not a chatbot: This is a base language model. It predicts next tokens, it doesn't answer questions.
• Tiny model: 11.5M parameters means limited capacity. It captures patterns and statistics of financial language, not deep reasoning.
• Narrow training data: Only financial company 10-K filings. Performance on other financial documents (earnings calls, proxy statements) is untested.
• No safety training: No RLHF, no content filtering. Not suitable for user-facing generation.

How to Use

import torch
from train import GPT, GPTConfig

# Load checkpoint
ckpt = torch.load("model.pt", map_location="cpu")
config = GPTConfig(**ckpt["config"])

model = GPT(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

# Run inference
tokens = torch.tensor([[1, 2, 3, ...]])  # your tokenized input
with torch.no_grad():
    logits = model(tokens)

Citation

If you use this model in your work, please cite:

@misc{10k-financial-slm-2026,
  title={10-K Financial SLM: A Domain-Specific Small Language Model for SEC Filings},
  year={2026},
  url={https://github.com/harryschaefer93/autoresearch-10k-macos}
}

Acknowledgments

• Andrej Karpathy / autoresearch-macos for the training framework
• Claude Code for autonomous experiment orchestration
• SEC EDGAR for the public filing data