zero-125m

zero-125m

zero-125m is a 124,680,960-parameter OLMo2-compatible base language model trained from scratch on 2,500,001,792 FineWeb-Edu tokens. It is the released dense model from the lm-from-zero project, which covers the complete path from byte-level BPE tokenization to local serving.

This repository includes the native Transformers model and tokenizer, F16 and Q8_0 GGUF derivatives, evaluation evidence, and the full technical report.

Technical report | Source and reproducibility | Capture evidence

Intended use

The model is intended for research, education, systems experiments, conversion tests, and local inference demonstrations. It is a pretrained base model. It has not been aligned for assistant use and should not be treated as a reliable chatbot, factual reference, or safety-critical system.

Training

• Architecture: OLMo2-compatible causal decoder
• Parameters: 124,680,960
• Layers: 12
• Hidden width: 768
• Query heads: 12
• Key/value heads: 4
• Context length: 2,048 tokens
• Vocabulary: 32,000 byte-level BPE tokens
• Training tokens: 2,500,001,792
• Data: FineWeb-Edu sample-10BT, immutable revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9
• Optimizer: AdamW with linear warmup and cosine decay
• Numerical format: bfloat16 training with fp32 optimizer state
• Hardware: one NVIDIA RTX 4080 SUPER

The tokenizer, shard manifest, model configuration, checkpoint, evaluation, and converted package are connected by recorded hashes. The native Transformers export has an exact maximum fp32 logit error of 0.0 against the project checkpoint in the recorded parity evaluation.

model.safetensors stores float32 tensors and is 498,738,888 bytes. That precision is deliberate: it is what makes the exact logit-parity check above meaningful, because the released file is bit-identical in value to the project checkpoint rather than a re-quantized copy of it. Load with dtype=torch.bfloat16 for inference, or take one of the GGUF files below if a smaller download matters more than parity.

Evaluation

Evaluation uses fixed, non-wrapping FineWeb-Edu windows.

These measurements describe next-token likelihood under this tokenizer and window protocol. They are not instruction-following, factuality, or safety scores.

The figures/ directory also carries the 20M architecture study, the dense ablation screening, and the local judge calibration from the wider project.

Transformers usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "tarpous/zero-125m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("Once upon a time,", return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

This is a base model, so continuations may be repetitive, abrupt, or inaccurate. A prompt formatted as a conversation does not turn it into an instruction-following model.

GGUF files

The Q8_0 artifact produced 76.5 decoded tokens per second in the recorded bounded CUDA llama.cpp smoke. Throughput is hardware and build dependent. Conversion details and the pre-tokenizer compatibility patch are documented in the project repository.

Broader project evidence

The project also trained a 20M dense FineWeb model and separate TinyStories models near 20M parameters using dense attention, Mamba-2, and masked discrete diffusion. It includes dense ablations, supervised fine-tuning, DPO, and local judge calibration. Those results describe the research project and are not additional training stages of this released base checkpoint.

The full report is available in this model repository as lm-from-zero-technical-report.pdf and in the GitHub release.

Limitations

• The model is small and was trained for only 2.5 billion tokens.
• It is a base model without safety or instruction tuning.
• FineWeb-Edu may contain errors, bias, personal information, and other undesirable web content despite dataset-level filtering.
• The model can generate false, biased, offensive, repetitive, or incoherent text.
• Evaluation covers held-out language-model loss, not downstream capability or responsible-deployment criteria.
• The planned 125M FineWeb Mamba-2 and masked-diffusion runs were not completed.
• The planned 500-prompt cross-judge calibration was not completed.

Users are responsible for evaluating the model in their own setting and for adding safeguards appropriate to any application.

Citation

@software{tarpous2026lmfromzero,
  author = {tarpous},
  title = {lm-from-zero: A Local-First, Reproducible Language-Model Pipeline from Tokenization to Serving},
  year = {2026},
  url = {https://github.com/tarpous/lm-from-zero}
}

License

The model weights, tokenizer, GGUF files, and code are released under Apache-2.0. The technical report and its figures are released under CC BY 4.0.