ounce100m-v1

ounce100m-v1

A 106,194,240-parameter decoder-only transformer trained from scratch on 999,817,216 tokens of Cion-lab/ounce100m-mix-v1, in 3814 optimiser steps on 2×Nvidia Tesla T4 inside Kaggle notebook sessions. It is the deliverable of a project whose whole point was to find out what a ~100 M-parameter model reaches from ~1 B tokens of a curated public mix, and to report it honestly, including where that is near chance.

Measured facts

Data

Cion-lab/ounce100m-mix-v1: 1,109,714,831 training tokens in 139 shards + 22,934,043 held-out validation tokens, cut with HuggingFaceTB/SmolLM2-135M:tokenizer.json (49,152 vocab, Apache-2.0), from 15 public sources at ≥15 per shard. Format and loading code are in that dataset's card.

Contamination. Overlap was measured mechanically against benchmark train/validation/dev material of the eight evaluation tasks (13-token windows, counts only — no benchmark item text was ever read, and test splits stayed untouched until evaluation). audit.json in the dataset repo records overlap_total = 0 over 1,073,047 documents and 21,898 held-out documents, tasks_covered = 8/8. Building it required dropping 5878 documents (17,366,967 tokens) that matched reference material; the exact exclusion masks ship in filter/ so the mix is reproducible.

Evaluate it

pip install "lm-eval==0.4.13"
lm_eval --model hf --model_args pretrained=Cion-lab/ounce100m-v1,dtype=float16,trust_remote_code=False \
  --tasks arc_challenge,arc_easy,hellaswag,mmlu,piqa,truthfulqa_mc1,truthfulqa_mc2,winogrande,gsm8k \
  --batch_size 8 --seed 42 --log_samples

No benchmark scores are published with this model. The eight-task evaluation was dropped on 2026-09-21T14:13Z by the project owner before any task produced a score (the first eval kernel died at 1.77 s on a module-path bug and was not relaunched), so every number on this card comes from training or from the held-out validation set - perplexity 75.98 on 1,953 windows of val/, which the model never trained on. The protocol that was going to be used is frozen and public anyway: task ids, metric names, shot counts, splits and chance levels were all fixed in docs/05-eval-plan.md of Cion-lab/ounce100m-code before any score existed, with the driver in eval/run_benchmarks.py at Cion-lab/ounce100m-code. Treat the command above as a reproduction recipe, not as a claim that these results were measured.

Files

cursor.json is the data position this checkpoint corresponds to: samples consumed, step, seed, the corpus fingerprint and a hash of the visit-order permutation, so the resume can be checked rather than trusted.

Not in this model

No instruction tuning, no RLHF, no chat template — it is a base model, deliberately. It has ~1 B tokens of training against the ~300 B of the comparable published reference models, so expect scores at or near chance on several tasks; that is the experiment's result, not a defect to be engineered around.