llm-0.5b-fineweb-edu

llm-0.5b-fineweb-edu

A 489M-parameter Llama-style base model pretrained from scratch on 4.19B tokens of FineWeb-Edu, on a single rented RTX 5090 for $13.70 of GPU time.

It exists to document what that budget buys, and the numbers below are all measured rather than estimated. It is not a model you should deploy — see Limitations.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("osjayaprakash/llm-0.5b-fineweb-edu")
tok = AutoTokenizer.from_pretrained("osjayaprakash/llm-0.5b-fineweb-edu")

ids = tok("Photosynthesis is", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=48, do_sample=True, temperature=0.8, top_k=50)
print(tok.decode(out[0], skip_special_tokens=True))

This is a base model: no instruction tuning, no chat template, no alignment. It continues text and nothing else.

Architecture

Architecturally a small Llama, so it loads as LlamaForCausalLM with no custom code. The export was verified by asserting logit equality against the original implementation: max absolute difference 3.2e-05.

Training

FineWeb-Edu sample-10BT, 2M documents tokenized to 2.03B training tokens plus 50M held out. The run made about two passes over that corpus, in three phases on the same GPU.

AdamW (0.9, 0.95), weight decay 0.1, gradient clipping 1.0, bf16 autocast, torch.compile, 524,288 tokens per optimizer step (batch 4 x 64 gradient accumulation x 2048). Cosine schedule with warmup in each phase. Final perplexity 20.05.

Across all 8,000 steps: no loss spikes, no non-finite gradients, 85% model-FLOPs utilization sustained at ~48,400 tokens/second.

Evaluation

Zero-shot via lm-evaluation-harness, every model run on the same machine with the same task versions.

At a matched 2.1B tokens this model scores 9.7 points above Pythia-410M. It sits 4.8 points below the same model trained on 300B tokens — 72x more data — and well behind the modern small models, which is mostly a data-curation gap rather than an architectural one.

Accuracy against tokens, showing the run never flattened:

MMLU (5-shot): 0.272, against a chance baseline of 0.250. Almost all of that comes from social sciences (0.320); humanities is still at chance. Excluded from the average above.

Qwen2.5-0.5B has no WikiText figure because its 152k vocabulary exhausted GPU memory on that task's rolling windows; it was re-run without WikiText.

Limitations

• Undertrained by design. 4.19B tokens is ~8.5 tokens per parameter, under half the Chinchilla-optimal ratio. The loss curve was still descending when the budget ran out.
• It will state things that are false. At this scale it produces fluent, plausible, frequently wrong text. Do not use it as a source of facts.
• No code ability. FineWeb-Edu is filtered educational web text with essentially no source code, so the model cannot write code and will not learn to from prompting.
• Reasoning benchmarks sit near chance. ARC-Challenge 0.265 and WinoGrande 0.502 barely moved across the entire run.
• English only, 2048-token context, no safety filtering or alignment of any kind. The training corpus is web text and carries its biases.
• Not for production. It is a reference point for what a small budget buys.

Two things that did not work

Weight averaging did not help. An exponential moving average kept over the last 4,000 steps scored 0.4430 against the final checkpoint's 0.4448, and was slightly worse on perplexity. Averaging pays when training ends at a high learning rate and the weights still carry optimization noise; this run annealed to 1e-5, so the schedule had already done the averaging. The two are substitutes, not complements.

Restarting the cosine schedule cost 21% of the budget. Continuing a finished run means warming the learning rate back up, which undoes part of the previous annealing. Validation loss rose 3.3226 to 3.3827 at the phase-2 restart and took 600 steps to recover; phase 3 took 1,100, because its schedule is twice as long and holds the rate near peak for twice as long. 1,700 of 8,000 steps went on re-earning ground already taken. A warmup-stable-decay schedule avoids this entirely and is the right choice when the total budget is not known in advance.

Cost

One RTX 5090 (32 GB) rented on Vast.ai at $0.4785/hour.

Citation

@misc{llm05b_fineweb_edu_2026,
  title  = {llm-0.5b-fineweb-edu: a 489M-parameter model pretrained on one consumer GPU},
  author = {osjayaprakash},
  year   = {2026},
  url    = {https://huggingface.co/osjayaprakash/llm-0.5b-fineweb-edu}
}

License

MIT.