mpu-30m-base

mpu-30m-base

mpu-30m is a 30.3M-parameter GPT-style language model trained from scratch on 400M tokens of TinyStories. It is the first model in the mpu series and the proof-of-concept run for an elastic, checkpoint-driven training platform that migrates a single training job across free and disposable GPU sessions (Colab/Kaggle) without losing progress.

It writes short, simple children's stories. It is small on purpose: the goal of this run was to validate the training infrastructure end to end, not to compete on quality.

Model details

Training

The interesting part: how it was trained

This model was trained by an elastic training system in which compute is disposable and checkpoints are persistent. Every ~20 minutes the trainer writes a full checkpoint (weights, optimizer, scaler, RNG state) and promotes it atomically to this repo. A session can be killed at any time; any other GPU session resumes from the latest promoted checkpoint and — because data batches are a pure function of (seed, step) — continues bit-identically, verified by test. This run survived a mid-training kill-and-resume across sessions.

Usage

The checkpoint uses a custom (nanoGPT-style) architecture and is not loadable via transformers.AutoModel. Weights are standard safetensors (see config.json in the checkpoint folder for the architecture: 6 layers, 6 heads, width 384, GPT-2 BPE tokenizer). The training and inference code is not yet public; it will be released alongside a later model in the series.

Sample output

Once upon a time there was a little robot. He was very happy and liked to roll with his friends. But one day, he rolled too fast and fell into a big puddle. He tried to roll out of his wet puddle, but he couldn't. He was stuck and couldn't get out. Luckily, a kind little girl saw the robot and knew just what to do. [...] From then on, the robot was extra careful.

Limitations

• Trained only on synthetic children's stories: tiny vocabulary in practice, simple grammar, no factual knowledge, English only.
• At this scale the model loses track of characters and pronouns, and occasionally substitutes a wrong noun mid-story.
• No instruction tuning, no safety tuning, no formal evaluation. Not for any production use — this is an educational artifact.