GPT3-dev-125m-1005

You can find all code on GitHub

Note: This is a model with 125 million parameters (attempt to replicate GPT-3 Small). (it's very undertrained.)

Note 2: This is a model checkpoint released on 10/05 2026 (72 batch size, 4 grad accumulation and 50000 steps under Muon optimizer). It scores 25.49% on MMLU which is slightly higher than 25% (random guess)

Note 3: This model already demonstrates basic abilities in generating text. It's not perfect and I will continue working on it. Expect Instruct model soon.

Model description

This is a small GPT-style autoregressive language model. It is intended as a development checkpoint, not as a production-ready assistant. But you can try.

This time I used kernels and Flash Attention 4 and Flash Attention 2 with the fallback to SDPA. This allowed me to cut the time required for one step from nearly 60 seconds (on jetson) to 3.6 seconds (on the server) and then to 2.2 seconds (using Unsloth kernels)

Important notes

This model is still undertrained. Its benchmark results are close to random-choice level on multiple-choice academic benchmarks, so the checkpoint should be treated as experimental.

It can generate basic text, but it may produce incorrect, repetitive, incoherent, or non-readable outputs. It is not instruction-tuned, but it can produce several meaningful paragraphs.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained(k050506koch/GPT3-dev-125m-1009, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(k050506koch/GPT3-dev-125m-1009, trust_remote_code=True)

if tokenizer.pad_token_id is None:
    tokenizer.pad_token_id = tokenizer.eos_token_id

prompt = "He is a doctor. His main goal is"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=96,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.2,
    no_repeat_ngram_size=3,
    pad_token_id=tokenizer.pad_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation results

Evaluation was run locally on CPU with a custom evaluation script.

These results should not be compared directly with Open LLM Leaderboard results unless the same evaluation harness, prompt format, number of shots, and dataset splits are used.

Summary

MMLU

Limitations

As this is only the next word prediction model, it doesn't know how to interact with the user.

Training data

HuggingFaceFW/fineweb. Only this

Training metadata

Checkpoint date: 10.05.2026
Parameters: 125231616
Context length: 2048
Batch size: 72
Gradient accumulation: 4
Sequence length: 512
Training steps: 50000
Optimizer: Fused Muon with Hermes kernels
Learning rate schedule: cosine
Hardware: Frankenstein (2012 datacenter server with a RTX 5070Ti)

Contributing

Contributions are always welcome.

I am still a student, so the code and model may contain mistakes, bugs, or incorrect assumptions. If you find an issue or have an improvement, feel free to open an issue or submit a pull request. I will be happy.

Acknowledgements

Thanks to OpenAI, Hugging Face, PyTorch and Unsloth for making this kind of research and experimentation possible.

References:

• OpenAI GPT-3 paper
• Hugging Face Transformers
• Unsloth cut-cross-entropy
• PyTorch