BetterGPT-150M

BetterGPT-150M

BetterGPT-150M is a 150 million parameter decoder-only Transformer language model pretrained from scratch using PyTorch.

It is a base language model and has not been instruction tuned. The model is intended for continued pretraining, supervised fine-tuning, research, and downstream adaptation.

BetterGPT is developed as an end-to-end engineering project that implements the complete lifecycle of building a modern small language model, including tokenizer training, dataset preparation, large-scale pretraining, and Hugging Face Transformers integration.

Despite training on just ~15B tokens, BetterGPT-150M outperforms several established ~110–160M parameter baselines (GPT-2 Small, OPT-125M, Pythia-160M) on science-reasoning benchmarks like ARC — see Evaluation for full results.

🚀 Live Demo

You can try out BetterGPT-150M directly in your browser: 👉 Try the Interactive Space Demo

---

Repo https://github.com/harikrish2727/BetterGPT

---

Model Details

Evaluation

BetterGPT-150M was evaluated zero-shot (0-shot) on standard academic benchmarks using lm-evaluation-harness. Notably, on ARC-Easy and ARC-Challenge, BetterGPT-150M outperforms GPT-2 Small, OPT-125M, Pythia-160M, and Cerebras-GPT-111M despite being trained on roughly 15B tokens — a fraction of the 40B–300B+ tokens used for those baselines — suggesting the FineMath/Cosmopedia-heavy data mix is particularly effective for science-reasoning-style tasks relative to raw token count. Results are grouped by capability; acc_norm is reported where answer options vary in length (this corrects for a length bias in raw log-likelihood scoring — see lm-eval-harness docs for details).

Commonsense & Physical Reasoning

Knowledge & Science Reasoning

Language Modeling Quality

Math & Logical Reasoning

In progress — MathQA and LogiQA results will be added once complete.

Baseline Comparison (~110–160M Parameter Scale)

Baseline figures are drawn from published papers/reproductions using varying lm-eval-harness versions, which can introduce small discrepancies (ranges shown reflect this). BetterGPT-150M's own figures above are from a single consistent run.

Training

BetterGPT-150M was pretrained on approximately 15 billion tokens using a two-stage curriculum together with a Warmup–Stable–Decay (WSD) learning rate schedule implemented using PyTorch's LambdaLR.

Stage 1 — Stable Phase (~13B Tokens)

The first stage focuses on broad language acquisition using a diverse mixture of educational, web, mathematical, and programming datasets.

Stage 2 — Annealing Phase (~2B Tokens)

The second stage increases the sampling probability of mathematics, reasoning, instructional text, and Python programming data while training with a reduced learning rate.

This curriculum is designed to adapt the model toward reasoning-intensive domains while preserving the language capabilities learned during the stable phase.

---

Training Data

BetterGPT-150M was pretrained using publicly available datasets, including:

• FineWeb-Edu
• Cosmopedia
• FineMath
• StarCoder-Python

The datasets were streamed, interleaved, and converted into binary training shards for efficient large-scale pretraining.

Please refer to the original dataset repositories for licensing information, intended uses, and any applicable restrictions.

---

Architecture

BetterGPT-150M implements a modern decoder-only Transformer architecture including:

• Multi-Head Self Attention
• Rotary Position Embeddings (RoPE)
• RMSNorm
• SwiGLU Feed Forward Networks
• Weight Tying
• PyTorch Scaled Dot Product Attention (SDPA)

The repository provides two Hugging Face compatible model classes:

• BetterGPTModel – Base transformer model returning hidden states (AutoModel)
• BetterGPTForCausalLM – Causal language model for autoregressive text generation (AutoModelForCausalLM)

---

Intended Uses

BetterGPT-150M is intended for:

• Continued pretraining
• Supervised fine-tuning
• Preference optimization
• Research
• Education
• Building downstream NLP applications

---

Out-of-Scope Uses

BetterGPT-150M is not instruction tuned and is not intended to be used directly as a conversational assistant.

Users requiring instruction-following behavior should fine-tune the model using supervised instruction tuning or other alignment techniques.

---

Limitations

As a relatively small pretrained language model, BetterGPT-150M has several limitations:

• May generate factually incorrect information.
• May produce hallucinated or inconsistent responses.
• Limited reasoning ability compared to significantly larger language models.
• Limited multilingual capability.
• Limited coding performance compared to larger code-specialized models.

Evaluation benchmarks are currently in progress and will be released separately.

---

Usage

Load for text generation

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained(
    "Harikrish2727/BetterGPT-150M",
    trust_remote_code=True
)

model = AutoModelForCausalLM.from_pretrained(
    "Harikrish2727/BetterGPT-150M",
    trust_remote_code=True,
    device_map="auto"
)

Generate text

prompt = "The future of artificial intelligence is"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)


outputs = model.generate(
    **inputs,
    max_new_tokens=500,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Load as a base model

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "Harikrish2727/BetterGPT-150M",
    trust_remote_code=True
)

model = AutoModel.from_pretrained(
    "Harikrish2727/BetterGPT-150M",
    trust_remote_code=True
)

---

Citation

If you use BetterGPT-150M in your work, please cite the project.

@software{bettergpt2026,
  title={BetterGPT: Building a Small Language Model from Scratch},
  author={Harikrishnan Vijayan},
  year={2026},
  url={https://github.com/Harikrish2727/BetterGPT}
}

---

License

This model is released under the Apache License 2.0.

Please ensure that any downstream use also complies with the licenses of the datasets used during pretraining.

---

Acknowledgements

BetterGPT is an independent engineering project inspired by modern open-source language models and the Hugging Face Transformers ecosystem.

The project draws inspiration from the broader open-source LLM community, including work such as nanoGPT, llm.c, Llama, Gemma, and Qwen.