sologpt-v3-150m-base

SoloLLM v3 150M Base

SoloLLM v3 150M Base is a from-scratch GPT-style decoder-only language model trained on one RTX 3090 as part of the SoloLLM project. It is a base text completion model, not an instruction-tuned chatbot.

The project goal was to build a full small-LM engineering loop: dataset construction, PyTorch model implementation, single-GPU pretraining, checkpoint recovery, evaluation, ablation, and an honest comparison against GPT-2 small.

Headline Result

The final 150M model beats GPT-2 small overall on the fixed SoloLLM v3 evaluation suite. A smaller 123M ablation also beats GPT-2 on most external checks, but it does not beat GPT-2 across every metric.

The honest claim is:

SoloLLM v3 trains GPT-2-class base LMs from scratch on one RTX 3090. The final 150M model beats GPT-2 small overall on a fixed evaluation suite, while a slightly smaller 123M model beats GPT-2 on most external benchmarks but does not fully clear the strict across-board smaller-than-GPT-2 bar.

Model Details

Training Data

The model was trained on a curated 10B-token mixture:

The dataset was filtered, deduplicated by normalized document hash, and packed into 1024-token training shards.

Files

Usage

This repo supports Hugging Face AutoModelForCausalLM loading through custom remote code. Pass trust_remote_code=True when loading the model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "bmax16634/sologpt-v3-150m-base"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True).to(device)
model.eval()

prompt = "The future of artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=40,
        do_sample=True,
        temperature=0.8,
        top_k=40,
        use_cache=False,
        remove_invalid_values=True,
        renormalize_logits=True,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

For a runnable example, see load_example.py. For low-level state-dict loading, the raw PyTorch implementation is still included as model.py.

Intended Use

This model is intended for:

• educational inspection of a small from-scratch base LM,
• text-completion experiments,
• reproducing the SoloLLM v3 evaluation story,
• portfolio/research engineering review.

It is not intended for production use, high-stakes decisions, factual QA, or chat/instruction-following use without additional tuning and safety evaluation.

Limitations

• This is a small base model, not an assistant.
• It can generate incorrect, biased, repetitive, or unsafe text.
• It has no retrieval, tool use, or instruction tuning.
• The strict smaller-than-GPT-2 across-board claim is not proven by this model; the winning 150M checkpoint is larger than GPT-2 small.
• Training data came from broad public web/text sources and may contain undesirable content despite filtering.

License

The SoloLLM code and released weights are provided under the MIT License by the author. Training data sources retain their own licenses and terms.

Project Links

• Author: Benjamin Maxwell
• Final 150M model: https://huggingface.co/bmax16634/sologpt-v3-150m-base
• Smaller 123M ablation: https://huggingface.co/bmax16634/sologpt-v3-123m-base
• Public completion demo: https://huggingface.co/spaces/bmax16634/sologpt-v3-150m-demo
• Original v1 model: https://huggingface.co/bmax16634/sologpt-base-v1
• Main result artifact in this repo: docs/v3_final_gpt2_comparison.md