sologpt-v3-123m-base

SoloLLM v3 123M Base

SoloLLM v3 123M Base is the smaller-than-GPT-2 ablation from the SoloLLM v3 project. It is a from-scratch GPT-style decoder-only base language model trained on one RTX 3090.

This is not the final best SoloLLM checkpoint. The final best model is bmax16634/sologpt-v3-150m-base. This 123M model is published because it is slightly smaller than GPT-2 small and documents the strict smaller-model test.

Bottom Line

The 123M model is slightly smaller than GPT-2 small and beats GPT-2 on most external checks, but it does not beat GPT-2 across every metric. It loses the project held-out perplexity comparison and some fixed-prompt generation diversity/repetition diagnostics.

The honest read:

The 123M model is a strong smaller-than-GPT-2 ablation, but it does not prove that a smaller model beats GPT-2 small across the board.

Model Details

Training Data

The model was trained on the same curated 10B-token SoloLLM v3 dataset as the 150M final model:

Multiple-Choice Detail

Length-normalized accuracy:

Files

Usage

This repo supports Hugging Face AutoModelForCausalLM loading through custom remote code. Pass trust_remote_code=True when loading the model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "bmax16634/sologpt-v3-123m-base"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True).to(device)
model.eval()

prompt = "The future of artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=40,
        do_sample=True,
        temperature=0.8,
        top_k=40,
        use_cache=False,
        remove_invalid_values=True,
        renormalize_logits=True,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

For a runnable example, see load_example.py. For low-level state-dict loading, the raw PyTorch implementation is still included as model.py.

Intended Use

This model is intended for:

• educational inspection of a smaller GPT-2-class base LM,
• ablation comparison against sologpt-v3-150m-base,
• text-completion experiments,
• reproducing the SoloLLM v3 evaluation story.

It is not intended for production use, high-stakes decisions, factual QA, or chat/instruction-following use without additional tuning and safety evaluation.

Limitations

• This is a small base model, not an assistant.
• It can generate incorrect, biased, repetitive, or unsafe text.
• It has no retrieval, tool use, or instruction tuning.
• It does not beat GPT-2 small across every metric.
• Training data came from broad public web/text sources and may contain undesirable content despite filtering.

Related Artifacts

• Final best model: https://huggingface.co/bmax16634/sologpt-v3-150m-base
• Public completion demo: https://huggingface.co/spaces/bmax16634/sologpt-v3-150m-demo
• Legacy v1 baseline: https://huggingface.co/bmax16634/sologpt-base-v1

License

The SoloLLM code and released weights are provided under the MIT License by the author. Training data sources retain their own licenses and terms.