welmia-2.0-103m-base

welmia-2.0-103m-base

A 103M-parameter GPT-style language model (RoPE, RMSNorm, SwiGLU, tied embeddings) pretrained completely from scratch on ~5B tokens (OpenWebText).

This is the base (non-instruction-tuned) model — raw next-token prediction only. For an instruction-following version, see welmia-1.0-103m-instruct (uploaded separately).

Architecture

Tokenizer note: this model uses a custom 24k-vocab BPE tokenizer trained specifically for this model — it is not the GPT-2/tiktoken vocabulary used by welmia-1.0-81m. Checkpoints and tokenizers between the two models are not interchangeable.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("Welmia/welmia-2.0-103m-base", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("Welmia/welmia-2.0-103m-base")

ids = tok("The history of artificial intelligence began", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required — this is a custom architecture, defined in modeling_gpt.py in this repo.

Important notes

• Base model, not instruction-tuned. It will continue text rather than answer questions or follow instructions. Use the -instruct version for chat/Q&A behavior.
• No KV cache in this implementation — generate() recomputes the full forward pass each step, so it's slower than production-grade inference code at longer output lengths.
• 512-token context. Inputs longer than this are truncated from the left during generation.

Training

Trained from scratch on ~5B tokens of OpenWebText, causal LM objective, RoPE + SwiGLU + RMSNorm architecture, AdamW optimizer with cosine LR schedule.

License

Apache License 2.0