welmia-2.0-103m-instruct

welmia-2.0-103m-instruct

A 103M-parameter GPT-style language model (RoPE, RMSNorm, SwiGLU, tied embeddings), pretrained from scratch on ~5B tokens (OpenWebText), then instruction-tuned on the Alpaca dataset.

Base (non-instruct) version: Welmia/welmia-2.0-103m-base.

Architecture

Tokenizer note: custom 24k-vocab BPE tokenizer, not GPT-2/tiktoken. Not interchangeable with welmia-1.0-81m.

Prompt format

This model expects the Alpaca instruction format:

Instruction:

{your instruction}

Response:

or, with additional context:

Instruction:

{your instruction}

Input:

{context}

Response:

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("Welmia/welmia-2.0-103m-instruct", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("Welmia/welmia-2.0-103m-instruct")

prompt = "### Instruction:\nWhat is Machine Learning.\n\n### Response:\n"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=150, do_sample=True, temperature=0.7, top_k=40)
print(tok.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required — custom architecture defined in modeling_gpt.py in this repo.

Important notes

• Single-turn only. Trained on Alpaca, which has no multi-turn/chat structure — treat each prompt as independent, not a running conversation.
• No KV cache. generate() recomputes the full forward pass each step.
• 512-token context. Longer inputs are truncated from the left.
• Small model — expect basic instruction-following, not strong reasoning or world knowledge.

Training

Pretrained from scratch on ~5B tokens OpenWebText (causal LM), then fine-tuned on the Alpaca instruction dataset (~52k examples, 3 epochs, AdamW, cosine LR, loss masked over the prompt so only the response is trained on).

License

Apache License 2.0