llm-350m-instruct-v2

LLM-350M-Instruct-V2

This is V2 of a 350M parameter language model I trained from scratch as a personal learning project. V2 improves on V1 by replacing Alpaca-cleaned with OpenHermes-2.5 — 200K GPT-4 generated examples in ChatML format instead of 52K GPT-3.5 examples in Alpaca format. Same pretrained base, better finetuning data.

I'm not a researcher. I don't work at a big lab. I just wanted to understand how LLMs actually work by building one. The whole thing ran on a single rented GPU for under $500 total across both versions.

V1 model | Training code | W&B logs

---

⚠️ Important: Prompt Format

This model requires ChatML format. If you send raw text without the template, the model won't recognize it as an instruction and will produce poor output. This is the single most common issue people hit — configure your inference tool before you test.

The format looks like this:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Your question here<|im_end|>
<|im_start|>assistant

Setup instructions for each tool are in the Quick Start section below.

---

Benchmarks

Val loss: 1.3704 vs V1's 1.7189 (20.3% lower)

Benchmark deltas are small and mixed — expected at 350M scale, and sensitive to the prompt format change (Alpaca → ChatML affects loglikelihood scoring). Val loss is the more reliable signal here.

---

What Changed from V1

---

Architecture

Same as V1 — modern LLaMA-style architecture at 350M parameters: RoPE positional encoding, RMSNorm (pre-norm), SwiGLU activations, Grouped Query Attention (4 KV / 16 query heads), 2048 token context, Flash Attention 2, tied embeddings.

Full architecture and pretraining details in the V1 model card.

---

Quick Start

Prompt Format (ChatML)

All requests must be wrapped in ChatML tags. Raw questions without the template will not work correctly.

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Your question here<|im_end|>
<|im_start|>assistant

---

LM Studio

This is the most important setup step — skip it and the model will produce bad output.

1. Download llm-350m-instruct-v2-q4_k_m.gguf from the Files tab (~197MB)
2. Load it in LM Studio
3. Click the prompt format dropdown (top of the chat window) → select ChatML
   • If ChatML isn't in the list, click "Edit" and set manually:
     • User prefix: <|im_start|>user\n
     • User suffix: <|im_end|>\n
     • Assistant prefix: <|im_start|>assistant\n
     • Assistant suffix: <|im_end|>\n
     • System prefix: <|im_start|>system\n
     • System suffix: <|im_end|>\n
4. In Model Parameters, set repeat_penalty to 1.1 (prevents repetition loops)
5. Now type your question normally — LM Studio handles the wrapping automatically

---

Ollama

The included Modelfile configures ChatML automatically:

# Option 1: Run directly
ollama run sandbreak80sd/llm-350m-instruct-v2

# Option 2: Build from Modelfile (for customization)
# Download Modelfile from the Files tab, then:
ollama create llm-350m-v2 -f Modelfile
ollama run llm-350m-v2 "What is the sun?"

Ollama handles the ChatML template automatically when using the published model.

---

llama.cpp (CLI)

# Download llm-350m-instruct-v2-q4_k_m.gguf, then:
./llama-cli -m llm-350m-instruct-v2-q4_k_m.gguf \
  --prompt "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nWhat is the sun?<|im_end|>\n<|im_start|>assistant\n" \
  -n 256 --temp 0.7 --top-p 0.9 --repeat-penalty 1.1 \
  --stop "<|im_end|>"

The --stop "<|im_end|>" flag is required — without it the model won't know when to stop generating.

---

Python (HuggingFace Transformers)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("sandbreak80sd/llm-350m-instruct-v2")
model = AutoModelForCausalLM.from_pretrained(
    "sandbreak80sd/llm-350m-instruct-v2",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Always use ChatML format — do not send raw text
prompt = "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nWhat is the sun?<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True,
    repetition_penalty=1.1,
    eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

---

Limitations

Same fundamental limitations as V1 — 350M parameters trained on a hobbyist budget:

• Math: Unreliable beyond simple arithmetic. Do not trust numerical outputs.
• Code: May be structurally plausible but semantically wrong. Always verify.
• Repetition: Without repeat_penalty=1.1, the model can loop. Always set this.
• Prompt format sensitivity: Must use ChatML. Raw prompts produce degraded output.
• No safety alignment: SFT only, no RLHF or DPO. Not for production use.
• Knowledge cutoff: Limited to pretraining data; no real-time information.
• Context: 2,048 tokens maximum.

---

Training Cost

~$12 for V2 finetuning on g6e.xlarge (L40S GPU). Full project cost including pretraining and V1: ~$310.

---

Citation

@misc{llm-350m-instruct-v2,
  author = {Stoner, Brad},
  title  = {LLM-350M-Instruct-V2: A 350M LLM trained from scratch with OpenHermes-2.5},
  year   = {2026},
  url    = {https://huggingface.co/sandbreak80sd/llm-350m-instruct-v2},
  note   = {Training code: https://github.com/sandbreak80/llm-350m}
}