Brujula-15M

Brújula-15M

The tiny champion of the Brújula family — a 15.5M-parameter decoder-only language model, trained entirely on a single consumer GPU (one Intel Arc B580, ~5h16m) from scratch on FineWeb-Edu. Brújula ("compass" in Spanish) is a minimal DeepSeek-style architecture: Multi-head Latent Attention (MLA) + RoPE + SquaredReLU FFN, tied embeddings, hybrid Muon + AdamW optimizer.

A base completion model (not instruction-tuned). At 15M it's a research/education artifact — surprisingly fluent short continuations for its size, but not a knowledge source.

Results

Perplexity (lower is better), fixed local harness at context length 1024:

It won't compete with much larger models on absolute perplexity — the point is that this is a complete, from-scratch LM that fits and trains on one consumer GPU. See the family below.

The Brújula family

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Sakatepon/Brujula-15M"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()

ids = tok("The mitochondria is the", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_p=0.95, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))

It's tiny — use sampling + a continuation cue ("X is the … " rather than a bare noun); greedy tends to repetition-loop.

Architecture

Training

FineWeb-Edu (~1.4B tokens), 1 epoch, hybrid Muon + AdamW, bf16, ~5h16m on a single Intel Arc B580. Fully local — no cloud.

Limitations

• Base completion model — not instruction-tuned, no safety tuning.
• English only, educational-web distribution (FineWeb-Edu).
• At 15M it produces plausible short prose but unreliable facts; best on cued, definitional prompts.
• Short context (1024); no KV-cache in this reference implementation.

License & attribution

• Model + code: Apache-2.0. Training data: FineWeb-Edu (ODC-BY).
• Built on ideas from: DeepSeek-V2 (MLA), Muon optimizer, Primer (SquaredReLU), GPT-2 (BPE tokenizer).