Brujula-18M

Brújula-18M (G_stack)

A 18M-parameter decoder-only LM created by depth-growing Brújula-15M with the G_stack operator (Du et al., "Stacking Your Transformers", NeurIPS 2024): copy the trained 15M's 4 layers into an 8-layer model, then continue pre-training. The whole thing runs on a single consumer GPU (Intel Arc B580). Brújula ("compass" in Spanish) uses a minimal DeepSeek-style architecture (MLA + RoPE + SquaredReLU, tied embeddings, Muon).

The result: depth-doubling the 15M champion nearly halved perplexity on both metrics, for ~3h of extra local compute.

Results

Perplexity (lower is better), fixed local harness at context length 1024:

Honest note: the gain comes from added depth + warm-start (the grown model effectively saw ~2× the cumulative tokens of the base), not depth alone — no from-scratch-8-layer control was run. Either way, it's the best sub-50M model in the family.

The Brújula family

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Sakatepon/Brujula-18M"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()

ids = tok("The mitochondria is the", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_p=0.95, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))

Small base model — use sampling + a continuation cue; greedy tends to repetition-loop.

Architecture

How it was made (G_stack)

1. Train Brújula-15M from scratch (4 layers).
2. Stack: copy the 4 trained layers into an 8-layer model ([0,1,2,3] → [0,1,2,3,0,1,2,3]); keep the embedding / final-norm / tied head.
3. Continue pre-training 1 epoch on FineWeb-Edu (~1.4B tokens), batch 32, peak LR 1.2e-3, bf16, ~3h14m on one Intel Arc B580.

The post-stack loss spikes (copy-init isn't function-preserving), then recovers and surpasses the 15M base within the first ~12% of the epoch — consistent with G_stack's finding that violating function preservation is fine and even preferable.

Limitations

• Base completion model — not instruction-tuned, no safety tuning.
• English only, educational-web distribution (FineWeb-Edu); weaker out-of-distribution.
• ~18M params: plausible prose, unreliable facts; best on cued, definitional prompts.
• Short context (1024); no KV-cache in this reference implementation.

License & attribution

• Model + code: Apache-2.0. Training data: FineWeb-Edu (ODC-BY).
• Methods: G_stack (Du et al., 2024), DeepSeek-V2 (MLA), Muon, Primer (SquaredReLU), GPT-2 (BPE).