Brujula-450M

Brújula-450M

A 512M-parameter (448M non-embedding — hence "450M") DeepSeek-style decoder, trained from scratch by one hobbyist. It is the largest model in the Brújula family and the base of a small tree of fine-tunes that probe what a model this size can and cannot learn.

Pre-training ran on a rented H100 (~57 h, ~$200); everything else in the family — the fine-tunes, the evaluation, the export — was done locally on a single Intel Arc B580 (12 GB).

What you are getting, up front

This is a hobby base model. It beats GPT-2-small on WikiText and writes coherent paragraphs, and that is the whole of it:

• It hallucinates freely. No instruction tuning, no alignment. It completes text; it does not follow instructions or answer questions reliably.
• 1024-token context as shipped — this is not a long-context model out of the box. YaRN gets it to a usable 16K for free, but 32K/64K are coin-flips (see below).
• No KV cache in this export, so generation is slower than a comparable standard model.
• It scores 0% on document QA and 0% on BABILong retrieval before fine-tuning. That is not a typo, and it is the point: the fine-tunes below are where the capability comes from.

Compare it to other consumer-GPU from-scratch projects, not to lab models trained on 10–100× the data.

Results

Measured against the 150M flagship on the same validation set under the same local AMP protocol:

−20% validation perplexity and −28% WikiText against the 150M, and ~20% better WikiText than a real GPT-2-small run through the same harness. (The training box's own full-val was 2.68 / 14.59; the 2.63 is a 300-batch local measurement. Both are reported rather than picking the flatterer.)

The fine-tune tree

This base exists to be fine-tuned, and the children are the interesting part — including where they fail:

• Brújula-450M-DocQA — pass it a paper, it answers from it. 0.0% → 22.7% on held-out documents from ~4,800 training samples.
• Brújula-450M-Retrieval — BABILong needle retrieval, 0% → 93% at 16K and holding ~80% out to 64K. Read its limitations: it is a synthetic-needle specialist and fails on real documents.

The 0% figures are this base model, scored on the same rows with the same grader. They are honest floors, not rhetorical ones: on document QA the base emits fluent, non-empty, wrong text.

Architecture

• MLA (Multi-head Latent Attention, DeepSeek-V2-style low-rank KV/Q) + RoPE + SquaredReLU FFN + tied embeddings.
• v3 Block Attention-Residuals — instead of a plain residual sum, each sublayer takes a softmax attention over windowed block-sums of earlier layers (arXiv:2603.15031).
• n_embd 1280, 28 layers, 8 heads (head_dim 160), attnres_block_size 7, vocab 50257 (GPT-2 BPE).
• Pre-trained at block 1024 on FineWeb-Edu, 2 epochs (~44 tokens/param) under one continuous cosine schedule (peak LR ~3e-3 annealing to ~0 across both epochs — not per-epoch restarts). The second epoch plus the anneal is where it overtook the 150M.

Context length

This model ships at its native 1024-token window, unscaled — that is the regime its numbers above were measured in, and baking a RoPE warp into the config would silently change it.

It does extend training-free with YaRN. Measured single-needle passkey retrieval, no fine-tuning:

16K is genuinely usable for free. 32K and 64K are coin-flips and skew toward late positions — for reliable long context, use the Retrieval fine-tune, and read its caveats first. Note that passkey measures planted-string retrieval only: this base scores 0% on BABILong (bAbI facts in real narrative prose), which is a much harder and more honest long-context test.

Usage

Custom modeling code, so trust_remote_code=True:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Sakatepon/Brujula-450M"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16
).eval()

ids = tok("The mitochondria is the", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95,
                     repetition_penalty=1.3, use_cache=False)
print(tok.decode(out[0]))

use_cache=False is required: Block Attention-Residuals mix across layers, so the stock HF KV cache does not apply. (The training repo has a working cached-decode path; it is not part of this export.)

To extend context yourself, set rope_scaling_method="yarn_ms", rope_trained_len=1024 and rope_scale_len to your target length in the config before loading.

Limitations

• Hallucinates confidently. It is a 512M base model trained on ~20B tokens. It has no instruction tuning and no alignment of any kind.
• No KV cache in this export → generation is slower than an equivalently-sized standard model.
• Not a long-context model out of the box (1024 native; see above).
• English only, GPT-2 BPE, FineWeb-Edu domain (educational web text).
• Base-model behaviour: it completes text, it does not follow instructions.

Training data

FineWeb-Edu (ODC-By 1.0).

License

Apache-2.0.