metallum-1b

Metallum-1B

A 1.0B-parameter ML/LLM-engineering specialist, pretrained from random initialization on a single RTX 5090 — 16.0B tokens in ~13.5 days including gates and recoveries — then one bounded knowledge/format SFT stage. Not a general chatbot. By Home Brewed Labs (project codename MetaLLM).

The engineering core of the project is measurement discipline: every training stage was gated by frozen batteries with exact checkpoint identity, replay accounting (≈1.04B replayed tokens from two hardware incidents, fully receipted), independent adversarial review of evaluation machinery, and a sealed never-trained-on final holdout (H9) reserved for a single one-time qualification.

Architecture

26-layer decoder, d_model 1792, GQA 28Q/14KV, SwiGLU (d_ff 4864), RMSNorm pre-norm + QK-norm + z-loss, RoPE θ=500k with NoPE every 4th layer (length-generalizing retrieval heads), document-masked attention, 40k byte-level BPE with FIM sentinels, 2048 native context. Muon (2D) + AdamW, WSD schedule, curriculum with short-sequence early phases.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("HomeBrewedlabs/metallum-1b",
                                             trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("HomeBrewedlabs/metallum-1b")

trust_remote_code=True is required: this is a custom architecture (QK-norm, NoPE interleave, document masking). Do not map to a stock Llama config — that silently drops QK-norm/NoPE and produces a different model than the one measured below. Stock GGUF/vLLM converters reject this architecture by design; a faithful llama.cpp port is on the roadmap.

Evaluation (champion s2b075, frozen batteries, repeated where noted)

All numbers were produced by frozen evaluators with sealed checkpoint identity; the ineval suite was used for model selection during development (selection-aware caveat), which is why the untouched H9 result is the number that matters for release claims.

Structured output: by decoding, not by hope

Base weights score 0/60 on a strict whole-output format suite (JSON schema / tool-call / MCQ-letter), and three bounded SFT recipes plus a ReST-EM screen all failed to move that (all receipted). This model therefore ships with decode-time structure enforcement: serve_metallum.py (included) constrains generation with a token-level JSON grammar + schema-forced keys, and scores 60/60 on the same frozen suite. If you consume the raw weights without the wrapper, expect prose, not JSON.

python serve_metallum.py --port 8399   # localhost endpoint: /generate, modes free|json|tool|mcq

Intended use

ML/LLM-engineering assistant: PyTorch/training scaffolding, concept explanation, long-context retrieval over technical docs, structured-output endpoints via the wrapper. Out of scope: general chat, non-ML factual QA, general-purpose coding, safety-critical use, autonomous code execution.

Honest limitations

• neural_ops floor exception: this checkpoint scores 4/5 on the internal neural-ops category vs the 5/5 preregistered floor. The exception was adjudicated explicitly by the owner; it is disclosed, not hidden.
• MCF (250q multiple-choice) is permanently non-promotable: the teacher generation prompts embedded real MCF items, so no MCF number from this lineage is valid evidence. Do not report one.
• ineval is selection-aware: 7/150 was measured on a suite used during development. The sealed H9 result (1.3953 BPC on 400k post-cutoff Wikipedia characters, single run, contamination-verified at 0.0099% shingle containment) is the generalization claim.
• Free generation makes local factual slips; verify specifics.
• No preference/safety alignment; no RLHF.
• Format behavior requires the constrained decoder (above).

Training data

Pretraining: 19.399B effective tokens (8.473B unique): 53.6% code, 19.6% knowledge, 15.2% reasoning, 11.6% math. 1.24% (240.6M tokens) is provider-teacher-derived; full lineage map is maintained by the authors. SFT (12,496 rows): 40% locally self-hosted open-teacher MCQs (qwen2.5-coder:14b, Apache-2.0), 12% self-generated synthetic retrieval rows, 48% provider-teacher-derived replay (disposition: owner written-rights determination, on file and available on request).

Attribution — Stack Exchange (stack_ml_edu, 2.973B pack tokens, 15.3% of the pack): CC BY-SA 4.0. Attribution is owed to Stack Exchange contributors, and the share-alike obligation attaches to that portion of the training data.

License

Apache-2.0 (model weights + code). See the attribution note above regarding training-data share-alike obligations.

Citation

Technical report in preparation.