HyperNix.3-mini

hypernix.3-mini

A 48.7M-parameter decoder-only language model, pretrained from scratch on 0.87B tokens with a fully custom 32,000-token vocabulary, on one gtx1080 — a consumer card — in 12d 3h.

No distillation, no fine-tune of someone else's checkpoint, no borrowed tokenizer. Random init to final weights, on hardware you can buy for the price of a nice dinner.

This is a base model. It completes text. It has never been instruction-tuned, chat-tuned, RLHF'd, or aligned. It will not follow instructions, and at this size it will not reliably answer questions. See What it is not.

---

Benchmarks

Every number below was measured on the hardware named above, by bench_hypernix3_mini.py in this repo, through EleutherAI's lm-evaluation-harness v0.4.13. The baselines were run locally, by the same script, in the same session — none of these figures were copied from another model card.

Baselines are the base (not instruct) checkpoints, because comparing a base model to an instruction-tuned one measures the tuning rather than the pretraining.

Partial evaluation. These numbers come from a --limit 500 pass — 500 documents per task, not the full sets. Treat them as indicative.

Zero-shot. Multiple-choice tasks report length-normalised accuracy (acc_norm); winogrande and lambada report acc; wikitext reports word-level perplexity. ± is the harness's standard error.

<details> <summary><b>Reproduce it</b></summary>

pip install lm-eval transformers accelerate hypernix
python bench_hypernix3_mini.py \
  --model-dir ./hypernix.3-mini \
  --tokenizer-dir ./hypernix.3-mini \
  --device cuda

Full config, per-task metrics and standard errors are in bench_results.json. Run date: 2026-09-25T06:05:34Z; device: NVIDIA GeForce GTX 1080.

</details>

How to read this

hypernix.3-mini is several timesx smaller than the smallest baseline here and was trained on a tiny fraction of their token budgets. On knowledge and commonsense tasks it is not competitive with them, and this card is not going to pretend otherwise — at 48.7M parameters and 0.87B tokens, scores at or near the random baseline on the harder multiple-choice sets are the expected result, not a defect.

What the model is actually for is the other table: it is small enough to train end to end on one consumer GPU in days, small enough to run anywhere, and it is a clean, honest, fully-reproducible artifact of that process.

---

Usage

With hypernix

from hypernix.models.neo_oven import preheat_brewed

oven = preheat_brewed("ray0rf1re/hypernix.3-mini", tokenizer_source="ray0rf1re/hypernix.3-mini")
print(oven.complete("The reason the sky is blue is", max_new_tokens=64, temperature=0.8))

With plain PyTorch

import torch, json
from huggingface_hub import snapshot_download
from hypernix.training.brewer import BrewerConfig, BrewerModel
from transformers import AutoTokenizer

path = snapshot_download("ray0rf1re/hypernix.3-mini")
cfg = BrewerConfig.load(f"{path}/config.json")
model = BrewerModel(cfg).eval()
model.load_state_dict(torch.load(f"{path}/model.pt", map_location="cpu"))
tok = AutoTokenizer.from_pretrained(path)

ids = torch.tensor([tok.encode("Once upon a time")])
for _ in range(32):
    nxt = model(ids)[:, -1].argmax(-1, keepdim=True)
    ids = torch.cat([ids, nxt], dim=1)
print(tok.decode(ids[0]))

Note on sliding-window attention. This model was trained with use_sliding_window=false, and you should load it that way. The hypernix.training.brewer release this was trained against combines the causal and sliding-window masks with torch.maximum, which for additive masks keeps the less masked of the two — so enabling SWA lets every query on odd-numbered layers see up to sliding_window_size - 1 tokens of its own future. The training script in this repo ships a corrected attention forward and turns SWA off by default.

---

Architecture

hyperNix0x-v2 is a decoder-only transformer: RMSNorm (pre-norm), rotary position embeddings, grouped-query attention, and a SwiGLU feed-forward block, with the input embedding tied to the output head.

Training

The full training script is in this repo: train_hypernix3_mini.py. It streams the corpus, trains the vocabulary, packs tokens into a uint16 memmap, and trains — end to end, one command, resumable.

Tokenizer

A byte-level BPE trained from scratch on a 1.5 GB sample of the training corpus. 32,000 tokens, no unknown token, with <|endoftext|>, <|pad|> and the three fill-in-the-middle markers (<fim_prefix>, <fim_middle>, <fim_suffix>) reserved. Round-trips byte-exactly.

Data

Streamed from the sample-10BT shard of FineWeb-Edu — Common Crawl web text filtered by an educational-quality classifier. The stream was tokenized directly into a flat uint16 array and the raw text was never stored, so the whole corpus footprint on disk was the token file. Documents are separated by <|endoftext|> and packed contiguously; no additional filtering, deduplication or reweighting was applied beyond what the source dataset already does.

---

What it is not

• Not an instruction-following model. No SFT, no chat template, no alignment. Prompt it like a text completer, not an assistant.
• Not a knowledge model. 48.7M parameters and 0.87B tokens is not enough capacity or exposure to store reliable facts. It will state false things fluently.
• Not safety-tuned. It reproduces the distribution of its web-derived training data, including its biases. Filter its output before showing it to anyone.
• Not multilingual. The corpus and the vocabulary are English.
• Not long-context. Trained at 512 tokens; quality past that is undefined.

Limitations and bias

The training corpus is derived from Common Crawl web text filtered by an educational quality classifier. It carries the demographic, geographic and viewpoint skew of the English-language web, and no additional debiasing, deduplication beyond the source dataset's own, or toxicity filtering was applied. At this scale the model has no meaningful capacity for factual recall or reasoning, so treat any confident-sounding output as a fluent guess.

License

Apache-2.0 for the weights and code in this repo. The training data (HuggingFaceFW/fineweb-edu) carries its own license — see the source dataset.

Citation

@misc{hypernix3mini,
  title  = {hypernix.3-mini: a 48.7M-parameter language model pretrained from scratch on a single GTX 1080},
  author = {Rayla (ray0rf1re)},
  year   = {2026},
  url    = {https://huggingface.co/ray0rf1re/hypernix.3-mini}
}

Built with HyperNix.