Nexus-Erebus-3M

Nexus-Erebus-3M

A 3-million-parameter language model trained from scratch. It is an arithmetic specialist: it handles integer arithmetic well for its size, and stays near chance on general tasks, at a footprint that runs almost anywhere.

How it works

The model reads and writes numbers least-significant-digit first, with a digit-atomic tokenizer. Aligning carry propagation with reading order is what lets a model this small do arithmetic. The digit reversal happens inside the tokenizer, so you pass and receive normal text. This requires trust_remote_code=True.

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("MaliosDark/Nexus-Erebus-3M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("MaliosDark/Nexus-Erebus-3M")

Results

Full test sets, 0-shot. acc_norm for multiple choice, accuracy for ArithMark-2.

The model is built for integer arithmetic. Non-arithmetic tasks sit near chance, which is expected at this size.

Reproduce the ArithMark-2 score

python benchmark_nexus_arithmark.py MaliosDark/Nexus-Erebus-3M

Training

Pretrained from scratch with a digit-atomic 4k tokenizer, least-significant-digit order, and a large stream of freshly generated synthetic integer arithmetic (addition, subtraction, multiplication, exact division, mixed and parenthesised multi-operator expressions), mixed with TinyStories for language. Then a short fine-tune on public benchmark train splits plus more arithmetic. No evaluation items were used at any stage. Fresh generation, never reusing a fixed pool, is what stops the model from memorising instead of learning.

Limitations

• It is an arithmetic specialist. On non-math tasks it performs near chance, by design.
• It reasons over integers in the ArithMark-2 style; it is not a general assistant.

License

MIT. Trained from scratch: 100% original weights, own tokenizer, no base model.