supermix-v74

Supermix v74

A newer model is available. Kai9987kai/supermix-v80 adds twelve physics and chemistry tasks and scores 0.575 across 21 task types. Its number is not comparable to the 0.894 below, which was measured on 10 arithmetic tasks only — they are different benchmarks. v74 remains the stronger model on those 10 arithmetic tasks.

An 8.6M-parameter research model that solves arithmetic word problems by writing out its working. It is not a general chat model, and the sections below say plainly what it cannot do.

what is 47 times 6
  asked as : What is 47 x 6?  (multiplication)
  reply    : 40 x 6 = 240, 7 x 6 = 42, total 282
  check    : CORRECT (282.0)

What it scores

0.894 (447/500) on a held-out problem-solving benchmark of ten task types, against 0.756 for the previous version, z=5.74. Problems are generated fresh at evaluation time, so these are novel problems rather than recalled ones. 0 replies were unparseable.

Read that headline honestly

The 0.894 is flattered by the four new task types, which this model finds easy. Compared like-for-like on the five tasks both versions were measured on, it is 0.818 vs 0.756 (z=1.99) — over the significance line by a hair, and driven almost entirely by average improving from 0.16 to 0.59.

One task genuinely regressed: arithmetic fell 0.99 → 0.89 (z=−2.81). Adding six task types to the training mix cost accuracy on the simplest one.

Raw numbers are in results/problem_solving_n500.json.

Prompt format matters — a lot

The benchmark score measures this model on its own training distribution. It is not a measure of how it handles natural language, and the gap is total: typed the way a person writes, the first five arithmetic questions tried against it were all wrong.

Measured, varying one feature at a time:

• the operator token matters — x works, times and * do not
• a lead-in phrase must be present — bare 47 x 6 is parsed as algebra
• capitalisation and trailing ? do not matter

src/prompt_normaliser.py rewrites a naturally-typed question into the trained form, which takes it to 7/7 on the same kind of questions. It is presentation, not capability: it computes nothing, never alters a number, and questions the model gets wrong in the trained format stay wrong.

The trained format, one example per task:

It recites on conversation

The model produces fluent-looking conversational replies:

"Check the traceback first, then we can isolate the failing function."

Every one of them is reproduced verbatim from its training data. The verbatim rate on the held-out dialogue probes is 1.0. This is a measured property of the training corpus, 19.8% of whose dialogue portion is a single repeated fragment.

Do not use this as a chat model. Its conversational ability is recall.

Usage

python example_usage.py "what is 47 times 6"

import sys; sys.path.insert(0, "src")
import prompt_normaliser, answer_check
from train_mimomix_talk import load_talk_checkpoint, generate_reply

model, tokenizer, _ = load_talk_checkpoint("supermix_v74.pt")
model.eval()

asked = prompt_normaliser.normalise("what is 128 divided by 8")
reply = generate_reply(model, tokenizer, asked.prompt, max_new_tokens=64)["reply"]

verdict = answer_check.check(asked.prompt, reply)   # None => not checkable

Requires torch. CPU is fine — the model is 8.6M parameters and a reply takes well under a second.

Always check the answer. src/answer_check.py re-derives the result from the question for nine question shapes and returns None when it cannot verify one. A None means not checked, never correct.

Architecture

Hybrid sliding-window/global attention with a sparse mixture-of-experts feed forward and a recursive thinking core, plus a multi-token-prediction head used for self-speculative decoding.

Digit-level tokenisation is what makes arithmetic representable at all — a word-level vocabulary cannot spell an unseen number.

Generalisation

Sentences were withheld wholesale from training, not split by row, so the held-out tiers measure composition rather than recall of a template:

Perplexity ratio tier3/tier1 is 1.008x. Full report in results/generalisation_results.json.

Training notes

The run segfaulted at step 11,500 of 18,000 after 9.2 hours and was resumed mid-curve — the OneCycle schedule was restored at 11,500 and ran to exactly 18,000, rather than re-warming on a fresh curve.

That detail matters for interpreting the model: at the point of the crash it looked much worse than it is. Three shared tasks appeared to have regressed badly (arithmetic 0.55, algebra_one_step 0.61, word_problem 0.73), and two of the three were simply unfinished, recovering to 0.89 and 0.96 during the anneal.

Limitations

• Not a chat model. Conversational replies are verbatim recall.
• Format-brittle. Use the trained prompt format, or the included normaliser.
• No knowledge. It was trained on generated arithmetic and a small templated dialogue set. It knows no facts about the world.
• percent and average are weak (0.75 and 0.59). It reliably produces correct intermediate working and then fumbles the final combine — 15% of 240 decomposes to 24 + 12 and answers 26.0.
• 8.6M parameters. This is a research artifact for studying whether small models can be made to solve problems verifiably, not a useful assistant.

License

No license has been specified for this repository yet. Until one is added, treat it as all rights reserved.