supermix-v86

Supermix v86

A 15.3M-parameter research model that solves physics, chemistry and arithmetic problems by writing out its working. Every training example was checked by an exact symbolic solver before it was trained on.

In how many ways can 2 items be chosen from 18?
  → combinations = n x (n - 1) / 2, 18 - 1 = 17, 10 x 17 = 170,
    8 x 17 = 136, half of 306 = 153, there are 153 ways, total 153
  → CORRECT (153.0)

It is not a chat model, and the limitations section says plainly what it cannot do.

Scores

0.779 overall, 95% Wilson interval [0.745, 0.810], on 630 freshly generated novel problems across 21 task types. 0 unparseable, 0 truncated.

This one is comparable to v80

Kai9987kai/supermix-v80 carries a section headed "Do not compare this with Supermix v74's 0.894", because those two were scored on different benchmarks. That warning was right, and it applies to v80's own published 0.575 as well: it was measured at an unrecorded generation cap on a since-rewritten set of generators.

So v80 was re-scored from its own checkpoint under identical conditions — same 630 problems, same seed 65, same 96-token cap, same generator fingerprint 4077062251bc762c9716a730f3818ad2 — and that re-scored figure is what the table below compares against.

The intervals do not overlap. Both receipts are in results/.

⚛ = science task. science 0.894 · arithmetic 0.626. Eleven tasks reach 1.00, against one in v80.

Per-task rows are n = 30, where the 95% Wilson interval is ±17 points at its widest. Treat a per-task change smaller than that as noise; the overall row (n = 630, ±4 points) is the one that carries weight.

The finding this model exists to demonstrate

combination went from 0.000 to 0.933, and it is the one number here that cannot be explained by anything except the model.

That task read 0.000 on every generator version — it was not a measurement artifact. v80 had memorised the answer format and guessed the value:

asked:  In how many ways can 2 items be chosen from 18?     (truth 153)
v80:    combinations = n choose k, 18 choose 2 = 164, total 164
v86:    combinations = n x (n - 1) / 2, 18 - 1 = 17,
        10 x 17 = 170, 8 x 17 = 136, half of 306 = 153, total 153

The corpus change was to keep every intermediate inside what the model had already been shown to learn — two-digit by one-digit products and two-digit sums — rather than asking it to recall an entire nCk table. This is the same lesson v80 demonstrated for multiplication, applied one level further.

What did not work

average is unchanged at 0.033. It has been this line's worst task since v70 and survived a full retrain. Measuring where it breaks:

first addition correct                          1 / 120  = 0.008
every individual addition step                  7 / 479  = 0.015
division correct, given the model's own total  83 / 120  = 0.692

The division works. The accumulation fails on the very first two-digit addition, at every chain length — accuracy is flat at 0.05 / 0.00 / 0.00 for 4, 5 and 6 values. The format emits running totals (sum: 48 then 137 then 170) and never writes an addition as an equation, so each sum has to happen with no scratchpad. results/average_by_arity.json has the measurement.

two_step regressed 0.633 → 0.300. That is outside the ±17-point per-task interval, so it is real, and it is not explained. arithmetic and algebra_one_step also slipped about 7 points each. Something in the new corpus helped twenty tasks and hurt multi-step arithmetic.

Training

Selection runs on accuracy because loss and accuracy are not reliably related in either direction on this line — an earlier model posted better loss and scored far worse.

The mid-run probe previously capped generation at 64 tokens while arithmetic_series replies are 81–84 tokens long, so that task read 0.00 during training whatever the model had learned. The cap is now 112 and the trainer refuses to start when a task's median reply cannot fit.

Using it

python example_usage.py

src/answer_check.py re-derives the result from the question for 21 question shapes and returns None when it cannot. None means not checked, never correct. src/prompt_normaliser.py rewrites a naturally-typed question into the corpus format — "47 times 6" becomes "What is 47 x 6?" — because these models answer the trained form and not the natural one. The rewrite is always reported so you can see what was actually asked.

Limitations

• Not a chat model. It emits derivations, not conversation. Multi-turn context degrades it: turn-aligned packing means it never saw any.
• Twenty-one task types and nothing else. Accuracy here is not general problem solving. Every prompt comes from a handful of fixed phrasings.
• Answers are extracted as the last number in a reply, so a reply that reasons correctly and trails into another number scores wrong. Every score is a lower bound.
• A correct chat answer may be recalled rather than computed. The benchmark draws unseen operands from the same space, which is what makes it recitation-proof, but a single answer to a question that appears in training proves nothing on its own.
• average does not work. 0.033, and the reason is known and documented above.