supermix-v89

Supermix v89

A 30.2M-parameter research model that solves physics, chemistry, arithmetic and Python code-tracing problems by writing out its working. Every training row was checked before it was trained on: science by an exact symbolic solver, code by executing the snippet.

What power corresponds to 5712 joules in 48 seconds?
  → power = work / time, 48 into 57 = 1, 57 - 48 = 9, 48 into 91 = 1,
    91 - 48 = 43, 48 into 432 = 9, 9 x 48 = 432, the power is 119 watts,
    total 119
  → CORRECT (119.0)

It is not a chat model, and the limitations section says plainly what it cannot do.

Read this before comparing it with v88

Overall, this model is level with v88. Scored paired against Kai9987kai/supermix-v88 on the same problems, same seed 65, same 96-token cap:

v89 wins 27 problems, v88 wins 24. McNemar exact two-sided p = 0.78 — the difference is noise.

That null hides two large effects in opposite directions, and both were predicted before the run. One is the reason this model exists. The other is a mistake, diagnosed mid-run and stated here rather than smoothed over.

Per-task rows are n = 21, where the 95% Wilson interval is ±20 points at its widest. percent and two_step are inside that interval; power and average are well outside it.

On its own 30-task benchmark v89 scores 0.919 [0.895, 0.938] over 630 problems, 0 unparseable and 0 truncated. The nine code-tracing tasks score 0.963 (182/189).

What this model exists to demonstrate

Long division, and why the previous attempt failed

power was the worst task in the benchmark for eight versions. Classifying every one of v88's 50 wrong replies by the operation that first fails:

DIVISION step          29  (58%)   power 15, acceleration 6, average 5, molarity 3
subtraction step       13  (26%)
no false written step   5  (10%)

Division performed in one jump was 58% of everything v88 got wrong, and power alone was 30%. It wrote 5712 / 48 = 114 and could not know better.

v87 had already tried splitting this step and made it far worse — power fell from 0.333 to 0.048 — because its decompose_quotient back-computed each partial dividend from the answer: to write 1920 / 64 = 30 the model had to already know the digit was 30. The format was a valid presentation of a result, not a procedure anyone could execute forward.

Long division is different in exactly that respect. Every number in the trace is on the page or one bring-down from it, and test_long_division.py checks that property mechanically for every generated row. That test caught two bugs invisible in the output before anything trained on it — a quotient with an internal zero producing a trace that reads as 16 instead of 106, and a tail of rows that breached the sequence budget while the median looked comfortable.

power went from 0.286 to 0.810. It is the largest single-task gain this line has recorded.

The average regression, and what caused it

average divides by four, five or six, and division by six repeats whenever the sum is not a multiple of three, so 22% of its problems asked for a value like 59.333333333333336. v89's corpus was built with a flag that nudges one operand so no mean repeats.

That flag narrowed the training corpus and not the benchmark. The physics tasks share their generators with the evaluator, so a corpus flag changes both sides together. The arithmetic tasks do not — eval_problem_solving._average is its own function and was untouched. The model trained on a narrower distribution than it is tested on, and the replies show structural errors — an operand added twice, four numbers divided by five — that v88 did not make.

It was diagnosed at step 15,000 of 24,250, when average had sat at 0/4 on every probe. The run was allowed to finish because the long-division arm was the point and it touches different tasks. The flag now documents the condition for ever using it again.

Without this error, the paired comparison would read roughly 0.913 against 0.893 — but that is arithmetic on the table above, not a measured result.

Prompt understanding

This model was also trained on a widened bank of prompt phrasings — casual register, contractions, missing punctuation, filler openers — with three forms per task withheld from training so generalisation can be measured separately from recall. On v88, the same corpus arm gave:

The normaliser is a presentation fix: it rewrites choosing 2 out of 54, how many ways is that into the trained form. 0.6967 is what the model itself understands; 0.8567 is what a chat interface running the normaliser delivers. Both are honest numbers for different questions.

Training

The learning-rate note is a real finding: v88's first attempt on this architecture used the default 3e-3 and diverged by step 3,000. Halving it fixed it.

Using it

python example_usage.py "What power corresponds to 5712 joules in 48 seconds?"

src/answer_check.py re-derives the result from the question for 24 question shapes, including code — which it checks by running the snippet — and returns None when it cannot. None means not checked, never correct.

src/step_audit.py reads a reply's own working and reports the first step that disagrees with exact arithmetic. It is what located every failure above.

src/prompt_normaliser.py rewrites a naturally-typed question into the corpus format. The rewrite is always reported so you can see what was actually asked.

Limitations

• Not a chat model. It emits derivations, not conversation. Multi-turn context degrades it: it never saw any.
• Thirty task types and nothing else. Accuracy here is not general problem solving.
• average is worse than v88 at 0.429, for the reason documented above.
• two_step at 0.524 and percent at 0.714 are inside noise relative to v88 but are the weakest arithmetic tasks either way.
• The code tasks trace code; they do not write it. Nine fixed snippet shapes.
• Answers are extracted as the last number in a reply. Every score is a lower bound.
• A correct answer may be recalled rather than computed. The benchmark draws unseen operands from the same space, which is what makes it recitation-proof, but a single answer proves nothing alone.