codette-newton-star-hard

newton-star-hard — ⚠️ Research Artifact: Below-Baseline Result

Second arm of a controlled STaR study. Trained on 350 keep-correct reasoning chains from difficulty-matched MMLU-Pro STEM (61% keep yield — the data genuinely challenged the model). Scored 28.0% on GPQA-main vs the untrained baseline's 34.0%: harder data attenuated the degradation seen with easy data (25.0%) but did not eliminate it.

Notable provenance: trained free on a Kaggle T4 after working through six documented environment issues (all fixes committed to the repo), and one literal household power failure. Where there's a will.

The STaR Study (GPQA-main, reason mode, n=100 per arm)

Finding: keep-correct self-taught reasoning consolidates existing ability rather than extending it — training data difficulty orders the outcome perfectly, but no keep-correct arm matched the untrained baseline. Full methodology, controls, and changelogs: Codette-Reasoning (see docs/CHANGELOG_2026-07-09.md, docs/CHANGELOG_2026-07-11.md).

Created by Jonathan Harrison (Raiff1982) · Raiff's Bits LLC