jin10-sdft-qwen2.5-7b

jin10-sdft-qwen2.5-7b

Qwen2.5-7B-Instruct after twelve windows of rolling self-distillation fine-tuning (SDFT) on Jin10 prediction-market data. Given a market, its recent hourly prices and a pool of macro-wire headlines, it predicts how far the next hour's move will land above or below a mean-reversion baseline — not the price itself.

Task

The baseline is a frozen OLS fit on the current price, 0.064650 − 0.031089 × price, estimated once on the warmup window's 461 training rows and held fixed for the whole schedule. The model forecasts the residual against it, discretised into nine quantile buckets.

Input is a system message defining the buckets plus a user message with the market description, ~23 hourly price points and ~30 headlines. Output is a reasoning chain and one bucket:

<reasoning>
...which headlines, if any, actually bear on this market...
</reasoning>
<answer>
B4
</answer>

For scoring the bucket becomes a number as anchor_baseline + midpoint(bucket), with midpoints [-0.153, -0.085, -0.026, -0.011, +0.000, +0.014, +0.034, +0.059, +0.166].

Results

6323 forward rows, each scored by the checkpoint that had not yet trained on that window. Skill is 1 − MSE(model)/MSE(anchor); abstentions fall back to the anchor.

On the 2145 permanent holdout rows the model never trains on: +0.0251, the only model in the comparison clearly ahead of the anchor there (runner-up +0.0079).

What this model does not do

It has not learned to classify. Bucket accuracy is 0.111 against a nine-bucket chance rate of 0.111. It beats the anchor because being one bucket off still moves the prediction the right way, not because it picks the right bucket.

It barely moves. Predicted standard deviation is 0.0203 against a true 0.1038 — about a fifth as much as the target varies. Inspecting the predictions: it selects the near-zero bucket B4 on 78.7% of rows, where the offset is +0.00007, so on four rows in five the prediction is the anchor. It never selects the extreme buckets B0 or B8 at all, so the largest moves in the data are outside the range it can express. All of the squared-error improvement comes from the 21% of rows where it does move, and on those it is closer than the anchor 54.9% of the time.

Two zero-shot frontier models understand the task better than it does — higher Pearson, higher bucket accuracy — and still score worse, because they are badly calibrated in magnitude. This model wins on restraint, not on insight.

The rolling schedule is not what earns the result. Frozen after the warmup window alone — 461 rows, 9.5% of the training data — the model already scores +0.0170. The remaining eleven windows and 4378 rows add +0.0024. Freezing costs nothing over time either: the warmup-only checkpoint shows no decay when evaluated on windows up to ten months past its freeze point.

Training

Two settings matter and both were established by ablation. Taking the teacher from the previous window's checkpoint instead of the base model collapses the run — completion length falls from 609 tokens to 6 and the model abstains on every probe from the fifth window on. Freezing the teacher rigidly instead of resyncing keeps the loss at 0.19–0.20 every window instead of falling to 0.09, and damages the model's off-task behaviour from the first window.

Off-task behaviour

Twelve prompts unrelated to the task, greedy decoding, across all twelve checkpoints: generation length stays at or above the base model's 276 tokens with no downward trend. The cost is format — 8 to 11 of 12 off-task answers come back wrapped in <reasoning> tags, with normal correct prose inside. A variant at learning rate 1e-5 leaks nothing at all but its skill falls to +0.0006.

Limitations

• Evaluated on one market family over one twelve-month span. No claim outside it.
• Bucket accuracy and RMSE here are forecasting metrics, not trading P&L.
• The forward cells are prequential: the model has seen none of a window when scored on it, but 70% of those rows are trained on immediately afterwards. The 2145 holdout rows are never trained on and give +0.0251.
• The comparison against zero-shot models is not like-for-like: this model is constrained to nine bucket midpoints, they are free to emit any number.