surge-fm-v3

surge-fm-v3 — Chronos-2 fine-tuned for every US EIA-930 balancing authority

Full fine-tune of amazon/chronos-2 on 7 years (2019–2025) of hourly load across 53 balancing authorities — every BA that publishes a demand series to EIA-930, spanning the Eastern, Western and Texas interconnections.

What you feed it matters more than which checkpoint you use. With a real day-ahead weather forecast this checkpoint scores 0.536 MASE on the 7 RTOs; with no future weather, 0.594; and with a flat persisted temperature — which is what surge's own API shipped for months — 0.740, which is worse than sending nothing at all. Read Feeding covariates before deploying.

Results on the 2025 hold-out

Rolling 24 h-ahead forecasts, step 24, exactly 8,760 hours, store deduplicated on (ts_utc, ba), MASE denominator = per-BA train-split seasonal-naive (m=24). Causality is enforced by a runtime check that perturbs all values at and after the forecast origin and requires the future covariates not to move.

Macro MAE with forecast weather: 1,073 MW over the RTOs, 250 MW over all 53.

Given identical forecast covariates this checkpoint beats stock Chronos-2 by 5.1% on the 53-BA macro, with non-overlapping 95% intervals ([0.534, 0.548] against [0.563, 0.577]). Given no future weather the two are within noise of each other. The fine-tuning is worth something specifically because the model learned to trust temperature — which only pays off when the temperature you supply is accurate.

Feeding covariates (read this)

Past covariates — load history, observed temperature, calendar features, and optionally peer BAs' load history. All of it is genuinely observable.

Future covariates over the horizon — calendar features always, and temperature only if it is a real forecast. Concretely:

• ✅ An archived or live NWP day-ahead forecast. surge uses Open-Meteo's temperature_2m_previous_day1 (the value forecast ~24 h before the valid hour); measured skill 1.06–1.55 °C MAE, correlation 0.983–0.990.
• ❌ Observed / reanalysis temperature. This is perfect foresight. It inflates results and is not reproducible in operation.
• ❌ A flat persisted temperature. Measurably worse than omitting temperature (0.740 vs 0.594). A constant 24 h temperature implies no diurnal cycle, and this checkpoint trusts the covariate.
• ❌ Realized wind/solar generation. Never knowable ahead.

Keep the past and future temperature channels on the same source and location. Mixing an ASOS station's history with a model-grid forecast left a ~5 °C discontinuity at the forecast boundary for PJM, whose representative station (DCA) sits ~200 km from its load centroid.

Crude causal substitutes were tried and all lost to simply withholding temperature: same-hour-yesterday (0.614 val), a train-fitted month × hour climatology (0.624 val), and flat persistence (0.629 val), against 0.609 for no future temperature.

Correction notice (supersedes earlier metrics)

Figures previously published on this card — RTO 0.518, 53-BA 0.636 — were not forecasts and should not be cited. Three defects were fixed:

1. Covariate leakage. Observed ASOS temperature and EIA actual wind and solar generation were declared known-future and sliced over the forecast window, handing the model realized values for the hours it was predicting.
2. Undeduplicated store. The append-only parquet store was scanned without deduplication, leaving up to 4 rows per hour (PJM: 1,112 duplicates). Because the series is indexed positionally this corrupted lag arithmetic and the seasonal-naive MASE denominator.
3. Unpinned test window. The "2025 hold-out" had no upper bound and had grown to 2025-01-01 → 2026-04-19, so the published number drifted with every ingest.

The weights here are unaffected by (1) — the leak was in evaluation and inference, not training. They are affected by (2), having been trained on the duplicated series.

Limitations

• Calibration is the weak point, and fine-tuning made it worse. The nominal 80% interval covers ~70% for this checkpoint versus ~76% for stock Chronos-2. If you need calibrated intervals more than sharp medians, apply conformal calibration on a held-out split, or use the base model.
• Concept drift is real. Measured ~8% MASE degradation moving from Jan–Apr 2025 to Jan–Apr 2026, for this checkpoint and for zero-shot Chronos-2. Plan on periodic re-adaptation.
• Five BAs have inflated MASE denominators (BANC, SPA, LGEE, SEC, TEPC): surge's outlier filter used an absolute 200 GW threshold, so a 70 GW spike in a 1.8 GW BA survives. Excluding them raises the 53-BA macro.
• Weather forecast archive starts 2021-03 (GFS 2 m temperature), which bounds how far back forecast-consistent training can go.
• Research and reference use only. Not for trading, regulated bidding, or bankability-graded decisions.

Usage

import torch
from chronos import BaseChronosPipeline

pipe = BaseChronosPipeline.from_pretrained(
    "Tylerbry1/surge-fm-v3", device_map="cuda", torch_dtype=torch.bfloat16
)

quantiles, mean = pipe.predict_quantiles(
    [{
        "target": load_history,              # observed MW
        "past_covariates": {
            "temp_c": temp_history,           # observed
            **calendar_history,               # hour/dow sin-cos, weekend, holiday
        },
        "future_covariates": {
            "temp_c": temp_day_ahead_forecast,  # a REAL forecast, not observations
            **calendar_future,
        },
    }],
    prediction_length=24, quantile_levels=[0.1, 0.5, 0.9],
)

Full harness, the causality check that enforces the above, and the weather scraper are in tylergibbs1/surge (experiments/causal_guard.py, surge/scrapers/openmeteo.py).