tsfm-toto2-4m-vq-ce-native

TSFM tokenizer ablation — Toto-2-4m-recipe clone (v10)

VQ-VAE OPTIMIZATION ablation (codebook health + next-code objective vs raw patches) on a ~3.6M-param decoder-only patched transformer trained on the OFFICIAL TempoPFN synthetic prior generators (cloned from automl/TempoPFN; GP/KernelSynth families capped at 6,144 steps, so the long pool uses the cheap families only) with contiguous patch masking, a 9-level quantile head, NorMuon+AdamW, and index RoPE at time-scaled positions. Horizon decoding is fixed patch-32 in every arm; only the CONTEXT tokenizer varies:

Each subfolder is one (arm, seed) run: model.pt (final), ckpt_15000.pt (rank-stability snapshot), config.json, results.json (dev-GIFT CRPS + long-horizon probe). Dev metrics use a fixed 14-task GIFT-Eval subset — NOT the full leaderboard; treat numbers as ablation-internal, not comparable to published GIFT scores. Generated by the v10 experiment notebook.

Results

=== transfer curve: GM-CRPS by checkpoint (dev-14, mean over seeds) ===
                     gm_avg2  gm_avg2_std  gm@7500  gm@15000  gm@22500  gm@final  gm_final_std  long_season  params_m       perp  probe_amp  n
arm                                                                                                                                           
R3_k4096_nat          0.1795          NaN   0.2066    0.1889    0.1840    0.1750           NaN       0.2512    4.2743  1597.8119     0.2750  1
R4_softce_nat         0.1817       0.0048   0.2018    0.1960    0.1834    0.1800        0.0099       0.2508    3.8780   558.6021     0.2380  2
N3_vqce_nobonus_nat   0.1835       0.0021   0.1987    0.1881    0.1883    0.1787        0.0034       0.2485    3.8780   555.6374     0.1750  3
R1_w01_nat            0.1836          NaN   0.1908    0.1854    0.1862    0.1809           NaN       0.2538    3.8780   570.1680     0.2180  1
R2_k2048_nat          0.1864       0.0009   0.2180    0.1911    0.1898    0.1829        0.0035       0.2543    4.0101   970.0609     0.2610  3
R5_horizonce_nat      0.1879       0.0015   0.1976    0.1888    0.1901    0.1858        0.0034       0.2527    4.0430   547.4886     0.1997  3
N1_vqce_nat           0.1909       0.0109   0.2121    0.2052    0.1983    0.1835        0.0049       0.2550    3.8780   620.2301     0.2280  3
N5_vqce_w05_nat       0.1959          NaN   0.2092    0.2005    0.1968    0.1949           NaN       0.2593    3.8780   598.1758     0.2410  1
R6_k2048_hz_nat       0.1991       0.0074   0.2010    0.1928    0.2061    0.1921        0.0045       0.2540    4.3072   949.2821     0.2677  3
N0_raw_nat            0.2037       0.0042   0.2011    0.1938    0.2014    0.2061        0.0048       0.2635    3.5802        NaN     0.2253  3
N2_vqce_multi_nat     0.2039       0.0189   0.1913    0.1943    0.2154    0.1925        0.0034       0.2598    4.2743   597.4680     0.2235  2
P0_fullzone_nat       0.2131          NaN      NaN       NaN       NaN    0.2131           NaN       0.2832    3.5802        NaN        NaN  1
N4_rawce_nat          0.2404       0.0030   0.2043    0.2293    0.2371    0.2437        0.0028       0.2832    3.7452   461.1100     0.3755  2

--- tripwire checks ---
  R3_k4096_nat vs R4_softce_nat: final gap=0.0050 -> UNRESOLVED (< 2*sigma)
  R4_softce_nat vs N3_vqce_nobonus_nat: final gap=-0.0013 -> UNRESOLVED (< 2*sigma)
  N3_vqce_nobonus_nat vs R1_w01_nat: final gap=0.0022 -> UNRESOLVED (< 2*sigma)
  R1_w01_nat vs R2_k2048_nat: final gap=0.0020 -> UNRESOLVED (< 2*sigma)
  R2_k2048_nat vs R5_horizonce_nat: final gap=0.0028 -> UNRESOLVED (< 2*sigma)
  R5_horizonce_nat vs N1_vqce_nat: final gap=-0.0022 -> UNRESOLVED (< 2*sigma)
  N1_vqce_nat vs N5_vqce_w05_nat: final gap=0.0114 -> UNRESOLVED (< 2*sigma)
  N5_vqce_w05_nat vs R6_k2048_hz_nat: final gap=-0.0029 -> UNRESOLVED (< 2*sigma)
  R6_k2048_hz_nat vs N0_raw_nat: final gap=0.0140 -> UNRESOLVED (< 2*sigma)
  N0_raw_nat vs N2_vqce_multi_nat: final gap=-0.0136 -> UNRESOLVED (< 2*sigma)
  N2_vqce_multi_nat vs P0_fullzone_nat: final gap=0.0207 -> RESOLVED
  P0_fullzone_nat vs N4_rawce_nat: final gap=0.0306 -> RESOLVED
  N0_raw_nat: TRANSFER INVERSION — best at ckpt #2 (0.1938) vs final (0.2061); synthetic overfit persists
  N2_vqce_multi_nat: TRANSFER INVERSION — best at ckpt #1 (0.1913) vs final (0.1925); synthetic overfit persists
  N4_rawce_nat: TRANSFER INVERSION — best at ckpt #1 (0.2043) vs final (0.2437); synthetic overfit persists