keak-nova-GGUF

Nova v4 — results and recommendation (2026-08-23)

The question this run answers

NOVA-V2-RESULTS.md documented Nova v2's guard collapse (0.629 base -> 0.303 final) and, at the time, diagnosed it as fine-tuning generically eroding safety alignment. That diagnosis was later found to have the right direction but the wrong mechanism: mlx_vlm.lora's --steps N silently truncates the dataset from the front with no shuffling, and v2's training window never contained a single guard row (they were suite-grouped near the end of the file). Full details in train/mlx_vlm_patches/README.md, "THE BIG ONE" section.

The fix: shuffle the training file once (train/data/split-v9-shuffled/) and use --steps 0, which this codebase special-cases to a true full pass (len(dataset) // batch_size). This run is the test of whether that fix actually resolves the regression, versus needing to also oversample guard rows (the fallback split-v9-guardboost/ variant, built but unused).

Result: it does. Decisively.

Full keakbench.run, base model as control (out/nova-v4-bench/base.json) vs the epoch_0 checkpoint (out/nova-v4-bench/epoch_0_nova-v4.json), both against the live mlxvlm: provider with the base-vs-adapter control discipline run-nova.sh requires:

Guard's 2 remaining failures (2/66) are both capability (answered, wrong) — zero SAFETY (complied when it should refuse) failures. This is the number that matters: training on the shuffled full-pass data improved safety behaviour rather than eroding it, the opposite of what v2 showed. That confirms the truncation bug, not fine-tuning itself, was the cause of v2's collapse.

The four training attempts, briefly (full detail in the patches README)

1. Unclipped, 2 epochs in one process — nan at epoch 1 step 322, never recovered.
2. Gradient clipping added (clip_gradients=1.0) — delayed to step 794, still died. mx.clip cannot rescue an already-nan value; the real trigger is a forward-pass overflow, not a large but finite gradient.
3. Skip-on-nan guard added, but train_epoch was @mx.compile'd — the guard's Python branch was traced once on clean early batches and the "apply the update" path got baked in permanently; the guard's own warnings kept printing (a side effect outside the traced graph) while the compiled graph ignored the check entirely. Every batch from ~step 600 onward corrupted the model regardless of the "skip" logging.
4. @mx.compile removed from train_epoch — the guard genuinely works now (2302 real skips logged), but nan onset in the resumed second epoch still occurs (this time ~step 560-600), just later than before and non-destructively. Epoch 0 alone, from a zero-initialized adapter, has completed cleanly in all four attempts. Only the second-epoch continuation (resuming an already-adapted checkpoint) has ever failed — the failure step varies run to run (322, 794, ~560-600) despite identical starting checkpoint and identical data order, which is consistent with LoRA dropout's randomness interacting with a structural instability specific to continuing training on already-adapted weights, not a single bad training example (checked: no anomalies in the data around any of the three failure windows).

Recommendation: ship epoch_0_nova-v4.safetensors as-is

• It already beats the untrained base on every one of 8 suites, 4 of them hitting a perfect 1.000.
• Guard improved by +0.341 with zero safety failures, resolving the exact problem this whole investigation started from.
• The one partial second-epoch checkpoint that did survive (nova-v4-safe2.safetensors, ~560 additional good steps before the skip-guard correctly froze it) scored lower on guard (0.886) than epoch_0 alone — a second epoch is not obviously worth the risk here, at least not without further work (lower LR for the continuation, or genuine per-epoch reshuffling, are the next levers if a second epoch is revisited later).
• Do not chase a fifth training attempt for marginal gains against a checkpoint this strong. If more capability is wanted later, the more promising path is a fresh v5 on an improved/larger dataset, not further passes over the same one.