readout-guidance-audio-tfg

Readout Guidance on TFG's audio benchmark (declipping / inpainting)

Readout head, diagnostics and full sweep for training-free guidance of audio declipping and audio inpainting, steering the frozen teticio/audio-diffusion-256 mel-spectrogram diffusion model under the TFG (Ye et al. 2024) protocol.

This is a negative / boundary result, and is published as such.

Summary

A y-agnostic multi-scale head (929,795 params) reads frozen UNet activations and predicts E[x₀ | xₜ]; guidance is -‖mask ⊙ x̂₀ − y‖₂. Both tasks are masked linear inverse problems.

Guidance works (FAD 2.43 → 0.25), but the readout head does not beat guiding with the denoiser's own x̂₀:

(N = 256, guidance scale 4, η = 1, 100 DDIM steps.)

Why

For a linear inverse problem the guidance term depends only on x̂₀, which the denoiser already computes for free. Measured on held-out mels (x0_compare.json, MSE against true x₀):

The denoiser wins below t ≈ 850; the head wins above it, by 77% at t = 990, because the ε-parameterized x̂₀ divides by a vanishing √ᾱ while the head regresses x₀ directly.

Restricting the head to that high-noise band (--head_t_lo 800 --head_t_hi 900) recovers most of the gap on declipping — recover −10.51 → −8.19, FAD 0.438 → 0.283 at N = 128, reaching parity with eps — but hurts on inpainting, where more of the signal is pinned by the measurement.

Conclusion: readout guidance pays off when the sampling-time target is not the posterior mean (a class label, a molecular property, a style). On a linear inverse problem the required quantity is the posterior mean, so the head competes with its own backbone and at best ties.

Contents

head/            trained x_0 readout head (929,795 params, val MSE 0.0336)
runs/            N=256 runs: config + metrics (none / eps / readout, both tasks)
sweep/           N=128 sweep: guidance scale, t-schedule, iter_steps, x0_scale
x0_compare.json  per-t MSE of head vs denoiser x_0 -- the diagnostic above
*_table.txt      rendered result tables
code/            training, sampling, evaluation and diagnostic scripts
RESULTS.md       full write-up, including TFG-repo bugs found

Requires the TFG codebase on PYTHONPATH. Three fixes are needed before their audio tasks run at all (scipy private-symbol import in torch_sqrtm.py, a best-of-N block in pipeline.py that only type-checks for images, and a mask mismatch in audio_inpainting.py) — all documented in RESULTS.md.