Downloads · 30 days
13
37% of all-time downloads
jatshi/LLM-Guided-Speech-Enhancement-v4-SFT
LLM-Guided-Speech-Enhancement-v4-SFT is a audio-text-to-text model from jatshi. Use it for the audio-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as mit.
This release contains the evidence-selected SFT LoRA adapter and audio projector from the 4.0 native-audio training run. It pairs a frozen openai/whisper-small encoder with Qwen/Qwen2.5-1.5B-Instruct: Whisper hidden s…
Downloads · 30 days
13
37% of all-time downloads
All-time downloads
35
Public
Repo size
92.5 MB
Likes
0
Public
Click a slice to open those files.
.pt44.1 MB · 45%
From the Hugging Face model README
This release contains the evidence-selected SFT LoRA adapter and audio projector from the 4.0
native-audio training run. It pairs a frozen openai/whisper-small encoder with
Qwen/Qwen2.5-1.5B-Instruct: Whisper hidden states are compressed into 16 continuous prefix tokens,
and the language model emits a validated JSON DSP prescription.
The language model does not directly synthesize a clean waveform. A deterministic DSP executor validates and applies its prescription, computes objective metrics against the available clean reference, and returns the original input whenever the candidate fails the safety gate.
Source and reproducibility code: Jatshi/llm-guided-speech-enhancement.
All three training stages completed and their artifacts remain auditable. The final checkpoint was not selected merely because it was last:
| Stage | Training status | Held-out structural result | Release decision |
|---|---|---|---|
| SFT | 1,500 steps completed | 1,416/1,416 valid JSON | selected |
| conservative DPO | 800 steps completed | canary output contains invalid numeric placeholders | rejected |
| GRPO | 300 steps completed | 0/1,416 valid JSON; mean reward 0 | rejected |
GRPO reported 1,200 saturated groups and zero mean reward. The 1,416 GRPO failures are preserved as negative evidence; this model card does not claim preference-optimization or RL improvement.
train-clean-100 through a documented ModelScope mirror.Training ran on one NVIDIA RTX 4080 SUPER with 32,760 MiB VRAM using QLoRA/NF4, BF16 compute,
gradient accumulation, and DeepSpeed ZeRO-2 at world_size=1.
| Stage | Steps | Initial loss | Final loss | Runtime |
|---|---|---|---|---|
| SFT | 1,500 | 2.314612 | 0.087327 | 3,017.39 s |
| DPO | 800 | 0.406933 | 0.325083 | 3,525.89 s |
| GRPO | 300 | 1.3355e-5 | 4.6299e-5 | 9,689.05 s |
Single-GPU ZeRO-2 is integration and optimizer-state evidence, not multi-GPU scaling evidence. A ZeRO-3 CPU-offload configuration is included for memory experiments but is not described as single- GPU parallelism.
The release evaluation covers all 1,416 test records. Cached Whisper embeddings were reused and generation was performed in deterministic batches of 16.
| Metric | Result |
|---|---|
| Valid JSON | 1,416/1,416 |
| Failed predictions | 0 |
| Inference wall time | 843.95 s |
| Throughput | 1.6778 samples/s |
| Amortized mean latency | 595.97 ms/sample |
| p50 / p95 latency | 586.20 / 667.53 ms/sample |
The executable DSP benchmark reports candidate deltas before rollback:
| Objective result | Mean delta |
|---|---|
| SI-SDR | +0.4509 dB |
| PESQ | +0.04068 |
| STOI | +0.002198 |
| Safety accept rate | 66.74% |
| Safety rollback rate | 33.26% |
All four SNR buckets (<5, 5–10, 10–20, and >=20 dB) have positive mean deltas for all three
available metrics. DNSMOS, WER, and speaker similarity are not reported because their backends were
not configured; missing metrics are not replaced with fabricated values.
This repository contains:
audio_projector.pt, which maps Whisper hidden states into the 16-token language prefix;It does not redistribute Qwen2.5-1.5B-Instruct or Whisper-small. Load those base models from their
original repositories and use lse_v2.native_inference.NativeAudioPlanner from the source repo.
diagnosis, actions, rationale, and confidence.The 33.26% rollback rate is an observed safety behavior, not a hidden failure count.
Use configs/native_audio_autodl_32gb.json and follow the
4.0 AutoDL runbook.
The release bundle includes machine-readable reports and SHA-256 checksums. Exact package versions
are recorded in environment.freeze.txt.
The source code is MIT licensed. Use of the adapter and reconstructed data remains subject to the Qwen, Whisper, LibriSpeech, and ESC-50 terms.