QaDiT-160

<p align="center"> <a href="https://github.com/sidharth72/QaDiT"> <img src="https://img.shields.io/badge/GitHub-Repository-181717?logo=github&logoColor=white" alt="GitHub"> </a> <a href="https://www.quarkml.com"> <img src="https://img.shields.io/badge/Website-QuarkML-blue?logo=googlechrome&logoColor=white" alt="Website"> </a> <a href="https://huggingface.co/QuarkML"> <img src="https://img.shields.io/badge/Hugging%20Face-Models-yellow?logo=huggingface&logoColor=black" alt="Hugging Face"> </a> </p>

QaDiT — text-to-audio latent Diffusion Transformer

A ~159M-parameter latent Diffusion Transformer that turns a text caption into 10.24 s of 16 kHz mono audio: FLAN-T5 conditioning → DiT denoising of AudioLDM KL-VAE latents → VAE decode → HiFi-GAN vocoder.

Example: Prompt: A small waterfall flows through a forest while insects buzz and birds sing.

Output:

<audio controls src="https://cdn-uploads.huggingface.co/production/uploads/64054e5e0ab5e22719fc179f/_RGLUAPxdImTitbAAD0g6.wav"></audio>

---

---

1. Big picture — three pipelines

---

2. Offline pre-compute (frozen models, run once)

Heavy frozen models run once; the training loop never loads T5, the VAE, or the REPA encoder.

---

3. Training step (what is actually optimized)

Only the DiT and its small glue layers receive gradients.

---

4. Inside one DiT block

---

5. Inference / sampling (caption → waveform)

---

6. Component ownership

---

Training objective (DDIM + v-prediction)

Forward process

$$ z_t = \sqrt{\bar{\alpha}_t},z_0

• \sqrt{1-\bar{\alpha}_t},\varepsilon $$

Network target

$$ v = \sqrt{\bar{\alpha}_t},\varepsilon

• \sqrt{1-\bar{\alpha}_t},z_0 $$

At sample time the DiT predicts v; we recover \hat{z}_0 and \hat{\varepsilon}, then step with DDIM (\eta = 0). CFG is applied in v-space with default scale (s = 4.0). After DDIM, latents are divided by latent_scale ≈ 0.95035 before VAE decode — that whole chain is what model.generate() runs.

---

Dataset

AudioCaps is captioned environmental / everyday sound — not speech or music. Those domains are out of distribution for this checkpoint.

---

Training run (this checkpoint)

---

Training curves

Put W&B / TensorBoard screenshots (or exports) under assets/ using the filenames below. Until then the images show as broken links on the Hub — that is intentional so the slots are obvious.

Diffusion / total loss

REPA loss

---

Usage

pip install transformers diffusers soundfile sentencepiece

import soundfile as sf
import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("QuarkML/QaDiT", trust_remote_code=True)
model = model.to("cuda" if torch.cuda.is_available() else "cpu").eval()

out = model.generate(
    "A small waterfall flows through a forest while insects buzz and birds sing.",
    num_inference_steps=200,
    guidance_scale=16.0,
    seed=0,
)
sf.write("sample.wav", out.audios[0], out.sampling_rate)

First generate downloads the frozen helpers this run was trained with: google/flan-t5-large and the VAE + vocoder from cvssp/audioldm-s-full-v2.

Output types

Precomputed T5 states

out = model.generate(
    encoder_hidden_states=text_emb,      # [B, 64, 1024]
    encoder_attention_mask=text_mask,    # [B, 64]
)

Single denoising step

v = model(latents, timesteps, encoder_hidden_states, encoder_attention_mask).sample

---

Precision and devices

Runs on CPU and CUDA. With dtype=torch.float16 or torch.bfloat16 the DiT backbone runs in half precision; DDIM schedule math stays in float32. Keep T5 / VAE / vocoder in float32 (FLAN-T5 overflows easily in fp16).

---

Important details

• config.latent_scale (0.9503493) must match training precompute. generate divides by it before VAE decode.
• Every sample is fixed length: 10.24 s @ 16 kHz.
• repa_layer exists for REPA fine-tuning; inference ignores it.
• Sampling always uses the EMA weights packaged here.

---

Limitations and intended use

Intended use: education, reproduction of a small latent DiT audio stack, ablations, and a starting checkpoint for longer / wider training.

Not intended for: production SFX libraries, speech synthesis, music generation, or safety-critical audio.

Known limits of this checkpoint

• ~24k steps on ~45k AudioCaps clips — undertrained vs public SOTA systems
• Weak on speech, music, and densely described scenes
• Inherits caption biases and coverage holes of AudioCaps
• Prefer the default 50 DDIM steps for demos; low step counts sound coarse

---

Research artifact — how to improve this

This release is a research artifact, not a production host model. The architecture and sampling path are solid enough to build on; the ceiling is mostly data and compute:

1. Train longer — continue past 24k steps with the same recipe (or lower LR).
2. Scale the dataset — mix in larger captioned audio corpora beyond AudioCaps.
3. Retune sampling — CFG scale, DDIM step count, and prompt formatting.
4. Keep measuring — log diffusion loss, REPA loss, and listening tests.

Those levers will move quality more than inventing a new backbone for this size of model. Contributions and longer runs are welcome; treat this Hub page as a reproducible baseline, not a finished product.

---

Citation

@misc{qadit2026,
  title        = {QaDiT: A Text-to-Audio Latent Diffusion Transformer},
  author       = {Sidharth GN},
  year         = {2026},
  note         = {Research artifact. Weights and transformers remote-code loading.}
}