acestep-vocals-lora

ACE-Step 1.5 — vocals-only (a cappella) LoRA

A LoRA for ACE-Step 1.5 acestep-v15-base (2B) that makes the model sing unaccompanied when nothing in the prompt told it to. The un-adapted base model, given a prompt that describes only a voice, still builds a backing track. This adapter removes most of it without touching diction.

---

⚠️ Read this before you generate anything

ACE-Step's dcw_enabled parameter defaults to True, and that default destroys the CFG checkpoints — including the one this LoRA attaches to.

With the default left alone, a 30-second clip comes back transcribing as the single word "so". Measured against a Whisper transcript:

Pass dcw_enabled=False in every generation call. infer.py in this repo forces it. If you wire the adapter into your own pipeline and get gibberish, this is why — the adapter is not the problem.

---

Usage

infer.py in this repo runs as-is and pulls the adapter from the Hub on first use.

pip install acestep                # or: git clone ACE-Step-1.5 && pip install -e .
wget https://huggingface.co/TTS-AGI/acestep-vocals-lora/resolve/main/infer.py

python infer.py \
  --caption "A solo female voice sings an Andean huayno, high and keening, with a bright nasal placement, ornamented turns and a piercing sustained upper register." \
  --lyrics "[verse]. The quarry filled with rain the year I left. I watched it take the ladder and the crane." \
  --duration 30 --seed 770001 --out out.flac

Useful flags: --adapter DIR (local copy instead of the Hub), --scale 0.5 (weaker), and --no-lora (the un-adapted baseline, for A/B).

Settings — exactly what the numbers on this page were produced with

50 steps at guidance 7.0 is the CFG-model pairing. The 8-step / guidance-1.0 pairing you will see in ACE-Step examples belongs to the turbo checkpoints and produces mush on base.

Attaching it yourself

from acestep.handler import AceStepHandler
dit = AceStepHandler()
dit.initialize_service(project_root=".", config_path="acestep-v15-base", device="cuda")
dit.load_lora("path/to/adapter")     # directory basename must contain NO dots (see below)
dit.set_use_lora(True)
dit.set_lora_scale(1.0)
assert dit.get_lora_status()["active"]

load_lora() uses the adapter directory basename as the PEFT adapter name, and PEFT rejects names containing dots. Stage the adapter into a dot-free directory first — infer.py does this for you. This is what makes raw training checkpoints (epoch_2_loss_0.7641) unloadable as written.

get_lora_status() returns {'loaded', 'active', 'scale'} — not lora_loaded / use_lora. Asserting the wrong keys silently disables the check, not the adapter.

---

Files

Each variant is bound to the checkpoint it was trained on — C and D will not do anything sensible on acestep-v15-base. Variant E (XL/4B base) is absent: it produced NaNs at step 70, inside warmup, and no usable weights. That says XL needs a lower LR, not that XL cannot work.

---

Training

Data — 1,314 clips, 60 genres

Selected from a pool of 8,582 captioned singing clips by a deliberately strict filter:

The last cut is the large one and it is not arbitrary: only one of the source runs carries lyrics, and training a vocal adapter on empty lyric conditioning teaches wordless singing, so those 659 rows were dropped rather than padded. The result is 1,314 clips over 60 genre labels (min 3 / median 20 / max 48 per genre), 25–30 s each, 48 kHz stereo, with [verse]/[chorus] markers and long prose captions describing the voice — register, vibrato, attack, mic distance — rather than tag lists.

genre_ratio = 0: captions are used in full. The Side-Step docs suggest genre_ratio 90 (swap the caption for a bare genre tag on 90% of samples) for style LoRAs, but the prompts this adapter is meant to fire on are ~300-character prose. Variant H tests that choice and does not overturn it. No trigger word — the adapter should improve vocals-only rendering unconditionally.

Synthetic-source caveat. The training audio is generated, not recorded: all 1,314 clips are LeVo 2 output, captioned by an audio LLM. This adapter distils LeVo 2's a cappella behaviour into ACE-Step. It learns the vocal character of a synthetic singer and inherits that generator's artefacts and biases. It is not trained on, and does not clone, any real performer.

Hyperparameters (variant A)

python train.py --plain -y fixed \
    --checkpoint-dir <ckpts> --model-variant acestep-v15-base \
    --dataset-dir <preprocessed-tensors> --output-dir <out> \
    --batch-size 1 --gradient-accumulation 4 \
    --optimizer-type adamw --scheduler-type cosine --warmup-steps 100 \
    --weight-decay 0.01 --max-grad-norm 1.0 --seed 42 --save-every 2 \
    --rank 128 --alpha 256 --lr 1e-4 --epochs 12 --dropout 0.1 \
    --cfg-ratio 0.15 --adapter-type lora --attention-type both

All eight variants were trained as independent single-GPU processes, four to a node — Lightning Fabric was not installed and the trainer degrades to a single-device loop with only a warning, so four different variants per node was a better use of whole-node billing than one faster run. Total 13.93 GPU-hours for all eight.

What was held out

No training clips were held out. All 1,314 went into training. Evaluation is instead on 40 prompts written specifically for it — 20 style captions × 2 lyric takes — and those captions and lyrics appear nowhere in the training corpus (verified: 0 of 20 captions and 0 of 2 lyric texts occur in the training set). So the evaluation measures generalisation to unseen prompts, not held-out audio, and it cannot detect memorisation of training clips.

---

Evaluation

Metric. accomp_frac = RMS(instrumental stem) / (RMS(instrumental) + RMS(vocal)) after BS-Roformer separation (ep.317, SDR 12.98). 0 = pure a cappella, 0.5 = instruments as loud as the voice. Separation artefacts are real but identical across both arms of a pair, so the trustworthy quantity is the within-pair difference, not the absolute level.

Design. 40 prompts (n = 40), each generated twice with identical caption, lyrics and seed in both arms, so every comparison is paired. p-values are two-sided exact sign tests over the 40 paired differences; CIs are 10k-resample bootstraps of the mean difference.

These prompts describe only the voice and never mention instruments. That condition matters: on the other grid, whose captions all say "no instruments are present", the un-adapted model already scores 0.02 and there is no headroom for any adapter to show anything (22/40, p = 0.64). Building the neutral grid was the difference between a null result and a measured one.

Variant A (recommended)

At scale 1.0 that is a 55% relative reduction in accompaniment energy. The effect grows monotonically with scale over the range tested. Lyric word recall (Parakeet ASR) is flat — 0.861 → 0.859 — with 0 collapsed clips out of 160, so the a cappella behaviour is not bought by degrading diction.

All variants

Each is paired against the un-adapted arm of its own base checkpoint, n = 40:

Why A at 1.0 is the shipped choice: it has the largest paired effect of any arm on acestep-v15-base (−0.104, roughly double every scale-0.5 arm) at p = 0.017. The honest qualification is that only A was evaluated at scale 1.0 — H is the most consistent variant at 0.5 (30/40, p = 0.002) with a smaller effect, and it was never tested at 1.0, so "A is best" is a statement about the arms that were actually measured, not a proof that A dominates H.

Secondary findings: rank 128 buys nothing over 64; LR 5e-5 is the weakest setting; genre_ratio 90 did not hurt, contradicting the prediction made when the dataset was built; and un-prompted a cappella compliance runs base < SFT < turbo, so the choice of base checkpoint matters more than the adapter does.

---

Limitations

• Style flattening at scale 1.0 is unmeasured. It is the documented failure mode of timbre LoRAs at high merge scale, and an energy-ratio metric is structurally blind to it. This is the main open risk on the "ship at 1.0" recommendation — judge it by ear on the demo page, which carries scale 1.0 next to the un-adapted baseline.
• The evaluation uses 2 distinct lyric texts across its 40 clips (20 captions × 2 takes). Prompt diversity is real; lyric diversity is not.
• No human listening study. Every number here is automatic (source separation + ASR).
• 2B only. Transfer to the XL/4B checkpoints is untested; the one XL run NaN'd in warmup.
• Nothing else improved. The adapter does not improve pitch, musicality or diction — it removes accompaniment nobody asked for, and that is the whole claim.
• Prompts that already say "no instruments are present" get no measurable benefit: the base model already complies there.

Provenance

Trained and evaluated on JUWELS/JUPITER GH200 nodes. Training audio is LeVo 2 output (see the synthetic-source caveat above). Apache-2.0, following the base model.