LTX-2.3-22b-IC-LoRA-Audio-Only-Context

LTX-2.3-22B IC-LoRA: Audio-Only In-Context Context (experimental, proof of concept)

Last updated: 2026-06-04

An IC-LoRA for LTX-2.3-22B (distilled) where the in-context reference is only audio: no image, no video, no init frame. The model still generates audio and video jointly; the only thing you hand it as a reference is an audio file. We trained it on natural talking-head clips where the reference is a 3.5 s window from a different clip of the same speaker — in practice a different window of the same recording session, so the pair shares session cues (room tone, mic, scene) beyond the voice; see data_recipe.md. In practice the audio reference transfers the vibe of the audio: the generated speech, the on-screen energy and mannerisms, and potentially identity cues riding along in the voice. It is the successor to the pitch "Helium" proof-of-concept, this time on natural (un-perturbed) data and with the cross-modal bridge added.

Status: early. The transfer is clearly visible in single takes, but it has not been through a controlled, quantitative eval, and the narrow trained voice-conditioned mapping is not yet cleanly separated from broad emergent audio-to-AV transfer. Treat the examples as illustrative single takes, not benchmarks. This needs more evals. If anyone wants to test and see what interesting things come, please do so and share! The point of releasing it is the process writeup below, so others can reproduce, adapt it to a different task, or continue where we left off.

Model files (two checkpoints, same recipe, different cut)

Both are 1000-step adapters trained identically except for which modules they adapt:

• cross_modal_step_01000.safetensors (~290 MB, 1728 bf16 tensors): adapts the audio stack and the cross-modal bridges (audio_to_video_attn, video_to_audio_attn). The bridge gives the audio reference a direct path into the video stream, so its effect on the video (mannerisms, energy, expression timing) is stronger. The default pick if you want the audio to move the picture.
• audio_only_step_01000.safetensors (~164 MB, 960 tensors): adapts the audio stack only (audio_attn1/2, audio_ff), no cross-modal. The reference shapes the generated audio; the video follows via the frozen base coupling, so the video effect is subtler / more emergent and the base video path is perturbed the least.

Each seems to have its own strengths and weaknesses; we have not pinned down which is better for which use (that is one of the open questions below). Try both.

• train_config.yaml: the training config (cross-modal; the audio-only variant just drops the two bridge modules). data_recipe.md: how the dataset was built.
• Training fork: fblissjr/LTX-2 @ audio-guidance-iclora-vtv (the audio_reference strategy).

What it is

• Base: LTX-2.3-22B distilled (ltx-2.3-22b-distilled-1.1), a joint audio-video model (this matters for the caveats). It is guidance-distilled, so it runs at CFG = 1 (no classifier-free guidance; strength is the only inference knob).
• Control: one in-context audio reference, appended clean at negative RoPE positions as out-of-timeline context (the ID-LoRA convention). No image, no video, no init frame. The model generates both modalities; only the reference is audio-only.
• Format: plain LoRA safetensors, PEFT convention (lora_A/lora_B under diffusion_model., no alpha keys), bf16. ComfyUI's LTX IC-LoRA loader reads it directly, no conversion.

How this differs from other IC-LoRAs (and from ID-LoRA)

Most IC-LoRAs for LTX-2.3 (Union Control canny/pose/depth, Motion-Track, video-reference adapters) hand the model a visual control stream and steer structure or motion. Here the in-context reference is only audio — no image, no video, no init frame. What transfers is the vibe of the audio: the generated speech, the energy and mannerisms on screen, and whatever cues ride along in the voice.

The closest prior work is ID-LoRA (cited below) — we built directly on it, trained on the same talking-head dataset, and inherited its negative-RoPE audio-reference convention and its module targets. The differences are deliberate ablations:

• No image anchor. ID-LoRA trains with first-frame conditioning on ~90% of steps (first_frame_conditioning_p: 0.9) and takes a --first-frame image at inference — the image carries the subject, the audio drives the performance. We set first-frame conditioning to 0.0: the audio reference is the only non-caption input, so anything that shows up in the picture had to come through the audio.
• Distilled base. We train on ltx-2.3-22b-distilled-1.1, so the adapter drops straight into the standard 8-step CFG=1 ComfyUI path. ID-LoRA's released checkpoints train on the dev (non-distilled) base.
• Two cuts to isolate the bridge. ID-LoRA adapts the audio stack + both cross-modal bridges as one checkpoint. We ship that cut and an audio-only cut (no bridges), so you can separate "shape the generated audio" from "couple the audio into the video."
• Consumer-scale recipe. Rank 32 / 1000 steps / batch 1 on a single 24 GB card, vs ID-LoRA's rank 128 / 6000 steps / batch 4.

Using in ComfyUI

1. Copy the LoRA into models/loras.
2. Install the custom nodes from ComfyUI-AudioLoopHelper (recommended). They add a key-match trust gate (a mis-bound LoRA fails loud instead of silently doing nothing), a Compose Reference Audio loader (visual waveform editor for picking one or more slices of the reference clip, with an Auto-find hook button that drops a slice on the loudest sustained window — no graph queue needed), a per-stream loader (separate strength for the cross-modal bridge vs the audio modules), and an Advanced guide (reference window + scale). They are not strictly required — the core path also runs on stock nodes (see "Using without the custom nodes" below).
3. Load the LoRA, load + window your reference with the Compose Reference Audio node (or feed raw audio straight into the guide), keep the caption neutral, and generate audio and video together from an empty latent (text-to-video; no image or video input). Example workflow: audio-ic-lora_single-pass.json.

Usage notes

• Strength: start ~0.5. Both checkpoints garble at high strength; the working band is roughly 0.3 to 0.75 for most references, but the ceiling depends on the reference and the generation (some of the examples below run at 0.9). If it breaks up, back off.
• With the per-stream loader you can push bridge_strength above audio_strength to amplify the audio-to-video coupling specifically.
• Don't double-trim. If you window the reference with the Compose Reference Audio node, set the Advanced guide's reference_window_sec = 0 so the two don't trim twice. Keep the total composed duration to a few seconds (~3.5 s matches training).
• Caption: keep it neutral — don't describe the voice, mood, or style in text, so the audio reference (not the caption) is what drives them.
• The reference audio matters a lot, in ways we do not fully understand yet — level/loudness, quality, and content all seem to change the result. Expect to experiment.
• Audio VAE precision: nothing to set in ComfyUI. ComfyUI runs the LTX audio VAE in fp32 automatically (it forces float32 for this VAE; the weights ship bf16 and get upcast), which matches the training encode — so the bf16 filename/loader widget doesn't change this. The video VAE runs bf16, also what training used. (You'd only need to handle fp32 yourself if you encode the reference outside ComfyUI, since the weights ship bf16.)

Using without the custom nodes

You do not strictly need this repo — our loader and guide are instrumented twins of stock nodes, so the core path runs on stock ComfyUI-LTXVideo + core ComfyUI. This stock path is untested for this model and may not behave the same way as the custom nodes. The attach and the LoRA load are the same code underneath, so it should match — but if it appears to do nothing (no change to the generated audio), the usual cause is that the native loader silently no-op'd a LoRA that bound nothing and gave no warning. So if you go this route: make sure the LoRA is loaded onto LTX-2.3-22B distilled (the exact base it was trained on) and confirm it actually has an effect. Our loader fails loud when nothing binds; the native one does not. With that said:

1. Load the LoRA with LTX IC-LoRA Loader (Model Only) (LTXICLoRALoaderModelOnly), strength ~0.5–0.9. One strength for all modules — you lose the cross-modal bridge's separate strength (the only thing the per-stream loader adds).
2. Load the audio VAE (LTXV Audio VAE Loader) and encode the reference (LTXV Audio VAE Encode). Trim your clip to ~3.5 s first — stock has no windowing.
3. Attach it with LTXV Set Audio Ref Tokens (LTXVSetAudioRefTokens) onto the positive + negative conditioning. This is byte-identical to our guide's attach; the model applies the negative-RoPE offset either way.
4. Generate audio + video from empty latents (Empty LTXV Latent Video + LTXV Empty Latent Audio) with a neutral caption, CFG = 1, and the usual LTX-2 distilled sampler.

Two things to watch for parity: (a) the reference encode should run in fp32 — ComfyUI does this automatically for the LTX audio VAE (it forces float32; the bf16 weights are upcast), so there is nothing to set in-graph. You only need to ensure fp32 yourself if you encode the reference outside ComfyUI. (b) keep the reference ~3.5 s (the training length). What you give up vs the custom nodes: the trust gate (stock silently no-ops on the wrong base, so confirm the LoRA actually has an effect), the per-stream bridge strength, and the reference windowing (the Compose editor + the Advanced guide's window).

Additional Examples

Same reference dialogue audio, different mediums / text prompts:

Animated cartoon <video controls width="480"><source src="https://huggingface.co/fbjr/LTX-2.3-22b-IC-LoRA-Audio-Only-Context/resolve/main/assets/cartoon_example.mp4" type="video/mp4"></video>

Sitcom <video controls width="480"><source src="https://huggingface.co/fbjr/LTX-2.3-22b-IC-LoRA-Audio-Only-Context/resolve/main/assets/sitcom_example.mp4" type="video/mp4"></video>

Music/song passed in as reference with the same text dialogue pattern:

<video controls width="480"><source src="https://huggingface.co/fbjr/LTX-2.3-22b-IC-LoRA-Audio-Only-Context/resolve/main/assets/music_example.mp4" type="video/mp4"></video>

Examples

Six single takes — three references (a dialogue clip, a music clip, and a spoken-word clip), each fed into both checkpoints. For the first two the seed is fixed across the pair, so the checkpoint (and its strength) is the only thing that changes — a like-for-like look; the spoken-word pair uses different seeds. Audio-only input (no image or video; the video is generated from noise). Each clip carries its generated audio (use the player controls). The references are kept generic on purpose.

T2V prompt (all six): a fixed restaurant-dialogue scene beginning In a medium shot at a dim restaurant table, an animated man leans in, grinning, and says ....

Reference 1 — a dialogue clip (warning: profanity).

Audio-only checkpoint (audio_only_step_01000), audio strength 0.9:

<video controls width="480"><source src="https://huggingface.co/fbjr/LTX-2.3-22b-IC-LoRA-Audio-Only-Context/resolve/main/assets/example_dialogue_audio_only.mp4" type="video/mp4"></video>

Cross-modal checkpoint (cross_modal_step_01000), audio strength 0.9, bridge strength 0.5 (per-stream loader):

<video controls width="480"><source src="https://huggingface.co/fbjr/LTX-2.3-22b-IC-LoRA-Audio-Only-Context/resolve/main/assets/example_dialogue_cross_modal.mp4" type="video/mp4"></video>

Reference 2 — a music clip (warning: profanity).

Audio-only checkpoint (audio_only_step_01000), audio strength 0.75:

<video controls width="480"><source src="https://huggingface.co/fbjr/LTX-2.3-22b-IC-LoRA-Audio-Only-Context/resolve/main/assets/example_music_audio_only.mp4" type="video/mp4"></video>

Cross-modal checkpoint (cross_modal_step_01000), strength 0.75:

<video controls width="480"><source src="https://huggingface.co/fbjr/LTX-2.3-22b-IC-LoRA-Audio-Only-Context/resolve/main/assets/example_music_cross_modal.mp4" type="video/mp4"></video>

Reference 3 — a spoken-word clip (warning: profanity).

Audio-only checkpoint (audio_only_step_01000), audio strength 0.6:

<video controls width="480"><source src="https://huggingface.co/fbjr/LTX-2.3-22b-IC-LoRA-Audio-Only-Context/resolve/main/assets/example_spokenword_audio_only.mp4" type="video/mp4"></video>

Cross-modal checkpoint (cross_modal_step_01000), audio strength 0.6, bridge strength 0.9 (per-stream loader):

<video controls width="480"><source src="https://huggingface.co/fbjr/LTX-2.3-22b-IC-LoRA-Audio-Only-Context/resolve/main/assets/example_spokenword_cross_modal.mp4" type="video/mp4"></video>

Training

Throughput is about 11.5 s/step on a 4090; batch_size > 1 OOMs at 512x512 (block-swap is already near its floor), so single-sample steps are the practical ceiling at this resolution.

Dataset (brief)

Natural talking-head clips with audio, un-perturbed (unlike the pitch model). For each target clip the reference is a 3.5 s voice window from a different clip of the same speaker, so the reference and target share the speaker but not the content. The caption is a constant neutral string, so whatever the voice carries has to come from the audio, not the text. Split is held out by identity. Full recipe: data_recipe.md.

What we learned, what we are still unsure of, and how to take it further

What seems to work

• An audio-only in-context reference does steer a distilled, joint audio-video model: at default strength, with no special knobs, the reference moves the generated speech and the speaker's mannerisms/energy. Confirmed only as single takes so far.
• The cross-modal bridge is the path the audio reference uses to reach the video. Adapting it (the cross-modal checkpoint) couples the audio's influence into the video more than the audio-only cut, which leaves voice-to-video to the frozen base.
• The base generates audio and video jointly, so the video tends to follow the generated audio. The reference's clearest, most reliable effect is on the audio; the video effect rides on that coupling.

What we are still unsure of

• Whether the narrow trained mapping (a voice steering the look that co-occurred with it in training) is doing anything beyond the broad emergent audio-to-AV transfer you see when you feed it out-of-distribution audio (songs, movie clips). The examples are the emergent kind; a controlled swap eval (fixed prompt + seed, swap only the reference, measure what carries over against a seed-swap noise floor) is still to do.
• The dependence on the reference audio itself: level/loudness, quality, and content all seem to matter, and we cannot yet say how. This deserves a systematic sweep.
• Which checkpoint (audio-only vs cross-modal) is better for which use, and the exact strength band before garbling, both of which depend on the audio.

A measurement honesty note (useful if you build observability): during training we watch a held-out loss plus a "reference-attribution gap" (loss with the correct reference minus loss with a wrong one). For a task like this one — where the target video already shows the thing the audio is supposed to control — that gap reads ~neutral, and that is expected, not a failure: reconstruction does not need the reference when the target already shows the answer. That is exactly why the real test is generation from noise (swap only the reference and watch the output), not a reconstruction-loss metric.

Where we think it could be better next time

• The target video gives the answer away during training (the speaker is right there on screen), so the model has little pressure to use the reference. The most promising next lever is to mask the target face region and/or bias training toward high noise levels, forcing the model to rely on the reference. (We did not do this yet.)
• More speaker diversity, and a quantitative swap eval: fixed prompt and seed, swap only the reference, measure the change against a seed-swap noise floor — using reference audio with no public footprint, so a positive result cannot be explained by base-model recall.
• A real reference-strength dial would need a small model-side change (the audio reference is read with no scalar applied); a timestep-range gate (apply the reference only over the noise band where it bites) works through stock ComfyUI nodes and is the cheapest thing to try.

How to pick it up (the easy button)

• Reproduce / continue: use the fork above (audio_reference strategy), precompute latents + the reference windows, and run train.py with train_config.yaml. The two checkpoints differ only by the target-module list.
• Adapt to a different task: the recipe is general. Pick the one thing you want the audio to control, then build pairs where the reference and target share only that thing and differ in everything else, with a neutral caption. Voice/accent/emotion/genre/era/scene are all candidates. Same strategy, new data.

Citation

The trick of placing the audio reference at negative RoPE positions comes from Lightricks' LipDub and the ID-LoRA work below. Our read of how the conditioning behaves is from inspecting it in ComfyUI and the model code, not a formal ablation.

@misc{dahan2026idlora,
  title  = {ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRA},
  author = {Dahan, Aviad and Yanuka, Moran and Kraicer, Noa and Wolf, Lior and Giryes, Raja},
  year   = {2026},
  eprint = {2603.10256},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD}
}

Acknowledgments

Thanks to WepeNerd (HF) for the idea of an audio-only IC-LoRA and for thinking through how an IC-LoRA learns; to the LTX-2 community trainer, Musubi Tuner, and Kijai's work for the infrastructure; and to Throttlekitty for pulling me into LTX-2.

License

Base model is by Lightricks under the LTX-2 community license. See https://github.com/Lightricks/LTX-2/blob/main/LICENSE for the full terms.