joyai-echo-multishot-workflow

JoyAI-Echo GGUF nodes — multishot fixes + automation patch

Before you dive in - a word on expectations. This is a community-built, bleeding-edge pipeline: a 22B audio+video model with cross-shot memory, running on consumer hardware. It is powerful, but it is not one-click - your first clean render will likely take some tuning to YOUR machine (VRAM, system RAM and pagefile, and which model build: bf16 / fp8 / GGUF / INT8). No two rigs behave identically. INSTRUCTIONS.md covers install, per-VRAM settings, and every failure mode reported so far. **If you get stuck, open an issue

• I answer, I troubleshoot, and most of the fixes in this pack exist because someone reported something.** You will not be left hanging.

New in 2.2 — LoRAs finally work on a GGUF DiT

Download ComfyUI_JoyAI_Echo_v2.2_COMPLETE.zip. This one is built from the repo tree rather than hand-patched over the 2.0 archive, so the zip, the repo and this README agree for the first time since 2.0.

Runtime LoRA on the GGUF path (joyecho_gguf_runtime_lora.py)

Until now, every LoRA you selected was silently dropped whenever the DiT was a GGUF. The stacker showed three LoRAs, the console printed one warning, and the render came out LoRA-free. If you batch overnight on a GGUF, everything you rendered ran without them.

The reason was real, not a bug: JoyEcho's only LoRA mechanism was fusion (W' = W + s·B·A baked into the weight tensors at load), and a GGUF keeps its weights packed in quantized blocks that are dequantized per layer at compute time. Fusing would mean dequantize → add → requantize the whole model, and a rank-32 delta often falls below the quantization step anyway.

So this release stops trying to fuse and applies the LoRA at runtime with forward hooks instead:

y = module(x) + Σᵢ scaleᵢ · (x @ Aᵢᵀ) @ Bᵢᵀ

The GGUF weights are never touched. On a quantized base this is higher fidelity than fuse-and-requantize, because the delta stays bf16 instead of being crushed into the quant grid.

• Costs about 1–2 % of a hooked layer's FLOPs (two rank-32 matmuls against hidden 4096).
• Key handling is shared with the fused path — PEFT (lora_A/lora_B) and kohya (lora_down/lora_up) naming, alpha/rank scaling when .alpha keys are present, plain strength when they are not (which is the case for every LoRA I ship). The runtime path and the fused path can't disagree about what a LoRA file means.
• It reports what it did, per LoRA: 576/576 pairs attached (N audio-touching), and it warns loudly if zero pairs matched — the key-naming failure that used to be completely silent.
• It is never fatal. A LoRA that won't load, or an attach that throws, falls back to the old drop-behaviour with a warning; the render continues.

Verified two ways. Unit: the hook output matches an explicitly fused reference to 9.5e-07 max absolute difference. In situ: on a live Q8_0 GGUF, 576/576 pairs attached for each LoRA in the stack and the render differed from the no-LoRA control (a strength-0 attach — hook present, scale zero — reproduces the control exactly, which is what proves the hook math rather than just the skip branch).

The VRAM cost, measured

Hooks cache their A/B on the compute device on first call and keep them there, and the offloader never streams them — they are deliberately kept off state_dict and out of .to() sweeps, the same way GGUFLinear hides _qweight. So a LoRA stack is a permanent pin on top of your model.

Measured from the actual tensor headers, in bf16:

Three full-size LoRAs would be ≈1.06 GB. On a 32 GB card that is noise. On a 12 GB card running sequential_offload — which gets the DiT itself down to 2–3 GB — it is roughly a third of your headroom. Budget for it, run fewer LoRAs, or turn the feature off.

Kill switch

JOYECHO_GGUF_LORA=0

Set that environment variable and the loader restores the old behaviour exactly: LoRAs are dropped on the GGUF path, nothing is pinned, and it says so in the console. Use it if you are VRAM-starved, or to A/B a render against the pre-2.2 result.

JoyEcho Plate Picker (joyecho_plate_picker.py) — new node

Ground materials are one of the places the base model reliably disappoints: ask for grass and you tend to get a dry wiry mat. Feeding a real Z-Image surface plate in as reference_image fixes it — measured to beat both the stock baseline and a retrained surfaces LoRA on vegetation, replicating across 4 seeds out of 4. The problem was never the mechanism, it was remembering to do it, and picking the right plate.

The Plate Picker reads the ground/surface phrases only out of the prompt and auto-injects a matching plate from a folder of <id>.png + <id>.txt pairs. In the Studio workflow it is wired to JoyEcho_RefPicker.fallback_image, so it only fires when no character reference took priority.

• Matching is deliberately dumb and deterministic — token overlap between the prompt's surface words and the plate's material clause, with a bonus for exact material hits. No embeddings: a lookup you cannot predict is a lookup you cannot debug. The chosen plate is printed every run.
• Use vision-model re-captions, not generation prompts. Captions written from the prompt lie — Z-Image renders concrete or asphalt about a third of the time regardless of what was asked for, and three of four keyword picks did not show their claimed material at all. Captions written by a model looking at the image say what is actually there.
• Season and weather conflicts are penalised. A reference_image drags its whole look across, so a frost plate would put winter into a summer scene. Right material, wrong world is still wrong.
• plate_denylist.txt excludes plates containing people, limbs or cast human shadows — otherwise the picker will happily put a figure into a scene that asked for an empty one. Ships with the list from the reviewed sheets; it is explicitly incomplete, and the file is plain text so you can add to it.
• Below min_score it returns nothing and the render proceeds exactly as it would have without the node. A silent wrong plate is worse than no plate.

Also in this pack

• RiftCast_Studio.json now carries the Plate Picker, wired into RefPicker's fallback, and ships with empty LoRA defaults (see 2.1 — the trap is much less dangerous now that GGUF LoRAs actually apply, which is exactly why the defaults must stay empty).
• The Character Designer widget fix (web/js/riftcast_style_shadow.js) — described under 2.1 below. It reached the 2.1 zip as a late hand-patch; 2.2 is the first build where it is properly part of the package.

New in 2.1 — LoRA defaults cleared

Download ComfyUI_JoyAI_Echo_v2.1_COMPLETE.zip. Same pack as 2.0 with three corrections:

• The RiftCast Studio workflow shipped with three LoRAs enabled by default, two of which were the same LoRA at different versions — accent_american_v2 in the stacker and accent_american_v1 on the loader's own lora_file, both fusing at once, plus surfaces_v1. That was never intended. All defaults are now empty; pick LoRAs deliberately.
• LoRAs are silently ignored on a GGUF DiT — see section 10. Empty defaults make that trap much harder to fall into.
• Character Designer's duplicate widgets now hide themselves. hair_color, hair_style, wardrobe and distinguishing exist on both the Designer and the Style node; when a Style node is wired the Designer's copies are discarded, which used to happen invisibly. They now disappear on connect and return, with their values, on disconnect.

New in 2.0 — RiftCast

Characters are now portable files. A .riftcast cartridge carries a character's voice anchor, reference stills, canonical description, and optionally LoRAs, environments, and a CharacterCardV3 persona — drop it in input/riftcast/ and the character casts by speaker tag in any script, no training. Format spec: RIFTCAST_SPEC.md; riftcast.py packs, inspects, and cuts cartridges from finished renders; demo cartridges (WREN, HANK) ship in the repo.

The bundled RiftCast Studio workflow is a full character creator: a Character Designer node for identity and a Style + Wardrobe node for appearance (71 styles across 10 families, plus hair colour, hair shape, makeup, accessories, demeanor and a wardrobe override) → audition tape → automatically packed cartridge. A style never injects its own label; each entry expands into concrete renderable descriptors, with separate masculine and feminine wardrobe readings. The workflow also carries a source switch between designer and classic LPFF/JSON prompt files, and a Render Clock node that drives every fps/frames socket from one place with 8n+1 frame snapping.

Also in 2.0: video_fps defaults to 24 per the accent finding (Feature #22 below — off-24 fps drifts every voice toward British/Australian and overrides accent wording), widget values persist by NAME across node-layout changes, temporal_upscale 2x latent upsampling to ~48 fps masters, and a 1441-frame (60 s) single-take ceiling.

Quick fixes — read this first

Nearly every problem reported with this pack is one of these.

Also here: Multishot Lite (core ComfyUI + one KJNodes node)

A second, much simpler workflow for chained talking shots, with no node pack to install.

Its own repo: https://github.com/jlucasmcrell/ltx23-multishot-lite (standalone docs + issues) · HF: https://huggingface.co/joeygambino/ltx23-multishot-lite

Mirrored here: MULTISHOT_LITE.md · Releases.

• Core ComfyUI nodes plus one from ComfyUI-KJNodes (LTXVAudioVideoMask, which powers the extension). Nothing else to install.
• Shot 2 is a true audio+video extension of shot 1, not a cut: the last ~3 s of shot 1's video and audio become latent context, so the model generates forward from an ongoing utterance and the voice carries over by construction.
• Mode 1 (default): character speaks in a reference voice you supply. Mode 2: bypass one node per shot; the model invents a voice.
• Both shots joined and refined into one FINAL file.

v1.x is superseded. It chained shots on shot 1's decoded last frame. That guide is pixel-continuable, so the sampler reproduced it instead of lip-syncing — chained shots came out as voiceover over a barely-moving face, and it got worse on more strongly distilled checkpoints. No setting fixed it. If you are on v1.x, update.

Not a replacement for the patch below. Lite has no memory bank — identity continuity comes only from the extension context, so it drifts over many shots. That is exactly why the node pack exists.

---

Start here - which download do I need?

Download the zip from Releases. That is the whole thing: every patch file, the workflow, an example prompt file, and step-by-step instructions. You do not need to clone this repo.

The files in this repo are the same files, unpacked, for browsing and diffing.

This is a patch, not a standalone node pack. Install RealRebelAI's ComfyUI_JoyAI_Echo_GGUF_Nodes first, then MERGE these files over it (replace when prompted) - do not delete or replace the whole folder. The pack prints a clear startup error if it detects a replace-instead-of-merge install.

Models are not here; they are on Hugging Face: https://huggingface.co/joeygambino

---

This is a patch, not a one-click ComfyUI Manager install. It layers on top of an existing ComfyUI_JoyAI_Echo_GGUF_Nodes install (RealRebelAI's Rebels GGUF loader stack). Install that first, then merge these files over it - do not replace the folder. The pack prints a clear startup error if it detects a replace-instead-of-merge install.

Models are not in this repo. They live on Hugging Face: https://huggingface.co/joeygambino

A set of bug fixes and features layered on top of the community ComfyUI_JoyAI_Echo_GGUF_Nodes pack (the Rebels GGUF loader stack around JoyAI-Echo). Everything here targets the multi-shot path (JoyEcho_Generate

• the discrete Rebels loaders / JoyEcho_ModelLoader).

This is a patch drop, not a standalone pack: copy these files over a working install of the same pack (back up first). The files are interdependent — in particular nodes.py calls new signatures added to the two libs/ files, so apply them together.

Tested on an RTX 5090 (32 GB) and a 3090 (24 GB), ComfyUI 0.26–0.27, torch 2.8–2.11, with the JoyAI-Echo bf16 release and self-built Q8 GGUFs.

---

Files in this package

nodes.py                                     # JoyEcho_TextEncode / _Generate / _ModelLoader / _LLMEnhance
__init__.py                                  # registrations for the new nodes
rebels_loaders.py                            # discrete GGUF loaders (text-encoder fixes)
joyecho_gguf_runtime_lora.py                 # runtime LoRA hooks for the GGUF DiT path (2.2)
joyecho_plate_picker.py    (new node)        # auto surface-plate reference by ground material (2.2)
plate_denylist.txt                           # plates with people/limbs/shadows, excluded from auto-injection
joyecho_prompt_source.py   (new node)        # one dropdown: .txt briefs + .json scripts
joyecho_ref_picker.py      (new node)        # auto reference-image picker by character name
joyecho_ref_batch.py       (new node)        # None-tolerant image batcher
joyecho_script_picker.py   (new node)        # JSON dropdown (superseded by PromptSource)
libs/ltx_distillation/models/ltx_wrapper.py  # fp8 quantization passthrough
libs/ltx_core/loader/fuse_loras.py           # kohya-LoRA fusion + alpha scaling + fuse telemetry
libs/ltx_core/quantization/policy.py         # fp8_scaled_mm_torch policy (+ sm_89 gate)
libs/ltx_core/quantization/fp8_torch_mm.py   # native-fp8 Linear forward (torch._scaled_mm)
libs/ltx_distillation/utils.py               # tiled VAE decode
libs/ltx_distillation/inference/memory_multishot.py            # memory bank TRIM FIX (critical)
libs/ltx_distillation/inference/bidirectional_pipeline.py      # dtype hardening vs fp8 params
libs/ltx_distillation/inference/memory_bidirectional_pipeline.py # dtype hardening vs fp8 params
prompts/long_story_writer_system_prompt.md   # (optional) de-musicked + character-age edits

The files are interdependent - apply the whole set together, never cherry-pick (a nodes.py newer than its libs/ raises AttributeError at load).

---

Bug fixes

0. RoPE clock hardcoded to 24 fps — the ~10 s lip-sync cliff (CRITICAL)

Lip sync held for the first several seconds of a shot then progressively fell apart, the mouth running steadily ahead of the audio, with the break crossing visibility around 9.6 s into every shot regardless of prompt, model, or reference. This is the reason the pack's practical dialogue limit was believed to be ~241 frames.

Root cause: LTX2DiffusionWrapper.VIDEO_FPS was a hardcoded class constant of 24.0, used to convert the video RoPE temporal coordinate from frames into seconds. The audio RoPE is built in true seconds. Rendering at 25 fps therefore ran the video positional clock 25/24 ≈ 4 % fast against audio — a linear divergence of ~0.04 s per second of runtime, i.e. roughly half a frame of drift per second, accumulating without bound. At ~9–10 s it passes the threshold where a viewer reads it as "not lip syncing".

Fix: the generate nodes now stamp the actual render fps onto the generator before sampling — JoyEcho_Generate (main pass and the hires refine pass) and JoyEcho_SingleShotGenerate. Any render at a consistent fps is now rope-coherent end to end.

Consequence: there is no ~241-frame shot limit. Verified with 69 s and 105 s multishot masters. This was a pipeline bug, not a model property — no LTX-2.3 or JoyAI-Echo checkpoint carries a short training-length cap here; the temporal RoPE range is positional_embedding_max_pos[0] = 20 (seconds), identical in JoyAI-Echo, ltx-2.3-22b-dev, and ltx-2.3-22b-distilled-1.1. Because the fix is in coordinate math rather than weights, it applies to every checkpoint loaded through these nodes, merges included. (nodes.py + libs/ltx_distillation/models/ltx_wrapper.py)

1. enable_audio_memory=False silently disabled ALL cross-shot memory

The pack computed audio_memory_latent=None when audio memory was off, and the video memory-bank save was gated on that latent being non-None — so with audio memory off (the standard anti-drone setting) the bank never filled and cross-shot identity silently died (symptom: memory_size=0 every shot even with memory_max_size=7; a new face each shot). Fix: memory storage is now unconditional; enable_audio_memory gates only the audio-memory injection path. Verify: console memory_size= should climb 0,1,2,… capped at your memory_max_size. (nodes.py)

1b. Memory bank trim was a NO-OP whenever memory_max_size <= num_fix_frames (CRITICAL)

PairedAudioVideoMemoryBank._trim() computed tail[-keep_tail:] - and when keep_tail == 0 (e.g. the common max_size=3 / num_fix_frames=3 combo), tail[-0:] is the WHOLE list, so the bank grew unbounded: every shot conditioned on EVERY prior shot. Symptom: console memory_size= climbing 0,1,2,...,N-1 past your cap, and severe compounding quality degradation over long runs (waxy skin, contrast crush, smearing by the late shots - the "gets worse as it goes" failure). Fixed with a proper zero-tail branch + anchor clamp; memory_size= now freezes at your cap. This one fix eliminated the entire long-run degradation in our tests. (libs/.../memory_multishot.py)

2. GGUF text-encoder loader (RebelsJE_TextEncoder)

Two fixes so a text-only Gemma-3 GGUF loads cleanly:

• meta-strip: drop vision_tower / multi_modal_projector / lm_head (the text-only GGUF has no weights for them → "Cannot copy out of meta tensor").
• device-unify: pin the embeddings-processor to the encoder's actual device (GGUF Gemma runs on CPU while the connector was on cuda → addmm device mismatch).
• fp8 gemma scale-key layouts: the our_fp8 swap only recognized its own export layout (bare module names + .scale_weight); standard HF/comfy-style fp8 gemma files (<module>.weight + .weight_scale, e.g. community abliterated builds) silently loaded with zero modules swapped — the encoder stayed bf16 with no indication. Both layouts are now accepted (per-tensor scalar scales; per-channel scales are skipped and those modules stay bf16), and a loud warning prints if a file matches neither. (rebels_loaders.py)

3. comfy_quant fp8 checkpoints crashed the state-dict loader (v1.7)

The INT8 ConvRot support intercepted EVERY .comfy_quant marker at the state-dict layer and raised on formats it didn't own — which broke loading of fp8-mixed single-file gemma encoders ("comfy_quant format 'float8_e4m3fn' is not supported"). The interception is now scoped to int8_tensorwise only; any other quant format passes through raw for the downstream fp8 swap to consume, exactly as before INT8 support landed. (libs/ltx_core/loader/sft_loader.py)

4. Console windows flashing at the end of every render (v1.7)

The AutoFinish worker was spawned with DETACHED_PROCESS, i.e. with no console at all — so every ffmpeg/ffprobe it ran allocated its own visible console window: a burst of half a dozen focus-stealing flashes as each master was assembled. The worker now launches with CREATE_NO_WINDOW: it gets an invisible console that all its children inherit silently. (joyecho_autofinish.py)

---

Features

3. Split per-domain negative lever (JoyEcho_TextEncode)

The DMD pipeline has no CFG, so the only steering lever is embedding-space. Instead of one negative_prompt/negative_scale that steers both branches, this splits it:

• negative_prompt_video / negative_scale_video — kills burned-in captions/subtitles. Working value ~0.5. Above ~0.8 it over-rotates the video context and locks every shot to shot 1's composition (scene-lock).
• negative_prompt_audio / negative_scale_audio — kills invented music/score. Keep ≤ ~0.4 or dialogue suffers. Steering is norm-preserving (RescaleCFG-style): cond' = renorm(cond + s*(cond − neg)). Old single-widget names still work as a fallback. (nodes.py)

4. Passthrough mode (JoyEcho_LLMEnhance)

mode = "passthrough (raw JSON, skip LLM)" — feed a finished {"prompts":[...]} script straight through with no LLM call / no API key. Auto-detects when story_idea already parses as that JSON. (nodes.py)

5. Reference-image conditioning — I2V-as-reference (JoyEcho_Generate)

New reference_image (IMAGE batch, up to 4). Identity references are prepended as video-only conditioning clips at the memory-encode step — they are never written into the paired audio/video bank. (An earlier attempt that seeded refs into the bank with zero-filled audio latents injected loud background noise with 2+ refs; video-only conditioning avoids it entirely.) Also new: head_trim_frames (auto 8 with refs) drops the first N frames of each shot, where the model morphs out of the reference/memory content. The trim is applied once right after decode, so the final output, the per-shot preview files, and any external concat of them stay frame-identical. (nodes.py)

6. Shot transitions (JoyEcho_Generate)

transition: cut (original) / dissolve (overlap cross-dissolve + equal-power audio crossfade) / vhs_glitch (analog static burst at each boundary: snow, tear bands, dropout lines + a raised-cosine tape-noise audio bed). transition_frames, glitch_intensity tune it. (nodes.py)

7. fp8 transformer quantization (JoyEcho_ModelLoader)

New fp8_transformer toggle. Quantizes the DiT's attention/FF linear weights to float8_e4m3fn at load, from the normal bf16 checkpoint (uses the vendored ltx_core.quantization.QuantizationPolicy.fp8_cast() — upcasts per-layer at inference). Roughly halves DiT weight memory and halves sequential-offload PCIe traffic; keeps memory training + all tensors; VAEs/text-encoder/non-linears stay bf16. Ignored when a GGUF DiT is selected (already quantized). (nodes.py + libs/ltx_distillation/models/ltx_wrapper.py — new quantization param; the quantized build path skips the post-load dtype cast that would otherwise silently upcast fp8 back to bf16.)

8. Tiled VAE decode (JoyEcho_Generate)

Decoding a long high-res shot (e.g. 241f @ 1280×736) in one pass hard-aborts the VAE decode on a 24–32 GB card (fatal cuDNN abort mid-conv, not a catchable OOM). New decode_tiling (auto/on/off) routes decode through the vendored VideoDecoder.tiled_decode — temporal-only 64-frame chunks with 24-frame blended overlap (no spatial tiles → no spatial seams), streaming each chunk to CPU. auto engages only above a size threshold, so small renders keep the original single-pass decode bit-for-bit. (nodes.py + libs/ltx_distillation/utils.py — decode_benchmark_sample gains a video_tiling_config kwarg + _decode_video_tiled_uint8.)

9. Model dropdown (JoyEcho_ModelLoader)

New model_file combo lists every .safetensors / .gguf under the ComfyUI checkpoints / diffusion_models / unet dirs. Pick a .safetensors → full checkpoint (replaces checkpoint_path); pick a .gguf → DiT loaded from GGUF while checkpoint_path still supplies the VAEs / vocoder / text connectors. "(use checkpoint_path)" keeps the old typed-path behavior. A matching lora_file dropdown lists every .safetensors under models/loras (applied at lora_strength on the safetensors DiT path; ignored for GGUF). Plus a clear early error if gemma_path is a .gguf/file/sidecar-less dir (this loader needs the HF gemma-3-12b-it folder; GGUF Gemma only works via RebelsJE_TextEncoder). (nodes.py)

10. LoRA loading hardening (JoyEcho_ModelLoader + libs/.../fuse_loras.py)

• A lora_file dropdown picks LoRAs from models/loras (existing lora_strength widget applies).
• LoRAs on the GGUF DiT path — fixed in 2.2. Fusion computes W' = W + alpha*B*A and bakes the delta into the weight tensors at load. That needs real bf16/fp8 tensors. A GGUF keeps weights packed in quantized blocks, memory-mapped and dequantized per layer at compute time — which is exactly why it is fast and RAM-cheap, and exactly why a delta cannot be added without dequantizing the whole model into RAM (and a rank-32 delta is often smaller than the quantization step anyway). Up to 2.1 the loader simply dropped every LoRA behind one console warning, so a GGUF batch rendered LoRA-free no matter what the stacker showed. From 2.2 they are applied at RUNTIME instead via forward hooks, with no change to the GGUF weights — see New in 2.2 at the top of this README for the mechanism, the validation, the ~881 MB three-LoRA VRAM pin, and the JOYECHO_GGUF_LORA=0 kill switch.
• Fusion now supports kohya naming (lora_down/lora_up) in addition to PEFT (lora_A/lora_B), with standard alpha/rank scaling — previously a kohya-named LoRA silently did NOTHING (zero keys matched, no warning).
• Fusion prints how many weights fused, and WARNS LOUDLY when a provided LoRA matched zero keys.
• The loader refuses ComfyUI-quantized checkpoints (.comfy_quant marker tensors, e.g. "fp8mixed learned" builds) with a clear error: this loader never applies their weight scales (the model would silently load mis-scaled) and LoRA fusion on them crashes with shape errors. Use bf16 checkpoints.

11. Automation / batching nodes (new)

• JoyEcho_PromptSource — one dropdown listing LPFF-style .txt briefs (from the inspire-pack prompts tree) and passthrough .json scripts (input/joyecho_prompts/). Multi-block briefs fan out like LoadPromptsFromFile. Emits story_idea (→ LLMEnhance) + character (→ RefPicker) + count. Replaces the LPFF→UnzipPrompt chain and lets you switch prompt sources with one dropdown instead of rewiring.
• JoyEcho_RefPicker — auto-selects a character reference image from a folder tree keyed by character name (a character_pick dropdown of the folder names, a typed/wired character string, or a prompt scan — dialogue mentions are stripped so only the on-screen subject wins). The dropdown survives model refreshes, an explicitly named character that matches no folder refuses to fall back to the prompt scan (a wiped/typo'd name can't silently become the wrong character's face), and the cache signature includes the prompt text (without it, ComfyUI could serve a cached pick from a previous queue item). on_no_match=no_reference returns nothing so a batch keeps running.
• JoyEcho_RefBatch — None-tolerant image batcher: combines up to 4 optional IMAGE inputs (e.g. two RefPickers for a two-character shot), skips missing refs, resizes mismatched sizes to the first image, outputs None if all are missing (Generate then just skips identity seeding). The stock KJNodes ImageBatchMulti crashes with 'NoneType' has no attribute 'shape' on a missing ref; this replaces it.
• JoyEcho_ScriptPicker — JSON dropdown (superseded by PromptSource; kept for compatibility).

12. GPU encode hot-swap (JoyEcho_TextEncode)

With low_vram the Gemma encoder used to encode every shot on CPU (~10s+ per shot). The encode pass now borrows the (idle) GPU when the encoder fits free VRAM - with a fits-check, an OOM fallback to CPU, and a move-back before the denoise phase. 20-shot encodes drop from minutes to seconds. (nodes.py)

13. encoder_fp8 (JoyEcho_ModelLoader)

Stores the Gemma encoder's linear weights as float8_e4m3fn with per-layer upcast at encode (encode runs once per item, so the upcast tax that makes fp8 slow on the DiT is irrelevant here). Wrapper drops ~24GB -> ~21GB and the GPU hot-swap engages on 32GB cards; JD's connector projections stay bf16.

14. fp8_scaled_mm (JoyEcho_ModelLoader) - native fp8 compute

Stores the DiT's attention/FF linears as fp8 AND runs the matmuls natively via torch._scaled_mm - no per-layer upcast tax (measured x2.8 raw kernel / x1.5 end-to-end vs bf16 on an RTX 5090). ~22GB resident enables sequential_offload=False at moderate resolutions. REQUIREMENTS: sm_89+ GPU (RTX 40/50 - clear error on older cards, with a per-device runtime fallback to upcast), and a bf16 source checkpoint (an fp8 FILE would load every tensor fp8 with the cast skipped and crash the noise path - guarded with a clear error). Tensorwise dynamic activation quant: A/B your content before adopting.

15. resident_blocks (JoyEcho_Generate)

Sequential offload middle ground: pin the first N of 48 transformer blocks permanently on GPU, stream the rest. N=24 halves the per-step PCIe traffic; raise until VRAM is nearly full. Composes with fp8 modes (fp8 blocks are half the bytes both resident and streamed).

16. Hires-fix second pass (JoyEcho_Generate)

hires_factor (>1.0) + hires_denoise: after all shots render, each shot is bicubic-upscaled, VAE re-encoded, re-noised at a tail sigma and re-denoised through the DMD ladder at the TARGET resolution - the model synthesizes real detail (RTX-class upscalers only sharpen what exists). Runs in 65-frame windows with cross-fade (a 24GB card survives 1920x1088 refines); memory bank and per-shot previews stay base-res; failures fall back to the base frames. Audio is untouched.

17. Reference scheduling upgrades (JoyEcho_RefPicker + _Generate)

• Script-carried ref pinning: {"prompts": [...], "refs": {"zara": "zara_file.png"}} pins a scene-matched reference per character (a full-scene ref SETS the render's scene - match it to the script).
• Re-entry injection: a character returning after a 3+-shot absence gets their ref re-injected at the return shot automatically (the rolling memory window is 4; long absences otherwise re-invent the character).
• Generate's ref dedup is schedule-aware (the same image scheduled at two shots survives; cap 6 scheduled entries).

18. Robustness

• Pipelines no longer derive their working dtype from next(parameters()).dtype (an fp8 first-param crashed torch.randn); fp8 dtypes are skipped with a bfloat16 fallback.
• fp8 gemma swap accepts both .scale_weight and .weight_scale layouts and warns loudly on zero matches instead of silently staying bf16.

20. Voice casting - per-character voices from files (NEW, 2026-07-29)

The memory bank guarantees voice CONSISTENCY, not correctness: shot 1 rolls its voice from text conditioning alone, and whatever it rolls, the bank then carries faithfully. This release makes the voice a CASTING decision instead of a roll, with zero per-run typing:

• Folder casting. Put a clip of the character speaking (>=4 s, mp4 or wav) in ComfyUI/input/joyecho_voices/<speaker-tag-lowercase>/. Any script whose speaker tag matches the folder gets that voice seeded into the memory bank as a character-tagged anchor slot BEFORE shot 1 - the first shot continues the cast voice instead of auditioning a new one, and the same file re-casts the same voice in every future render. The pick is deterministic (alphabetically first file). Replace the file to recast.
• Script-carried casting. A "voice_refs": {"Alice": "path/clip.mp4"} key in the script JSON overrides the folder scan per character.
• Audio-only anchors. A bare wav/flac anchor pairs its voice with the character's ref image from joyecho_refs/<tag>/ so the slot keeps its face+voice contract.
• Speaker order is derived from the script - an explicit "speakers": [...] array, or the "<Name> is talking" attribution in each shot. It now travels INSIDE the conditioning (and its disk cache), so it can never go stale or leak between graphs. The speaker_order widget remains as a manual override only.
• Anchor + latest policy. A speaker's audio context is their anchor slot(s) plus their most recent shot only - one drifted shot can no longer accumulate a majority and take over the rest of the video.
• Cold-start fix. A speaker's first line falls back to the unfiltered bank instead of a zeroed (silent) one - the regression that previously made per-character filtering unusable.

Two-character staging note that saves you a night: a2v cross-attention has no spatial addressing - audio at time t drives EVERY face in frame, however small or distant. Stage ONE face per shot (shot-reverse-shot) and put only the visible character's description in that shot's prompt.

21. Correctness fixes from a full sampler-path audit (2026-07-29)

• Hires refine now respects your seed. Its re-noise fields were seeded from a hardcoded constant - every render's refine detail layer was identical regardless of the seed widget, for months.
• Chained SingleShot graphs no longer condition on the PREVIOUS queue run's output. The memory bank object was mutated in place through ComfyUI's output cache; incoming banks are now cloned.
• The conditioning disk cache key includes the checkpoint - a model swap can no longer be served the previous model's conditioning tensors. (Existing cache files are invalidated once; they rebuild on first render.)

22. The 24 fps rule — frame rate steers ACCENTS (2026-07-30)

The most consequential finding in this pack, discovered chasing a week of "why is everyone suddenly Australian":

LTX-2.3's joint audio-video prior is 24 fps-native, and the render fps is a hidden accent dial. The model experiences fps as video-token density per second of audio RoPE time. Move it off 24 and voices slide monotonically into the Commonwealth vowel space — same text, same seed, same everything:

Two practical consequences:

• Off-24 fps overrides your prompt. At 25 fps, even a line carrying "speaking in a casual American accent" renders British — a few conditioning tokens cannot outvote a geometric signal present in every attention operation at every step. At 24 fps the same wording is obeyed.
• This interacts with Bug fix #0. Pre-patch, the rope clock was hardcoded 24, so 25 fps renders had broken lip-sync but American voices. The sync fix made the clock honest — and surfaced the accent drift that the bug had been masking. If your voices "changed" after applying the patch, this is why: you were rendering at 25. Drop to 24 and you get sync AND your accents.

As of v1.7 the node default is 24, the bundled workflow ships at 24, and the Generate node prints a warning if you dial anything else. The flip side is a free feature: if you want an authentic British or Australian character, render their scenes at 25 or 30 fps instead of writing accent prose — it is more consistent than any wording.

Voice prior note: with NO accent stated, young-female characters lean Australian even at 24 fps (the base model's lean). State the accent on every spoken line regardless — fps sets whether the model listens; the prompt still has to do the asking.

19. Finishing: who builds your master (READ THIS before touching hires)

hires_factor is a ROUTING switch, not a quality slider - it decides which pipeline builds your final video:

• Judge and publish only the *_MASTER.mp4. The in-graph SaveVideo output (prefixed PREVIEW in the shipped workflow) is a convenience preview: its re-encode is bit-starved and shows artifacts the master does not have. This is a ComfyUI limitation, not a render problem.
• Resolutions: everything works at any /32 size; only the spatial hires mode additionally needs the /32 result EVEN on both axes (1280x768 and 1344x768 qualify; 1280x736 and 1536x864 do not - spatial will smear one edge and warn in the console).
• All other widgets are genuinely free: seed, num_frames (long shots are fine - the old ~10s lip-sync limit was the fps bug, now fixed), fps, memory sizes, head_trim.
• The master pipeline re-encodes with bf 0 + tune grain end to end, so masters never reintroduce B-frame pumping.

---

Applying

1. Back up your existing pack folder.
2. Copy each file over the same relative path in ComfyUI/custom_nodes/ComfyUI_JoyAI_Echo_GGUF_Nodes/.
3. Restart ComfyUI. New widgets append at the end of existing nodes, so saved graphs keep their values; the four new nodes appear under the JoyAI-Echo category. Press R after adding model files to refresh the model_file dropdown.

The libs/ files must match the vendored ltx_core / ltx_distillation in your pack (same JoyAI-Echo release). If your libs/ differ substantially, cherry-pick the changes described above rather than overwriting.

Not included (intentionally): model weights, the gemma_assets/ tokenizer binaries, .bak snapshots, and __pycache__.

---

Where the models live

Nothing in this repo is a model. Weights are on Hugging Face:

Civitai mirrors: GGUF builds - workflow

License and scope

This repository contains only my patch files - it is not a redistribution of the underlying pack or of any model weights.

• The patched pack wraps JoyAI-Echo, which is research / non-commercial. That is the strictest term in the stack and it governs your outputs.
• LTX-2 / LTX-2.3 is under the LTX-2 Community License.
• Gemma 3 - the text encoder, and the gemma_assets/ tokenizer sidecars bundled here because the loader requires them - is subject to the Gemma Terms of Use.
• AI-generated content produced with this stack must be disclosed as such.

Not affiliated with Lightricks, JD, RealRebelAI, Comfy-Org, or Google.

Credits

This patch stands on other people's work:

• JD Joy Future Academy - JoyAI-Echo, the multishot memory model this whole stack serves (research / non-commercial license).
• Lightricks - LTX-2 / LTX-2.3 (LTX-2 Community License).
• TenStrip - the LTX2.3 DMD LoRAs; the hires strong (tenstrip 4-step) mode uses his published upscale sigma ladder verbatim.
• RealRebelAI - the Rebels GGUF loader stack this patches, and the Q6_K_RM GGUF whose tensor canon the self-built GGUFs mirror.
• Comfy-Org - the comfy-quants int8_tensorwise + ConvRot export recipe behind the INT8 checkpoint.
• Google - Gemma 3 12B, the text encoder (Gemma license).

Support

Everything here is free and stays free. If it saved you time, you can sponsor me on GitHub, buy me a coffee, or support me on Liberapay.