The refine-default comment claimed Condition variants are excluded from two-stage
refine, but supports_two_stage_refine has no such carve-out and ltx_process builds
a second condition set so conditioning survives the upsample. Two file references
had also drifted: ltx_process.py:179 moved, and the offload hook keying now lives
in sd_offload_balanced. Name the symbols instead of the line numbers.
Its 2.x sibling always did, so only the 0.9 path hit a cpu weight against cuda
latents. Also logs why a run is refused, which until now travelled only in the
raised error.
run_ltx reported failure by yielding a string, which is why LTX had no API.
run() is the core underneath: keyword arguments named as video_run.run names
them, a VideoResult back, VideoError out with 499 for an interrupt. The lock,
progress and summary stay in the adapter, whose signature is unchanged since
callers bind to it by keyword. Failure now closes the processing object and
deactivates networks, which abort never did.
2.5 reuses the LTX-2 pipeline classes, so it is described through the
capability table: Gemma 4 text encoder, cross-timestep conditioning, and the
upsampler and stage 2 LoRA that now ship inside the model repo. The repo
carries the distilled scheduler config, so Dev rows restore the terminal
shift, and the Dev transformer sits in transformer_full.
Distilled rows stop forcing dynamic shifting on, which remapped their sigma
schedule. Auto duration hands the clip length to the duration head.
Some paths left both to pipeline defaults, which track the current upstream
model: distilled picked up guidance it already bakes in, and 2.0 could land on
2.3's joint-sigma path.
The LTX tab already collected a last-frame image but anchored every condition at index 0, so it never acted as a last frame. Build a separate condition for it at the final frame: index -1 for the 2.x family (latent index, negatives wrap) and num_frames-1 for 0.9 (pixel index). The Last image input now shows only for Condition models, the pipelines that accept multi-frame conditioning.
- monkey-patch LTX2ConnectorTransformer1d.forward to restore pre-#13564
padding logic when the upstream torch.flip pattern is detected; fixes
word-order scrambling in audio dialogue tracks
- reorganize LTX model entries into version-group separators (2.3 v1.1,
2.3 v1.0, 2.0, 0.9.x) with base/distilled subgroups; separators are
selectable no-ops handled in run_ltx
Upstream LTX2 encode_prompt is not @torch.no_grad-decorated; only __call__
is. Calling encode_prompt directly from ltx_process recorded the full
Gemma3 forward graph.
review feedback from vladmandic on commit 80fde086f.
- run_ltx: drop the inline random.seed() / randrange resolve and use
processing.fix_seed(p) right after the StableDiffusionProcessingVideo
construction. p.seed carries the resolved int across every stage
(latent_pass, both upsample paths, refine, post-refine vae_decode).
matches the existing fix_seed(p) call sites in img2img.py:30,
video_run.py:101, xyz_grid.py:266 and 8 others.
- process_decode: add AudioFrameList(list) subclass and attach_audio
helper to carry output.audio onto the returned frame list. mirrors
the existing output.bytes early-return contract: samples.audio
survives downstream so process_images_inner can collect it via the
Processed.audio kwarg.
- drop the p.audio_capture transit in process_diffusers and the
fallback read in process_images_inner. p is input params, not
output state.
- resolve seed=-1 once at top of run_ltx and thread the int through every
stage (StableDiffusionProcessingVideo, _latent_pass, upsample 0.9/2.x,
refine, post-refine vae_decode). get_generator(-1) reseeds globally per
call so each stage was rolling an uncorrelated value; p.seed now carries
the resolved seed so reruns reproduce.
- pre-encode prompts via shared.sd_model.encode_prompt before the latent
path; park the four tensors on CPU and pass them as prompt_embeds /
*_attention_mask kwargs through _latent_pass and refine_args. Stage 2
reuses Stage 1's embeds instead of re-running the text encoder. The
manual encode is outside pipe.__call__ so the post-forward offload hook
never fires; apply_balanced_offload(force=True) re-anchors the device
map so the text encoder doesn't stay pinned through Stage 1.
- skip audio_vae.decode + vocoder on refine when audio_enable=False:
output_type='latent' bypasses the internal audio + video decode and
hands off to the post-refine vae_decode block. Per-step audio
cross-attention still runs for video conditioning.
drop the project-specific stage 1 direct audio decode (a6870f7d2). smoke
testing showed broadband tinniness on distilled is BWE-bound, not stage 2
corruption, so the deviation didn't fix the underlying issue.
revert to canonical:
- _latent_pass returns video latents only; result.audio is unused.
- stage 2 receives audio_latents=None (default), prepare_audio_latents
generates fresh gaussian noise, audio scheduler runs the 3 stage-2
sigmas under identity guidance, video<->audio cross-attention
conditions the audio branch.
- capture stage 2 result.audio[0].float().cpu() for save.
upstream evidence: pipeline_ltx2.py:937-940 documents audio_latents as
pre-generated noisy latents (initial gaussian, not stage 1 output); no
caller in diffusers threads stage outputs into the kwarg.
non-latent path is unchanged and continues to work via p.audio_capture.
process_decode strips video pipeline output to a flat list of frames at the
PIL early-return (processing_diffusers.py:461-465), so any output.audio is
lost before processing.process_images returns. video pipelines that produce
synchronized audio (LTX-2 audio-capable models) were getting silent mp4s on
the non-latent path.
stash output.audio on p.audio_capture before process_decode runs and let
processing read it back as a fallback when samples is a flat list.
ltx_process non-latent branch strips the (B, 2, N) batch dim with [0] so
write_audio's .T+contiguous() path produces interleaved bytes for AAC s16.
stop threading stage 1 audio_latents into stage 2 refine. Lightricks/LTX-2#126
reports the two-stage pipeline degrades audio quality, confirmed locally as
clean speech with tinny ambient/foley/music on the threaded path.
root cause: stage 2 prepare_audio_latents calls _create_noised_state at
noise_scale=0.909 (pipeline_ltx2.py:704-714, 598-603), keeping ~9% of stage 1
signal. 3 refine steps recover speech via video<->audio cross-attention but
not broadband content.
new path: _latent_pass decodes audio_latents to waveform via audio_vae +
vocoder mirroring pipeline_ltx2.py:1471-1473 exactly (input cast to
audio_vae.dtype, no module dtype mutation). stage 2 result.audio is
discarded; cross-attention still runs each block for video conditioning.
LTX refine pipe uses output_type='pil' so result.frames[0] returns a list, not a 5-D tensor. Convert via images_to_tensor before the helper sees it, mirroring what save_video already does for the same input.
LTX, video_run, and framepack_worker bypass process_images_inner, so
they call apply_video_interpolation explicitly before save_video. Save
receives already-inflated frames; the sentinel guard skips its own pass.
- LTX and video_run scale mp4_fps by interpolation_factor(p) so duration
is preserved instead of stretched (LTX is conditioned on source fps)
- FramePack pre-divides at gen time per get_latent_paddings, so save fps
stays at mp4_fps; worker passes p=None so save call uses
mp4_interpolate=0 to skip directly
- replaces the inline (mp4_interpolate+1) fps math at LTX with the
helper-driven equivalent
LTX conditions the model on mp4_fps as the source frame rate, then
RIFE inflates frame count at save. Without compensation, save fps
stays equal to source fps and the video becomes (mp4_interpolate+1)x
slow-motion (e.g. 121 frames at 24 fps with mp4_interpolate=1 saved
as 242 frames at 24 fps = 10s instead of the intended 5s). Scale the
saved fps by the interpolation factor at the LTX call site so output
duration matches user intent.
FramePack already pre-divides mp4_fps at generation time (compute-
saving semantic baked into UI math), so it does not need this fix.
The pipelines now produce duration-correct output via two different
mechanisms; standardization is tracked for the RIFE-in-processing
follow-up.
Distilled refine ran without the scheduler swap and identity guidance
kwargs prescribed by huggingface/diffusers#13217; only Dev got that
setup via the supports_canonical_stage2 branch. Distilled is already
trained at identity but still needs the recipe applied to avoid the
four-way composition double-dipping on top of the distilled sigma
schedule (oversaturation/striping).
Unify the branches under family == '2.x'; gate only the LoRA load and
unload on supports_canonical_stage2.
Conditions were built once at the user's chosen resolution, so condition
variants either ran same-res for both stages (blocked by the previous
gate) or fed base-res conditions to a final-res refine on the manual
upsample+refine path. Build conditions twice when staging is active:
once at base dims for Stage 1, once at final dims for Stage 2. Drop the
condition-class exclusion from supports_two_stage_refine.
auto_refine_upsample gated on supports_canonical_stage2 (Dev-only LoRA
path), so Distilled refine ran same-res for both stages and oversaturated.
Split the gate: supports_two_stage_refine covers any 2.x non-condition
variant and drives the staging plan plus UI Refine default;
supports_canonical_stage2 stays Dev-only for the LoRA branch.
Modules already inventoried at load time; repeating the six-line dump at
each upsample or refine boundary is redundant. silent=True suppresses
the per-module DEBUG lines; op=init and Model class= INFO stay intact.
The upsample pipes receive shared.sd_model.vae as a constructor formality;
the forward pass is pure latent to latent. Main pipe already owns that
VAE's accelerate hook lifecycle, so walking it again from the upsample
pipe raises "Cannot copy out of meta tensor" when the main pipe has
offloaded params to meta. Skip it in the walk.
Collapses the scheduler + shared.opts snapshot/mutate/restore block in
run_ltx into a single `with temp_scheduler_opts(...)` statement. Same
five pieces of state are snapshotted and restored as before; the
mechanism is unchanged per PR #4783 thread 2 feedback, only readability.
LTXCaps gains a `variant` field ('0.9', '2.0', '2.3') replacing
`is_ltx_2_3`; variant-specific branches check `caps.variant == '2.3'`
instead of grepping the model name.
ltx_util.load_upsample_2x mirrors load_upsample's contract so the 2.x
path owns a module-level cache and stops reloading ~2.3 GB every run.
The cached pipe is stamped with a synthetic
`CheckpointInfo('ltx-upsampler-2.x')` so it gets its own OffloadHook
slot and can go through apply_balanced_offload without invalidating
the main pipe's module map. The hardcoded `.to('cpu')` and post-pass
torch_gc are gone; the second apply_balanced_offload handles spill.
video_overrides comment on OzzyGT LTX-2.3 connectors states plainly
that mirrors pack weights twice by design; ignore_patterns is the
surgical workaround, not an hf_hub bug.
Drive-by: load_upsample log line used `__class__.__name__` (always
'type'); switched to `__name__`.
Each engine tab (Generic, FramePack, LTX) now owns its own size, seed,
frame count, and init imagery. The previously shared Size and Inputs
subtabs are removed; persistent encoding settings move to a single
Output subtab constructed first so engine tabs can consume its widgets.
FramePack drops its local fps/interpolate sliders and consumes the
shared Output widgets; LTX drops the redundant shared init_image
parameter and keeps only its own ltx_init_image, with run_ltx's
condition_image fallback removed in turn.
Implement the Lightricks two-stage recipe (diffusers PR #13217) for the
LTX-2.x Dev family: Stage 1 at half-res with full four-way guidance,
2x latent upsample, Stage 2 with distilled LoRA + scheduler swap + identity
guidance on STAGE_2_DISTILLED_SIGMA_VALUES.
Extends to both LTX-2.0 and LTX-2.3 Dev via per-family distilled-LoRA
repos carried on the caps; Distilled variants take the same flow minus
the LoRA swap. Auto-couples Refine with a fixed 2x upsample on any Dev
variant with a known LoRA when the user enables Refine without Upsample.
- caps: is_ltx_2_3, use_cross_timestep, default_dynamic_shift,
stage2_dev_lora_repo, supports_canonical_stage2, modality_default_scale,
guidance_rescale_default; LTX-2.x defaults realigned to canonical
cfg=3.0 / steps=30; per-variant STG block and four-way guidance wired
for non-distilled 2.x
- process: canonical Stage 1/Stage 2 helpers, scheduler + opts snapshot
under try/finally, per-family upsampler repo, audio latents threaded
from Stage 1 into Stage 2, use_cross_timestep gated per caps
- overrides: skip the redundant unsharded LTX-2.3 connectors blob and
share LTX2TextConnectors weights across 2.3 variants when te_shared_t5
- load: Gemma3 shared-TE path for LTX-2.3; gate use_dynamic_shifting=False
override to 0.9.x only so LTX-2.x stays on its canonical token-count
dynamic shift
Rework the LTX Video tab so one UI handles every registered variant
(0.9.0 through 2.3, Dev/Distilled/SDNQ-4Bit, T2V/I2V/Condition). Per-
variant behavior is driven from a single capability lookup rather than
substring matching on model names scattered across the backend.
- modules/ltx/ltx_capabilities.py: new module computing family, is_i2v,
distilled, supports_input_media, supports_multi_condition,
supports_image_cond_noise_scale, supports_decode_timestep,
supports_stg, supports_audio, supports_frame_rate_kwarg, and the
default CFG / steps / sampler_shift for a given model name by reading
its registered repo_cls in models_def.
- modules/ltx/ltx_ui.py: capability-gated UI. Selecting a model rewires
accordion visibility, slider interactivity, and defaults via a single
model.change handler. New controls: dedicated image input slot inside
the LTX tab (replaces the disconnected shared init_image for I2V),
condition strength slider, CFG / sampler shift / dynamic shift
sliders that were previously unreachable. Input media accordion
restructured so the image slot is always-visible while the video /
gallery prefix tabs only appear on Condition pipelines.
- modules/ltx/ltx_process.py: route the base pass through
processing.process_images(p) so LTX inherits standard scheduler
wiring, extra_networks activation, VAE handling, and error plumbing
from StableDiffusionProcessingVideo. The multi-pass latent path
(upsample / refine) stays on direct pipeline calls for latent
re-entry. Refine noise control gets family-specific kwargs:
denoise_strength for 0.9.x LTXConditionPipeline, noise_scale for all
2.x pipelines; the prior strength= injection crashed on 2.x and only
affected conditioning intensity on 0.9.x. Add torch_gc between every
stage boundary (base to upsample to refine to vae decode) so the
CUDA allocator cache does not retain the prior pass's allocations
across stages. Remove the TypeError fallback that silently passed
raw latents to save_video when VAE decode returned None on OOM;
those errors now surface cleanly.
- modules/ltx/ltx_util.py: get_conditions grows a family parameter and
builds LTX2VideoCondition (frames, index, strength) for 2.x or
LTXVideoCondition (image, video, frame_index, strength) for 0.9.x.
get_bucket floors to max(32, vae_spatial_compression_ratio) since LTX
pipelines validate divisibility by 32 regardless of family.
- modules/video_models/video_overrides.py: extend the I2V generator
reset to cover LTX2ImageToVideoPipeline and both Condition classes.
Keep the strength= kwarg injection gated to 0.9.x
LTXConditionPipeline only; LTX2ConditionPipeline.__call__ does not
accept it (per-condition strength lives on the LTX2VideoCondition
dataclass instead).
CalamitousFelicitousness
Vladimir Mandic
Ryan Meador