- resolve seed=-1 once at top of run_ltx and thread the int through every
stage (StableDiffusionProcessingVideo, _latent_pass, upsample 0.9/2.x,
refine, post-refine vae_decode). get_generator(-1) reseeds globally per
call so each stage was rolling an uncorrelated value; p.seed now carries
the resolved seed so reruns reproduce.
- pre-encode prompts via shared.sd_model.encode_prompt before the latent
path; park the four tensors on CPU and pass them as prompt_embeds /
*_attention_mask kwargs through _latent_pass and refine_args. Stage 2
reuses Stage 1's embeds instead of re-running the text encoder. The
manual encode is outside pipe.__call__ so the post-forward offload hook
never fires; apply_balanced_offload(force=True) re-anchors the device
map so the text encoder doesn't stay pinned through Stage 1.
- skip audio_vae.decode + vocoder on refine when audio_enable=False:
output_type='latent' bypasses the internal audio + video decode and
hands off to the post-refine vae_decode block. Per-step audio
cross-attention still runs for video conditioning.
drop the project-specific stage 1 direct audio decode (a6870f7d2). smoke
testing showed broadband tinniness on distilled is BWE-bound, not stage 2
corruption, so the deviation didn't fix the underlying issue.
revert to canonical:
- _latent_pass returns video latents only; result.audio is unused.
- stage 2 receives audio_latents=None (default), prepare_audio_latents
generates fresh gaussian noise, audio scheduler runs the 3 stage-2
sigmas under identity guidance, video<->audio cross-attention
conditions the audio branch.
- capture stage 2 result.audio[0].float().cpu() for save.
upstream evidence: pipeline_ltx2.py:937-940 documents audio_latents as
pre-generated noisy latents (initial gaussian, not stage 1 output); no
caller in diffusers threads stage outputs into the kwarg.
non-latent path is unchanged and continues to work via p.audio_capture.
process_decode strips video pipeline output to a flat list of frames at the
PIL early-return (processing_diffusers.py:461-465), so any output.audio is
lost before processing.process_images returns. video pipelines that produce
synchronized audio (LTX-2 audio-capable models) were getting silent mp4s on
the non-latent path.
stash output.audio on p.audio_capture before process_decode runs and let
processing read it back as a fallback when samples is a flat list.
ltx_process non-latent branch strips the (B, 2, N) batch dim with [0] so
write_audio's .T+contiguous() path produces interleaved bytes for AAC s16.
stop threading stage 1 audio_latents into stage 2 refine. Lightricks/LTX-2#126
reports the two-stage pipeline degrades audio quality, confirmed locally as
clean speech with tinny ambient/foley/music on the threaded path.
root cause: stage 2 prepare_audio_latents calls _create_noised_state at
noise_scale=0.909 (pipeline_ltx2.py:704-714, 598-603), keeping ~9% of stage 1
signal. 3 refine steps recover speech via video<->audio cross-attention but
not broadband content.
new path: _latent_pass decodes audio_latents to waveform via audio_vae +
vocoder mirroring pipeline_ltx2.py:1471-1473 exactly (input cast to
audio_vae.dtype, no module dtype mutation). stage 2 result.audio is
discarded; cross-attention still runs each block for video conditioning.
relabel toggle to 'LTX save audio' (default true) since audio always
generates on 2.x audio-capable models; the toggle gates mux only. hint
added to locale_en.json.
split add_audio_stream from write_audio. avformat_write_header runs on
first container.mux() and freezes the stream set, so audio added after
video packets has time_base=0/0 and raises 'Cannot rebase to zero time.'
atomic_save_video registers the audio stream before the encode loop.
applied_layers is cleared and re-populated on every network_activate call.
With lora_apply_te=True the second activate (TE-only pass) finds all
modules already at the target state and skips them all, leaving
applied_layers empty and breaking the restore trigger on the next gen.
native_active is set from loaded_networks at the end of activate, so it
survives idempotent re-runs and only flips false after the restore call
clears loaded_networks.
Flux2/Klein loads LoRAs as native modules through the diffusers method
path. network_activate() was only called when new native modules existed,
so removing a LoRA from the prompt left backed-up weights unrestored.
Check applied_layers to detect previously active native modules and
trigger network_activate() for the restore path.
When a video script (animatediff, text2video, image2video, stablevideodiffusion)
runs via Control tab, both the script's save and control_run's end-of-run save
fired. The latter crashed silently because the local `video` cv2 capture name at
control_run:584 shadowed the modules.video import, so the duplicate was hidden
and the gallery video link never propagated.
- alias import as video_module to bypass the shadow
- p.video_saved marker set by each script
- control_run skips its end-of-run save when the marker is set
LTX refine pipe uses output_type='pil' so result.frames[0] returns a list, not a 5-D tensor. Convert via images_to_tensor before the helper sees it, mirroring what save_video already does for the same input.
Set p.video_interpolate at run() entry so process_images_inner picks up
the helper. Save calls keep their existing kwargs; the sentinel guard
skips re-interpolation when the helper already ran.
- animatediff, text2video, image2video, stablevideodiffusion route
mp4_interpolate into p.video_interpolate
- modules/control/run.py routes the request video_interpolate arg
- xyz_grid intentionally untouched: its end-of-axis save_video stitches
the cell slideshow, not per-cell videos; a future video_interpolate
axis would set p.video_interpolate per cell via axis_options
LTX, video_run, and framepack_worker bypass process_images_inner, so
they call apply_video_interpolation explicitly before save_video. Save
receives already-inflated frames; the sentinel guard skips its own pass.
- LTX and video_run scale mp4_fps by interpolation_factor(p) so duration
is preserved instead of stretched (LTX is conditioned on source fps)
- FramePack pre-divides at gen time per get_latent_paddings, so save fps
stays at mp4_fps; worker passes p=None so save call uses
mp4_interpolate=0 to skip directly
- replaces the inline (mp4_interpolate+1) fps math at LTX with the
helper-driven equivalent
Promote RIFE interpolation from a save-time kwarg to a real stage of the
processing pipeline so per-frame work (detailer, color correction,
postprocess scripts) operates on source-rate frames and the inflated
stream becomes the saved output.
- new modules/processing_video.py with apply_video_interpolation,
interpolation_factor, expand_infotexts; PIL/tensor/numpy dispatch
- video_interpolate, video_interpolate_scale, video_interpolated fields
on StableDiffusionProcessingVideo
- process_images_inner runs the helper after the batch loop and inflates
infotexts in lockstep
- save_video in modules/video.py and modules/video_models/video_save.py
short-circuit re-interpolation when p.video_interpolated is set; the
user-facing kwarg still flows into metadata
read_info_from_image had three issues that together corrupted
metadata in the Process tab and img2img batch:
- items aliased image.info, so items.pop('parameters', ...) mutated
the source PIL image's info dict in place
- after popping, parameters was never restored to items, so callers
that re-stamp items onto a downstream image (postprocessing.py,
img2img.py) silently dropped the original generation params for
any input without EXIF UserComment
- synthetic width/height/mode keys were stuffed into items and ended
up persisted as PNG tEXt chunks even though no caller reads them
Result for the Process tab: upscaled outputs lacked the original
parameters chunk and instead had useless width/height/mode chunks
holding the source's pre-upscale dimensions. With this fix items is
a copy, parameters is restored after parsing, and width/height/mode
no longer leak into saved files.
Image.fromarray returns a fresh PIL image with empty .info, so the
source's parameters/UserComment metadata was silently dropped on the
grading roundtrip. In the Process tab this caused upscale + color
grading to save PNGs without the original generation parameters,
while upscale alone (via PIL resize/filter ops that preserve .info)
still worked.
torchvision 0.26 removed the torchvision.io.video submodule entirely
(only image I/O remains). The monkey-patch in install_requirements
(`torchvision.io.video.av = av`) was attempting to swap PyAV into a
torchvision-internal that no longer exists, raising AttributeError on
every FramePack run. The patch is fully vestigial against modern
torchvision; PyAV is still installed by `install('av')`, and the
sdnext save path uses PyAV directly via av.open().
LTX conditions the model on mp4_fps as the source frame rate, then
RIFE inflates frame count at save. Without compensation, save fps
stays equal to source fps and the video becomes (mp4_interpolate+1)x
slow-motion (e.g. 121 frames at 24 fps with mp4_interpolate=1 saved
as 242 frames at 24 fps = 10s instead of the intended 5s). Scale the
saved fps by the interpolation factor at the LTX call site so output
duration matches user intent.
FramePack already pre-divides mp4_fps at generation time (compute-
saving semantic baked into UI math), so it does not need this fix.
The pipelines now produce duration-correct output via two different
mechanisms; standardization is tracked for the RIFE-in-processing
follow-up.
video_save passes [-1,1]-range pixels to rife.interpolate_nchw, but
RIFE v4.25's IFNet explicitly clamps inputs to [0,1] (Head and IFNet
forward pass), turning every negative pixel value into zero. v3.9
silently extrapolated and produced soft artifacts; v4.25 produces
washout. Convert to [0,1] before the RIFE call and back to [-1,1] for
downstream save.
- Vendor IFNet_HDv3 v4.25 (5 IFBlocks, Head encoder, feat channel) and
v4 warplayer with explicit (tenFlow_div, backwarp_tenGrid) signature
- Rewrite RifeModel.inference for the new forward signature with
per-(H,W,device,dtype) caching of tenFlow_div and backwarp_tenGrid
- Force fp32 inference: bf16 produced visible checkerboard at the new
IFNet's depth (was hidden by v3.9's shallower architecture)
- Crop padded frames in interpolate_nchw before output (was missing,
produced gray bar on non-128-aligned inputs)
- Drop training scaffolding (AdamW, EPE/SOBEL, update method)
- Log obsolete legacy v3.9 weights file on first v4.25 load instead of
silently deleting user data
- Default download URL is HolyWu vs-rife mirror (MIT, byte-identical
upstream weights); swap to project-hosted URL before merge
- new autocomplete_translations setting (default off); checkbox in script accordion + locale hint
- get_cached gates companion load on the opt; toggle-flip auto-syncs missing files in a background thread
- list_all_sync, local_names and config_json filter out *.translations.json so companion files never surface as selectable dicts
- one-time per-name warning when opt is on but companion is missing locally
Distilled refine ran without the scheduler swap and identity guidance
kwargs prescribed by huggingface/diffusers#13217; only Dev got that
setup via the supports_canonical_stage2 branch. Distilled is already
trained at identity but still needs the recipe applied to avoid the
four-way composition double-dipping on top of the distilled sigma
schedule (oversaturation/striping).
Unify the branches under family == '2.x'; gate only the LoRA load and
unload on supports_canonical_stage2.
Both 2.3-1.1 I2V Distilled entries (full and SDNQ-4Bit) advertised
LTX2Pipeline. ltx_capabilities derives is_i2v partly from the cls_name
check, so I2V code paths (input media UI, supports_input_media, latent
prep) were not engaged for these models.
CalamitousFelicitousness
Vladimir Mandic
awsr