The modular branch returned the pipe instead of assigning it, so the video
tab and the API failed with "model not loaded" and the whole post-load tail
was skipped along with it.
The registry helpers filtered only the 'None' placeholder, so the eleven
LTX separator rows resolved as models and a POST naming one reached
load_model with a null repo. is_model() now covers both sentinel kinds.
DiffusionPipeline.download derives ignore_patterns from the passed components and
never reads the caller's, so the kwarg set in load_override was inert and diffusers
logged it as an unexpected keyword. The unsharded connectors duplicate it was meant
to skip was fetched on every LTX-2.3 mirror load.
Resolve the repo to a snapshot built from those patterns and hand from_pretrained
the folder instead of the id. That skips its download path, so the passed component
folders it would have pruned are pruned here instead. The connectors override drops
its own copy of the kwarg, which ModelMixin never read either.
Selected from the specs the pipe already declares, update_components first and
load_components for the rest, so only what the workflow declares gets fetched. Weights
land in hfcache. Covers the TODO on component names.
Encoder sharing left off: shared_te_map matches a substring of the repo name and 4b hits
_dynamic_4bit, redirecting Qwen3-VL to another repo.
load_components leaves a failed component as None and carries on, so the pipe reaches
generation short of one and fails somewhere unrelated. Compare against what the workflow
declares and refuse a pipe missing any of it. The conditioner fallback goes too, since
loading it another way just defers the failure into generation.
The ref2va checkpoint partition conditions on reference images instead
of keyframes, so it gets its own registry row and reference card, and
the video core marshals PIL images into task_args as
MiniMaxH3ImageReference. Images are converted to RGB first, since the
reference encoder reads the array raw. The keyframe path is unchanged.
Validation runs before the model load in one funnel shared by the tab
and the API, so a rejected request costs nothing: references on a
non-reference model, a reference model with nothing to condition on,
more than nine images, non-images, and aspect outside 1:4 to 4:1 all
return 400. The image path rejects a reference pipe without references
instead of reaching a transformer that was never loaded.
Add POST /sdapi/v1/video plus GET /sdapi/v1/video/models and
GET /sdapi/v1/video/file. The generation body is extracted from the
gradio handler into a keyword-only core, video_run.run, which returns a
structured result and raises typed errors; the positional generate
signature is unchanged and now adapts to the core. Omitting engine and
model drives the currently loaded checkpoint when it is video-capable,
which covers models loaded from local folders without a registry entry.
- registry helpers in models_def (find, engines, pipeline_classes,
workflow_for_class); validate_pipeline reuses the shared class set
- modular pipes stamp their workflow so out-of-registry loads dispatch
onto the modular branch
- disk switches (mp4_*) and wire switches (send_*) are independent;
artifacts above the base64 cap fall back to path plus the file route,
which is jailed to the video output directory and serves video/mp4
with range support
- always-on video scripts get bootstrapped default args, matching the
txt2img handler; missing bootstrap raised a TypeError per frame
- checkpoint overrides are rejected with a pointer to the checkpoint
endpoint; unknown engine, model and sampler names return 404 with the
valid choices
- cli/api-video.py client, test/test-video-api.py suite and a
full-test.sh entry; video mimetypes registered; rate-limit cost set
- remove the unreferenced video_ui.run_video dispatcher
The modular override forces Default instead of the None sentinel:
Default restores the model's own scheduler, which is the bespoke pair.
The reference entries request sampler: Default to match.
Modular pipelines run every stage inside one pipeline call, leaving
the ui on a single inference label. Forward hooks on the text encoder,
transformer and vae decoders now surface the current stage through
shared state, and the interrupt check runs in every stage so encodes
and tiled decodes abort promptly.
- saving a model registers a job instead of appearing idle
- group offload logs each component before the pin step instead of
only after completion
The modular text encoder quant config now excludes the vision tower.
The default skip pool covers diffusion module names, so qwen-vl vision
blocks quantized alongside the language layers; vision blocks have no
validated quantization precedent and run only for keyframe workflows.
Modular pipelines flagged as base models in the video registry cache
into the diffusers folder, so the folder scan and the model dropdown
pick them up once downloaded and the reference card reports them as
ready; video-only models keep caching into the shared cache and stay
out of the dropdown. The snapshot scan accepts modular_model_index.json
when the plain index is absent.
The modular loader passes sdnq quantization configs to load_components
as a per-component dict: transformers take the model config, the text
encoder takes the te config, and components without an entry load
unquantized. Pre-quantized repositories keep their own config, which
diffusers detects before a passed config applies.
First natively modular model: the pipeline is driven directly through
ModularPipeline, with components fetched per workflow (fl2va covers
text and first/last-frame conditioning).
- per-generation overrides snap the canvas to /32, align frames to
the 17n+5 grid and duration window, and keep the bespoke scheduler
pair
- group offload for modular pipelines applied per component in
sd_offload; re-application is a guarded no-op
- audio checkbox pops the audio decode block so decode and muxing are
skipped
- frames=1 renders a single still image: the duration floor lifts per
instance and sub-floor latents pad at the vae decoder
- progress and interrupt handling via a transformer forward pre-hook
- vae scale factor override, tuple-safe patch size
The I2V path forwards a last-frame image to the pipeline when one is supplied and the loaded pipeline accepts it, turning the run into first-last-frame interpolation. supports_last_frame() gates on the pipeline taking a last_image argument and not running expand_timesteps, which conditions on the first frame only, so a model that cannot use a last frame logs a warning instead of silently ignoring it.
Wan 2.2 A14B ships a per-model boundary_ratio (0.9 I2V, 0.875 T2V) that selects the high- or low-noise expert per step. The video and base-model image loaders both load the shipped value; the slider override is applied at generation time in set_pipeline_args, the one point both paths pass through before invoking the pipeline.
The denoising loop reads config.boundary_ratio each call, so tuning takes effect with no reload for video and base-model images alike. The slider defaults to -1, meaning use the model's value; 0 to 1 set the boundary explicitly. Single-expert stages stay load-time because they drop a transformer to free VRAM.
- monkey-patch LTX2ConnectorTransformer1d.forward to restore pre-#13564
padding logic when the upstream torch.flip pattern is detected; fixes
word-order scrambling in audio dialogue tracks
- reorganize LTX model entries into version-group separators (2.3 v1.1,
2.3 v1.0, 2.0, 0.9.x) with base/distilled subgroups; separators are
selectable no-ops handled in run_ltx
relabel toggle to 'LTX save audio' (default true) since audio always
generates on 2.x audio-capable models; the toggle gates mux only. hint
added to locale_en.json.
split add_audio_stream from write_audio. avformat_write_header runs on
first container.mux() and freezes the stream set, so audio added after
video packets has time_base=0/0 and raises 'Cannot rebase to zero time.'
atomic_save_video registers the audio stream before the encode loop.
LTX, video_run, and framepack_worker bypass process_images_inner, so
they call apply_video_interpolation explicitly before save_video. Save
receives already-inflated frames; the sentinel guard skips its own pass.
- LTX and video_run scale mp4_fps by interpolation_factor(p) so duration
is preserved instead of stretched (LTX is conditioned on source fps)
- FramePack pre-divides at gen time per get_latent_paddings, so save fps
stays at mp4_fps; worker passes p=None so save call uses
mp4_interpolate=0 to skip directly
- replaces the inline (mp4_interpolate+1) fps math at LTX with the
helper-driven equivalent
Promote RIFE interpolation from a save-time kwarg to a real stage of the
processing pipeline so per-frame work (detailer, color correction,
postprocess scripts) operates on source-rate frames and the inflated
stream becomes the saved output.
- new modules/processing_video.py with apply_video_interpolation,
interpolation_factor, expand_infotexts; PIL/tensor/numpy dispatch
- video_interpolate, video_interpolate_scale, video_interpolated fields
on StableDiffusionProcessingVideo
- process_images_inner runs the helper after the batch loop and inflates
infotexts in lockstep
- save_video in modules/video.py and modules/video_models/video_save.py
short-circuit re-interpolation when p.video_interpolated is set; the
user-facing kwarg still flows into metadata
video_save passes [-1,1]-range pixels to rife.interpolate_nchw, but
RIFE v4.25's IFNet explicitly clamps inputs to [0,1] (Head and IFNet
forward pass), turning every negative pixel value into zero. v3.9
silently extrapolated and produced soft artifacts; v4.25 produces
washout. Convert to [0,1] before the RIFE call and back to [-1,1] for
downstream save.
Both 2.3-1.1 I2V Distilled entries (full and SDNQ-4Bit) advertised
LTX2Pipeline. ltx_capabilities derives is_i2v partly from the cls_name
check, so I2V code paths (input media UI, supports_input_media, latent
prep) were not engaged for these models.
LTXCaps gains a `variant` field ('0.9', '2.0', '2.3') replacing
`is_ltx_2_3`; variant-specific branches check `caps.variant == '2.3'`
instead of grepping the model name.
ltx_util.load_upsample_2x mirrors load_upsample's contract so the 2.x
path owns a module-level cache and stops reloading ~2.3 GB every run.
The cached pipe is stamped with a synthetic
`CheckpointInfo('ltx-upsampler-2.x')` so it gets its own OffloadHook
slot and can go through apply_balanced_offload without invalidating
the main pipe's module map. The hardcoded `.to('cpu')` and post-pass
torch_gc are gone; the second apply_balanced_offload handles spill.
video_overrides comment on OzzyGT LTX-2.3 connectors states plainly
that mirrors pack weights twice by design; ignore_patterns is the
surgical workaround, not an hf_hub bug.
Drive-by: load_upsample log line used `__class__.__name__` (always
'type'); switched to `__name__`.
CalamitousFelicitousness
Vladimir Mandic
Dity0
QualiaRain
Ryan Meador