Some paths left both to pipeline defaults, which track the current upstream
model: distilled picked up guidance it already bakes in, and 2.0 could land on
2.3's joint-sigma path.
The modular branch returned the pipe instead of assigning it, so the video
tab and the API failed with "model not loaded" and the whole post-load tail
was skipped along with it.
The registry helpers filtered only the 'None' placeholder, so the eleven
LTX separator rows resolved as models and a POST naming one reached
load_model with a null repo. is_model() now covers both sentinel kinds.
DiffusionPipeline.download derives ignore_patterns from the passed components and
never reads the caller's, so the kwarg set in load_override was inert and diffusers
logged it as an unexpected keyword. The unsharded connectors duplicate it was meant
to skip was fetched on every LTX-2.3 mirror load.
Resolve the repo to a snapshot built from those patterns and hand from_pretrained
the folder instead of the id. That skips its download path, so the passed component
folders it would have pruned are pruned here instead. The connectors override drops
its own copy of the kwarg, which ModelMixin never read either.
Selected from the specs the pipe already declares, update_components first and
load_components for the rest, so only what the workflow declares gets fetched. Weights
land in hfcache. Covers the TODO on component names.
Encoder sharing left off: shared_te_map matches a substring of the repo name and 4b hits
_dynamic_4bit, redirecting Qwen3-VL to another repo.
None from a family loader means unhandled, so the chain falls through to the folder
loader, which for a modular pipe builds an object holding only its from_config helpers
and installs it. Refuse it when none of the from_pretrained specs materialized.
load_components leaves a failed component as None and carries on, so the pipe reaches
generation short of one and fails somewhere unrelated. Compare against what the workflow
declares and refuse a pipe missing any of it. The conditioner fallback goes too, since
loading it another way just defers the failure into generation.
The ref2va checkpoint partition conditions on reference images instead
of keyframes, so it gets its own registry row and reference card, and
the video core marshals PIL images into task_args as
MiniMaxH3ImageReference. Images are converted to RGB first, since the
reference encoder reads the array raw. The keyframe path is unchanged.
Validation runs before the model load in one funnel shared by the tab
and the API, so a rejected request costs nothing: references on a
non-reference model, a reference model with nothing to condition on,
more than nine images, non-images, and aspect outside 1:4 to 4:1 all
return 400. The image path rejects a reference pipe without references
instead of reaching a transformer that was never loaded.
Add POST /sdapi/v1/video plus GET /sdapi/v1/video/models and
GET /sdapi/v1/video/file. The generation body is extracted from the
gradio handler into a keyword-only core, video_run.run, which returns a
structured result and raises typed errors; the positional generate
signature is unchanged and now adapts to the core. Omitting engine and
model drives the currently loaded checkpoint when it is video-capable,
which covers models loaded from local folders without a registry entry.
- registry helpers in models_def (find, engines, pipeline_classes,
workflow_for_class); validate_pipeline reuses the shared class set
- modular pipes stamp their workflow so out-of-registry loads dispatch
onto the modular branch
- disk switches (mp4_*) and wire switches (send_*) are independent;
artifacts above the base64 cap fall back to path plus the file route,
which is jailed to the video output directory and serves video/mp4
with range support
- always-on video scripts get bootstrapped default args, matching the
txt2img handler; missing bootstrap raised a TypeError per frame
- checkpoint overrides are rejected with a pointer to the checkpoint
endpoint; unknown engine, model and sampler names return 404 with the
valid choices
- cli/api-video.py client, test/test-video-api.py suite and a
full-test.sh entry; video mimetypes registered; rate-limit cost set
- remove the unreferenced video_ui.run_video dispatcher
The modular override forces Default instead of the None sentinel:
Default restores the model's own scheduler, which is the bespoke pair.
The reference entries request sampler: Default to match.
The local-folder skip in hf_auth_check keyed on model_index.json only,
so modular pipeline folders fell through to a hub auth check with a
filesystem path as the repo id.
Saving a modular pipeline now rewrites the component references in its
index to the destination folder, so a reload uses the saved quantized
weights instead of following the specs back to the source repositories.
The folder scan accepts modular_model_index.json for direct folders,
matching the snapshot branch, so saved pipelines list in the model
dropdown.
The progress response carries a stage label in textinfo, but the
button and hint composed their text from the job name alone, so
staged pipelines sat on a bare inference label. The stage now takes
precedence in the button label and appears next to the job in the
performance hint.
An empty prompt tokenizes to zero tokens and the conditioner fails
reshaping an empty sequence; a single space keeps the generate-on-empty
behavior other models have.
Modular pipelines run every stage inside one pipeline call, leaving
the ui on a single inference label. Forward hooks on the text encoder,
transformer and vae decoders now surface the current stage through
shared state, and the interrupt check runs in every stage so encodes
and tiled decodes abort promptly.
- saving a model registers a job instead of appearing idle
- group offload logs each component before the pin step instead of
only after completion
The modular text encoder quant config now excludes the vision tower.
The default skip pool covers diffusion module names, so qwen-vl vision
blocks quantized alongside the language layers; vision blocks have no
validated quantization precedent and run only for keyframe workflows.
Modular pipelines flagged as base models in the video registry cache
into the diffusers folder, so the folder scan and the model dropdown
pick them up once downloaded and the reference card reports them as
ready; video-only models keep caching into the shared cache and stay
out of the dropdown. The snapshot scan accepts modular_model_index.json
when the plain index is absent.
The modular loader passes sdnq quantization configs to load_components
as a per-component dict: transformers take the model config, the text
encoder takes the te config, and components without an entry load
unquantized. Pre-quantized repositories keep their own config, which
diffusers detects before a passed config applies.
Reference entries for the bf16 repo and the sdnq uint4 quant load the
modular pipeline through the standard dispatch. Image tabs run the
model in still mode with audio off; the video tab keeps its own
overrides through the shared per-generation hook. Detailer is not
supported and is disabled with a warning.
First natively modular model: the pipeline is driven directly through
ModularPipeline, with components fetched per workflow (fl2va covers
text and first/last-frame conditioning).
- per-generation overrides snap the canvas to /32, align frames to
the 17n+5 grid and duration window, and keep the bespoke scheduler
pair
- group offload for modular pipelines applied per component in
sd_offload; re-application is a guarded no-op
- audio checkbox pops the audio decode block so decode and muxing are
skipped
- frames=1 renders a single still image: the duration floor lifts per
instance and sub-floor latents pad at the vae decoder
- progress and interrupt handling via a transformer forward pre-hook
- vae scale factor override, tuple-safe patch size
CalamitousFelicitousness
Vladimir Mandic
Oleksandr Liutyi
Dity0