Tab runners pre-move the vae module to the gpu before generation,
which parks an on-demand component on the accelerator for the whole
denoise. Moves of a stamped module toward the accelerator now return
early; the entry bridge onloads it when its encode or decode runs.
The per-component stats block printed only from the balanced offload
path, so group mode loads showed no component classes, sizes or
quantization. Group and modular applies now print the same block once
per loaded model, with sizes measured directly when no balanced hook
map exists.
A pipeline-level move to the accelerator carries on-demand components
along with it, since modular pipelines skip only group-hooked modules;
the load path then left a resting vae on the gpu until its first use.
Bulk moves now re-evict stamped components, whose entry points onload
them when needed.
Vae-class components never take group hooks, so group mode kept them
resident on the gpu; a MiniMax-class video vae holds about 10GB that
way while running only seconds per generation. Components above 1GB
now rest in system memory: the apply_forward_hook bridge on encode and
decode fires an on-demand hook that moves the whole module to the
device, so tiled calls find every weight already loaded, and the
processing seams return it to cpu once outputs are materialized. Small
vaes stay resident since the transfer would cost more than it frees.
- placement is decided per component by measured size and requires the
entry bridge; components without it stay resident
- move_model no longer forces on-demand vaes to the gpu for
non-txt2img tasks, and full_vae_encode onloads before binding the
input, which otherwise lands on the resting device
- mode switches clear the stamp and hook in both directions
Group offload hooks report the onload device at module level while the
weights rest on cpu, so every native apply took the parameter
replacement branch in assign_weight and detached the written layers
from the hook's group bookkeeping. The activation and deactivation
walks now remove a component's group hooks before its first weight
write and reapply offload at the end of the pass: writes land in place
on the resting tensors and fresh groups snapshot the result.
- hooks come off lazily, only for components with a covered layer or a
pending backup or factor-stash restore; repeat activations with an
unchanged set leave the hooks untouched
- remove_group_offload_component follows wrapper components to the
inner model that carries the hooks
Group offload is applied per component through one engine shared by
regular and modular pipelines. Each component carries a config
signature: re-application with unchanged settings is a no-op instead
of raising before the first forward or silently keeping a stale
config, and changed settings remove the hooks and reapply. Switching
offload modes cleans up the previous mode's hooks in both directions.
- text encoders always offload at leaf level without streams, so
their weights are never held in pinned host memory
- the vae never takes group hooks and stays resident: the hooks are
forward-scoped, while pipelines enter through encode/decode and
tiled calls re-enter per tile
- new pin offload memory option: disabled pins one group at a time
instead of holding the whole module in non-pageable memory, and
modules larger than half of system memory degrade automatically
- record stream is clamped to stream mode; the standalone
combination is rejected upstream
Entries in the always and never lists are matched against the pipeline
component name (text_encoder, vae) as well as the model class name, so
one entry covers every architecture instead of needing a new class name
per model. Class entries keep working unchanged.
Never is still tested first, so a class entry there exempts a single
model from a component entry in the always list.
Autotune sweeps and kernel compiles run inside the first forward pass at a
new shape and can take minutes with no indication in the log or UI. Add
listeners on the triton autotuning and compilation knobs plus a wrap of the
per-candidate benchmark: a sweep draws a console progress bar over its
candidates and mirrors the count in the live progress text, the completion
line records the kernel, shape, duration and compile share, and standalone
compiles over 1s are logged.
- fix timer_sdnq reading bench_time from the python wrapper functions
instead of the autotuner kernel objects, which left the two matmul
autotune timers permanently empty
- restore the pre-tuning progress text across chained sweeps, so an
abandoned sweep cannot strand its own tuning label in the UI
The audio waveform rides as an attribute on the decoded sample list, but
the batch script hooks rewrap samples into a plain list before the
capture ran, so joint audio-video models muxed silent files whenever a
script runner was attached. Capture the attribute before the hooks run.
The keep-current-scheduler sentinel leaked into user-facing surfaces:
infotexts and filenames recorded Sampler: None, restoring such an
infotext pushed an invalid value into the dropdown, and the API
samplers list never offered Default at all.
- infotext always records the sampler, mapping a stray None to Default
- parse maps legacy Sampler: None infotexts to Default
- api samplers list leads with Default; the remaining excluded config
keys are shared templates, not samplers
Dynamo tracks a lifetime recompile counter per compiled function that
freed models leave climbing while their graphs and guards die, and the
compiled dequant runs fullgraph, so crossing the accumulated limit is a
hard FailOnRecompileLimitHit instead of an eager fallback; enough model
or quant switches in one process got there. unload_model_weights now
calls reset_compile_caches when the compiled dequant is active, dropping
the dead graphs and the counters in the same sweep as the unload gc.
Raised limits only move the wall; the reset removes it.
- scoped to the model unload branch: the reset is global and must only
run when the graphs' owner is being discarded
- regression test trips the wall under a lowered limit and recovers
through the same helper the unload path calls
NATIVE_DISPATCH is the documented registration surface for per-arch
native loaders and is read cross-module by the fidelity analyzer, so
the private marker signaled the opposite of its role and enforced
nothing.
safe_open exposes keys() but implements no __iter__, so the dict idiom applied
in d11a619a6 raises TypeError on every non-streamer load of a pre-quantized
checkpoint. The threaded method reaches the same loop, since load_threaded
delegates one file at a time.
Cached networks are shared objects, and network_load overwrote their
multipliers before network_deactivate ran, so fuse-mode removal recomputed
the subtraction delta with the new values: a strength edit froze at its
first applied value and a later removal left residue in the model weights.
network_load now stages the values on the net and network_activate promotes
them, so the removal pass always subtracts the delta that was applied.
Backup mode restores from stored tensors and was unaffected.
Backup-mode apply and restore installed fresh Parameters. Matmul kernel
selection is sensitive to operand placement, so the first load/remove cycle
shifted otherwise deterministic renders once per process even though every
weight restored byte-exact: bit-identical inputs entered the first post-cycle
unet forward and a different output left it. Copying into the existing
parameter keeps each touched module on its load-time allocation and drops the
per-layer transient of holding old and new weights side by side.
- assign_weight writes weight and bias installs in place when shape, dtype
and device match; quantized fallback layers keep their rebuild path
- regression test pins storage stability across the activate walk
Under a pressed balanced offload, dispatched modules hold meta tensors
whose data lives in the accelerate offload map. The factor path raised
trying to move a meta svd tensor and aborted activation mid-pass; the
legacy requantize path silently skipped those layers. Both left the
model with a partially applied network.
Rebuild the offload state with apply_balanced_offload(force) at
activate and deactivate entry: modules come back real on cpu with
hooks intact and the execution device unchanged, so both paths see
usable tensors and the next forward re-onloads under the watermark.
network_load seeded net.dyn_dim with extra_networks_default_multiplier
when no dyn_dims list was passed, so a float multiplier landed where
consumers expect a rank and slice with it. The prompt path always builds
a per-network list of ints or None, which is why the crash never fired
from the UI; any direct network_load caller hits it in both
rebuild_conventional and the sdnq factor path.
create_module built each up/down module with the default constructor,
which kaiming-initializes the parameter, then copied the stored weight
over the whole thing. The init is thrown away every time and costs about
four times the copy: 22.1ms per module against 2.5ms, or 5.8s against
0.7s over a 264-module lora, on every load.
skip_init constructs on meta and materializes uninitialized, so the copy
still fully defines the parameter. Dtype, device and values are
unchanged, including the fp32 upcast of bf16 files that the copy performs.
CalamitousFelicitousness
Dity0
Vladimir Mandic