Group offload ignored the offload exclusion settings and its options took
effect only at the next model load. Components named by the never-offload
list or the model-type exclusion now stay resident on the accelerator,
with matching shared between the balanced hook and the group roles. The
offload options re-place loaded components when changed: group modes
re-run the placement pass, and balanced mode rebuilds its hook, which
snapshots the exclusion lists at construction.
The force sweep enumerated pipeline components and moved every module to
cpu, including components with no hook to bring them back. It now moves
only modules stamped for on-demand placement, walks the pipe variants
through the shared enumeration instead of the components property that can
raise, keeps going past a component that fails to move, and collects
garbage once per sweep rather than per module.
Group offload placed components by name matching, which sent every modular
text encoder through a second apply with the denoiser profile and missed
vae-class components not named vae. Placement now derives from the component:
an encode or decode entry bridge and an upstream group-offload opt-out both
take whole-module on-demand placement, an undecorated encode or decode entry
stays resident with a warning, denoiser slot names take the per-step profile,
and everything else takes the leaf profile. One pass serves classic and
modular pipelines, enumerating modular components from their specs, with
hooks landing on text encoder wrappers rather than their inner models. A
signature sentinel marks a component mid-apply; a failed apply is torn down
on the next pass. The pass logs the placement of every component.
Modular pipelines run every stage inside one pipeline call, leaving
the ui on a single inference label. Forward hooks on the text encoder,
transformer and vae decoders now surface the current stage through
shared state, and the interrupt check runs in every stage so encodes
and tiled decodes abort promptly.
- saving a model registers a job instead of appearing idle
- group offload logs each component before the pin step instead of
only after completion
First natively modular model: the pipeline is driven directly through
ModularPipeline, with components fetched per workflow (fl2va covers
text and first/last-frame conditioning).
- per-generation overrides snap the canvas to /32, align frames to
the 17n+5 grid and duration window, and keep the bespoke scheduler
pair
- group offload for modular pipelines applied per component in
sd_offload; re-application is a guarded no-op
- audio checkbox pops the audio decode block so decode and muxing are
skipped
- frames=1 renders a single still image: the duration floor lifts per
instance and sub-floor latents pad at the vae decoder
- progress and interrupt handling via a transformer forward pre-hook
- vae scale factor override, tuple-safe patch size
The guard compared component size against half of total system memory and
silently dropped to unpinned streaming above that. Total is the wrong
quantity: a component that fits comfortably in free memory gets denied, and
unpinned leaf streaming is slow out of proportion to its size because every
per-leaf transfer pays staging cost.
Budget against memory free at apply time less a reserve, and cache the
verdict on the module since a granted pin lowers the same reading it derives
from. A denied pin now also drops streams and falls back to block_level,
since leaf plus stream is only the right shape when weights are pinned, and
logs the per-step transfer volume so the cost of the slow path is visible up
front.
Model switches kept most of the previous model resident, and the next
load could stall in kernel reclaim while the freed memory was still held.
- strip group offload hooks in disable_offload so the meta move at unload
actually frees component weights; hook removal resolves wrapper
components that carry hooks on the inner model
- flush the torch pinned host cache in torch_gc so freed streaming
buffers return to the OS instead of staying cached in-process
- skip the pipe-level accelerator move for group-managed pipes: the
offload engine already placed every component, and the move only
dragged on-demand components to the accelerator for the trailing
eviction to undo
The per-component stats block printed only from the balanced offload
path, so group mode loads showed no component classes, sizes or
quantization. Group and modular applies now print the same block once
per loaded model, with sizes measured directly when no balanced hook
map exists.
Vae-class components never take group hooks, so group mode kept them
resident on the gpu; a MiniMax-class video vae holds about 10GB that
way while running only seconds per generation. Components above 1GB
now rest in system memory: the apply_forward_hook bridge on encode and
decode fires an on-demand hook that moves the whole module to the
device, so tiled calls find every weight already loaded, and the
processing seams return it to cpu once outputs are materialized. Small
vaes stay resident since the transfer would cost more than it frees.
- placement is decided per component by measured size and requires the
entry bridge; components without it stay resident
- move_model no longer forces on-demand vaes to the gpu for
non-txt2img tasks, and full_vae_encode onloads before binding the
input, which otherwise lands on the resting device
- mode switches clear the stamp and hook in both directions
Group offload hooks report the onload device at module level while the
weights rest on cpu, so every native apply took the parameter
replacement branch in assign_weight and detached the written layers
from the hook's group bookkeeping. The activation and deactivation
walks now remove a component's group hooks before its first weight
write and reapply offload at the end of the pass: writes land in place
on the resting tensors and fresh groups snapshot the result.
- hooks come off lazily, only for components with a covered layer or a
pending backup or factor-stash restore; repeat activations with an
unchanged set leave the hooks untouched
- remove_group_offload_component follows wrapper components to the
inner model that carries the hooks
Group offload is applied per component through one engine shared by
regular and modular pipelines. Each component carries a config
signature: re-application with unchanged settings is a no-op instead
of raising before the first forward or silently keeping a stale
config, and changed settings remove the hooks and reapply. Switching
offload modes cleans up the previous mode's hooks in both directions.
- text encoders always offload at leaf level without streams, so
their weights are never held in pinned host memory
- the vae never takes group hooks and stays resident: the hooks are
forward-scoped, while pipelines enter through encode/decode and
tiled calls re-enter per tile
- new pin offload memory option: disabled pins one group at a time
instead of holding the whole module in non-pageable memory, and
modules larger than half of system memory degrade automatically
- record stream is clamped to stream mode; the standalone
combination is rejected upstream
Entries in the always and never lists are matched against the pipeline
component name (text_encoder, vae) as well as the model class name, so
one entry covers every architecture instead of needing a new class name
per model. Class entries keep working unchanged.
Never is still tested first, so a class entry there exempts a single
model from a component entry in the always list.
Vladimir Mandic
CalamitousFelicitousness
Disty0
awsr
Ryan Meador