The rules the walk depends on were spread across the comments that
happened to need them, and the attributes it keeps on the model's modules
were written from four files with the ownership recorded nowhere. Both
are stated once in the module docstring, including the identity the
factor cache keys its pass entry on and the three writers that share the
svd tensors.
Deactivate carried its own shorter copy of the component list, missing
text_encoder_4 and transformer_2. Nothing depends on the difference
today: layer names are stamped only on text_encoder, text_encoder_2,
unet, transformer and llm_adapter, so modules in the other components
are skipped by both passes. Sharing the list keeps the two from drifting
apart if that stamping ever widens.
The mode shown in the load and unload lines was derived at print time
from the live fuse setting, so a set applied under one setting was
reported under whatever the setting said later, and the unload line
described the pass that was about to replace it rather than the one
being removed. The pass records the mode it actually used.
last_backup_size was created on lora_common by assignment from
networks.py and read back through a getattr default; both fields are
declared where they live now.
The error limiter halts a pass by raising, and nothing between the raise
and the caller put the model back. A halted pass left group offload hooks
stripped from every component the walk had reached, left a sequential
model on the cpu with offload disabled, and left the counters other
modules read describing the pass before it.
The epilogue moves into finish_pass under a finally, so the model returns
to its offload mode and the counters describe the pass that just ran. The
abort still reaches the caller.
Pass state is reset in one place in lora_sdnq now. Two of the six
accumulators were not being cleared at the start of a pass, and a stale
routed layer suppresses the fallback count for that layer next time.
The per-module walk carried four mechanisms inline, each repeating the
same tail: count the layer, stamp the pair that marks it current, advance
the bar, continue. Five copies of that tail and three of the backup probe
put the deepest arm nine levels in.
Each mechanism is now a function that either takes the layer or declines
to the next, and the walk reads as the four of them in order. The pass
state they share moves onto one object built before the walk starts, with
the accept tail, the stamp and the bar tick as its methods. That takes
network_activate from 218 lines to 55, none of it deeper than the module
loop.
Two shapes are deliberately not folded into that tail: the weight path
counts weights and bias separately and tracks what the module refused,
and the factor-strip restore stamps without counting. Hosting hands a
declined delta back rather than leaving it in a flag, so a pair of Nones
still reads as assembled and no layer is calculated twice.
Both passes opened with the same four steps written twice: bring the
model into a writable state, enumerate the components to walk, open a
progress bar, and probe a weight backup before restoring it. Pull each
into a helper and call it from both entry points. The component
collector keeps the two lists it is given, so deactivate still walks its
own shorter set, and promotion now clears the staged config it consumed.
The restore field had no writer and add() had no caller. Deactivate
assigned its elapsed time where activate accumulates, so a generation
that unloaded more than once reported only the last pass; both are
zeroed together when the generation ends.
The mechanism gate tests assert select_candidate declines under
requantize, and the apply-method hint names the cache option among
those the requantize choice disables.
<lora:name:1.0:lbw=VALUE> scales each targeted layer's delta by a slot of a
per-architecture block vector. VALUE is a preset name, a scalar, or a comma
vector; presets stretch onto the block count of the current model and the
a1111 17-slot and 12-slot layouts are accepted on sd and sdxl. The factor
enters through the module multiplier, so every apply path carries it: the
exact factor channel, hosting, requantize routing, dense stack combines and
select scoring.
- modules/lora/lora_blocks.py: slot classification from network_layer_mapping
(namespace-first, anchored chain prefixes), preset resolution reusing the
merge block-weight tables with BASE forced neutral, generated classic
segment names plus DOUBLE/SINGLE chain names, per-model memoization
- the raw spec stages through pending_config and promotes with the other
multipliers, keeping fuse removal consistent
- block weights join the activation signature, the per-module apply stamp
and the factor cache identity; entries without block weights keep their
existing signature bytes
- non-native load methods warn once and ignore the argument
Select scoring staged full fp32 copies, an abs copy and top-k workspace per
layer (hundreds of MB of transients that collide with block swapping on
offloaded denoisers) and recomputed scores from freshly assembled deltas on
every apply, which kept select modes out of the factor-cache fast path.
- score_pair: row-chunked fp32 interiors, fp64 accumulators, one device sync
- select scores persist in the factor cache as additive per-layer records
under the existing configuration signature
- apply_select_cached and register_weight_pair_cached replay a pair without
assembling deltas; the weight-kind winner is still computed at schedule
time
The weight-kind select branch counted its backup at its own call site
and again at the shared backup call when registration fell through, so
the reported backup size double-counted those layers; the size now
lands with whichever path keeps the layer. A pair the svd channel
cannot carry (bias delta or malformed member) previously dropped to
the sum paths with no trace; the fallthrough now says so once.
A restore-only activation walk carries zero networks, so the
networks=0 required=2 fallback warning fired on every network-free
generation whenever a select stack mode was set. The gate now
short-circuits at zero; the 1-and-3-network warnings that remain
meaningful are unchanged.
The klora and estlora balance factors accumulated across registrations
without ever resetting, so a multiplier change or pair swap blended the
previous registration into every later schedule. Balances now sum over
the live entries at finalize time, which keeps drop and re-register
consistent by construction.
- key flips one step early: step callbacks fire after the denoise, so
the winner is now live during the crossover step forward and a
final-step crossover engages instead of expiring
- include the calibration toggle in the factor cache signature so a hit
never replays factors computed under the other setting
- fall back to summation with a warning when hosting is disabled on a
quantized model instead of registering schedules that cannot flip
- drop the unused score_topk helper
Select modes left no trace distinguishable from plain summation: apply_select
counted its layers on the exact path, and the mode field in the load summary
reflects the requested setting rather than what executed. A flip count can only
come from a populated schedule.
- report layers, initial style picks, flips, steps and gamma from finalize
- deduplicate on content, since the schedule rebuilds on every pass
- give select its own apply counter instead of inflating apply=exact
Rebuilding balanced offload before touching weights reconstructs the
OffloadHook, whose constructor prints the op=init banner and module
inventory meant for model load, so every network switch replayed the
full load-time announcement. The hook constructor and the model summary
now honor the silent flag and the network activate, deactivate, and
selection paths pass it; real model loads keep the full output.
The mode field printed the configured fuse-or-backup strategy, which predates
the factor path and reads as mode=backup on loads that took no backup at all.
It now reports what the load actually holds: backup when weight backups were
taken, fuse when fusing is active, factor when the whole load rode the svd
channel and unload just drops factors.
Select pairs now ride the svd side channel on any SDNQ linear, not
only sub-8-bit ones: quantized backups are packed tensors, so the
weight rewrite path cannot recompute a winner from them and left
layers stripped mid-requantize. The sub-8-bit gate stays for dense
hosting, where requantize retains the delta at 8 bits and above.
Weight selection now only serves unquantized modules: finalize
iterates a snapshot so dead entries drop cleanly, materializes
balanced-offload modules before rewriting weights and skips modules
with stripped or quantized weights instead of corrupting the layer.
A select mode forced backup mode whenever it was merely set, so a
leftover setting changed behavior for ordinary single-network loads.
The fuse gate now engages only when the loaded set could actually
select (exactly two networks, compile permitting) or while selection
segments are still live on model layers. Re-application drops any
stale per-layer schedule so a later pass reset can never replay an
old winner over freshly applied weights. Fallback notices log per
activation instead of once per session; only the in-loop flip notice
stays latched.
The exact factor path was the only apply route with no log line; its
success read as silence. Track layers taking it beside the hosted and
fallback lists and report all three as key=value apply lines
(apply=exact/hosted/requantize); the stack fallback notices use the
same form. The suite pins its stack-mode baseline to sum so a mode
left set in user config cannot reroute tests that assume plain
summation.
Two-network subject+style sets select a winner per layer instead of
summing: scores are top-K magnitude sums (klora) or Frobenius energies
(estlora), and a timestep ramp shifts layers from the subject network
toward the style network across sampling, reduced to at most one
precomputed flip per layer per pass. On sub-8-bit SDNQ the pair rides
the side-channel as separate segments flipped in place; other layers
recompute the winner from the pristine backup, so select modes force
backup mode. Selection resets per pass from the callback setup and is
gated off under model compile. estlora's measured style-discrepancy
term is exposed as an option. Adds XYZ axes for the stack settings.
Add lora_stack_mode with ties, dare_ties, dare_linear and
magnitude_prune combination of per-network deltas when several loaded
networks target one layer; sum stays the default and the exact factor
path. Combined deltas ride the existing tail: hosted svd on sub-8-bit
SDNQ, requantize at int8 and above, direct add elsewhere. Text-encoder
layers and single-network sets keep plain summation. DARE masks draw
from per-layer sha256 seeds so re-applies and cache entries stay
deterministic; the stack settings join the activation and factor-cache
signatures so settings changes re-apply without a reload.
A cache hit still paid full calc_updown per layer, the dominant cost of a
replayed apply. The walk now probes the pass's cache entry first through
apply_cached, which evaluates the requantize routing rule from the delta
rms stored in the entry and attaches the cached factors without assembling
the delta; layers the rule declines fall through with the assembled delta
reused for the requantize path instead of recomputing it. The entry format
gains the rms and bumps to fmt 5, so older files recompute once and
rewrite. Hit accounting stays single-count through a plain lookup plus an
explicit hit note.
- replayed krea2 LoKR apply drops calc from 9.1s to 0.2s and activate
from 39.5s fresh to 4.8s replayed, hosted/routed split identical
- three cache tests pin the calc skip on plain, mixed and dense-pair sets
New lora_sdnq_apply radio (exact, requantize) in the lora settings.
requantize keeps the previous behavior: every quantized layer takes the
dequantize-add-requantize path, with factor attach and svd hosting gated
off. A settings-only flip re-applies loaded networks: the mechanism
rides a per-module apply stamp and the network-changed signature, and
the activate fallthrough strips factors a closed gate leaves attached.
Requantize chosen by the setting logs as info instead of the
reduced-fidelity warning.
- locale hint covers fidelity and memory tradeoffs of both methods
- suite: gate, legacy routing and flip-transition tests
Non-additive families (lokr, loha, oft, dora, full) merged into the
quantized weight and lost most of their delta on low-bit formats. On
sub-8-bit layers the set's calc_updown delta now rides the svd
side-channel as its top singular directions instead: factorable members
are subtracted out and appended exactly, so only the non-factorable
remainder is truncated. Truncation keeps the dominant part of the
effect and drops an orthogonal residual, where requantize keeps the
grid extrema and adds grid-shift noise of the delta's own magnitude;
on real lokr files retention rises from 0.04 to about 0.5 at the
default rank.
Hosted layers take no weight backup and unload bit-exactly. The svd
runs under a forked rng so generation seeds are unaffected. At 8 bits
and above requantize retains most of the delta and remains the path.
lora_sdnq_host_rank caps the hosted rank; 0 disables hosting.
- restore stashed svd factors onto the layer's current device; the
stash tuple does not follow module device moves, so an offload
between apply and remove left restored factors on a stale device
- recheck factor shapes for layers already in factor mode, so a
malformed stacked network downgrades to the legacy path instead of
raising in the concat
- clear the fallback log at activate entry so a raise mid-pass cannot
leak stale entries into the next report
- pin both behaviors in the suite and state the compute-dtype fidelity
floor in the module docstring
Baking a lora into a quantized weight requantizes it, and on low-bit
formats round-to-nearest erases sub-step deltas (uint4 retains roughly
2/group_size of the signal). Plain lora deltas now ride the sdnq svd
side-channel: factors append to svd_up/svd_down with the down factor
hadamard-rotated, applied by the dequantizer at full precision in every
forward mode. Apply and remove are exact and take no weight backup.
- non-factorable families (dora, lokr, loha, oft, cp mid, dense bias)
fall back to requantize with a per-pass summary warning
- native fuse now honors the quantized-model guard; fuse requantized in
place on every network swap and accumulated drift
- layers that fell back on a mixed set restore from backup before
re-entering the factor path; untargeted quantized layers are no
longer flagged
- test/test-sdnq-lora-factors.py pins the erasure law, factor-path
exactness, memory accounting and set transitions
A delta that does not fit its target module cannot apply, and applying only
the layers that do fit leaves the model in a state nothing was trained for,
so try_load_chain drops the whole file when any family reports a mismatch.
Bias deltas were never checked against the target bias and could only surface
at apply time; a module with no bias stays a non-mismatch, since whole
architectures are built bias=False.
- check bias deltas against the module bias in the lora, norm and full loaders
- carry the mismatch count on the network so the chain can refuse the file
- record refused writes in the infotext so a partial apply is not read as clean
- point the krea2 full-diff test at a module that has a bias
network_add_weights defaulted its base tensor to self.weight for the bias
delta as well, so in fuse mode a diff_b was added to the weight matrix and
the result written into the bias. Layers where in and out differ threw a
shape error and had the weight matrix installed as their bias, square layers
broadcast silently, and either way the summary still counted the delta as
applied.
- pick the base tensor from the bias flag
- name the layer, target and both shapes in the mismatch error
- return which of (weight, bias) took a write, count the rest as refused
- report refused= on partially applied and partially removed networks
- cover both apply paths in test/test-lora-apply.py
Group offload hooks report the onload device at module level while the
weights rest on cpu, so every native apply took the parameter
replacement branch in assign_weight and detached the written layers
from the hook's group bookkeeping. The activation and deactivation
walks now remove a component's group hooks before its first weight
write and reapply offload at the end of the pass: writes land in place
on the resting tensors and fresh groups snapshot the result.
- hooks come off lazily, only for components with a covered layer or a
pending backup or factor-stash restore; repeat activations with an
unchanged set leave the hooks untouched
- remove_group_offload_component follows wrapper components to the
inner model that carries the hooks
Cached networks are shared objects, and network_load overwrote their
multipliers before network_deactivate ran, so fuse-mode removal recomputed
the subtraction delta with the new values: a strength edit froze at its
first applied value and a later removal left residue in the model weights.
network_load now stages the values on the net and network_activate promotes
them, so the removal pass always subtracts the delta that was applied.
Backup mode restores from stored tensors and was unaffected.
Under a pressed balanced offload, dispatched modules hold meta tensors
whose data lives in the accelerate offload map. The factor path raised
trying to move a meta svd tensor and aborted activation mid-pass; the
legacy requantize path silently skipped those layers. Both left the
model with a partially applied network.
Rebuild the offload state with apply_balanced_offload(force) at
activate and deactivate entry: modules come back real on cpu with
hooks intact and the execution device unchanged, so both paths see
usable tensors and the next forward re-onloads under the watermark.
Network activation ran after prompt encoding, so text encoder lora
weights never affected embeds on the first generation and the stale
result was then served from the embed cache. The trailing unfiltered
activate in network_load also overrode the te exclude filter, so the
lora_apply_te setting was never honored.
- parse and activate networks in process_base before pipeline args are built
- activate_filtered gates text encoder components on per-request or global
lora_apply_te; used by base, hires, detailer and faceid call sites
- network_load accepts activate=False for callers that run their own
deactivate/activate sequence with include/exclude
- network_activate walks excluded components in restore-only mode so a
filtered text encoder reverts to backup instead of keeping stale deltas
- loaded_loras cache is single-entry since per-filter entries go stale when
the setting toggles
- prompt embed cache key includes the effective lora_apply_te value
applied_layers is cleared and re-populated on every network_activate call.
With lora_apply_te=True the second activate (TE-only pass) finds all
modules already at the target state and skips them all, leaving
applied_layers empty and breaking the restore trigger on the next gen.
native_active is set from loaded_networks at the end of activate, so it
survives idempotent re-runs and only flips false after the restore call
clears loaded_networks.
Dispatch anima loras through a dedicated native loader covering kohya,
bfl/ai-toolkit, and hybrid (bfl with alpha plus qwen3 text encoder)
formats. Cosmos 2.0 path rename is mirrored from diffusers in flat
(underscore) form so rewritten paths match network_layer_mapping keys
without further conversion.
Split model_type from cosmos to anima so a future base-cosmos2 lora
path stays separable. Update flow_models, taesd supported list, and the
taesd wanvideo bucket so samplers and preview decoding keep working
after the split.
Extend assign_network_names_to_compvis_modules to walk pipe.llm_adapter
under the lora_llm_adapter_ prefix, and add llm_adapter to
default_components so activate and deactivate include it for anima
models while staying inert elsewhere via the existing getattr guards.
CalamitousFelicitousness
Vladimir Mandic
awsr
Disty0