The active stack mode was only visible inside the stack= token of the
trace-level network check line. Add it to the load summary as its own
stack= field alongside method, mode, te and unet, carrying the mode and
its tuning (ties:0.50, klora:1.50:0.50, sum). Non-native loads report
sum, since those paths always combine as sum regardless of the setting.
EST-LoRA scores each layer by squared Frobenius energy, so a magnitude gap
between the two networks enters squared and the louder network wins nearly
every layer, starving the quieter one. The style side is now scaled by the
total-energy ratio (mirroring klora's gamma), making selection scale-invariant
so a network cannot take layers on magnitude alone. On the krea2 subject+style
pair this lifts the style network from 18% to 65% of the layer-step budget.
- lora_stack: accumulate per-mode energy totals, apply the balance in the est ramp
- test: content-louder est pair now hands over mid-schedule where raw scoring never would
- locale: note the est magnitude balance, and that a select mode gives each layer
to one network so both can be under-applied, while dense modes blend more fully
Rebuilding balanced offload before touching weights reconstructs the
OffloadHook, whose constructor prints the op=init banner and module
inventory meant for model load, so every network switch replayed the
full load-time announcement. The hook constructor and the model summary
now honor the silent flag and the network activate, deactivate, and
selection paths pass it; real model loads keep the full output.
The mode field printed the configured fuse-or-backup strategy, which predates
the factor path and reads as mode=backup on loads that took no backup at all.
It now reports what the load actually holds: backup when weight backups were
taken, fuse when fusing is active, factor when the whole load rode the svd
channel and unload just drops factors.
Requantizing a dense-combined delta into 8-bit weights is checkpoint-fragile:
on some checkpoints the round trip visibly damages the render while the same
combination hosted on the svd channel is clean. Dense-mode sets with two or
more contributing networks on a layer now ride the hosted path regardless of
bit width; single-set behavior at 8 bits and above is unchanged.
- host_candidate: dense multi-net layers qualify at any width
- suite: dense pair at int8 hosts; single non-factorable set at int8 keeps
the requantize fallback
Select pairs now ride the svd side channel on any SDNQ linear, not
only sub-8-bit ones: quantized backups are packed tensors, so the
weight rewrite path cannot recompute a winner from them and left
layers stripped mid-requantize. The sub-8-bit gate stays for dense
hosting, where requantize retains the delta at 8 bits and above.
Weight selection now only serves unquantized modules: finalize
iterates a snapshot so dead entries drop cleanly, materializes
balanced-offload modules before rewriting weights and skips modules
with stripped or quantized weights instead of corrupting the layer.
A select mode forced backup mode whenever it was merely set, so a
leftover setting changed behavior for ordinary single-network loads.
The fuse gate now engages only when the loaded set could actually
select (exactly two networks, compile permitting) or while selection
segments are still live on model layers. Re-application drops any
stale per-layer schedule so a later pass reset can never replay an
old winner over freshly applied weights. Fallback notices log per
activation instead of once per session; only the in-loop flip notice
stays latched.
The exact factor path was the only apply route with no log line; its
success read as silence. Track layers taking it beside the hosted and
fallback lists and report all three as key=value apply lines
(apply=exact/hosted/requantize); the stack fallback notices use the
same form. The suite pins its stack-mode baseline to sum so a mode
left set in user config cannot reroute tests that assume plain
summation.
Two-network subject+style sets select a winner per layer instead of
summing: scores are top-K magnitude sums (klora) or Frobenius energies
(estlora), and a timestep ramp shifts layers from the subject network
toward the style network across sampling, reduced to at most one
precomputed flip per layer per pass. On sub-8-bit SDNQ the pair rides
the side-channel as separate segments flipped in place; other layers
recompute the winner from the pristine backup, so select modes force
backup mode. Selection resets per pass from the callback setup and is
gated off under model compile. estlora's measured style-discrepancy
term is exposed as an option. Adds XYZ axes for the stack settings.
Add lora_stack_mode with ties, dare_ties, dare_linear and
magnitude_prune combination of per-network deltas when several loaded
networks target one layer; sum stays the default and the exact factor
path. Combined deltas ride the existing tail: hosted svd on sub-8-bit
SDNQ, requantize at int8 and above, direct add elsewhere. Text-encoder
layers and single-network sets keep plain summation. DARE masks draw
from per-layer sha256 seeds so re-applies and cache entries stay
deterministic; the stack settings join the activation and factor-cache
signatures so settings changes re-apply without a reload.
Hosted truncation kept the full rank cap even when the spectrum ends in
numerical zeros, padding exactly low-rank deltas (low-rank LyCORIS,
full-family diffs) up to the cap. Slice the kept factors where
cumulative capture reaches 1 - 1e-6 of the sketch total, and trim
trailing all-zero columns when attaching cache entries written before
the slice, so they collapse the same way without a format bump. The
hosted log line reports the realized rank spread when it sits below
the cap.
- flat spectra keep the cap; a rank-8 delta under cap 256 stores 8 ranks
- select segments follow the effective rank
- suite pins the collapse, the flat-spectrum guard and the padded-entry trim
A cache hit still paid full calc_updown per layer, the dominant cost of a
replayed apply. The walk now probes the pass's cache entry first through
apply_cached, which evaluates the requantize routing rule from the delta
rms stored in the entry and attaches the cached factors without assembling
the delta; layers the rule declines fall through with the assembled delta
reused for the requantize path instead of recomputing it. The entry format
gains the rms and bumps to fmt 5, so older files recompute once and
rewrite. Hit accounting stays single-count through a plain lookup plus an
explicit hit note.
- replayed krea2 LoKR apply drops calc from 9.1s to 0.2s and activate
from 39.5s fresh to 4.8s replayed, hosted/routed split identical
- three cache tests pin the calc skip on plain, mixed and dense-pair sets
Hosting truncates every non-factorable set on sub-8-bit layers, but a
delta large against the grid step whose truncation capture is low is
retained better by the grid than by the rank cap. apply_hosted now
returns such layers to the requantize path when rms(delta)/mean(step)
exceeds 0.30 and the sketch capture falls below 0.90, thresholds sized
on 610 calibrated modules across krea2 and anima. Both terms must
agree: a thin delta rounds away on the grid however low its capture,
and a low-rank delta hosts exactly however fat it is.
- scoped to pure non-factorable sets; factorable members, dense-combined
deltas and svd-channel checkpoints keep hosting
- factor cache entries memoize the decision through their stored
capture, so replays route without re-running the sketch
- routed layers log as info apart from the forced-fallback warning
- the fidelity CLI applies the same rule so its reports track the loader
- seven routing tests, constants module-level and test-overridable
Persist now fires either when every hooked layer reaches the token quota
or at a fixed denoiser-forward deadline; layers under a small token floor
are omitted and stay on plain truncation. The all-done barrier alone
never fired on models whose modulation or pooled projections see a few
tokens per forward, so hooks stayed registered forever and statistics
recollected every session. eligible_modules walks the transformer or the
unet, so unet checkpoints collect statistics at all.
- lora_sdnq: drop the unused use_previous parameter; the factor cache
store now always runs inside a begin_pass
- tests: deadline persist, token-floor omission, unet root walk
The compiled dequant specializes per factor rank, so each distinct lora
set shape paid a compile stall on switch. Pad appended factors to a
power-of-two rank ladder (multiples of 64 past the hosted cap) with zero
fill: switches inside a bucket reuse the compiled graph, and common
trained ranks land on their bucket exactly so padding is usually a
no-op. Regression tests pin the factor add inside the single compiled
graph and the bucket reuse.
Sketch hosted deltas at rank+64 with eight power iterations and keep the
top rank columns; this lands within noise of the exact decomposition at
roughly twice a sketch cost the factor cache pays once per configuration.
Bump the cache format so narrower-sketch entries reload as misses.
Rowwise int8 with fp32 scales halves cache entries; measured in output
space on real hosted deltas the roundtrip is fidelity-free (within
0.0003 of fp32 factors, bf16 storage likewise). Factors are quantized
before first use and the dequantized roundtrip is what the apply
attaches, so a fresh compute and a later cache hit stay bit-identical;
the entry format is versioned and pre-int8 entries reload as misses.
- lora_factor_cache: quantize_rowwise/dequantize_rowwise, store returns
the applied pair, fmt guard on read
- test/test-sdnq-lora-factors.py: int8 quantization test, entry-size
assertion in the roundtrip test
Hosting a non-factorable set costs one truncated svd per layer on every
fresh apply. The factors are deterministic in the checkpoint, the loaded
set, the host rank and the calibration statistics, so they persist under
data/lora-factor-cache keyed by that identity and replay bit-identically
on later applies. lora_sdnq_host_cache sets the disk budget in GB
(0 disables), least-recently-used entries are evicted past it. With the
cost paid once per configuration, svd subspace iterations rise from 2 to
4: about two thirds of the captured-energy gap to an exact decomposition
for +45% one-time compute; an exact svd measures 200-400x slower at
these shapes and is not viable.
- modules/lora/lora_factor_cache.py: signature, store, flush, eviction
- lora_sdnq.apply_hosted: fetch before computing, store after, hits in
the load summary
- cli/lora-quant-fidelity.py: matching niter for the hosted mirror
- test/test-sdnq-lora-factors.py: factor-cache category, 3 tests
New lora_sdnq_apply radio (exact, requantize) in the lora settings.
requantize keeps the previous behavior: every quantized layer takes the
dequantize-add-requantize path, with factor attach and svd hosting gated
off. A settings-only flip re-applies loaded networks: the mechanism
rides a per-module apply stamp and the network-changed signature, and
the activate fallthrough strips factors a closed gate leaves attached.
Requantize chosen by the setting logs as info instead of the
reduced-fidelity warning.
- locale hint covers fidelity and memory tradeoffs of both methods
- suite: gate, legacy routing and flip-transition tests
Plain svd truncation of hosted deltas is optimal in weight space but not
in output space: activations concentrate energy in a few input channels,
so scaling the delta by per-channel input RMS before the svd spends the
rank budget on output error instead. Statistics stream from the model's
own forwards on sub-8-bit SDNQ checkpoints and cache per checkpoint;
measured on real LoKR files this raises output-delta retention by ~0.05
at rank 256 and ~0.09 at rank 64, most on MLP down projections.
- modules/lora/lora_calib.py: capture hooks, per-checkpoint cache under
data/sdnq-calib, statistics land on layers as sdnq_calib_rms; gated by
lora_sdnq_host_calib, skipped when the model is compiled
- lora_sdnq.apply_hosted: weighted truncation when statistics exist,
calib count in the load summary
- cli/sdnq-calibrate.py: complete calibration now against a live server
- cli/lora-quant-fidelity.py --calib: hosted rho scored in the
activation-weighted norm
- test/test-sdnq-lora-factors.py: calibration category, 5 tests
Non-additive families (lokr, loha, oft, dora, full) merged into the
quantized weight and lost most of their delta on low-bit formats. On
sub-8-bit layers the set's calc_updown delta now rides the svd
side-channel as its top singular directions instead: factorable members
are subtracted out and appended exactly, so only the non-factorable
remainder is truncated. Truncation keeps the dominant part of the
effect and drops an orthogonal residual, where requantize keeps the
grid extrema and adds grid-shift noise of the delta's own magnitude;
on real lokr files retention rises from 0.04 to about 0.5 at the
default rank.
Hosted layers take no weight backup and unload bit-exactly. The svd
runs under a forked rng so generation seeds are unaffected. At 8 bits
and above requantize retains most of the delta and remains the path.
lora_sdnq_host_rank caps the hosted rank; 0 disables hosting.
- restore stashed svd factors onto the layer's current device; the
stash tuple does not follow module device moves, so an offload
between apply and remove left restored factors on a stale device
- recheck factor shapes for layers already in factor mode, so a
malformed stacked network downgrades to the legacy path instead of
raising in the concat
- clear the fallback log at activate entry so a raise mid-pass cannot
leak stale entries into the next report
- pin both behaviors in the suite and state the compute-dtype fidelity
floor in the module docstring
Baking a lora into a quantized weight requantizes it, and on low-bit
formats round-to-nearest erases sub-step deltas (uint4 retains roughly
2/group_size of the signal). Plain lora deltas now ride the sdnq svd
side-channel: factors append to svd_up/svd_down with the down factor
hadamard-rotated, applied by the dequantizer at full precision in every
forward mode. Apply and remove are exact and take no weight backup.
- non-factorable families (dora, lokr, loha, oft, cp mid, dense bias)
fall back to requantize with a per-pass summary warning
- native fuse now honors the quantized-model guard; fuse requantized in
place on every network swap and accumulated drift
- layers that fell back on a mixed set restore from backup before
re-entering the factor path; untargeted quantized layers are no
longer flagged
- test/test-sdnq-lora-factors.py pins the erasure law, factor-path
exactness, memory accounting and set transitions
A delta that does not fit its target module cannot apply, and applying only
the layers that do fit leaves the model in a state nothing was trained for,
so try_load_chain drops the whole file when any family reports a mismatch.
Bias deltas were never checked against the target bias and could only surface
at apply time; a module with no bias stays a non-mismatch, since whole
architectures are built bias=False.
- check bias deltas against the module bias in the lora, norm and full loaders
- carry the mismatch count on the network so the chain can refuse the file
- record refused writes in the infotext so a partial apply is not read as clean
- point the krea2 full-diff test at a module that has a bias
network_add_weights defaulted its base tensor to self.weight for the bias
delta as well, so in fuse mode a diff_b was added to the weight matrix and
the result written into the bias. Layers where in and out differ threw a
shape error and had the weight matrix installed as their bias, square layers
broadcast silently, and either way the summary still counted the delta as
applied.
- pick the base tensor from the bias flag
- name the layer, target and both shapes in the mismatch error
- return which of (weight, bias) took a write, count the rest as refused
- report refused= on partially applied and partially removed networks
- cover both apply paths in test/test-lora-apply.py
Group offload hooks report the onload device at module level while the
weights rest on cpu, so every native apply took the parameter
replacement branch in assign_weight and detached the written layers
from the hook's group bookkeeping. The activation and deactivation
walks now remove a component's group hooks before its first weight
write and reapply offload at the end of the pass: writes land in place
on the resting tensors and fresh groups snapshot the result.
- hooks come off lazily, only for components with a covered layer or a
pending backup or factor-stash restore; repeat activations with an
unchanged set leave the hooks untouched
- remove_group_offload_component follows wrapper components to the
inner model that carries the hooks
NATIVE_DISPATCH is the documented registration surface for per-arch
native loaders and is read cross-module by the fidelity analyzer, so
the private marker signaled the opposite of its role and enforced
nothing.
Cached networks are shared objects, and network_load overwrote their
multipliers before network_deactivate ran, so fuse-mode removal recomputed
the subtraction delta with the new values: a strength edit froze at its
first applied value and a later removal left residue in the model weights.
network_load now stages the values on the net and network_activate promotes
them, so the removal pass always subtracts the delta that was applied.
Backup mode restores from stored tensors and was unaffected.
Backup-mode apply and restore installed fresh Parameters. Matmul kernel
selection is sensitive to operand placement, so the first load/remove cycle
shifted otherwise deterministic renders once per process even though every
weight restored byte-exact: bit-identical inputs entered the first post-cycle
unet forward and a different output left it. Copying into the existing
parameter keeps each touched module on its load-time allocation and drops the
per-layer transient of holding old and new weights side by side.
- assign_weight writes weight and bias installs in place when shape, dtype
and device match; quantized fallback layers keep their rebuild path
- regression test pins storage stability across the activate walk
Under a pressed balanced offload, dispatched modules hold meta tensors
whose data lives in the accelerate offload map. The factor path raised
trying to move a meta svd tensor and aborted activation mid-pass; the
legacy requantize path silently skipped those layers. Both left the
model with a partially applied network.
Rebuild the offload state with apply_balanced_offload(force) at
activate and deactivate entry: modules come back real on cpu with
hooks intact and the execution device unchanged, so both paths see
usable tensors and the next forward re-onloads under the watermark.
network_load seeded net.dyn_dim with extra_networks_default_multiplier
when no dyn_dims list was passed, so a float multiplier landed where
consumers expect a rank and slice with it. The prompt path always builds
a per-network list of ints or None, which is why the crash never fired
from the UI; any direct network_load caller hits it in both
rebuild_conventional and the sdnq factor path.
create_module built each up/down module with the default constructor,
which kaiming-initializes the parameter, then copied the stored weight
over the whole thing. The init is thrown away every time and costs about
four times the copy: 22.1ms per module against 2.5ms, or 5.8s against
0.7s over a 264-module lora, on every load.
skip_init constructs on meta and materializes uninitialized, so the copy
still fully defines the parameter. Dtype, device and values are
unchanged, including the fp32 upcast of bf16 files that the copy performs.
The native loader entry log repeated the name and full file path already
printed one line earlier by network_load. Remove it and fold cache-hit
status into the network_load announce line, so a native load emits one
starting line plus the result line instead of three with a duplicated
path.
The Krea 2 transformer keeps checkpoint-style module names while the
official krea/Krea-2-LoRA releases are saved with upstream-diffusers
names, so all 264 modules failed to bind and the LoRAs silently did
nothing. Krea 2 is the only native-LoRA arch with an sdnext-owned
transformer, so its module names diverge from the diffusers ecosystem.
- native_adapter.resolve_group_targets consults the arch resolver first
for passthrough prefixes, falling back to verbatim binding; a no-op
for arches that load the diffusers class
- krea2_lora maps diffusers attn/ff/text_fusion/embedder names onto the
checkpoint module tree; checkpoint-named LoRAs still bind verbatim
- add test/test-krea2-native-adapters.py
A full-weight extraction on Z-Image bound 308 modules and applied 172 of
them, silently dropping the rest, and left 71 more unmapped.
assign_network_names_to_compvis_modules puts every transformer module in
network_layer_mapping but skips stamping network_layer_name on norms,
which is the attribute the apply pass keys off. try_load_full bound those
modules through the mapping and they then never applied; stamp them
loader-locally, as try_load_norm already does.
Z-Image also names three module groups differently from the diffusers
tree: the qk-norms (q_norm/k_norm vs norm_q/norm_k), and the patch
embedder and final layer, which live in ModuleDicts keyed by
"{patch_size}-{f_patch_size}" and so carry a key the checkpoint has no
notion of. Read that key from the live model rather than hardcoding it.
The counts close exactly: 68 qk-norms plus 3 non-block targets are the 71
that went unmapped.
LyCORIS extraction with use_sparse_bias saves bias_indices/bias_values/
bias_size per module: the sparse weight-shaped remainder of the SVD
extraction, named bias for historical reasons. The keys were dropped by
both loader paths, so extracted adapters applied without the residual
correction; the dense-bias branch in finalize_updown that consumes it
was unreachable.
- rebuild the COO tensor in NetworkModule.__init__ (int16 indices cast
to long), shared by the native and generic loaders; kept sparse so
the dense += sparse in finalize_updown materializes per module at
apply instead of near-model-size densification at load
- accept the triplet suffixes in LORA_SUFFIXES; fused targets skip
with the weight-shaped-bias warning
- cover an extraction-faithful numeric round-trip and the fused skip
in the offline suite
ai-toolkit DoRA saves lora_A/B plus a 1-D per-output magnitude key in
place of alpha; PEFT and diffusers name the same quantity
lora_magnitude_vector. Neither key was in the suffix table, so such
adapters loaded as plain LoRA with the magnitude renormalization
silently missing. The semantics match LyCORIS wd_on_out=True row norms,
so both keys convert onto the existing dora_scale path.
- accept .magnitude and .lora_magnitude_vector in LORA_SUFFIXES and
convert at group level in try_load_lora
- reshape 1-D vectors to (out, 1): on square layers the apply-time
orientation detection would otherwise renormalize the wrong axis
- cover square-layer numeric equality, fused-qkv slicing and the PEFT
key form in the offline suite
finalize_updown ran apply_weight_decompose on the unscaled delta and
multiplied the result by alpha/rank afterward. LyCORIS and ComfyUI both
bake alpha/rank into the diff before computing the row norms, so any
DoRA with alpha != rank renormalized against the wrong merged weight
(64% relative delta error for kohya-style alpha=1 rank=8; exact only
when alpha == rank, which full-matrix LoKR forces).
- scale updown by calc_scale() before apply_weight_decompose; apply
only the multiplier afterward
- multiplier lerps the full merged delta (0 disables, 1 equals the
trainer output); LyCORIS weight-mode ratio interpolation leaves the
diff applied at multiplier 0 and is not used
- add a numeric regression test mirroring the LyCORIS forward reference
A diff_b bias delta on a fused BFL target passed through whole and
failed at apply with a shape mismatch. diff_b stores one value per
output feature, so it partitions with the fused rows exactly like the
up-weight; slice it with the chunk in the LoRA loader. The legacy
weight-shaped bias key (LyCORIS sparse-residual heritage) has no
defined partition on a fused target and no known emitter pairs it with
chunk-capable families, so the group is skipped with a warning in the
LoRA, LoKR and LoHA loaders.
- add slice_bias_delta beside slice_dora_scale; warn and skip
non-per-output diff_b shapes
- cover sliced diff_b flowing out as ex_bias and the legacy-bias skip
in the offline suite
A LoKR group whose Kronecker product does not fit the resolved module
previously bound anyway and failed at apply time as a caught per-module
error, leaving the adapter partially applied with only an error log.
Reject the group at load with a warning instead, matching the LoRA
path's shapes_match gate.
- lokr_kron_shape derives (out, in_flat) from full, rank-decomposed or
Tucker-rebuilt factors, folding conv kernel dims into in_flat
- lokr_shapes_match honors SDNQ original shapes and chunk partitions:
equal chunks need total * out rows, row-range slices an exact range;
the input dim is never chunked
- cover non-fused and fused rejection in the offline suite
LyCORIS wd=True saves a dora_scale companion for LoRA/LoHA/LoKR; on
fused BFL targets the chunk paths passed it through whole, so apply
failed with a shape mismatch and the module was dropped. Per-output
magnitudes (wd_on_out=True, the default) partition exactly with the
fused rows; per-input magnitudes couple the chunks through shared
column norms and have no exact split.
- slice per-output dora_scale rows with the chunk in the LoRA, LoKR
and LoHA loaders
- skip per-input DoRA on fused targets with a specific warning
- cover sliced and skipped orientations in the offline suite
The lycoris_ save format is arch-independent: LyCORIS standalone wraps
the loaded diffusers model and emits the wrapped module path with dots
as underscores, so verbatim passthrough is correct for any arch. Only
flux2 handled it; zimage, chroma, ernie and krea2 reported such files
as not loaded.
- add lycoris_ to KNOWN_PREFIXES_DEFAULT and PASSTHROUGH_PREFIXES_DEFAULT
- drop flux2's per-arch prefix append and resolve_targets branch
- add lycoris_ to ANIMA_PREFIXES (anima replaces the default tuple);
network_prefix_for already routes it to the transformer namespace
- cover the passthrough with a zimage loader test
CalamitousFelicitousness
Vladimir Mandic
Dity0