The parser no longer takes reference-name prefixes to tell bare
reference keys from bare diffusers keys. Any bare key carries the
sentinel and the arch resolver renames what it knows and passes the rest
through. Flux2 keeps its list for file-format detection only.
A bare key that matches no known prefix is parsed with the
bare-diffusers sentinel and handed to the resolver instead of being
dropped at parse time. The per-arch lists of bare diffusers prefixes are
gone, and a path that names no live module counts as unmapped.
A ChunkSpec can reorder equal row blocks of the rows it selects, so an
arch declares a swapped SwiGLU projection on the target instead of
permuting the state dict first. Only the LoRA family applies it; the
others skip a reordered target with a warning.
The reference fc1 is a fused [gate; value] SwiGLU projection and the
diffusers port stores [value; gate]. The native mapping did not swap the
halves, so gate and value deltas landed on each other's rows. The
mapping now also renames the standalone projections, reads a
metadata-only alpha, and accepts the musubi, peft dit and diffusers-named
layouts.
SDNQ codebook layers keep their Lloyd levels in the scale slot, so reading
scale.mean() as the grid step returned the levels' near-zero mean and sent
sub-step deltas to requantize, where the grid erases them. grid_step returns
the mean adjacent-level gap for those layers and the plain scale mean otherwise.
The seventeen lines that rebuild w1 and w2 from whatever the file stored
were copied into all three lokr variants, character for character. They
move to the base class; each variant keeps only the part that differs,
which is how it addresses the product.
The base class keeps its conv branch, which the two chunk variants
deliberately lack: those address 2-d fused weights.
Hosting asked select_candidate whether it could take a layer, and that
function reads the host rank, so each mechanism was gated through the
other one's name. The shared conditions move into channel_candidate,
which says what they actually test: the layer is quantized, a loaded
network covers it, and there is a rank budget to spend on it.
The rules the walk depends on were spread across the comments that
happened to need them, and the attributes it keeps on the model's modules
were written from four files with the ownership recorded nowhere. Both
are stated once in the module docstring, including the identity the
factor cache keys its pass entry on and the three writers that share the
svd tensors.
Deactivate carried its own shorter copy of the component list, missing
text_encoder_4 and transformer_2. Nothing depends on the difference
today: layer names are stamped only on text_encoder, text_encoder_2,
unet, transformer and llm_adapter, so modules in the other components
are skipped by both passes. Sharing the list keeps the two from drifting
apart if that stamping ever widens.
The mode shown in the load and unload lines was derived at print time
from the live fuse setting, so a set applied under one setting was
reported under whatever the setting said later, and the unload line
described the pass that was about to replace it rather than the one
being removed. The pass records the mode it actually used.
last_backup_size was created on lora_common by assignment from
networks.py and read back through a getattr default; both fields are
declared where they live now.
The error limiter halts a pass by raising, and nothing between the raise
and the caller put the model back. A halted pass left group offload hooks
stripped from every component the walk had reached, left a sequential
model on the cpu with offload disabled, and left the counters other
modules read describing the pass before it.
The epilogue moves into finish_pass under a finally, so the model returns
to its offload mode and the counters describe the pass that just ran. The
abort still reaches the caller.
Pass state is reset in one place in lora_sdnq now. Two of the six
accumulators were not being cleared at the start of a pass, and a stale
routed layer suppresses the fallback count for that layer next time.
The per-module walk carried four mechanisms inline, each repeating the
same tail: count the layer, stamp the pair that marks it current, advance
the bar, continue. Five copies of that tail and three of the backup probe
put the deepest arm nine levels in.
Each mechanism is now a function that either takes the layer or declines
to the next, and the walk reads as the four of them in order. The pass
state they share moves onto one object built before the walk starts, with
the accept tail, the stamp and the bar tick as its methods. That takes
network_activate from 218 lines to 55, none of it deeper than the module
loop.
Two shapes are deliberately not folded into that tail: the weight path
counts weights and bias separately and tracks what the module refused,
and the factor-strip restore stamps without counting. Hosting hands a
declined delta back rather than leaving it in a flag, so a pair of Nones
still reads as assembled and no layer is calculated twice.
Both passes opened with the same four steps written twice: bring the
model into a writable state, enumerate the components to walk, open a
progress bar, and probe a weight backup before restoring it. Pull each
into a helper and call it from both entry points. The component
collector keeps the two lists it is given, so deactivate still walks its
own shorter set, and promotion now clears the staged config it consumed.
The restore field had no writer and add() had no caller. Deactivate
assigned its elapsed time where activate accumulates, so a generation
that unloaded more than once reported only the last pass; both are
zeroed together when the generation ends.
The ramp read 1.5 when the option was absent while the option itself
defaults to 0.0, so a config without the key ran a ramp the settings
page said was off. Only reachable where the options registry is not
loaded, which is where the offline suites run.
The keys that mark a degradation as reported lived for the life of the
process, so a user who saw "flip=skipped weight=offloaded", changed the
offload mode and hit the same wall again was told nothing the second
time. Tie the set to the settings the warnings speak about: the stack
signature, the offload mode, the host rank and the checkpoint. Repeating
under one context still says it once.
The generic loader never offered a file to the boft type, so butterfly
OFT adapters reached the oft type instead, which claims any oft_blocks
key without checking its rank and then reads the block count as the lora
dim. Register boft ahead of oft; files with 3-d blocks still land on oft.
The nunchaku path replaced the loaded network list with the on-disk
entries it composed from, so reading a loaded network back hit an object
without the fields it expects: choosing the reported method reads
len(net.modules) and raised on every set change, costing that generation
its infotext and trigger tags. The adapter was already composed by then,
so the image was unaffected. Wrap the composed set in Network objects
and mutate the list in place.
The mechanism gate tests assert select_candidate declines under
requantize, and the apply-method hint names the cache option among
those the requantize choice disables.
<lora:name:1.0:lbw=VALUE> scales each targeted layer's delta by a slot of a
per-architecture block vector. VALUE is a preset name, a scalar, or a comma
vector; presets stretch onto the block count of the current model and the
a1111 17-slot and 12-slot layouts are accepted on sd and sdxl. The factor
enters through the module multiplier, so every apply path carries it: the
exact factor channel, hosting, requantize routing, dense stack combines and
select scoring.
- modules/lora/lora_blocks.py: slot classification from network_layer_mapping
(namespace-first, anchored chain prefixes), preset resolution reusing the
merge block-weight tables with BASE forced neutral, generated classic
segment names plus DOUBLE/SINGLE chain names, per-model memoization
- the raw spec stages through pending_config and promotes with the other
multipliers, keeping fuse removal consistent
- block weights join the activation signature, the per-module apply stamp
and the factor cache identity; entries without block weights keep their
existing signature bytes
- non-native load methods warn once and ignore the argument
The weight-kind schedule reset ran each winner's calc_updown on the target
weight's device, which on a block-swapped denoiser is the cpu; at hundreds
of layers per pass the cpu matmuls dominated every select generation. The
delta now computes on the accelerator and moves back, matching the activate
walk's convention.
- reset and flip execution log a debug timing line (materialize, select
loop, move/calc/apply split); the reset runs outside the activate walk,
so its cost was invisible to the load timers
Select scoring staged full fp32 copies, an abs copy and top-k workspace per
layer (hundreds of MB of transients that collide with block swapping on
offloaded denoisers) and recomputed scores from freshly assembled deltas on
every apply, which kept select modes out of the factor-cache fast path.
- score_pair: row-chunked fp32 interiors, fp64 accumulators, one device sync
- select scores persist in the factor cache as additive per-layer records
under the existing configuration signature
- apply_select_cached and register_weight_pair_cached replay a pair without
assembling deltas; the weight-kind winner is still computed at schedule
time
The weight-kind select branch counted its backup at its own call site
and again at the shared backup call when registration fell through, so
the reported backup size double-counted those layers; the size now
lands with whichever path keeps the layer. A pair the svd channel
cannot carry (bias delta or malformed member) previously dropped to
the sum paths with no trace; the fallthrough now says so once.
A restore-only activation walk carries zero networks, so the
networks=0 required=2 fallback warning fired on every network-free
generation whenever a select stack mode was set. The gate now
short-circuits at zero; the 1-and-3-network warnings that remain
meaningful are unchanged.
The klora and estlora balance factors accumulated across registrations
without ever resetting, so a multiplier change or pair swap blended the
previous registration into every later schedule. Balances now sum over
the live entries at finalize time, which keeps drop and re-register
consistent by construction.
- key flips one step early: step callbacks fire after the denoise, so
the winner is now live during the crossover step forward and a
final-step crossover engages instead of expiring
- include the calibration toggle in the factor cache signature so a hit
never replays factors computed under the other setting
- fall back to summation with a warning when hosting is disabled on a
quantized model instead of registering schedules that cannot flip
- drop the unused score_topk helper
Select modes left no trace distinguishable from plain summation: apply_select
counted its layers on the exact path, and the mode field in the load summary
reflects the requested setting rather than what executed. A flip count can only
come from a populated schedule.
- report layers, initial style picks, flips, steps and gamma from finalize
- deduplicate on content, since the schedule rebuilds on every pass
- give select its own apply counter instead of inflating apply=exact
The active stack mode was only visible inside the stack= token of the
trace-level network check line. Add it to the load summary as its own
stack= field alongside method, mode, te and unet, carrying the mode and
its tuning (ties:0.50, klora:1.50:0.50, sum). Non-native loads report
sum, since those paths always combine as sum regardless of the setting.
EST-LoRA scores each layer by squared Frobenius energy, so a magnitude gap
between the two networks enters squared and the louder network wins nearly
every layer, starving the quieter one. The style side is now scaled by the
total-energy ratio (mirroring klora's gamma), making selection scale-invariant
so a network cannot take layers on magnitude alone. On the krea2 subject+style
pair this lifts the style network from 18% to 65% of the layer-step budget.
- lora_stack: accumulate per-mode energy totals, apply the balance in the est ramp
- test: content-louder est pair now hands over mid-schedule where raw scoring never would
- locale: note the est magnitude balance, and that a select mode gives each layer
to one network so both can be under-applied, while dense modes blend more fully
Rebuilding balanced offload before touching weights reconstructs the
OffloadHook, whose constructor prints the op=init banner and module
inventory meant for model load, so every network switch replayed the
full load-time announcement. The hook constructor and the model summary
now honor the silent flag and the network activate, deactivate, and
selection paths pass it; real model loads keep the full output.
The mode field printed the configured fuse-or-backup strategy, which predates
the factor path and reads as mode=backup on loads that took no backup at all.
It now reports what the load actually holds: backup when weight backups were
taken, fuse when fusing is active, factor when the whole load rode the svd
channel and unload just drops factors.
Requantizing a dense-combined delta into 8-bit weights is checkpoint-fragile:
on some checkpoints the round trip visibly damages the render while the same
combination hosted on the svd channel is clean. Dense-mode sets with two or
more contributing networks on a layer now ride the hosted path regardless of
bit width; single-set behavior at 8 bits and above is unchanged.
- host_candidate: dense multi-net layers qualify at any width
- suite: dense pair at int8 hosts; single non-factorable set at int8 keeps
the requantize fallback
Select pairs now ride the svd side channel on any SDNQ linear, not
only sub-8-bit ones: quantized backups are packed tensors, so the
weight rewrite path cannot recompute a winner from them and left
layers stripped mid-requantize. The sub-8-bit gate stays for dense
hosting, where requantize retains the delta at 8 bits and above.
Weight selection now only serves unquantized modules: finalize
iterates a snapshot so dead entries drop cleanly, materializes
balanced-offload modules before rewriting weights and skips modules
with stripped or quantized weights instead of corrupting the layer.
A select mode forced backup mode whenever it was merely set, so a
leftover setting changed behavior for ordinary single-network loads.
The fuse gate now engages only when the loaded set could actually
select (exactly two networks, compile permitting) or while selection
segments are still live on model layers. Re-application drops any
stale per-layer schedule so a later pass reset can never replay an
old winner over freshly applied weights. Fallback notices log per
activation instead of once per session; only the in-loop flip notice
stays latched.
The exact factor path was the only apply route with no log line; its
success read as silence. Track layers taking it beside the hosted and
fallback lists and report all three as key=value apply lines
(apply=exact/hosted/requantize); the stack fallback notices use the
same form. The suite pins its stack-mode baseline to sum so a mode
left set in user config cannot reroute tests that assume plain
summation.
Two-network subject+style sets select a winner per layer instead of
summing: scores are top-K magnitude sums (klora) or Frobenius energies
(estlora), and a timestep ramp shifts layers from the subject network
toward the style network across sampling, reduced to at most one
precomputed flip per layer per pass. On sub-8-bit SDNQ the pair rides
the side-channel as separate segments flipped in place; other layers
recompute the winner from the pristine backup, so select modes force
backup mode. Selection resets per pass from the callback setup and is
gated off under model compile. estlora's measured style-discrepancy
term is exposed as an option. Adds XYZ axes for the stack settings.
Add lora_stack_mode with ties, dare_ties, dare_linear and
magnitude_prune combination of per-network deltas when several loaded
networks target one layer; sum stays the default and the exact factor
path. Combined deltas ride the existing tail: hosted svd on sub-8-bit
SDNQ, requantize at int8 and above, direct add elsewhere. Text-encoder
layers and single-network sets keep plain summation. DARE masks draw
from per-layer sha256 seeds so re-applies and cache entries stay
deterministic; the stack settings join the activation and factor-cache
signatures so settings changes re-apply without a reload.
Hosted truncation kept the full rank cap even when the spectrum ends in
numerical zeros, padding exactly low-rank deltas (low-rank LyCORIS,
full-family diffs) up to the cap. Slice the kept factors where
cumulative capture reaches 1 - 1e-6 of the sketch total, and trim
trailing all-zero columns when attaching cache entries written before
the slice, so they collapse the same way without a format bump. The
hosted log line reports the realized rank spread when it sits below
the cap.
- flat spectra keep the cap; a rank-8 delta under cap 256 stores 8 ranks
- select segments follow the effective rank
- suite pins the collapse, the flat-spectrum guard and the padded-entry trim
A cache hit still paid full calc_updown per layer, the dominant cost of a
replayed apply. The walk now probes the pass's cache entry first through
apply_cached, which evaluates the requantize routing rule from the delta
rms stored in the entry and attaches the cached factors without assembling
the delta; layers the rule declines fall through with the assembled delta
reused for the requantize path instead of recomputing it. The entry format
gains the rms and bumps to fmt 5, so older files recompute once and
rewrite. Hit accounting stays single-count through a plain lookup plus an
explicit hit note.
- replayed krea2 LoKR apply drops calc from 9.1s to 0.2s and activate
from 39.5s fresh to 4.8s replayed, hosted/routed split identical
- three cache tests pin the calc skip on plain, mixed and dense-pair sets
Hosting truncates every non-factorable set on sub-8-bit layers, but a
delta large against the grid step whose truncation capture is low is
retained better by the grid than by the rank cap. apply_hosted now
returns such layers to the requantize path when rms(delta)/mean(step)
exceeds 0.30 and the sketch capture falls below 0.90, thresholds sized
on 610 calibrated modules across krea2 and anima. Both terms must
agree: a thin delta rounds away on the grid however low its capture,
and a low-rank delta hosts exactly however fat it is.
- scoped to pure non-factorable sets; factorable members, dense-combined
deltas and svd-channel checkpoints keep hosting
- factor cache entries memoize the decision through their stored
capture, so replays route without re-running the sketch
- routed layers log as info apart from the forced-fallback warning
- the fidelity CLI applies the same rule so its reports track the loader
- seven routing tests, constants module-level and test-overridable
Persist now fires either when every hooked layer reaches the token quota
or at a fixed denoiser-forward deadline; layers under a small token floor
are omitted and stay on plain truncation. The all-done barrier alone
never fired on models whose modulation or pooled projections see a few
tokens per forward, so hooks stayed registered forever and statistics
recollected every session. eligible_modules walks the transformer or the
unet, so unet checkpoints collect statistics at all.
- lora_sdnq: drop the unused use_previous parameter; the factor cache
store now always runs inside a begin_pass
- tests: deadline persist, token-floor omission, unet root walk
The compiled dequant specializes per factor rank, so each distinct lora
set shape paid a compile stall on switch. Pad appended factors to a
power-of-two rank ladder (multiples of 64 past the hosted cap) with zero
fill: switches inside a bucket reuse the compiled graph, and common
trained ranks land on their bucket exactly so padding is usually a
no-op. Regression tests pin the factor add inside the single compiled
graph and the bucket reuse.
Sketch hosted deltas at rank+64 with eight power iterations and keep the
top rank columns; this lands within noise of the exact decomposition at
roughly twice a sketch cost the factor cache pays once per configuration.
Bump the cache format so narrower-sketch entries reload as misses.
Rowwise int8 with fp32 scales halves cache entries; measured in output
space on real hosted deltas the roundtrip is fidelity-free (within
0.0003 of fp32 factors, bf16 storage likewise). Factors are quantized
before first use and the dequantized roundtrip is what the apply
attaches, so a fresh compute and a later cache hit stay bit-identical;
the entry format is versioned and pre-int8 entries reload as misses.
- lora_factor_cache: quantize_rowwise/dequantize_rowwise, store returns
the applied pair, fmt guard on read
- test/test-sdnq-lora-factors.py: int8 quantization test, entry-size
assertion in the roundtrip test
Vladimir Mandic
CalamitousFelicitousness