The mechanism gate tests assert select_candidate declines under
requantize, and the apply-method hint names the cache option among
those the requantize choice disables.
The in-place weight installs, the promote-after-deactivate fuse
ordering, and the dynamo reset at model unload live in the shared
loader code; their regression pins belong in the campaign suite beside
the paths they protect.
String axis rewriting every lora tag in the prompt: an existing lbw=
argument is replaced, None removes it for a clean baseline cell. Choices
list the preset names; raw vectors go through csv mode with escaped
commas. Long values truncate in the grid legend.
<lora:name:1.0:lbw=VALUE> scales each targeted layer's delta by a slot of a
per-architecture block vector. VALUE is a preset name, a scalar, or a comma
vector; presets stretch onto the block count of the current model and the
a1111 17-slot and 12-slot layouts are accepted on sd and sdxl. The factor
enters through the module multiplier, so every apply path carries it: the
exact factor channel, hosting, requantize routing, dense stack combines and
select scoring.
- modules/lora/lora_blocks.py: slot classification from network_layer_mapping
(namespace-first, anchored chain prefixes), preset resolution reusing the
merge block-weight tables with BASE forced neutral, generated classic
segment names plus DOUBLE/SINGLE chain names, per-model memoization
- the raw spec stages through pending_config and promotes with the other
multipliers, keeping fuse removal consistent
- block weights join the activation signature, the per-module apply stamp
and the factor cache identity; entries without block weights keep their
existing signature bytes
- non-native load methods warn once and ignore the argument
At 1.5 the ramp hands nearly every layer to the style network mid-generation,
which on few-step models overwrites the forming subject before identity sets.
Alpha 0 freezes the schedule into a static per-layer split: the subject keeps
its layers for the whole generation and style keeps the layers where it is
more salient. Nonzero values remain the scheduled handover toward the second
network.
- locale hints describe both regimes; the mode hint no longer implies the
shift is always on
The weight-kind schedule reset ran each winner's calc_updown on the target
weight's device, which on a block-swapped denoiser is the cpu; at hundreds
of layers per pass the cpu matmuls dominated every select generation. The
delta now computes on the accelerator and moves back, matching the activate
walk's convention.
- reset and flip execution log a debug timing line (materialize, select
loop, move/calc/apply split); the reset runs outside the activate walk,
so its cost was invisible to the load timers
Select scoring staged full fp32 copies, an abs copy and top-k workspace per
layer (hundreds of MB of transients that collide with block swapping on
offloaded denoisers) and recomputed scores from freshly assembled deltas on
every apply, which kept select modes out of the factor-cache fast path.
- score_pair: row-chunked fp32 interiors, fp64 accumulators, one device sync
- select scores persist in the factor cache as additive per-layer records
under the existing configuration signature
- apply_select_cached and register_weight_pair_cached replay a pair without
assembling deltas; the weight-kind winner is still computed at schedule
time
The weight-kind select branch counted its backup at its own call site
and again at the shared backup call when registration fell through, so
the reported backup size double-counted those layers; the size now
lands with whichever path keeps the layer. A pair the svd channel
cannot carry (bias delta or malformed member) previously dropped to
the sum paths with no trace; the fallthrough now says so once.
A restore-only activation walk carries zero networks, so the
networks=0 required=2 fallback warning fired on every network-free
generation whenever a select stack mode was set. The gate now
short-circuits at zero; the 1-and-3-network warnings that remain
meaningful are unchanged.
The klora and estlora balance factors accumulated across registrations
without ever resetting, so a multiplier change or pair swap blended the
previous registration into every later schedule. Balances now sum over
the live entries at finalize time, which keeps drop and re-register
consistent by construction.
- key flips one step early: step callbacks fire after the denoise, so
the winner is now live during the crossover step forward and a
final-step crossover engages instead of expiring
- include the calibration toggle in the factor cache signature so a hit
never replays factors computed under the other setting
- fall back to summation with a warning when hosting is disabled on a
quantized model instead of registering schedules that cannot flip
- drop the unused score_topk helper
Select modes left no trace distinguishable from plain summation: apply_select
counted its layers on the exact path, and the mode field in the load summary
reflects the requested setting rather than what executed. A flip count can only
come from a populated schedule.
- report layers, initial style picks, flips, steps and gamma from finalize
- deduplicate on content, since the schedule rebuilds on every pass
- give select its own apply counter instead of inflating apply=exact
The active stack mode was only visible inside the stack= token of the
trace-level network check line. Add it to the load summary as its own
stack= field alongside method, mode, te and unet, carrying the mode and
its tuning (ties:0.50, klora:1.50:0.50, sum). Non-native loads report
sum, since those paths always combine as sum regardless of the setting.
EST-LoRA scores each layer by squared Frobenius energy, so a magnitude gap
between the two networks enters squared and the louder network wins nearly
every layer, starving the quieter one. The style side is now scaled by the
total-energy ratio (mirroring klora's gamma), making selection scale-invariant
so a network cannot take layers on magnitude alone. On the krea2 subject+style
pair this lifts the style network from 18% to 65% of the layer-step budget.
- lora_stack: accumulate per-mode energy totals, apply the balance in the est ramp
- test: content-louder est pair now hands over mid-schedule where raw scoring never would
- locale: note the est magnitude balance, and that a select mode gives each layer
to one network so both can be under-applied, while dense modes blend more fully
Rebuilding balanced offload before touching weights reconstructs the
OffloadHook, whose constructor prints the op=init banner and module
inventory meant for model load, so every network switch replayed the
full load-time announcement. The hook constructor and the model summary
now honor the silent flag and the network activate, deactivate, and
selection paths pass it; real model loads keep the full output.
The mode field printed the configured fuse-or-backup strategy, which predates
the factor path and reads as mode=backup on loads that took no backup at all.
It now reports what the load actually holds: backup when weight backups were
taken, fuse when fusing is active, factor when the whole load rode the svd
channel and unload just drops factors.
Requantizing a dense-combined delta into 8-bit weights is checkpoint-fragile:
on some checkpoints the round trip visibly damages the render while the same
combination hosted on the svd channel is clean. Dense-mode sets with two or
more contributing networks on a layer now ride the hosted path regardless of
bit width; single-set behavior at 8 bits and above is unchanged.
- host_candidate: dense multi-net layers qualify at any width
- suite: dense pair at int8 hosts; single non-factorable set at int8 keeps
the requantize fallback
Select pairs now ride the svd side channel on any SDNQ linear, not
only sub-8-bit ones: quantized backups are packed tensors, so the
weight rewrite path cannot recompute a winner from them and left
layers stripped mid-requantize. The sub-8-bit gate stays for dense
hosting, where requantize retains the delta at 8 bits and above.
Weight selection now only serves unquantized modules: finalize
iterates a snapshot so dead entries drop cleanly, materializes
balanced-offload modules before rewriting weights and skips modules
with stripped or quantized weights instead of corrupting the layer.
Text encoder networks now apply unconditionally in the native path.
- remove the option, the per-request parameter and the control threading
- collapse activate_filtered into plain activate at all call sites
- drop the toggle from the prompt embed cache key
- register the retired key so existing configs load without warnings
Restructure the seven hints to the settings pattern: lead definition,
tradeoff, scope, special values and stated default, with value and
cross-setting markup. Mode bullets and special values match loader
behavior: sub-8-bit hosting gate, rank 0 fallback, ramp 0 freezing
the balance.
A select mode forced backup mode whenever it was merely set, so a
leftover setting changed behavior for ordinary single-network loads.
The fuse gate now engages only when the loaded set could actually
select (exactly two networks, compile permitting) or while selection
segments are still live on model layers. Re-application drops any
stale per-layer schedule so a later pass reset can never replay an
old winner over freshly applied weights. Fallback notices log per
activation instead of once per session; only the in-loop flip notice
stays latched.
The exact factor path was the only apply route with no log line; its
success read as silence. Track layers taking it beside the hosted and
fallback lists and report all three as key=value apply lines
(apply=exact/hosted/requantize); the stack fallback notices use the
same form. The suite pins its stack-mode baseline to sum so a mode
left set in user config cannot reroute tests that assume plain
summation.
Two-network subject+style sets select a winner per layer instead of
summing: scores are top-K magnitude sums (klora) or Frobenius energies
(estlora), and a timestep ramp shifts layers from the subject network
toward the style network across sampling, reduced to at most one
precomputed flip per layer per pass. On sub-8-bit SDNQ the pair rides
the side-channel as separate segments flipped in place; other layers
recompute the winner from the pristine backup, so select modes force
backup mode. Selection resets per pass from the callback setup and is
gated off under model compile. estlora's measured style-discrepancy
term is exposed as an option. Adds XYZ axes for the stack settings.
Add lora_stack_mode with ties, dare_ties, dare_linear and
magnitude_prune combination of per-network deltas when several loaded
networks target one layer; sum stays the default and the exact factor
path. Combined deltas ride the existing tail: hosted svd on sub-8-bit
SDNQ, requantize at int8 and above, direct add elsewhere. Text-encoder
layers and single-network sets keep plain summation. DARE masks draw
from per-layer sha256 seeds so re-applies and cache entries stay
deterministic; the stack settings join the activation and factor-cache
signatures so settings changes re-apply without a reload.
Per-head selection is what breaks head-consistency sensitive models, measured on Anima across fourteen arms: the only clean sparse selections are the ones every head shares. The head sharing hint names itself as the first resort for banded output, and the exclusion hint carries why Cosmos stays listed.
sparse_attention_exclude is a comma separated denylist matched case insensitively against the architecture, the pipeline class and the denoiser class, so one entry works whichever name is to hand. It resolves once per model rather than per call and declines with a log line. Seeded with CosmosTransformer3DModel, the transformer Anima runs, which returns banded noise at every budget tested against a sound dense baseline; listing the class rather than the architecture covers the other models built on it, none of which have been checked.
get_block_mask_input takes only the mask now, so the helper pads to 4d itself, and the standalone nesting filter is inlined into prune_configs. The raise for a tile list that cannot nest moves into the prune_configs row, pinned with from_small so the fallback to the small configs cannot refill the list before the filter runs.
The SDP overrides and SDNQ Attention hints describe the block mask capability
without naming what consumes it, so the two backends that can drive sparse
attention now say so where the reader is choosing between them.
Each hint now carries its default, the tradeoff in both directions, and the
setting it depends on. The two schedule settings do nothing unless both are
above zero, which nothing said before. The sequence floor is given in
resolution as well as tokens, since the setting is read in pixels.
The option defaulted to 0 and a property turned that into 8192, which hid
the real default behind a sentinel and left no way to sparsify every
sequence. The default is now 8192 in the settings registry, 0 means no
floor, and the slider tops out at 32768 rather than a length nothing
reaches. A negative value is left unclaimed for a derived floor.
install_layout_hook took transformer or unet and stopped, so a pipeline
with a second denoiser left that one unhooked and its forwards ran against
whatever the first had published. The slot list follows the denoiser names
in sd_offload_state; the aux components on that list pack no attention
sequence, and a publication from one would clear the layout.
Fourteen axes drive the attention subsystem from the grid: the diffusers
method, the sdp override chain, the dispatcher kernel, five sdnq attention
knobs and six sparse settings. An axis writes shared.opts.data and rebuilds
the chain itself, since the onchange for these settings runs through the
queue lock a grid cell already holds. SharedSettingsStackHelper saves and
restores every attention setting around the grid, keys an axis introduced
included, and an axis for a setting a backend owns warns when that backend
is not in the active chain.
The classic tick test used a bare object as the pipeline's transformer,
which context.begin hands to register_forward_pre_hook whenever sparse
attention is enabled in the config.
Only the modular path had a per-forward kwargs hook, so a classic pipeline had
no layout and the selector sparsified its whole sequence, prompt conditioning
included. A denoiser pre-hook installed per generation now reads whichever
convention the model uses: the *_indices tensors, or the txt_ids and img_ids a
joint transformer is given.
Packing order is not derivable from the call and differs between architectures,
so only classes verified to pack text first publish; anything else falls back
as before rather than pinning the wrong half of the sequence dense.
The transformer is the only site that knows every segment length, since the
pad to a multiple of 256 happens there and the text stream is padded to a
fixed 512 whatever the prompt. It now publishes the live runs of the mask, so
the selector pins the conditioning dense and drops the padding instead of
sparsifying the whole sequence. At 2048x2048 with a short prompt that is 6 of
264 key blocks per query row moved from pinned to dropped.
The sparse stage declined every call carrying an attention mask, so a model
that passes a padding mask stayed dense at every length. It now serves those
calls when the serving backend declares `masked_block`: sdnq takes attn_mask
and block_mask together, flex would need a mask_mod and does not. A backend
that cannot compose says so once instead of skipping quietly.
A strided query or key changes what a backend's input prep costs, so the route trace records it and keeps strided and contiguous calls as separate routes. The trace also counts calls per route and logs the counts when a generation ends, so a per-step cost can be divided by the number of calls that paid it.
The sdnq Triton kernel gains a block mask input (on the submodule's feat/block-mask branch; the pointer here is unchanged), so the backend advertises the block_mask cap and passes the router's selection as block_mask plus its block sizes. Prepare probes the installed kernel's signature and narrows the cap when the input is missing, warning when sparse attention is on, so an older submodule stays dense instead of failing. PlanEntry carries the narrowed caps, and the router, the sparse stage and the report read those rather than the declaration.
- benchmark: int8-sparse100/50/30/15 and int8-radial30 rows feed the same producer as the flex rows into the quantized kernel, skipped on builds without block_mask; a --configs filter selects attention rows the way --block-configs selects block rows; sparse rows of either backend are held out of the settings advice
- test/test-attention-sdnq-sparse.py: the kernel contract against the token-expanded mask, the nesting filter, the public entry on the quantized paths, the backward against the same selection and against fp32 autograd, and the flex consumer on one selection, pinned to one tile per run with a --tiles sweep
- locale: the sparse attention hint names both capable backends
SD_SPARSE_PATTERN=radial swaps the content aware selection for a band
around the diagonal at the density the selector would have produced. It
is the control the selector has to beat, and the fallback the kill
criteria name if it does not.
The band honors the same layout pins. Without them it never keeps the
text columns, which sit far from the diagonal for late video rows, so
prompt conditioning is lost and it renders a different scene entirely:
a control that loses for the wrong reason proves nothing.
flex_attention called eagerly materializes the whole score matrix. At a
video sequence length that is tens of gigabytes, and it surfaces as a
CUDA driver error rather than a clean allocation failure: MiniMax H3 at
9505 tokens over 56 heads asks for about 20 GB for the scores alone and
the generation dies. The backend now calls the compiled entry point for
every path, which is also the only form that reads block lists.
Dense flex, the selector at four budgets, and a density matched static
radial band, at any shape without a mask or a causal rule. The selector
runs inside the timed region, because a producer measured beside the
attention it accelerates looks free and is not, and the budget 100
percent row keeps every tile so it isolates that cost. All of them are
reported and none can win an sdnq settings recommendation.
Selecting blocks synchronized with the accelerator three times per
attention call: once to size the top-k, once to decide whether the
budget covered every candidate, and once to compute the density for
reporting. At the sdxl shape those stalls cost 1.6 ms against a 1.2 ms
attention, so the selector lost to dense at every budget.
The parts that depend only on geometry and layout are now built once and
cached per layout, ranking replaces top-k so no host side k is needed,
and density became a method that reads back only when something asks.
The block mask also stops sorting a mask of zeros to fill partial slots
it leaves empty. Selection overhead at sdxl drops from 1.84 ms to
0.23 ms, and a 30 percent budget moves from 0.36x of dense to 1.19x.
Vladimir Mandic
CalamitousFelicitousness