A restore-only activation walk carries zero networks, so the
networks=0 required=2 fallback warning fired on every network-free
generation whenever a select stack mode was set. The gate now
short-circuits at zero; the 1-and-3-network warnings that remain
meaningful are unchanged.
The klora and estlora balance factors accumulated across registrations
without ever resetting, so a multiplier change or pair swap blended the
previous registration into every later schedule. Balances now sum over
the live entries at finalize time, which keeps drop and re-register
consistent by construction.
- key flips one step early: step callbacks fire after the denoise, so
the winner is now live during the crossover step forward and a
final-step crossover engages instead of expiring
- include the calibration toggle in the factor cache signature so a hit
never replays factors computed under the other setting
- fall back to summation with a warning when hosting is disabled on a
quantized model instead of registering schedules that cannot flip
- drop the unused score_topk helper
Select modes left no trace distinguishable from plain summation: apply_select
counted its layers on the exact path, and the mode field in the load summary
reflects the requested setting rather than what executed. A flip count can only
come from a populated schedule.
- report layers, initial style picks, flips, steps and gamma from finalize
- deduplicate on content, since the schedule rebuilds on every pass
- give select its own apply counter instead of inflating apply=exact
The active stack mode was only visible inside the stack= token of the
trace-level network check line. Add it to the load summary as its own
stack= field alongside method, mode, te and unet, carrying the mode and
its tuning (ties:0.50, klora:1.50:0.50, sum). Non-native loads report
sum, since those paths always combine as sum regardless of the setting.
EST-LoRA scores each layer by squared Frobenius energy, so a magnitude gap
between the two networks enters squared and the louder network wins nearly
every layer, starving the quieter one. The style side is now scaled by the
total-energy ratio (mirroring klora's gamma), making selection scale-invariant
so a network cannot take layers on magnitude alone. On the krea2 subject+style
pair this lifts the style network from 18% to 65% of the layer-step budget.
- lora_stack: accumulate per-mode energy totals, apply the balance in the est ramp
- test: content-louder est pair now hands over mid-schedule where raw scoring never would
- locale: note the est magnitude balance, and that a select mode gives each layer
to one network so both can be under-applied, while dense modes blend more fully
Rebuilding balanced offload before touching weights reconstructs the
OffloadHook, whose constructor prints the op=init banner and module
inventory meant for model load, so every network switch replayed the
full load-time announcement. The hook constructor and the model summary
now honor the silent flag and the network activate, deactivate, and
selection paths pass it; real model loads keep the full output.
The mode field printed the configured fuse-or-backup strategy, which predates
the factor path and reads as mode=backup on loads that took no backup at all.
It now reports what the load actually holds: backup when weight backups were
taken, fuse when fusing is active, factor when the whole load rode the svd
channel and unload just drops factors.
Requantizing a dense-combined delta into 8-bit weights is checkpoint-fragile:
on some checkpoints the round trip visibly damages the render while the same
combination hosted on the svd channel is clean. Dense-mode sets with two or
more contributing networks on a layer now ride the hosted path regardless of
bit width; single-set behavior at 8 bits and above is unchanged.
- host_candidate: dense multi-net layers qualify at any width
- suite: dense pair at int8 hosts; single non-factorable set at int8 keeps
the requantize fallback
Select pairs now ride the svd side channel on any SDNQ linear, not
only sub-8-bit ones: quantized backups are packed tensors, so the
weight rewrite path cannot recompute a winner from them and left
layers stripped mid-requantize. The sub-8-bit gate stays for dense
hosting, where requantize retains the delta at 8 bits and above.
Weight selection now only serves unquantized modules: finalize
iterates a snapshot so dead entries drop cleanly, materializes
balanced-offload modules before rewriting weights and skips modules
with stripped or quantized weights instead of corrupting the layer.
Text encoder networks now apply unconditionally in the native path.
- remove the option, the per-request parameter and the control threading
- collapse activate_filtered into plain activate at all call sites
- drop the toggle from the prompt embed cache key
- register the retired key so existing configs load without warnings
A select mode forced backup mode whenever it was merely set, so a
leftover setting changed behavior for ordinary single-network loads.
The fuse gate now engages only when the loaded set could actually
select (exactly two networks, compile permitting) or while selection
segments are still live on model layers. Re-application drops any
stale per-layer schedule so a later pass reset can never replay an
old winner over freshly applied weights. Fallback notices log per
activation instead of once per session; only the in-loop flip notice
stays latched.
The exact factor path was the only apply route with no log line; its
success read as silence. Track layers taking it beside the hosted and
fallback lists and report all three as key=value apply lines
(apply=exact/hosted/requantize); the stack fallback notices use the
same form. The suite pins its stack-mode baseline to sum so a mode
left set in user config cannot reroute tests that assume plain
summation.
Two-network subject+style sets select a winner per layer instead of
summing: scores are top-K magnitude sums (klora) or Frobenius energies
(estlora), and a timestep ramp shifts layers from the subject network
toward the style network across sampling, reduced to at most one
precomputed flip per layer per pass. On sub-8-bit SDNQ the pair rides
the side-channel as separate segments flipped in place; other layers
recompute the winner from the pristine backup, so select modes force
backup mode. Selection resets per pass from the callback setup and is
gated off under model compile. estlora's measured style-discrepancy
term is exposed as an option. Adds XYZ axes for the stack settings.
Add lora_stack_mode with ties, dare_ties, dare_linear and
magnitude_prune combination of per-network deltas when several loaded
networks target one layer; sum stays the default and the exact factor
path. Combined deltas ride the existing tail: hosted svd on sub-8-bit
SDNQ, requantize at int8 and above, direct add elsewhere. Text-encoder
layers and single-network sets keep plain summation. DARE masks draw
from per-layer sha256 seeds so re-applies and cache entries stay
deterministic; the stack settings join the activation and factor-cache
signatures so settings changes re-apply without a reload.
sparse_attention_exclude is a comma separated denylist matched case insensitively against the architecture, the pipeline class and the denoiser class, so one entry works whichever name is to hand. It resolves once per model rather than per call and declines with a log line. Seeded with CosmosTransformer3DModel, the transformer Anima runs, which returns banded noise at every budget tested against a sound dense baseline; listing the class rather than the architecture covers the other models built on it, none of which have been checked.
The option defaulted to 0 and a property turned that into 8192, which hid
the real default behind a sentinel and left no way to sparsify every
sequence. The default is now 8192 in the settings registry, 0 means no
floor, and the slider tops out at 32768 rather than a length nothing
reaches. A negative value is left unclaimed for a derived floor.
install_layout_hook took transformer or unet and stopped, so a pipeline
with a second denoiser left that one unhooked and its forwards ran against
whatever the first had published. The slot list follows the denoiser names
in sd_offload_state; the aux components on that list pack no attention
sequence, and a publication from one would clear the layout.
Fourteen axes drive the attention subsystem from the grid: the diffusers
method, the sdp override chain, the dispatcher kernel, five sdnq attention
knobs and six sparse settings. An axis writes shared.opts.data and rebuilds
the chain itself, since the onchange for these settings runs through the
queue lock a grid cell already holds. SharedSettingsStackHelper saves and
restores every attention setting around the grid, keys an axis introduced
included, and an axis for a setting a backend owns warns when that backend
is not in the active chain.
Only the modular path had a per-forward kwargs hook, so a classic pipeline had
no layout and the selector sparsified its whole sequence, prompt conditioning
included. A denoiser pre-hook installed per generation now reads whichever
convention the model uses: the *_indices tensors, or the txt_ids and img_ids a
joint transformer is given.
Packing order is not derivable from the call and differs between architectures,
so only classes verified to pack text first publish; anything else falls back
as before rather than pinning the wrong half of the sequence dense.
The transformer is the only site that knows every segment length, since the
pad to a multiple of 256 happens there and the text stream is padded to a
fixed 512 whatever the prompt. It now publishes the live runs of the mask, so
the selector pins the conditioning dense and drops the padding instead of
sparsifying the whole sequence. At 2048x2048 with a short prompt that is 6 of
264 key blocks per query row moved from pinned to dropped.
The sparse stage declined every call carrying an attention mask, so a model
that passes a padding mask stayed dense at every length. It now serves those
calls when the serving backend declares `masked_block`: sdnq takes attn_mask
and block_mask together, flex would need a mask_mod and does not. A backend
that cannot compose says so once instead of skipping quietly.
A strided query or key changes what a backend's input prep costs, so the route trace records it and keeps strided and contiguous calls as separate routes. The trace also counts calls per route and logs the counts when a generation ends, so a per-step cost can be divided by the number of calls that paid it.
The sdnq Triton kernel gains a block mask input (on the submodule's feat/block-mask branch; the pointer here is unchanged), so the backend advertises the block_mask cap and passes the router's selection as block_mask plus its block sizes. Prepare probes the installed kernel's signature and narrows the cap when the input is missing, warning when sparse attention is on, so an older submodule stays dense instead of failing. PlanEntry carries the narrowed caps, and the router, the sparse stage and the report read those rather than the declaration.
- benchmark: int8-sparse100/50/30/15 and int8-radial30 rows feed the same producer as the flex rows into the quantized kernel, skipped on builds without block_mask; a --configs filter selects attention rows the way --block-configs selects block rows; sparse rows of either backend are held out of the settings advice
- test/test-attention-sdnq-sparse.py: the kernel contract against the token-expanded mask, the nesting filter, the public entry on the quantized paths, the backward against the same selection and against fp32 autograd, and the flex consumer on one selection, pinned to one tile per run with a --tiles sweep
- locale: the sparse attention hint names both capable backends
SD_SPARSE_PATTERN=radial swaps the content aware selection for a band
around the diagonal at the density the selector would have produced. It
is the control the selector has to beat, and the fallback the kill
criteria name if it does not.
The band honors the same layout pins. Without them it never keeps the
text columns, which sit far from the diagonal for late video rows, so
prompt conditioning is lost and it renders a different scene entirely:
a control that loses for the wrong reason proves nothing.
flex_attention called eagerly materializes the whole score matrix. At a
video sequence length that is tens of gigabytes, and it surfaces as a
CUDA driver error rather than a clean allocation failure: MiniMax H3 at
9505 tokens over 56 heads asks for about 20 GB for the scores alone and
the generation dies. The backend now calls the compiled entry point for
every path, which is also the only form that reads block lists.
Selecting blocks synchronized with the accelerator three times per
attention call: once to size the top-k, once to decide whether the
budget covered every candidate, and once to compute the density for
reporting. At the sdxl shape those stalls cost 1.6 ms against a 1.2 ms
attention, so the selector lost to dense at every budget.
The parts that depend only on geometry and layout are now built once and
cached per layout, ranking replaces top-k so no host side k is needed,
and density became a method that reads back only when something asks.
The block mask also stops sorting a mask of zeros to fill partial slots
it leaves empty. Selection overhead at sdxl drops from 1.84 ms to
0.23 ms, and a 30 percent budget moves from 0.36x of dense to 1.19x.
A sequence below the minimum length left the feature enabled and idle
with nothing in the log. The stage now records why it declined and says
so once per length, so an enabled setting that cannot act is visible
rather than quiet.
Sparse attention is a stage over the chain rather than a member of it:
one switch, and the router hands the selection to whichever active
backend advertises that it consumes a block mask, currently flex. A
backend declares that through a capability set, so the quantized kernel
joins later without touching the router.
The stage gates on the component role, self attention, a minimum
sequence length defaulting to the measured 8192 token crossover, and the
absence of a token mask or causal flag, which flex cannot combine with a
block only mask. Budgets follow a precomputed per step schedule with at
most two distinct values. Enabling the feature with no capable backend
in the chain warns and leaves attention dense rather than doing nothing
quietly.
The modular pre-forward hook now receives kwargs and publishes whatever
token layout the pipeline passes by name, so a packed sequence gets its
conditioning pinned without any model specific code. Without a layout
the whole sequence is sparsified and that is logged once per length.
The selector mean-pools query and key tiles, scores the tile pairs and
keeps the highest scoring key tiles per query tile within a budget
expressed as a fraction of the sparsifiable candidates. No scale and no
softmax, since top-k is invariant under both. Scoring runs on query
heads so grouped attention needs no key expansion, the diagonal is
always kept so no query row is left empty, and a budget that covers
every candidate reports dense instead of building a full mask.
The layout says what a packed sequence holds. Only video and image
tokens are sparsifiable; text, conditioning, audio and anything
unrecognized pin their rows and columns dense, padding is dropped, and a
tile straddling a boundary pins. Layouts come from the *_indices tensors
a pipeline passes its transformer by name, from ordered segments where a
transformer packs the sequence itself, or from a leading prefix as a
fallback.
The flex consumer builds a BlockMask with every selected tile in the
full slots, so mask_mod is never invoked and no dense mask is
materialized, and calls flex_attention compiled: called eagerly it reads
mask_mod rather than the block lists, so a block only mask attends
densely and silently. test/test-attention-sparse.py covers this with a
row that fails if the selection stops changing the output, alongside
tile equivalence against sdpa fed the same tiles, measured against the
flex kernel floor rather than an absolute tolerance.
Hosted truncation kept the full rank cap even when the spectrum ends in
numerical zeros, padding exactly low-rank deltas (low-rank LyCORIS,
full-family diffs) up to the cap. Slice the kept factors where
cumulative capture reaches 1 - 1e-6 of the sketch total, and trim
trailing all-zero columns when attaching cache entries written before
the slice, so they collapse the same way without a format bump. The
hosted log line reports the realized rank spread when it sits below
the cap.
- flat spectra keep the cap; a rank-8 delta under cap 256 stores 8 ranks
- select segments follow the effective rank
- suite pins the collapse, the flat-spectrum guard and the padded-entry trim
A cache hit still paid full calc_updown per layer, the dominant cost of a
replayed apply. The walk now probes the pass's cache entry first through
apply_cached, which evaluates the requantize routing rule from the delta
rms stored in the entry and attaches the cached factors without assembling
the delta; layers the rule declines fall through with the assembled delta
reused for the requantize path instead of recomputing it. The entry format
gains the rms and bumps to fmt 5, so older files recompute once and
rewrite. Hit accounting stays single-count through a plain lookup plus an
explicit hit note.
- replayed krea2 LoKR apply drops calc from 9.1s to 0.2s and activate
from 39.5s fresh to 4.8s replayed, hosted/routed split identical
- three cache tests pin the calc skip on plain, mixed and dense-pair sets
Hosting truncates every non-factorable set on sub-8-bit layers, but a
delta large against the grid step whose truncation capture is low is
retained better by the grid than by the rank cap. apply_hosted now
returns such layers to the requantize path when rms(delta)/mean(step)
exceeds 0.30 and the sketch capture falls below 0.90, thresholds sized
on 610 calibrated modules across krea2 and anima. Both terms must
agree: a thin delta rounds away on the grid however low its capture,
and a low-rank delta hosts exactly however fat it is.
- scoped to pure non-factorable sets; factorable members, dense-combined
deltas and svd-channel checkpoints keep hosting
- factor cache entries memoize the decision through their stored
capture, so replays route without re-running the sketch
- routed layers log as info apart from the forced-fallback warning
- the fidelity CLI applies the same rule so its reports track the loader
- seven routing tests, constants module-level and test-overridable
Persist now fires either when every hooked layer reaches the token quota
or at a fixed denoiser-forward deadline; layers under a small token floor
are omitted and stay on plain truncation. The all-done barrier alone
never fired on models whose modulation or pooled projections see a few
tokens per forward, so hooks stayed registered forever and statistics
recollected every session. eligible_modules walks the transformer or the
unet, so unet checkpoints collect statistics at all.
- lora_sdnq: drop the unused use_previous parameter; the factor cache
store now always runs inside a begin_pass
- tests: deadline persist, token-floor omission, unet root walk
The compiled dequant specializes per factor rank, so each distinct lora
set shape paid a compile stall on switch. Pad appended factors to a
power-of-two rank ladder (multiples of 64 past the hosted cap) with zero
fill: switches inside a bucket reuse the compiled graph, and common
trained ranks land on their bucket exactly so padding is usually a
no-op. Regression tests pin the factor add inside the single compiled
graph and the bucket reuse.
Sketch hosted deltas at rank+64 with eight power iterations and keep the
top rank columns; this lands within noise of the exact decomposition at
roughly twice a sketch cost the factor cache pays once per configuration.
Bump the cache format so narrower-sketch entries reload as misses.
Rowwise int8 with fp32 scales halves cache entries; measured in output
space on real hosted deltas the roundtrip is fidelity-free (within
0.0003 of fp32 factors, bf16 storage likewise). Factors are quantized
before first use and the dequantized roundtrip is what the apply
attaches, so a fresh compute and a later cache hit stay bit-identical;
the entry format is versioned and pre-int8 entries reload as misses.
- lora_factor_cache: quantize_rowwise/dequantize_rowwise, store returns
the applied pair, fmt guard on read
- test/test-sdnq-lora-factors.py: int8 quantization test, entry-size
assertion in the roundtrip test
Hosting a non-factorable set costs one truncated svd per layer on every
fresh apply. The factors are deterministic in the checkpoint, the loaded
set, the host rank and the calibration statistics, so they persist under
data/lora-factor-cache keyed by that identity and replay bit-identically
on later applies. lora_sdnq_host_cache sets the disk budget in GB
(0 disables), least-recently-used entries are evicted past it. With the
cost paid once per configuration, svd subspace iterations rise from 2 to
4: about two thirds of the captured-energy gap to an exact decomposition
for +45% one-time compute; an exact svd measures 200-400x slower at
these shapes and is not viable.
- modules/lora/lora_factor_cache.py: signature, store, flush, eviction
- lora_sdnq.apply_hosted: fetch before computing, store after, hits in
the load summary
- cli/lora-quant-fidelity.py: matching niter for the hosted mirror
- test/test-sdnq-lora-factors.py: factor-cache category, 3 tests
process_images swallows the interrupt assertion, so a cancel reaches the shared
video core as an empty result and was raised as 'processing failed' with 500,
leaving clients unable to tell a cancel from a crash. Mirrors the LTX path,
which already returns 499.
Queue.__exit__ returned _queue_lock, and a truthy __exit__ return suppresses the
exception in flight, so every `with queue_lock:` block discarded exceptions and
resumed with locals from the aborted block unassigned. The bare threading.Lock it
replaced returned None.
The refine-default comment claimed Condition variants are excluded from two-stage
refine, but supports_two_stage_refine has no such carve-out and ltx_process builds
a second condition set so conditioning survives the upsample. Two file references
had also drifted: ltx_process.py:179 moved, and the offload hook keying now lives
in sd_offload_balanced. Name the symbols instead of the line numbers.
New lora_sdnq_apply radio (exact, requantize) in the lora settings.
requantize keeps the previous behavior: every quantized layer takes the
dequantize-add-requantize path, with factor attach and svd hosting gated
off. A settings-only flip re-applies loaded networks: the mechanism
rides a per-module apply stamp and the network-changed signature, and
the activate fallthrough strips factors a closed gate leaves attached.
Requantize chosen by the setting logs as info instead of the
reduced-fidelity warning.
- locale hint covers fidelity and memory tradeoffs of both methods
- suite: gate, legacy routing and flip-transition tests
CalamitousFelicitousness
Vladimir Mandic
Alejandro Alberto Aguilar Morales