Fourteen axes drive the attention subsystem from the grid: the diffusers
method, the sdp override chain, the dispatcher kernel, five sdnq attention
knobs and six sparse settings. An axis writes shared.opts.data and rebuilds
the chain itself, since the onchange for these settings runs through the
queue lock a grid cell already holds. SharedSettingsStackHelper saves and
restores every attention setting around the grid, keys an axis introduced
included, and an axis for a setting a backend owns warns when that backend
is not in the active chain.
The classic tick test used a bare object as the pipeline's transformer,
which context.begin hands to register_forward_pre_hook whenever sparse
attention is enabled in the config.
Only the modular path had a per-forward kwargs hook, so a classic pipeline had
no layout and the selector sparsified its whole sequence, prompt conditioning
included. A denoiser pre-hook installed per generation now reads whichever
convention the model uses: the *_indices tensors, or the txt_ids and img_ids a
joint transformer is given.
Packing order is not derivable from the call and differs between architectures,
so only classes verified to pack text first publish; anything else falls back
as before rather than pinning the wrong half of the sequence dense.
The transformer is the only site that knows every segment length, since the
pad to a multiple of 256 happens there and the text stream is padded to a
fixed 512 whatever the prompt. It now publishes the live runs of the mask, so
the selector pins the conditioning dense and drops the padding instead of
sparsifying the whole sequence. At 2048x2048 with a short prompt that is 6 of
264 key blocks per query row moved from pinned to dropped.
The sparse stage declined every call carrying an attention mask, so a model
that passes a padding mask stayed dense at every length. It now serves those
calls when the serving backend declares `masked_block`: sdnq takes attn_mask
and block_mask together, flex would need a mask_mod and does not. A backend
that cannot compose says so once instead of skipping quietly.
A strided query or key changes what a backend's input prep costs, so the route trace records it and keeps strided and contiguous calls as separate routes. The trace also counts calls per route and logs the counts when a generation ends, so a per-step cost can be divided by the number of calls that paid it.
The sdnq Triton kernel gains a block mask input (on the submodule's feat/block-mask branch; the pointer here is unchanged), so the backend advertises the block_mask cap and passes the router's selection as block_mask plus its block sizes. Prepare probes the installed kernel's signature and narrows the cap when the input is missing, warning when sparse attention is on, so an older submodule stays dense instead of failing. PlanEntry carries the narrowed caps, and the router, the sparse stage and the report read those rather than the declaration.
- benchmark: int8-sparse100/50/30/15 and int8-radial30 rows feed the same producer as the flex rows into the quantized kernel, skipped on builds without block_mask; a --configs filter selects attention rows the way --block-configs selects block rows; sparse rows of either backend are held out of the settings advice
- test/test-attention-sdnq-sparse.py: the kernel contract against the token-expanded mask, the nesting filter, the public entry on the quantized paths, the backward against the same selection and against fp32 autograd, and the flex consumer on one selection, pinned to one tile per run with a --tiles sweep
- locale: the sparse attention hint names both capable backends
SD_SPARSE_PATTERN=radial swaps the content aware selection for a band
around the diagonal at the density the selector would have produced. It
is the control the selector has to beat, and the fallback the kill
criteria name if it does not.
The band honors the same layout pins. Without them it never keeps the
text columns, which sit far from the diagonal for late video rows, so
prompt conditioning is lost and it renders a different scene entirely:
a control that loses for the wrong reason proves nothing.
flex_attention called eagerly materializes the whole score matrix. At a
video sequence length that is tens of gigabytes, and it surfaces as a
CUDA driver error rather than a clean allocation failure: MiniMax H3 at
9505 tokens over 56 heads asks for about 20 GB for the scores alone and
the generation dies. The backend now calls the compiled entry point for
every path, which is also the only form that reads block lists.
Dense flex, the selector at four budgets, and a density matched static
radial band, at any shape without a mask or a causal rule. The selector
runs inside the timed region, because a producer measured beside the
attention it accelerates looks free and is not, and the budget 100
percent row keeps every tile so it isolates that cost. All of them are
reported and none can win an sdnq settings recommendation.
Selecting blocks synchronized with the accelerator three times per
attention call: once to size the top-k, once to decide whether the
budget covered every candidate, and once to compute the density for
reporting. At the sdxl shape those stalls cost 1.6 ms against a 1.2 ms
attention, so the selector lost to dense at every budget.
The parts that depend only on geometry and layout are now built once and
cached per layout, ranking replaces top-k so no host side k is needed,
and density became a method that reads back only when something asks.
The block mask also stops sorting a mask of zeros to fill partial slots
it leaves empty. Selection overhead at sdxl drops from 1.84 ms to
0.23 ms, and a 30 percent budget moves from 0.36x of dense to 1.19x.
The block section runs before the attention section, so the split table
read an empty attention report and instrument B never engaged. The splits
now render from the stored block results once both sections are in.
- the agreement gate admits the measured 0.1-2.5% systematic offset between the two instruments (strided projection views vs contiguous standalone tensors); a noise-only threshold rejected every row
- presets and block geometries can declare their own config timeout; autotune cost grows with the square of the sequence and the 300 s default cut every sdnq row at 109k tokens
Extend cli/sdnq-attention-benchmark.py ahead of block-sparse attention work:
- BenchBlock takes head_dim and a swiglu mlp variant, so geometries whose
attention is wider than the residual stream (MiniMax H3: 56x128 over a
5376 stream) are expressible; existing geometries are unchanged
- attention masks move from a hardcoded preset ladder into mask_fn preset
keys with an element guard, since h3-scale dense masks would allocate
gigabytes; per-preset iters/warmup/ref_head_chunk overrides and a
head-sliced fp32 reference bound the cost of very large shapes
- new presets: h3 (38222 tokens) and h3-long (109574) from the released
canvas and vae geometry, gate-2k..64k crossover probes at h3 geometry,
an h3 block geometry, and sparse/gate run aliases for --shapes
- --block-geometries selects block-section geometries; h3 runs only when
selected
- each block config is re-timed with an identity attention_fn and the
compute split is reported per config, cross-checked against the
standalone attention table when the same run benched a matching shape;
speedup ceilings per kv budget are stated only where the instruments
agree
A sequence below the minimum length left the feature enabled and idle
with nothing in the log. The stage now records why it declined and says
so once per length, so an enabled setting that cannot act is visible
rather than quiet.
Sparse attention is a stage over the chain rather than a member of it:
one switch, and the router hands the selection to whichever active
backend advertises that it consumes a block mask, currently flex. A
backend declares that through a capability set, so the quantized kernel
joins later without touching the router.
The stage gates on the component role, self attention, a minimum
sequence length defaulting to the measured 8192 token crossover, and the
absence of a token mask or causal flag, which flex cannot combine with a
block only mask. Budgets follow a precomputed per step schedule with at
most two distinct values. Enabling the feature with no capable backend
in the chain warns and leaves attention dense rather than doing nothing
quietly.
The modular pre-forward hook now receives kwargs and publishes whatever
token layout the pipeline passes by name, so a packed sequence gets its
conditioning pinned without any model specific code. Without a layout
the whole sequence is sparsified and that is logged once per length.
The selector mean-pools query and key tiles, scores the tile pairs and
keeps the highest scoring key tiles per query tile within a budget
expressed as a fraction of the sparsifiable candidates. No scale and no
softmax, since top-k is invariant under both. Scoring runs on query
heads so grouped attention needs no key expansion, the diagonal is
always kept so no query row is left empty, and a budget that covers
every candidate reports dense instead of building a full mask.
The layout says what a packed sequence holds. Only video and image
tokens are sparsifiable; text, conditioning, audio and anything
unrecognized pin their rows and columns dense, padding is dropped, and a
tile straddling a boundary pins. Layouts come from the *_indices tensors
a pipeline passes its transformer by name, from ordered segments where a
transformer packs the sequence itself, or from a leading prefix as a
fallback.
The flex consumer builds a BlockMask with every selected tile in the
full slots, so mask_mod is never invoked and no dense mask is
materialized, and calls flex_attention compiled: called eagerly it reads
mask_mod rather than the block lists, so a block only mask attends
densely and silently. test/test-attention-sparse.py covers this with a
row that fails if the selection stops changing the output, alongside
tile equivalence against sdpa fed the same tiles, measured against the
flex kernel floor rather than an absolute tolerance.
Hosted truncation kept the full rank cap even when the spectrum ends in
numerical zeros, padding exactly low-rank deltas (low-rank LyCORIS,
full-family diffs) up to the cap. Slice the kept factors where
cumulative capture reaches 1 - 1e-6 of the sketch total, and trim
trailing all-zero columns when attaching cache entries written before
the slice, so they collapse the same way without a format bump. The
hosted log line reports the realized rank spread when it sits below
the cap.
- flat spectra keep the cap; a rank-8 delta under cap 256 stores 8 ranks
- select segments follow the effective rank
- suite pins the collapse, the flat-spectrum guard and the padded-entry trim
A cache hit still paid full calc_updown per layer, the dominant cost of a
replayed apply. The walk now probes the pass's cache entry first through
apply_cached, which evaluates the requantize routing rule from the delta
rms stored in the entry and attaches the cached factors without assembling
the delta; layers the rule declines fall through with the assembled delta
reused for the requantize path instead of recomputing it. The entry format
gains the rms and bumps to fmt 5, so older files recompute once and
rewrite. Hit accounting stays single-count through a plain lookup plus an
explicit hit note.
- replayed krea2 LoKR apply drops calc from 9.1s to 0.2s and activate
from 39.5s fresh to 4.8s replayed, hosted/routed split identical
- three cache tests pin the calc skip on plain, mixed and dense-pair sets
Hosting truncates every non-factorable set on sub-8-bit layers, but a
delta large against the grid step whose truncation capture is low is
retained better by the grid than by the rank cap. apply_hosted now
returns such layers to the requantize path when rms(delta)/mean(step)
exceeds 0.30 and the sketch capture falls below 0.90, thresholds sized
on 610 calibrated modules across krea2 and anima. Both terms must
agree: a thin delta rounds away on the grid however low its capture,
and a low-rank delta hosts exactly however fat it is.
- scoped to pure non-factorable sets; factorable members, dense-combined
deltas and svd-channel checkpoints keep hosting
- factor cache entries memoize the decision through their stored
capture, so replays route without re-running the sketch
- routed layers log as info apart from the forced-fallback warning
- the fidelity CLI applies the same rule so its reports track the loader
- seven routing tests, constants module-level and test-overridable
Persist now fires either when every hooked layer reaches the token quota
or at a fixed denoiser-forward deadline; layers under a small token floor
are omitted and stay on plain truncation. The all-done barrier alone
never fired on models whose modulation or pooled projections see a few
tokens per forward, so hooks stayed registered forever and statistics
recollected every session. eligible_modules walks the transformer or the
unet, so unet checkpoints collect statistics at all.
- lora_sdnq: drop the unused use_previous parameter; the factor cache
store now always runs inside a begin_pass
- tests: deadline persist, token-floor omission, unet root walk
The compiled dequant specializes per factor rank, so each distinct lora
set shape paid a compile stall on switch. Pad appended factors to a
power-of-two rank ladder (multiples of 64 past the hosted cap) with zero
fill: switches inside a bucket reuse the compiled graph, and common
trained ranks land on their bucket exactly so padding is usually a
no-op. Regression tests pin the factor add inside the single compiled
graph and the bucket reuse.
Sketch hosted deltas at rank+64 with eight power iterations and keep the
top rank columns; this lands within noise of the exact decomposition at
roughly twice a sketch cost the factor cache pays once per configuration.
Bump the cache format so narrower-sketch entries reload as misses.
Rowwise int8 with fp32 scales halves cache entries; measured in output
space on real hosted deltas the roundtrip is fidelity-free (within
0.0003 of fp32 factors, bf16 storage likewise). Factors are quantized
before first use and the dequantized roundtrip is what the apply
attaches, so a fresh compute and a later cache hit stay bit-identical;
the entry format is versioned and pre-int8 entries reload as misses.
- lora_factor_cache: quantize_rowwise/dequantize_rowwise, store returns
the applied pair, fmt guard on read
- test/test-sdnq-lora-factors.py: int8 quantization test, entry-size
assertion in the roundtrip test
Hosting a non-factorable set costs one truncated svd per layer on every
fresh apply. The factors are deterministic in the checkpoint, the loaded
set, the host rank and the calibration statistics, so they persist under
data/lora-factor-cache keyed by that identity and replay bit-identically
on later applies. lora_sdnq_host_cache sets the disk budget in GB
(0 disables), least-recently-used entries are evicted past it. With the
cost paid once per configuration, svd subspace iterations rise from 2 to
4: about two thirds of the captured-energy gap to an exact decomposition
for +45% one-time compute; an exact svd measures 200-400x slower at
these shapes and is not viable.
- modules/lora/lora_factor_cache.py: signature, store, flush, eviction
- lora_sdnq.apply_hosted: fetch before computing, store after, hits in
the load summary
- cli/lora-quant-fidelity.py: matching niter for the hosted mirror
- test/test-sdnq-lora-factors.py: factor-cache category, 3 tests
process_images swallows the interrupt assertion, so a cancel reaches the shared
video core as an empty result and was raised as 'processing failed' with 500,
leaving clients unable to tell a cancel from a crash. Mirrors the LTX path,
which already returns 499.
Queue.__exit__ returned _queue_lock, and a truthy __exit__ return suppresses the
exception in flight, so every `with queue_lock:` block discarded exceptions and
resumed with locals from the aborted block unassigned. The bare threading.Lock it
replaced returned None.
The refine-default comment claimed Condition variants are excluded from two-stage
refine, but supports_two_stage_refine has no such carve-out and ltx_process builds
a second condition set so conditioning survives the upsample. Two file references
had also drifted: ltx_process.py:179 moved, and the offload hook keying now lives
in sd_offload_balanced. Name the symbols instead of the line numbers.
CalamitousFelicitousness
Vladimir Mandic
Dity0
Alejandro Alberto Aguilar Morales