sparse_attention_exclude is a comma separated denylist matched case insensitively against the architecture, the pipeline class and the denoiser class, so one entry works whichever name is to hand. It resolves once per model rather than per call and declines with a log line. Seeded with CosmosTransformer3DModel, the transformer Anima runs, which returns banded noise at every budget tested against a sound dense baseline; listing the class rather than the architecture covers the other models built on it, none of which have been checked.
The option defaulted to 0 and a property turned that into 8192, which hid
the real default behind a sentinel and left no way to sparsify every
sequence. The default is now 8192 in the settings registry, 0 means no
floor, and the slider tops out at 32768 rather than a length nothing
reaches. A negative value is left unclaimed for a derived floor.
Only the modular path had a per-forward kwargs hook, so a classic pipeline had
no layout and the selector sparsified its whole sequence, prompt conditioning
included. A denoiser pre-hook installed per generation now reads whichever
convention the model uses: the *_indices tensors, or the txt_ids and img_ids a
joint transformer is given.
Packing order is not derivable from the call and differs between architectures,
so only classes verified to pack text first publish; anything else falls back
as before rather than pinning the wrong half of the sequence dense.
The transformer is the only site that knows every segment length, since the
pad to a multiple of 256 happens there and the text stream is padded to a
fixed 512 whatever the prompt. It now publishes the live runs of the mask, so
the selector pins the conditioning dense and drops the padding instead of
sparsifying the whole sequence. At 2048x2048 with a short prompt that is 6 of
264 key blocks per query row moved from pinned to dropped.
The sparse stage declined every call carrying an attention mask, so a model
that passes a padding mask stayed dense at every length. It now serves those
calls when the serving backend declares `masked_block`: sdnq takes attn_mask
and block_mask together, flex would need a mask_mod and does not. A backend
that cannot compose says so once instead of skipping quietly.
SD_SPARSE_PATTERN=radial swaps the content aware selection for a band
around the diagonal at the density the selector would have produced. It
is the control the selector has to beat, and the fallback the kill
criteria name if it does not.
The band honors the same layout pins. Without them it never keeps the
text columns, which sit far from the diagonal for late video rows, so
prompt conditioning is lost and it renders a different scene entirely:
a control that loses for the wrong reason proves nothing.
Selecting blocks synchronized with the accelerator three times per
attention call: once to size the top-k, once to decide whether the
budget covered every candidate, and once to compute the density for
reporting. At the sdxl shape those stalls cost 1.6 ms against a 1.2 ms
attention, so the selector lost to dense at every budget.
The parts that depend only on geometry and layout are now built once and
cached per layout, ranking replaces top-k so no host side k is needed,
and density became a method that reads back only when something asks.
The block mask also stops sorting a mask of zeros to fill partial slots
it leaves empty. Selection overhead at sdxl drops from 1.84 ms to
0.23 ms, and a 30 percent budget moves from 0.36x of dense to 1.19x.
A sequence below the minimum length left the feature enabled and idle
with nothing in the log. The stage now records why it declined and says
so once per length, so an enabled setting that cannot act is visible
rather than quiet.
Sparse attention is a stage over the chain rather than a member of it:
one switch, and the router hands the selection to whichever active
backend advertises that it consumes a block mask, currently flex. A
backend declares that through a capability set, so the quantized kernel
joins later without touching the router.
The stage gates on the component role, self attention, a minimum
sequence length defaulting to the measured 8192 token crossover, and the
absence of a token mask or causal flag, which flex cannot combine with a
block only mask. Budgets follow a precomputed per step schedule with at
most two distinct values. Enabling the feature with no capable backend
in the chain warns and leaves attention dense rather than doing nothing
quietly.
The modular pre-forward hook now receives kwargs and publishes whatever
token layout the pipeline passes by name, so a packed sequence gets its
conditioning pinned without any model specific code. Without a layout
the whole sequence is sparsified and that is logged once per length.
The selector mean-pools query and key tiles, scores the tile pairs and
keeps the highest scoring key tiles per query tile within a budget
expressed as a fraction of the sparsifiable candidates. No scale and no
softmax, since top-k is invariant under both. Scoring runs on query
heads so grouped attention needs no key expansion, the diagonal is
always kept so no query row is left empty, and a budget that covers
every candidate reports dense instead of building a full mask.
The layout says what a packed sequence holds. Only video and image
tokens are sparsifiable; text, conditioning, audio and anything
unrecognized pin their rows and columns dense, padding is dropped, and a
tile straddling a boundary pins. Layouts come from the *_indices tensors
a pipeline passes its transformer by name, from ordered segments where a
transformer packs the sequence itself, or from a leading prefix as a
fallback.
The flex consumer builds a BlockMask with every selected tile in the
full slots, so mask_mod is never invoked and no dense mask is
materialized, and calls flex_attention compiled: called eagerly it reads
mask_mod rather than the block lists, so a block only mask attends
densely and silently. test/test-attention-sparse.py covers this with a
row that fails if the selection stops changing the output, alongside
tile equivalence against sdpa fed the same tiles, measured against the
flex kernel floor rather than an absolute tolerance.
Vladimir Mandic
CalamitousFelicitousness