docs(settings): cross-reference sparse attention from the sdp hints

The SDP overrides and SDNQ Attention hints describe the block mask capability
without naming what consumes it, so the two backends that can drive sparse
attention now say so where the reader is choosing between them.
This commit is contained in:
CalamitousFelicitousness
2026-08-24 23:05:53 +01:00
parent f0531e1c88
commit 8138424bbe
+2 -2
View File
@@ -1462,8 +1462,8 @@
{"id":"","label":"Sections","localized":"","hint":"","ui":"video"},
{"id":"","label":"Samplers","localized":"","hint":"Samplers/schedulers advanced settings","ui":"tab_txt2img"},
{"id":"","label":"SDP kernels","localized":"","hint":"Which of torch's built-in attention kernels torch is allowed to choose from. These are permissions rather than a selection: torch picks one per call from whatever is left enabled, preferring <b>Flash</b>, dropping to <b>Memory</b> for calls flash cannot serve such as those carrying an arbitrary attention mask, and to <b>Math</b> when neither fits. Clearing a box removes a candidate; it never pins the remaining one to every call.<br><br><b>Flash</b> is torch's own build of the FlashAttention kernel. It is not the same thing as the <b>Flash attention</b> entry in <b><i>SDP overrides</i></b>, which calls the separately installed flash-attn package and bypasses torch entirely.<br><b>Memory</b> is the memory-efficient kernel, which accepts arbitrary masks that flash does not.<br><b>Math</b> is the unfused reference path, the widest in what it accepts and the least optimized. Leaving it enabled keeps a fallback for calls the other two decline.<br><br>Applies while <b><i>Attention method</i></b> is <b>Scaled-Dot-Product</b>, and continues to govern the calls that an enabled override declines.<br><br>All three by default. ZLUDA starts with <b>Math</b> alone.","ui":"settings_cuda"},
{"id":"","label":"SDP overrides","localized":"","hint":"Replaces torch attention with another implementation. Each entry declares the shapes, dtypes and mask conditions it can serve; a call that fails them moves to the next entry and finally back to torch, so several can be enabled together and the chain resolves per call.<br><br><b>Flash attention</b> installs and calls the flash-attn package directly, for calls with no attention mask, half precision inputs and a head dimension of 128 or less.<br><b>Sage attention</b> computes attention with quantized matmuls, for head dimensions of 64, 96 or 128 and no attention mask.<br><b>SDNQ attention</b> is SD.Next's own quantized Triton kernel, configured in the section below. It takes attention masks, which the other quantized backends do not.<br><b>Flex attention</b> uses torch's compiled flex_attention.<br><b>Dynamic attention</b> slices attention to fit available memory and serves every call the others decline, standing in for the torch fallback.<br><b>Triton Flash attention</b> is a Triton implementation for ROCm and ZLUDA, listed only on those backends.<br><br>The quantized and compiled backends trade some numerical accuracy for throughput. How much of each arrives depends on the model, the sequence length and the GPU, so comparing them on the actual workload settles it faster than picking by reputation.<br><br>None by default on CUDA. ZLUDA, CPU and MPS start with <b>Dynamic attention</b>, as do ROCm GPUs older than RDNA3.","ui":"settings_cuda"},
{"id":"","label":"SDNQ Attention","localized":"","hint":"Settings for the <b>SDNQ attention</b> entry in <b><i>SDP overrides</i></b>. They do nothing until that override is enabled.<br><br>The kernel quantizes the two matmuls inside attention, computing them on lower precision operands and rescaling the result. It is written in Triton, so it needs a working Triton for the active device.<br>Short sequences and single-head calls are left to the rest of the chain, so text encoders and the VAE keep ordinary attention.","ui":"settings_cuda"},
{"id":"","label":"SDP overrides","localized":"","hint":"Replaces torch attention with another implementation. Each entry declares the shapes, dtypes and mask conditions it can serve; a call that fails them moves to the next entry and finally back to torch, so several can be enabled together and the chain resolves per call.<br><br><b>Flash attention</b> installs and calls the flash-attn package directly, for calls with no attention mask, half precision inputs and a head dimension of 128 or less.<br><b>Sage attention</b> computes attention with quantized matmuls, for head dimensions of 64, 96 or 128 and no attention mask.<br><b>SDNQ attention</b> is SD.Next's own quantized Triton kernel, configured in the section below. It takes attention masks, which the other quantized backends do not, and it is one of the two backends <b><i>Sparse Attention</i></b> can drive.<br><b>Flex attention</b> uses torch's compiled flex_attention, the other backend <b><i>Sparse Attention</i></b> can drive.<br><b>Dynamic attention</b> slices attention to fit available memory and serves every call the others decline, standing in for the torch fallback.<br><b>Triton Flash attention</b> is a Triton implementation for ROCm and ZLUDA, listed only on those backends.<br><br>The quantized and compiled backends trade some numerical accuracy for throughput. How much of each arrives depends on the model, the sequence length and the GPU, so comparing them on the actual workload settles it faster than picking by reputation.<br><br>None by default on CUDA. ZLUDA, CPU and MPS start with <b>Dynamic attention</b>, as do ROCm GPUs older than RDNA3.","ui":"settings_cuda"},
{"id":"","label":"SDNQ Attention","localized":"","hint":"Settings for the <b>SDNQ attention</b> entry in <b><i>SDP overrides</i></b>. They do nothing until that override is enabled.<br><br>The kernel quantizes the two matmuls inside attention, computing them on lower precision operands and rescaling the result. It is written in Triton, so it needs a working Triton for the active device.<br>It takes an attention mask and a block mask together, which is what lets <b><i>Sparse Attention</i></b> use it.<br>Short sequences and single-head calls are left to the rest of the chain, so text encoders and the VAE keep ordinary attention.","ui":"settings_cuda"},
{"id":"","label":"SDNQ Attention use Smooth K","localized":"","hint":"Subtracts the mean of the keys before quantizing them. Keys carry a large offset that is shared across the sequence, which spends most of the quantized range representing a value identical for every key and leaves little of it for the differences that decide the attention.<br>Softmax ignores a constant shift applied to every score in a row, so removing that offset changes the quantization error and not the attention.<br><br>Costs one mean and one subtraction per call.<br>Enabled by default.","ui":"settings_cuda"},
{"id":"","label":"SDNQ Attention use Hadamard","localized":"","hint":"Rotates queries and keys by a Hadamard transform before quantizing them. The rotation spreads a few oversized channels across all of them, which is the error shape quantization handles worst. The transform is orthogonal, so the scores it produces are the ones the unrotated tensors would produce, minus the quantization error it removes.<br>With <b><i>SDNQ Attention PV MatMul type</i></b> also set, the values are rotated as well and the output is rotated back.<br><br>Costs a rotation pass on every attention call, so it is worth enabling where a model shows quantization artifacts without it.<br>Idle while <b><i>SDNQ Attention MatMul type</i></b> is <b>disabled</b>, since nothing is quantized then.<br><br>Disabled by default.","ui":"settings_cuda"},
{"id":"","label":"SDNQ Attention use FP16 Accumulation","localized":"","hint":"Accumulates the floating point matmuls in fp16 rather than fp32. Some tensor cores run fp16 accumulation at a higher rate than fp32, and on those the kernel is cheaper for it.<br>Operands are pre-scaled to keep products inside the fp16 range, which covers ordinary activations with less headroom than fp32 leaves.<br><br>Reaches the parts of the kernel that run in floating point. An int8 matmul accumulates in int32 and is unaffected, so at the default <b><i>SDNQ Attention MatMul type</i></b> this applies to the probability-value matmul alone.<br><br>Disabled by default.","ui":"settings_cuda"},