mirror of
https://github.com/vladmandic/automatic
synced 2026-09-19 09:14:35 +02:00
docs(settings): name the sparse attention sequence floor
The hint deferred to an unnamed built in threshold where every comparable hint in the file gives the value. It is a single static constant.
This commit is contained in:
@@ -1473,7 +1473,7 @@
|
||||
{"id":"","label":"SDNQ Attention Quantize FP32","localized":"","hint":"Upcasts queries, keys and values to fp32 for the quantization step, meaning the mean subtraction, scale and rounding that produce the low precision operands. The matmuls themselves are unaffected, and the kernel applies the scales in fp32 either way.<br>Turned off, that arithmetic runs in the model's own precision. bf16 carries eight mantissa bits, so a scale derived in it is coarser than one derived in fp32, and <b><i>SDNQ Attention use Smooth K</i></b> loses the most from it, since a mean across the whole sequence is exactly the kind of sum that wants the extra bits.<br><br>Whether the upcast costs anything depends on how the GPU runs fp32 vector work against fp16 and bf16. NVIDIA and AMD run them at the same rate here, so there is nothing to save; Intel runs fp32 slower and takes a noticeable hit.<br><br>Enabled by default.","ui":"settings_cuda"},
|
||||
{"id":"","label":"Sparse Attention","localized":"","hint":"Attention is computed on a subset of key tiles instead of all of them, which lowers cost on long sequences. Requires an attention backend that accepts a block mask, <b>SDNQ attention</b> or <b>Flex attention</b> in <b><i>SDP overrides</i></b>. Below the minimum sequence the setting stays inactive and attention is dense.","ui":"settings_cuda"},
|
||||
{"id":"","label":"Sparse Attention KV budget","localized":"","hint":"Percentage of the eligible key tiles each query tile keeps. Lower budgets are faster and coarser. Text, conditioning and audio tokens are always kept, as are the tiles on the diagonal, so the budget applies only to the bulk image or video tokens.","ui":"settings_cuda"},
|
||||
{"id":"","label":"Sparse Attention minimum sequence","localized":"","hint":"Shortest sequence that is sparsified. Below it attention stays dense, because the selection costs more than it saves. <b>0</b> uses the built in threshold.","ui":"settings_cuda"},
|
||||
{"id":"","label":"Sparse Attention minimum sequence","localized":"","hint":"Shortest sequence that is sparsified. Below it attention stays dense, because the selection costs more than it saves. <b>0</b> uses <b>8192</b>.","ui":"settings_cuda"},
|
||||
{"id":"","label":"Sparse Attention dense steps","localized":"","hint":"Number of steps at the start and end of sampling that receive a larger budget, where composition and detail are set. <b>0</b> applies one budget to every step.","ui":"settings_cuda"},
|
||||
{"id":"","label":"Sparse Attention dense step bonus","localized":"","hint":"Percentage points added to the budget on the dense steps.","ui":"settings_cuda"},
|
||||
{"id":"","label":"Sparse Attention share selection across heads","localized":"","hint":"One selection is computed for all attention heads rather than one per head. Cheaper to select and coarser in what it keeps.","ui":"settings_cuda"}
|
||||
|
||||
Reference in New Issue
Block a user