Public/automatic
1
0
Fork 0
mirror of https://github.com/vladmandic/automatic synced 2026-09-20 01:31:13 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat(attention): sparse attention settings and router stage

Browse Source 
Sparse attention is a stage over the chain rather than a member of it:
one switch, and the router hands the selection to whichever active
backend advertises that it consumes a block mask, currently flex. A
backend declares that through a capability set, so the quantized kernel
joins later without touching the router.

The stage gates on the component role, self attention, a minimum
sequence length defaulting to the measured 8192 token crossover, and the
absence of a token mask or causal flag, which flex cannot combine with a
block only mask. Budgets follow a precomputed per step schedule with at
most two distinct values. Enabling the feature with no capable backend
in the chain warns and leaves attention dense rather than doing nothing
quietly.

The modular pre-forward hook now receives kwargs and publishes whatever
token layout the pipeline passes by name, so a packed sequence gets its
conditioning pinned without any model specific code. Without a layout
the whole sequence is sparsified and that is logged once per length.
This commit is contained in:
CalamitousFelicitousness
2026-08-22 23:32:18 +01:00
parent aa4aa57fe8
commit d75abce642
10 changed files with 286 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/ui_definitions.py
+8
View File
@@ -262,6 +262,14 @@ def create_settings(cmd_opts):
"sdnq_attention_pv_matmul_type": OptionInfo("disabled", "SDNQ Attention PV MatMul type", gr.Radio, {"choices": sdnq_matmul_modes}),
"sdnq_attention_hadamard_group_size": OptionInfo(256, "SDNQ Attention Hadamard Group Size", gr.Slider, {"minimum": 4, "maximum": 1024, "step": 1}),
"sparse_attention_sep": OptionInfo("<h2>Sparse Attention</h2>", "", gr.HTML),
"sparse_attention_enabled": OptionInfo(False, "Sparse Attention", gr.Checkbox),
"sparse_attention_budget": OptionInfo(30, "Sparse Attention KV budget", gr.Slider, {"minimum": 5, "maximum": 100, "step": 5}),
"sparse_attention_min_tokens": OptionInfo(0, "Sparse Attention minimum sequence", gr.Slider, {"minimum": 0, "maximum": 65536, "step": 1024}),
"sparse_attention_schedule_steps": OptionInfo(0, "Sparse Attention dense steps", gr.Slider, {"minimum": 0, "maximum": 10, "step": 1}),
"sparse_attention_schedule_bump": OptionInfo(30, "Sparse Attention dense step bonus", gr.Slider, {"minimum": 0, "maximum": 70, "step": 5}),
"sparse_attention_head_shared": OptionInfo(False, "Sparse Attention share selection across heads", gr.Checkbox),
"hf_attention_sep": OptionInfo("<h2>Attention Dispatcher</h2>", "", gr.HTML),
"hf_attention": OptionInfo('', "Attention dispatcher kernel", gr.Textbox),
}))