The linear layers bind their scaled-mm function at import, so
SDNQ_USE_TRITON_MM freezes the backend per process and comparing triton
against the torch fallback meant two runs, carrying clock drift into the
delta. --mm-backends rebinds the function on the consuming modules
between benches, so both rows measure the same quantized layer under the
same clock state.
- swap targets are the four linear modules that from-import a scaled-mm
function; every swap resets dynamo, since the layer forwards are
compiled and the traced graph pins the previous function
- the torch row is captured from kernel_wrappers rather than
reimplemented, so it is unavailable where triton is the device default
(rerun with SDNQ_USE_TRITON_MM=0) and says so
- rounds alternate order and keep the fastest per row, so drift cancels
instead of favouring whichever backend runs second
- flag output error when backends disagree past 1e-4: they are meant to
be numerically equivalent
probe_fp8 and the broken-compile bench path forced eager input prep by
toggling torch._dynamo.config.disable. torch 2.13+ raises 'found no
compiled frames' when a fullgraph-compiled function is called inside a
disable window, so the probe failed before reaching the attention kernel
and reported an environment error instead of the kernel's real verdict,
on fp8-capable gpus too. Swap the module-global prep function for its
eager inner instead, matching the correctness section.
The environment panel and json now carry python, platform, backend and
the backend runtime versions (cuda/cudnn/driver, hip, ipex, openvino,
directml via devices.get_gpu_info), so shared reports identify the stack
without inferring it from the torch version string.
The webui bootstrap log is gated behind an fd-level suppression for the
import window: the bootstrap reconfigures its loggers during import and
onnxruntime warns from c++, so logger levels alone cannot silence it.
Import failures still print after the gate lifts.
After the correctness section aborts on a sticky kernel fault, the
remaining sections cannot run; flush outputs and exit instead of
cascading into the next cuda call's traceback.
A kernel fault (misaligned address, illegal memory access) is sticky:
every later cuda call in the process fails, so one faulting check killed
the whole run through the next reference computation and lost the --save
and --json outputs. Compute the correctness reference inside the
per-check handler, detect a dead context and abort the section with a
clear message, and flush partial outputs from a top-level handler on any
crash or interrupt.
The fp8 attention probe can fail for reasons other than missing hardware
support (torch or triton compile issues); reporting those as 'not supported
on this gpu' is wrong on gpus where fp8 is native. Match the triton
pre-sm_89 signature and label anything else as an environment failure.
Toggling torch._dynamo.config.disable to force eager input prep fails on
newer torch: a fullgraph-compiled function called inside a disable window
raises 'found no compiled frames', failing every check and poisoning the
first compiled call afterwards. Swap the module-global prep function for
its eager inner instead, matching the dequant section's mechanism.
- store per-check errors and exceptions in the json report, so remote
reports carry the failure mode
Benchmark weight dequantization alongside attention: eager vs compiled
dequantization, measured standalone and through the full linear forward,
plus the quantized matmul forward, across int/uint 8-6-4-2 and the
pre-quantized float formats (float8_e4m3fn, float8_e4m3fn_sdnq,
float4_e2m1fn group 16) at krea 2 layer geometry, against a bf16 nn.Linear
baseline and a true-fp32 output reference. Each row carries measured
storage size and one-shot quantize time; an svd/hadamard variants table
measures rotation and low-rank costs on top of the base dtypes.
A combined block section measures complete configurations (weights dtype x
matmul path x attention) end to end through a dit-style transformer block,
with output error at depth one and four against an fp32 reference block,
because component speedups and errors do not compose multiplicatively.
- recommendations weigh error against speed: a faster option is rejected
when it multiplies measured output error beyond 2x, and reason strings
cite both numbers; notes include measured size/error and speed/error
frontiers
- robustness: extreme-activation stress rows in the correctness matrix
(pass on finite output), max-token-error columns beside norm error, and
non-finite outputs labeled as verdicts; norm metrics alone hide
token-level corruption on outlier-heavy inputs
- probe compiled weight dequant for e4m3 and e5m2 storage and report the
fp8 compile gate status; on ampere e5m2 compiles while e4m3 does not
- new attention presets: sdxl-cross (cross-attention), qwen3-te (causal
gqa text encoder), wan22-cfg (batched cfg video); cross-attention
correctness check
- --sections, --dequant-dtypes, --dequant-variants and --block-configs
selectors, --json structured results output
sdnext enables tf32 globally; a math-backend dispatch fallback (gqa,
some mask shapes) degrades the fp32 reference to tf32 precision,
measured 4.2e-4 vs 5.0e-7 relative to fp64. the default mem-efficient
backend is unaffected.
Recommendations must hold on any gpu, so every verdict now comes from
measuring the configuration the advice points to.
- run benchmarks in the webui's configured dtype, --dtype overrides
- bench the prep mode matching the environment: compiled, verified
dynamic=false workaround, or eager; label each with the webui change
the numbers correspond to
- add a prep column for the q/k/v quantization cost outside the
kernel; eager prep measured 1.9 ms vs 0.2 ms fused at sdxl,
inverting the quantized matmul verdict
- compare Use Quantized MatMul against the unquantized sdnq row so
quantization pays for its own prep
- star the lowest-error config within 5% of the fastest sdnq time;
the same rule drives the worth-enabling verdict vs torch sdpa
- drop the (auto, default) suffix; document the settings mapping and
star rule in the notes
- wrap correctness and shape tables in panels to separate sections
The pv matmul recommendation trusted the float8 hardware probe alone,
so gpus where fp8 pv compiles but runs slower than unquantized pv got
a recommendation that degrades generation speed and error. Measured on
an rtx 2000 ada, fp8 pv lands at x0.93 of the sdpa baseline while int8
qk alone reaches x1.26.
- add fp8 pv and fp8 qk bench configs, gated on the eager float8
probe so unsupported gpus keep clean tables
- include fp8 pv in the video presets since the pv stage matters most
at long sequence lengths
- recommend a quantized pv type only when the fastest measured pv
variant beats plain int8 qk by at least 5%, otherwise recommend auto
and name the variants that measured no gain
- cite the measured fp8 qk ratio in the matmul type reason instead of
the static compile claim
The environment panel reported triton as missing and float8 as
unsupported on ada gpus. The triton lookup used the distribution name,
which varies by platform (triton-windows on windows), and the fp8 probe
ran through the torch.compile input prep, so a broken compile toolchain
read as a hardware limitation with the actual error discarded.
- read the module version for the triton line instead of the
distribution name
- probe float8 with eager input prep so the result reflects gpu
capability; show the failure reason when genuinely unsupported
- probe the compiled input prep separately and cold, before any eager
kernel run: inductor lowers part of the dynamic-shape prep to a cpu
helper kernel, so a missing host c++ compiler fails every sdnq
attention call at generation, and a prior eager run in the same
process masks the cold-start failure the webui hits
- on failure, verify the SDNQ_COMPILE_KWARGS dynamic:false workaround
live and report it in the environment panel and recommendation notes
- benchmark with eager input prep when the compiled path is broken so
kernel numbers still fill in
- replace the unicode star marker with ascii: rich's legacy windows
renderer crashes with a cp1252 UnicodeEncodeError on piped output
Benchmark and validate sdnq attention on the local gpu: correctness
checks for mask, causal, gqa and padding code paths, float8 support
probe, per-shape speed and error comparison against torch sdpa and
sageattention, and measured recommendations for the settings section.
Tables fill in live with per-config compile/warmup/timing progress,
current settings are compared against recommended values, and --save
writes a plain-text transcript of the results. --config-timeout turns
stuck compiles into failed rows; sd15 skips its hadamard configs since
compiling hadamard with a non pow2 head dim currently hangs inductor.
- shape presets follow real model geometries: sd15, sdxl, anima,
flux2 klein, wan22 a14b, ltx 2.3, plus a masked joint-attention case
- runs the shipped kernel from modules/sdnq/kernels/triton_atten.py
directly, with stock sdpa restored for baselines and references
- Remove remote manifest fetch from startup dropdown population;
list only locally available tag files via files_cache.list_files()
- Rename CLI utilities to tags-fetch, tags-prune, tags-manifest
- Replace create/destroy mirror div with persistent offscreen mirror
for caret position measurement
- Add ResizeObserver to reposition dropdown on textarea resize
- Add autocomplete-mirror classname to measurement div
- Remove superfluous SimpleNamespace import in cli/api-caption.py, use Map instead
- Drop _ prefix from internal helper functions in modules/api/caption.py
- Move DeepDanbooru model path to top-level models folder instead of nesting under CLIP
- Rename shadowing import in waifudiffusion batch to avoid F823/E0606
- Fix import order in cli/api-caption.py (stdlib before third-party)
- Rename local variable shadowing function name in cli/api-caption.py
- Remove unnecessary global statement in devices.bypass_sdpa_hijacks
- Add use_safetensors=True to all 16 model from_pretrained calls to
avoid downloading redundant .bin files alongside safetensors
- Add device property to JoyTag VisionModel so move_model can relocate
it to CUDA (fixes 'ViT object has no attribute device')
- Fix Pix2Struct dtype mismatch by casting float inputs to model dtype
while preserving integer tensor types
- Patch AutoConfig.register with exist_ok=True during Ovis loading to
handle duplicate aimv2 registration on model reload
- Detect Qwen VL fine-tune architecture from config model_type instead
of repo name, fixing ToriiGate and similar third-party fine-tunes
- Change UI default task from Short Caption to Normal Caption, and
preserve it on model switch instead of resetting to Use Prompt
- Add dual-prefill testing across 5 VQA test methods using a shared
_check_prefill helper
- Fix pre-existing ruff W605 in strip_think_xml_tags docstring
- Remove caption_openclip_min_length from settings, API models, endpoints, and UI
(clip_interrogator library has no min_length support; parameter was never functional)
- Split vlm_prompts_florence into base Florence prompts and PromptGen-only prompts
(GENERATE_TAGS, Analyze, Mixed Caption require MiaoshouAI PromptGen fine-tune)
- Add 'promptgen' category to /vqa/prompts API endpoint
- Fix gaze detection: move DETECT_GAZE check before generic 'detect ' prefix
to prevent "Detect Gaze" matching as detect target="Gaze"
- Update test suite: remove min_length tests, fix min_flavors to use mode='best',
add acceptance-only notes, fix thinking trace detection, improve bracket/OCR tests,
split Florence/PromptGen test coverage
Update cli/test-caption-api.py:
- Update test structure for new caption API endpoints
- Fix Moondream gaze detection test prompt to use 'Detect Gaze'
instead of 'Where is the person looking?' to match handler trigger
- Improve test result categorization and tracking
CalamitousFelicitousness
Disty0
Vladimir Mandic
QualiaRain
awsr