The triton autotune hijack draws its own rich progress bar on the
logger console during sweeps; a second live display on the same
terminal tramples the benchmark's live tables and leaves stale panel
border lines in scrollback. Disarm the bar in the benchmark process
and keep the hijack's bookkeeping.
Write the text and json reports by default, named to the archive
convention <gpu>-t<torch>-<date> with a collision suffix for the pair,
into $SDNQ_BENCH_DIR, --outdir, or benchmarks/ under the sdnext root.
'none' disables either output and explicit paths behave as before.
Add accum configs, value-side stress rows, a recommendation row, block
specs, and a triton-mm variant for the new use_fp16_accum surface. The
unsafe unquantized-pv mode is measured but never starred or recommended;
accum rows skip on sdnq builds without the kwarg. Also move the dequant
compile verdict off the removed sdnq_dequantize_compile option to a note
keyed on SDNQ_USE_TORCH_COMPILE, fixing the crash after 2250d8052.
The ref2va checkpoint partition conditions on reference images instead
of keyframes, so it gets its own registry row and reference card, and
the video core marshals PIL images into task_args as
MiniMaxH3ImageReference. Images are converted to RGB first, since the
reference encoder reads the array raw. The keyframe path is unchanged.
Validation runs before the model load in one funnel shared by the tab
and the API, so a rejected request costs nothing: references on a
non-reference model, a reference model with nothing to condition on,
more than nine images, non-images, and aspect outside 1:4 to 4:1 all
return 400. The image path rejects a reference pipe without references
instead of reaching a transformer that was never loaded.
Add POST /sdapi/v1/video plus GET /sdapi/v1/video/models and
GET /sdapi/v1/video/file. The generation body is extracted from the
gradio handler into a keyword-only core, video_run.run, which returns a
structured result and raises typed errors; the positional generate
signature is unchanged and now adapts to the core. Omitting engine and
model drives the currently loaded checkpoint when it is video-capable,
which covers models loaded from local folders without a registry entry.
- registry helpers in models_def (find, engines, pipeline_classes,
workflow_for_class); validate_pipeline reuses the shared class set
- modular pipes stamp their workflow so out-of-registry loads dispatch
onto the modular branch
- disk switches (mp4_*) and wire switches (send_*) are independent;
artifacts above the base64 cap fall back to path plus the file route,
which is jailed to the video output directory and serves video/mp4
with range support
- always-on video scripts get bootstrapped default args, matching the
txt2img handler; missing bootstrap raised a TypeError per frame
- checkpoint overrides are rejected with a pointer to the checkpoint
endpoint; unknown engine, model and sampler names return 404 with the
valid choices
- cli/api-video.py client, test/test-video-api.py suite and a
full-test.sh entry; video mimetypes registered; rate-limit cost set
- remove the unreferenced video_ui.run_video dispatcher
The settings rows were verdicted one at a time against int8 qk alone, so
the table could assemble a tuple no row ever measured and contradict its
own stars: pv lost a solo 10 percent margin test while the starred best
config carried pv at four of nine shapes, and the emitted smooth-only
stack was strictly dominated by measured rows at three shapes.
- select the recommendation as the best measured quantized config by the
star rule (lowest error within 5 percent of the fastest, error cap vs
int8), then gate that one config against unquantized at a z adjusted
for having selected the best of the pool
- decompose the winner into the settings rows; each reason cites the
measured sibling that isolates its setting, replacing the blanket no
gain wording that hid an 8 percent measured win
- drop the additive stack estimate, composition check and float8 rescue,
which the joint selection covers by construction on measured rows
- name each shape's selected config in the per-shape verdict note and
compute sidak z exactly for any candidate count
Hoisting the function-local imports to module scope wrote
`import importlib.import_module`, which cannot resolve because
import_module is a function in importlib, not a submodule, so the
benchmark failed at startup with ModuleNotFoundError.
- import importlib, which is what the module-scope call sites need
The toggle changes no output, so the verdict is a symmetric
faster/slower test on the layer forward it gates, voted across every
measured dequant shape; the standalone kernel ratio stays in the notes.
Every dequant shape and block geometry comes from a real transformer config,
one source of truth per model.
- flux.1: attention linears 3072x3072, feed-forward 12288x3072, block at
3072 wide, 24 heads, 12288 ff, 4608 joint tokens
- krea 2: standalone wq 6144x6144, swiglu gate 16384x6144, block at 6144
wide, 48 heads, 4608 joint tokens
- the block section measures every geometry in block_geometries; buyback
costs are judged at the block matching the reference shape, with the
geometry named in the reason
- te shape lookup keys on its label instead of a list index
Krea 2 runs joint attention over one text plus image stream with a segment
mask: text is padded to a fixed 512 tokens and the padded tail is masked for
queries and keys both, so fully masked query rows yield nan under sdpa. The
preset carries the transformer's exact (B, 1, L, L) mask, nan-guards the
error path the way the model does, and uses the 48 kernel-level heads left
after gqa expansion at 4608 joint tokens.
- fix the drift fields crashing when save_report rebuilt the run info
Per-preset include lists trimmed video and masked shapes for runtime, which
left the composition check without its smooth_hadamard row there. Only hard
technical exclusions remain, as an exclusion map: sd15 skips hadamard configs
because compiling hadamard with a non pow2 head dim hangs torch inductor.
- wan22, wan22-cfg, ltx2 and masked gain smooth_hadamard, fp16qk, fp8qk and
fp8full rows; availability gates and the per-config timeout still apply
Repeat-pair runs measured 2-4% between-run drift on one machine, enough to
flip threshold verdicts near the margin on every rerun.
- bench keeps its iteration samples: rows carry a median sigma, and per-shape
sentinel re-measurements sample run-level clock drift
- on/off verdicts are three-zone at the run's own noise level; too close to
the margin keeps the current setting and says so
- pv candidates (now including fp16) are tested independently against the
margin at a sidak-adjusted z instead of min-then-threshold
- unmeasured toggle stacks are estimated additively in the composition check
- per-shape qk verdicts print alongside the reference-shape verdict
- cross-gpu error sanity bands flag corrupted measurements
- on/off verdicts share one speed margin (recommend_speed_margin, 10%) across
attention, dequant, compile, te and conv rows
- re-check the recommended toggle stack against unquantized; individually passing
buybacks can eat a marginal qk gain
- judge smooth k and hadamard cost at block scope when measured: kernel rows hand
the prep contiguous q/k/v, real models hand it strided views from the fused qkv
projection; reasons cite both scopes
- give bare float8 qk its own shot at the margin before disabling, the verdict
must not hinge on int8 alone
- compare triton flash against the recommended config, not always int8
- note self-attention shapes that disagree with the reference verdict
- fold the enable rows into the type rows: disabled, enabled or an explicit dtype
- pv keep-unquantized value is disabled; enabled means int8 pv
- add a text encoder row judged at te geometry
- update locale hints, drop the orphaned checkbox hint
"sage" is a different kernel per gpu: sageattention dispatches the pv dtype and
accumulator by arch and cuda version, and modules/attention.py forces the fp16-pv
cuda kernel on sm86, so one label across shared reports compares unlike kernels.
- resolve the row label from the dispatch: sm89 with cuda 12.8+ reads
"sage int8 qk + fp8 pv, fp32+fp16 accum", sm86 reads "fp16 pv, fp32 accum"
- name the sm86-only fp16-accum baseline for what it is
Notes ran to paragraph length and wrapped in the terminal, burying the numbers.
- lead with the topic, keep the numbers and the fix, drop the restatement
- one line per note at 140 columns, except the sd15 compile warning
sdnq no longer skips compile for fp8 storage, it upcasts e4m3 to the scale dtype
before the compiled dequant, so the tool's mirror of the deleted skip_fp8_compile
gate went dead and fed the raw e4m3 weight to its compiled dequant: on pre-ada gpus
that reports a compile failure for rows the webui runs fine.
- mirror the upcast in dequant_args, keep the hardware probe on the raw weight
- drop dequant_gated_to_eager and the gated-to-eager row, fp8 rows are measured now
- say upcast, not eager fallback, in the environment panel and dequant notes
Hiding the sdnext startup log by pointing fds 1 and 2 at a file kills the process
when stdout is a real console: sys.stdout writes through the win32 console api on
the handle behind fd 1, so the first bootstrap log line raises OSError 'the handle
is invalid', and with stderr broken the same way the interpreter aborts with no
message, no traceback and no results.
- swap sys.stdout/sys.stderr to a buffer along with the fd redirect
- capture the startup log instead of discarding it to devnull, replay it on failure
- catch SystemExit: loader.py and installer.py exit on fatal startup errors
The linear layers bind their scaled-mm function at import, so
SDNQ_USE_TRITON_MM freezes the backend per process and comparing triton
against the torch fallback meant two runs, carrying clock drift into the
delta. --mm-backends rebinds the function on the consuming modules
between benches, so both rows measure the same quantized layer under the
same clock state.
- swap targets are the four linear modules that from-import a scaled-mm
function; every swap resets dynamo, since the layer forwards are
compiled and the traced graph pins the previous function
- the torch row is captured from kernel_wrappers rather than
reimplemented, so it is unavailable where triton is the device default
(rerun with SDNQ_USE_TRITON_MM=0) and says so
- rounds alternate order and keep the fastest per row, so drift cancels
instead of favouring whichever backend runs second
- flag output error when backends disagree past 1e-4: they are meant to
be numerically equivalent
probe_fp8 and the broken-compile bench path forced eager input prep by
toggling torch._dynamo.config.disable. torch 2.13+ raises 'found no
compiled frames' when a fullgraph-compiled function is called inside a
disable window, so the probe failed before reaching the attention kernel
and reported an environment error instead of the kernel's real verdict,
on fp8-capable gpus too. Swap the module-global prep function for its
eager inner instead, matching the correctness section.
The environment panel and json now carry python, platform, backend and
the backend runtime versions (cuda/cudnn/driver, hip, ipex, openvino,
directml via devices.get_gpu_info), so shared reports identify the stack
without inferring it from the torch version string.
The webui bootstrap log is gated behind an fd-level suppression for the
import window: the bootstrap reconfigures its loggers during import and
onnxruntime warns from c++, so logger levels alone cannot silence it.
Import failures still print after the gate lifts.
After the correctness section aborts on a sticky kernel fault, the
remaining sections cannot run; flush outputs and exit instead of
cascading into the next cuda call's traceback.
A kernel fault (misaligned address, illegal memory access) is sticky:
every later cuda call in the process fails, so one faulting check killed
the whole run through the next reference computation and lost the --save
and --json outputs. Compute the correctness reference inside the
per-check handler, detect a dead context and abort the section with a
clear message, and flush partial outputs from a top-level handler on any
crash or interrupt.
The fp8 attention probe can fail for reasons other than missing hardware
support (torch or triton compile issues); reporting those as 'not supported
on this gpu' is wrong on gpus where fp8 is native. Match the triton
pre-sm_89 signature and label anything else as an environment failure.
Toggling torch._dynamo.config.disable to force eager input prep fails on
newer torch: a fullgraph-compiled function called inside a disable window
raises 'found no compiled frames', failing every check and poisoning the
first compiled call afterwards. Swap the module-global prep function for
its eager inner instead, matching the dequant section's mechanism.
- store per-check errors and exceptions in the json report, so remote
reports carry the failure mode
Benchmark weight dequantization alongside attention: eager vs compiled
dequantization, measured standalone and through the full linear forward,
plus the quantized matmul forward, across int/uint 8-6-4-2 and the
pre-quantized float formats (float8_e4m3fn, float8_e4m3fn_sdnq,
float4_e2m1fn group 16) at krea 2 layer geometry, against a bf16 nn.Linear
baseline and a true-fp32 output reference. Each row carries measured
storage size and one-shot quantize time; an svd/hadamard variants table
measures rotation and low-rank costs on top of the base dtypes.
A combined block section measures complete configurations (weights dtype x
matmul path x attention) end to end through a dit-style transformer block,
with output error at depth one and four against an fp32 reference block,
because component speedups and errors do not compose multiplicatively.
- recommendations weigh error against speed: a faster option is rejected
when it multiplies measured output error beyond 2x, and reason strings
cite both numbers; notes include measured size/error and speed/error
frontiers
- robustness: extreme-activation stress rows in the correctness matrix
(pass on finite output), max-token-error columns beside norm error, and
non-finite outputs labeled as verdicts; norm metrics alone hide
token-level corruption on outlier-heavy inputs
- probe compiled weight dequant for e4m3 and e5m2 storage and report the
fp8 compile gate status; on ampere e5m2 compiles while e4m3 does not
- new attention presets: sdxl-cross (cross-attention), qwen3-te (causal
gqa text encoder), wan22-cfg (batched cfg video); cross-attention
correctness check
- --sections, --dequant-dtypes, --dequant-variants and --block-configs
selectors, --json structured results output
sdnext enables tf32 globally; a math-backend dispatch fallback (gqa,
some mask shapes) degrades the fp32 reference to tf32 precision,
measured 4.2e-4 vs 5.0e-7 relative to fp64. the default mem-efficient
backend is unaffected.
Recommendations must hold on any gpu, so every verdict now comes from
measuring the configuration the advice points to.
- run benchmarks in the webui's configured dtype, --dtype overrides
- bench the prep mode matching the environment: compiled, verified
dynamic=false workaround, or eager; label each with the webui change
the numbers correspond to
- add a prep column for the q/k/v quantization cost outside the
kernel; eager prep measured 1.9 ms vs 0.2 ms fused at sdxl,
inverting the quantized matmul verdict
- compare Use Quantized MatMul against the unquantized sdnq row so
quantization pays for its own prep
- star the lowest-error config within 5% of the fastest sdnq time;
the same rule drives the worth-enabling verdict vs torch sdpa
- drop the (auto, default) suffix; document the settings mapping and
star rule in the notes
- wrap correctness and shape tables in panels to separate sections
The pv matmul recommendation trusted the float8 hardware probe alone,
so gpus where fp8 pv compiles but runs slower than unquantized pv got
a recommendation that degrades generation speed and error. Measured on
an rtx 2000 ada, fp8 pv lands at x0.93 of the sdpa baseline while int8
qk alone reaches x1.26.
- add fp8 pv and fp8 qk bench configs, gated on the eager float8
probe so unsupported gpus keep clean tables
- include fp8 pv in the video presets since the pv stage matters most
at long sequence lengths
- recommend a quantized pv type only when the fastest measured pv
variant beats plain int8 qk by at least 5%, otherwise recommend auto
and name the variants that measured no gain
- cite the measured fp8 qk ratio in the matmul type reason instead of
the static compile claim
The environment panel reported triton as missing and float8 as
unsupported on ada gpus. The triton lookup used the distribution name,
which varies by platform (triton-windows on windows), and the fp8 probe
ran through the torch.compile input prep, so a broken compile toolchain
read as a hardware limitation with the actual error discarded.
- read the module version for the triton line instead of the
distribution name
- probe float8 with eager input prep so the result reflects gpu
capability; show the failure reason when genuinely unsupported
- probe the compiled input prep separately and cold, before any eager
kernel run: inductor lowers part of the dynamic-shape prep to a cpu
helper kernel, so a missing host c++ compiler fails every sdnq
attention call at generation, and a prior eager run in the same
process masks the cold-start failure the webui hits
- on failure, verify the SDNQ_COMPILE_KWARGS dynamic:false workaround
live and report it in the environment panel and recommendation notes
- benchmark with eager input prep when the compiled path is broken so
kernel numbers still fill in
- replace the unicode star marker with ascii: rich's legacy windows
renderer crashes with a cp1252 UnicodeEncodeError on piped output
Benchmark and validate sdnq attention on the local gpu: correctness
checks for mask, causal, gqa and padding code paths, float8 support
probe, per-shape speed and error comparison against torch sdpa and
sageattention, and measured recommendations for the settings section.
Tables fill in live with per-config compile/warmup/timing progress,
current settings are compared against recommended values, and --save
writes a plain-text transcript of the results. --config-timeout turns
stuck compiles into failed rows; sd15 skips its hadamard configs since
compiling hadamard with a non pow2 head dim currently hangs inductor.
- shape presets follow real model geometries: sd15, sdxl, anima,
flux2 klein, wan22 a14b, ltx 2.3, plus a masked joint-attention case
- runs the shipped kernel from modules/sdnq/kernels/triton_atten.py
directly, with stock sdpa restored for baselines and references
CalamitousFelicitousness
Vladimir Mandic
Dity0
QualiaRain