Plain svd truncation of hosted deltas is optimal in weight space but not
in output space: activations concentrate energy in a few input channels,
so scaling the delta by per-channel input RMS before the svd spends the
rank budget on output error instead. Statistics stream from the model's
own forwards on sub-8-bit SDNQ checkpoints and cache per checkpoint;
measured on real LoKR files this raises output-delta retention by ~0.05
at rank 256 and ~0.09 at rank 64, most on MLP down projections.
- modules/lora/lora_calib.py: capture hooks, per-checkpoint cache under
data/sdnq-calib, statistics land on layers as sdnq_calib_rms; gated by
lora_sdnq_host_calib, skipped when the model is compiled
- lora_sdnq.apply_hosted: weighted truncation when statistics exist,
calib count in the load summary
- cli/sdnq-calibrate.py: complete calibration now against a live server
- cli/lora-quant-fidelity.py --calib: hosted rho scored in the
activation-weighted norm
- test/test-sdnq-lora-factors.py: calibration category, 5 tests
Non-factorable modules on sub-8-bit formats report the hosted figure:
the seeded svd truncation at --host-rank realized through the bf16
materialize, mirroring the loader. The requantize figure stays in
requant_rho; --host-rank 0 restores the old scoring.
The analyzer scored factor-eligible modules applied_rho=1.0 by
construction. The side-channel stores the delta losslessly, but the
dequantizer materializes base + factors in the result dtype, so small
deltas round at the bf16 ulp of the base weight. Score the realized
delta through that rounding; sub-ulp loras now report the same floor an
unquantized bf16 model gives them instead of a false 1.0.
Also survive a broken file and keep completed work: per-lora failures
are recorded and skipped, the report json rewrites after every file,
and a complete flag marks a finished run.
Three defects surfaced by running the analyzer over a full local
collection against shipped checkpoints.
The requantize path hardcoded use_svd=False, but network_add_weights
requantizes with the layer's own svd setting and rank. On an svd
checkpoint the dequantized weight is not on the plain integer grid, so
requantizing without svd produced an error dominated by the discarded
correction and roughly orthogonal to the delta, which read back as a
retention near 1.0 on a grid where the delta is 0.005 of a step. Thread
use_svd, svd_rank and svd_steps through and reuse the returned factors.
Targets whose weight is not a matrix (norm and scale parameters, 205 of
them in one z-image extraction) reached the quantizer and the module stub,
both of which unpack two dimensions and raised. Skip and count them before
either. An all-zero delta (some full-rank extractions ship empty .diff)
divided by its own norm and reported nan; its retention is undefined
rather than erased, so it is excluded from the medians and counted.
Kohya-style files carry an already-underscored base (lora_unet_layers_0_
mlp_gate_proj), which the loader resolves by comparing network_prefix +
path.replace('.', '_') against each module's stamped name, so both sides
are underscored and the file loads. The analyzer instead looked the base
up as a literal dotted module path, so every module of such a file was
reported unmatched: 76 files in a local collection, including 36 of 57
anima and 5 of 10 chroma.
Fall back to a stamped-name index when the direct lookup misses. Dotted
bases are unaffected.
The analyzer only mapped plain-lora groups, so a file carrying no plain
lora (a pure lokr, for example) analyzed zero modules and fell through to
a 1.0 default: it reported perfect fidelity for exactly the files that
degrade most. Measured on the shipped krea 2 uint4 checkpoint, those
files land between 0.04 and 0.34.
Every targeted module is now rebuilt with the loader's own module class
and its delta read from the production calc_updown, so lokr, loha, oft,
full, ia3, glora, norm and the dora / dense-bias / diff_b variants are
measured as they apply; factor-path eligibility is decided by calling the
loader's own predicate. Modules carrying several families sum their
deltas the way the loader stacks them, and a family the tool cannot
rebuild is reported instead of counting as clean.
- report per-module applied fidelity (1.0 on the factor path, measured
rho on the requantize path) as a median and an energy-weighted mean
- add --dtype bf16 to measure the unquantized reference rather than
assert it
- drop the per-module empty_cache: it cost 16ms per module against 1ms
of reuse, and the caching allocator already reuses the buffers
- keep shard handles open across modules
Offline analyzer for a (model, lora) pair: maps lora modules onto the
transformer, measures per-module delta-to-step ratio and requantize
retention, and reports factor-path eligibility. Loads pre-quantized
sdnq repos or simulates quantization on bf16 repos; supports --json
and --fail-under for scripted checks.
The triton autotune hijack draws its own rich progress bar on the
logger console during sweeps; a second live display on the same
terminal tramples the benchmark's live tables and leaves stale panel
border lines in scrollback. Disarm the bar in the benchmark process
and keep the hijack's bookkeeping.
Write the text and json reports by default, named to the archive
convention <gpu>-t<torch>-<date> with a collision suffix for the pair,
into $SDNQ_BENCH_DIR, --outdir, or benchmarks/ under the sdnext root.
'none' disables either output and explicit paths behave as before.
Add accum configs, value-side stress rows, a recommendation row, block
specs, and a triton-mm variant for the new use_fp16_accum surface. The
unsafe unquantized-pv mode is measured but never starred or recommended;
accum rows skip on sdnq builds without the kwarg. Also move the dequant
compile verdict off the removed sdnq_dequantize_compile option to a note
keyed on SDNQ_USE_TORCH_COMPILE, fixing the crash after 2250d8052.
The ref2va checkpoint partition conditions on reference images instead
of keyframes, so it gets its own registry row and reference card, and
the video core marshals PIL images into task_args as
MiniMaxH3ImageReference. Images are converted to RGB first, since the
reference encoder reads the array raw. The keyframe path is unchanged.
Validation runs before the model load in one funnel shared by the tab
and the API, so a rejected request costs nothing: references on a
non-reference model, a reference model with nothing to condition on,
more than nine images, non-images, and aspect outside 1:4 to 4:1 all
return 400. The image path rejects a reference pipe without references
instead of reaching a transformer that was never loaded.
Add POST /sdapi/v1/video plus GET /sdapi/v1/video/models and
GET /sdapi/v1/video/file. The generation body is extracted from the
gradio handler into a keyword-only core, video_run.run, which returns a
structured result and raises typed errors; the positional generate
signature is unchanged and now adapts to the core. Omitting engine and
model drives the currently loaded checkpoint when it is video-capable,
which covers models loaded from local folders without a registry entry.
- registry helpers in models_def (find, engines, pipeline_classes,
workflow_for_class); validate_pipeline reuses the shared class set
- modular pipes stamp their workflow so out-of-registry loads dispatch
onto the modular branch
- disk switches (mp4_*) and wire switches (send_*) are independent;
artifacts above the base64 cap fall back to path plus the file route,
which is jailed to the video output directory and serves video/mp4
with range support
- always-on video scripts get bootstrapped default args, matching the
txt2img handler; missing bootstrap raised a TypeError per frame
- checkpoint overrides are rejected with a pointer to the checkpoint
endpoint; unknown engine, model and sampler names return 404 with the
valid choices
- cli/api-video.py client, test/test-video-api.py suite and a
full-test.sh entry; video mimetypes registered; rate-limit cost set
- remove the unreferenced video_ui.run_video dispatcher
The settings rows were verdicted one at a time against int8 qk alone, so
the table could assemble a tuple no row ever measured and contradict its
own stars: pv lost a solo 10 percent margin test while the starred best
config carried pv at four of nine shapes, and the emitted smooth-only
stack was strictly dominated by measured rows at three shapes.
- select the recommendation as the best measured quantized config by the
star rule (lowest error within 5 percent of the fastest, error cap vs
int8), then gate that one config against unquantized at a z adjusted
for having selected the best of the pool
- decompose the winner into the settings rows; each reason cites the
measured sibling that isolates its setting, replacing the blanket no
gain wording that hid an 8 percent measured win
- drop the additive stack estimate, composition check and float8 rescue,
which the joint selection covers by construction on measured rows
- name each shape's selected config in the per-shape verdict note and
compute sidak z exactly for any candidate count
Hoisting the function-local imports to module scope wrote
`import importlib.import_module`, which cannot resolve because
import_module is a function in importlib, not a submodule, so the
benchmark failed at startup with ModuleNotFoundError.
- import importlib, which is what the module-scope call sites need
The toggle changes no output, so the verdict is a symmetric
faster/slower test on the layer forward it gates, voted across every
measured dequant shape; the standalone kernel ratio stays in the notes.
Every dequant shape and block geometry comes from a real transformer config,
one source of truth per model.
- flux.1: attention linears 3072x3072, feed-forward 12288x3072, block at
3072 wide, 24 heads, 12288 ff, 4608 joint tokens
- krea 2: standalone wq 6144x6144, swiglu gate 16384x6144, block at 6144
wide, 48 heads, 4608 joint tokens
- the block section measures every geometry in block_geometries; buyback
costs are judged at the block matching the reference shape, with the
geometry named in the reason
- te shape lookup keys on its label instead of a list index
Krea 2 runs joint attention over one text plus image stream with a segment
mask: text is padded to a fixed 512 tokens and the padded tail is masked for
queries and keys both, so fully masked query rows yield nan under sdpa. The
preset carries the transformer's exact (B, 1, L, L) mask, nan-guards the
error path the way the model does, and uses the 48 kernel-level heads left
after gqa expansion at 4608 joint tokens.
- fix the drift fields crashing when save_report rebuilt the run info
Per-preset include lists trimmed video and masked shapes for runtime, which
left the composition check without its smooth_hadamard row there. Only hard
technical exclusions remain, as an exclusion map: sd15 skips hadamard configs
because compiling hadamard with a non pow2 head dim hangs torch inductor.
- wan22, wan22-cfg, ltx2 and masked gain smooth_hadamard, fp16qk, fp8qk and
fp8full rows; availability gates and the per-config timeout still apply
Repeat-pair runs measured 2-4% between-run drift on one machine, enough to
flip threshold verdicts near the margin on every rerun.
- bench keeps its iteration samples: rows carry a median sigma, and per-shape
sentinel re-measurements sample run-level clock drift
- on/off verdicts are three-zone at the run's own noise level; too close to
the margin keeps the current setting and says so
- pv candidates (now including fp16) are tested independently against the
margin at a sidak-adjusted z instead of min-then-threshold
- unmeasured toggle stacks are estimated additively in the composition check
- per-shape qk verdicts print alongside the reference-shape verdict
- cross-gpu error sanity bands flag corrupted measurements
- on/off verdicts share one speed margin (recommend_speed_margin, 10%) across
attention, dequant, compile, te and conv rows
- re-check the recommended toggle stack against unquantized; individually passing
buybacks can eat a marginal qk gain
- judge smooth k and hadamard cost at block scope when measured: kernel rows hand
the prep contiguous q/k/v, real models hand it strided views from the fused qkv
projection; reasons cite both scopes
- give bare float8 qk its own shot at the margin before disabling, the verdict
must not hinge on int8 alone
- compare triton flash against the recommended config, not always int8
- note self-attention shapes that disagree with the reference verdict
- fold the enable rows into the type rows: disabled, enabled or an explicit dtype
- pv keep-unquantized value is disabled; enabled means int8 pv
- add a text encoder row judged at te geometry
- update locale hints, drop the orphaned checkbox hint
"sage" is a different kernel per gpu: sageattention dispatches the pv dtype and
accumulator by arch and cuda version, and modules/attention.py forces the fp16-pv
cuda kernel on sm86, so one label across shared reports compares unlike kernels.
- resolve the row label from the dispatch: sm89 with cuda 12.8+ reads
"sage int8 qk + fp8 pv, fp32+fp16 accum", sm86 reads "fp16 pv, fp32 accum"
- name the sm86-only fp16-accum baseline for what it is
Notes ran to paragraph length and wrapped in the terminal, burying the numbers.
- lead with the topic, keep the numbers and the fix, drop the restatement
- one line per note at 140 columns, except the sd15 compile warning
sdnq no longer skips compile for fp8 storage, it upcasts e4m3 to the scale dtype
before the compiled dequant, so the tool's mirror of the deleted skip_fp8_compile
gate went dead and fed the raw e4m3 weight to its compiled dequant: on pre-ada gpus
that reports a compile failure for rows the webui runs fine.
- mirror the upcast in dequant_args, keep the hardware probe on the raw weight
- drop dequant_gated_to_eager and the gated-to-eager row, fp8 rows are measured now
- say upcast, not eager fallback, in the environment panel and dequant notes
Hiding the sdnext startup log by pointing fds 1 and 2 at a file kills the process
when stdout is a real console: sys.stdout writes through the win32 console api on
the handle behind fd 1, so the first bootstrap log line raises OSError 'the handle
is invalid', and with stderr broken the same way the interpreter aborts with no
message, no traceback and no results.
- swap sys.stdout/sys.stderr to a buffer along with the fd redirect
- capture the startup log instead of discarding it to devnull, replay it on failure
- catch SystemExit: loader.py and installer.py exit on fatal startup errors
The linear layers bind their scaled-mm function at import, so
SDNQ_USE_TRITON_MM freezes the backend per process and comparing triton
against the torch fallback meant two runs, carrying clock drift into the
delta. --mm-backends rebinds the function on the consuming modules
between benches, so both rows measure the same quantized layer under the
same clock state.
- swap targets are the four linear modules that from-import a scaled-mm
function; every swap resets dynamo, since the layer forwards are
compiled and the traced graph pins the previous function
- the torch row is captured from kernel_wrappers rather than
reimplemented, so it is unavailable where triton is the device default
(rerun with SDNQ_USE_TRITON_MM=0) and says so
- rounds alternate order and keep the fastest per row, so drift cancels
instead of favouring whichever backend runs second
- flag output error when backends disagree past 1e-4: they are meant to
be numerically equivalent
probe_fp8 and the broken-compile bench path forced eager input prep by
toggling torch._dynamo.config.disable. torch 2.13+ raises 'found no
compiled frames' when a fullgraph-compiled function is called inside a
disable window, so the probe failed before reaching the attention kernel
and reported an environment error instead of the kernel's real verdict,
on fp8-capable gpus too. Swap the module-global prep function for its
eager inner instead, matching the correctness section.
The environment panel and json now carry python, platform, backend and
the backend runtime versions (cuda/cudnn/driver, hip, ipex, openvino,
directml via devices.get_gpu_info), so shared reports identify the stack
without inferring it from the torch version string.
The webui bootstrap log is gated behind an fd-level suppression for the
import window: the bootstrap reconfigures its loggers during import and
onnxruntime warns from c++, so logger levels alone cannot silence it.
Import failures still print after the gate lifts.
After the correctness section aborts on a sticky kernel fault, the
remaining sections cannot run; flush outputs and exit instead of
cascading into the next cuda call's traceback.
A kernel fault (misaligned address, illegal memory access) is sticky:
every later cuda call in the process fails, so one faulting check killed
the whole run through the next reference computation and lost the --save
and --json outputs. Compute the correctness reference inside the
per-check handler, detect a dead context and abort the section with a
clear message, and flush partial outputs from a top-level handler on any
crash or interrupt.
The fp8 attention probe can fail for reasons other than missing hardware
support (torch or triton compile issues); reporting those as 'not supported
on this gpu' is wrong on gpus where fp8 is native. Match the triton
pre-sm_89 signature and label anything else as an environment failure.
Toggling torch._dynamo.config.disable to force eager input prep fails on
newer torch: a fullgraph-compiled function called inside a disable window
raises 'found no compiled frames', failing every check and poisoning the
first compiled call afterwards. Swap the module-global prep function for
its eager inner instead, matching the dequant section's mechanism.
- store per-check errors and exceptions in the json report, so remote
reports carry the failure mode
Benchmark weight dequantization alongside attention: eager vs compiled
dequantization, measured standalone and through the full linear forward,
plus the quantized matmul forward, across int/uint 8-6-4-2 and the
pre-quantized float formats (float8_e4m3fn, float8_e4m3fn_sdnq,
float4_e2m1fn group 16) at krea 2 layer geometry, against a bf16 nn.Linear
baseline and a true-fp32 output reference. Each row carries measured
storage size and one-shot quantize time; an svd/hadamard variants table
measures rotation and low-rank costs on top of the base dtypes.
A combined block section measures complete configurations (weights dtype x
matmul path x attention) end to end through a dit-style transformer block,
with output error at depth one and four against an fp32 reference block,
because component speedups and errors do not compose multiplicatively.
- recommendations weigh error against speed: a faster option is rejected
when it multiplies measured output error beyond 2x, and reason strings
cite both numbers; notes include measured size/error and speed/error
frontiers
- robustness: extreme-activation stress rows in the correctness matrix
(pass on finite output), max-token-error columns beside norm error, and
non-finite outputs labeled as verdicts; norm metrics alone hide
token-level corruption on outlier-heavy inputs
- probe compiled weight dequant for e4m3 and e5m2 storage and report the
fp8 compile gate status; on ampere e5m2 compiles while e4m3 does not
- new attention presets: sdxl-cross (cross-attention), qwen3-te (causal
gqa text encoder), wan22-cfg (batched cfg video); cross-attention
correctness check
- --sections, --dequant-dtypes, --dequant-variants and --block-configs
selectors, --json structured results output
sdnext enables tf32 globally; a math-backend dispatch fallback (gqa,
some mask shapes) degrades the fp32 reference to tf32 precision,
measured 4.2e-4 vs 5.0e-7 relative to fp64. the default mem-efficient
backend is unaffected.
Recommendations must hold on any gpu, so every verdict now comes from
measuring the configuration the advice points to.
- run benchmarks in the webui's configured dtype, --dtype overrides
- bench the prep mode matching the environment: compiled, verified
dynamic=false workaround, or eager; label each with the webui change
the numbers correspond to
- add a prep column for the q/k/v quantization cost outside the
kernel; eager prep measured 1.9 ms vs 0.2 ms fused at sdxl,
inverting the quantized matmul verdict
- compare Use Quantized MatMul against the unquantized sdnq row so
quantization pays for its own prep
- star the lowest-error config within 5% of the fastest sdnq time;
the same rule drives the worth-enabling verdict vs torch sdpa
- drop the (auto, default) suffix; document the settings mapping and
star rule in the notes
- wrap correctness and shape tables in panels to separate sections
Vladimir Mandic
CalamitousFelicitousness
Dity0