New lora_sdnq_apply radio (exact, requantize) in the lora settings.
requantize keeps the previous behavior: every quantized layer takes the
dequantize-add-requantize path, with factor attach and svd hosting gated
off. A settings-only flip re-applies loaded networks: the mechanism
rides a per-module apply stamp and the network-changed signature, and
the activate fallthrough strips factors a closed gate leaves attached.
Requantize chosen by the setting logs as info instead of the
reduced-fidelity warning.
- locale hint covers fidelity and memory tradeoffs of both methods
- suite: gate, legacy routing and flip-transition tests
Plain svd truncation of hosted deltas is optimal in weight space but not
in output space: activations concentrate energy in a few input channels,
so scaling the delta by per-channel input RMS before the svd spends the
rank budget on output error instead. Statistics stream from the model's
own forwards on sub-8-bit SDNQ checkpoints and cache per checkpoint;
measured on real LoKR files this raises output-delta retention by ~0.05
at rank 256 and ~0.09 at rank 64, most on MLP down projections.
- modules/lora/lora_calib.py: capture hooks, per-checkpoint cache under
data/sdnq-calib, statistics land on layers as sdnq_calib_rms; gated by
lora_sdnq_host_calib, skipped when the model is compiled
- lora_sdnq.apply_hosted: weighted truncation when statistics exist,
calib count in the load summary
- cli/sdnq-calibrate.py: complete calibration now against a live server
- cli/lora-quant-fidelity.py --calib: hosted rho scored in the
activation-weighted norm
- test/test-sdnq-lora-factors.py: calibration category, 5 tests
Non-additive families (lokr, loha, oft, dora, full) merged into the
quantized weight and lost most of their delta on low-bit formats. On
sub-8-bit layers the set's calc_updown delta now rides the svd
side-channel as its top singular directions instead: factorable members
are subtracted out and appended exactly, so only the non-factorable
remainder is truncated. Truncation keeps the dominant part of the
effect and drops an orthogonal residual, where requantize keeps the
grid extrema and adds grid-shift noise of the delta's own magnitude;
on real lokr files retention rises from 0.04 to about 0.5 at the
default rank.
Hosted layers take no weight backup and unload bit-exactly. The svd
runs under a forked rng so generation seeds are unaffected. At 8 bits
and above requantize retains most of the delta and remains the path.
lora_sdnq_host_rank caps the hosted rank; 0 disables hosting.
- restore stashed svd factors onto the layer's current device; the
stash tuple does not follow module device moves, so an offload
between apply and remove left restored factors on a stale device
- recheck factor shapes for layers already in factor mode, so a
malformed stacked network downgrades to the legacy path instead of
raising in the concat
- clear the fallback log at activate entry so a raise mid-pass cannot
leak stale entries into the next report
- pin both behaviors in the suite and state the compute-dtype fidelity
floor in the module docstring
Checkpoints quantized without hadamard must attach factors unrotated;
checkpoints carrying their own svd correction must keep it under apply
and get the original factors back on remove. Both pinned in both svd
layouts.
Baking a lora into a quantized weight requantizes it, and on low-bit
formats round-to-nearest erases sub-step deltas (uint4 retains roughly
2/group_size of the signal). Plain lora deltas now ride the sdnq svd
side-channel: factors append to svd_up/svd_down with the down factor
hadamard-rotated, applied by the dequantizer at full precision in every
forward mode. Apply and remove are exact and take no weight backup.
- non-factorable families (dora, lokr, loha, oft, cp mid, dense bias)
fall back to requantize with a per-pass summary warning
- native fuse now honors the quantized-model guard; fuse requantized in
place on every network swap and accumulated drift
- layers that fell back on a mixed set restore from backup before
re-entering the factor path; untargeted quantized layers are no
longer flagged
- test/test-sdnq-lora-factors.py pins the erasure law, factor-path
exactness, memory accounting and set transitions
Batch matrix-matrix and Dynamic Attention BMM applied a legacy Attention
processor to pipe.unet, which a diffusion transformer does not have, so
they served unet models alone and said nothing elsewhere. The choices, the
processor and its slice helper are removed, an unrecognized method now
warns rather than selecting nothing, and a stored value is rewritten to
Scaled-Dot-Product on load.
attention_slicing holds one of Default, Enabled or Disabled, so testing the
string for truth sent Disabled down the enable branch and left the disable
call unreachable, while the log line below it reported the choice rather
than the action taken.
bypass_sdpa_hijacks and llm_context restore the pinned original over the
router and put the router back, including when the body raises. Captioners
and detailers run inside them, so the router must be removable.
The sdnq backend read six settings on every call; it now captures them
when the chain is built. Each backend declares the settings its call
captures, and webui registers one onchange over those names plus the
override set and the torch kernel flags, so a change rebuilds the chain
between jobs. When a compiled model is resident the rebuild also resets
dynamo, since its graphs hold the previous router.
SD_ATTN_DEBUG logs each distinct route once: backend, component role,
step, shapes, dtype and mask presence. The router takes an optional
observer for it, so the clean path carries one pointer check. report()
returns the active chain and generation context, and torch_info records
the whole chain as one string instead of the last prepared backend.
A module-level context tells attention consumers what is running: the
component role (transformer, text encoder, vae), the index of the
denoiser forward about to run, the pass length, and the model. It is
opened and closed around process_images, reset per denoising pass beside
the callback setup, and advanced by both step sources: the classic
callback passes the completed step plus one, the modular pre-forward
hook counts forwards. Roles come from the existing text encoder and vae
hijacks and the modular phase hooks. The step also lives in a device
scalar updated in place, so a compiled reader keeps its graph across
steps.
The flex backend never called the sdpa it replaced, so any backend
stacked before it was unreachable and every call it could not serve,
cpu or 3d inputs included, failed inside flex_attention. It is now an
ordinary entry gated on what flex_attention accepts: 4d tensors on one
non-cpu device. The mask path drops the 2d special case, which indexed
attn_mask.size and reshaped the mask onto the wrong axis; expanding to
(batch, heads, q, kv) already follows sdpa broadcast semantics.
Replace the six closure hijacks stacked in devices.set_sdpa_params with
a registry of declarative backends and one router installed in their
place. Each backend declares the constraints its closure carried as a
predicate, a priority matching its old stacking position, and a prepare
step that imports and configures the implementation; the router walks
the prepared entries by priority and hands declined calls to the
terminal backend (dynamic, flex) or the original sdpa, so fallback is
the router's job rather than each closure's.
- parity held: gates transcribed literally, the same kernel kwargs,
enable_gqa passed to the original only when set, torch_info keeps the
last prepared backend, the dynamic pin still set
- a backend enabled on a platform without it warns instead of silently
doing nothing
- the legacy set_* entry points are gone; devices.py installs the router
- test/test-attention-router.py checks every override subset against the
old stacking order, gate parity over 16,000 shape cases, dispatch,
terminal handoff and prepare isolation, offline
A delta that does not fit its target module cannot apply, and applying only
the layers that do fit leaves the model in a state nothing was trained for,
so try_load_chain drops the whole file when any family reports a mismatch.
Bias deltas were never checked against the target bias and could only surface
at apply time; a module with no bias stays a non-mismatch, since whole
architectures are built bias=False.
- check bias deltas against the module bias in the lora, norm and full loaders
- carry the mismatch count on the network so the chain can refuse the file
- record refused writes in the infotext so a partial apply is not read as clean
- point the krea2 full-diff test at a module that has a bias
network_add_weights defaulted its base tensor to self.weight for the bias
delta as well, so in fuse mode a diff_b was added to the weight matrix and
the result written into the bias. Layers where in and out differ threw a
shape error and had the weight matrix installed as their bias, square layers
broadcast silently, and either way the summary still counted the delta as
applied.
- pick the base tensor from the bias flag
- name the layer, target and both shapes in the mismatch error
- return which of (weight, bias) took a write, count the rest as refused
- report refused= on partially applied and partially removed networks
- cover both apply paths in test/test-lora-apply.py
run_ltx reported failure by yielding a string, which is why LTX had no API.
run() is the core underneath: keyword arguments named as video_run.run names
them, a VideoResult back, VideoError out with 499 for an interrupt. The lock,
progress and summary stay in the adapter, whose signature is unchanged since
callers bind to it by keyword. Failure now closes the processing object and
deactivates networks, which abort never did.
Pins the sentinel contract, row uniqueness, and the equivalence between
dispatch_mode and the ladder it replaced. A row that declares neither a name
marker nor a mapped pipeline class now fails here instead of reaching a runner
that would generate it as text to video.
Pins the cases that separate a complete slice from a truncated one, including
the zero-argument script whose empty slice is complete, and asserts no hook
runner slices the vector on its own.
The tab held the only code that built video and audio references, sniffed the
file type itself, and dropped anything it did not recognize: an unknown
extension, a file that had gone missing, and any decode failure were all skipped
without a word, leaving a request that generated from fewer references than were
uploaded.
Reference marshalling now goes through the same funnel the api path uses, and
runs before the load, so a rejected file costs nothing and says which file and
why. The workflow comes from the registry row, which is where the loader reads
it from as well.
- a rejected input returns its reason to the output box, since the general
handler only reaches the log
- references uploaded against a keyframe workflow warn instead of vanishing:
the accordion hides on a row change but the files it held do not
- guard p.close() in the finally, which the model-not-loaded return has always
reached before p exists
parse_options reads : and , as separators and = as the only assignment, so a
segment without = becomes a valueless flag set to 1. The shipped strings used
ffmpeg command line spelling, which parses without error into other values.
- crf:16 parsed to {'crf': '1', '16': '1'}, encoding every api, framepack and
seedvr video near lossless rather than at crf 16
- crf=23:b:v=0 pinned the generic bitrate option to 1 bit per second on vp8 and
vp9, collapsing their output
- qscale:v=3 reached mpeg4 and mjpeg as nothing at all, replaced by an explicit
quantizer range
- test-video-codecs.py asserts every preset segment carries an assignment
The core took reference images only, so no api caller could send the video and
audio references the ref2va workflow conditions on, and the marshalling that
handles them existed solely in the MiniMax tab.
validate_references now gates on the workflow and hands the entries to the
architecture that owns them, which accepts decoded images and local file paths
in any mix and preserves their order, since order fixes the labels a prompt
addresses. reference_caps exposes the same limits the validation enforces, so a
client reads them instead of mirroring the numbers.
- MAX_IMAGE_REFERENCES is gone: the limits now cover all three kinds and a total
- the run body no longer builds reference objects or knows their class
- an image is converted where it is built rather than at the call site, so a
reference decoded from a file and one posted as base64 arrive the same way
- pipeline args summarize a reference list by kind, since a decoded video would
otherwise print its frames into the per-generation log line
- the video endpoint documents what it actually accepts: images alone, because
video and audio decode from files rather than from the wire, and an upload
reference only where an extension provides the store that resolves one
MiniMax-H3 conditions on image, video and audio references, and the limits it
enforces on them are constructor defaults on a block class the package does not
re-export, so they cannot be imported and are mirrored here instead.
The resolver takes decoded images and local file paths and returns the reference
objects the pipeline reads, checking cheapest first: classification and counts
open no files, container headers are read without decoding, and only then is the
media decoded. Everything runs before the model load, so a rejected request costs
nothing.
- reference limits as a frozen dataclass, keyed by workflow since the rows that
carry ref2va differ only in which repo they load
- media classification and container probing as generic video helpers
- a url is refused before construction: the reference classes fetch one and
decode whatever comes back
- a video is bounded by duration and by what it decodes to, since the pipeline
truncates it to the generated length and the decode is held across the load
- the frame floor is counted on the decoded video at the rate it resamples to,
which is what the conditioner measures
- torchaudio and av are checked up front rather than surfacing as an import
failure once the weights are resident
Static inventory of every registered pipeline: component classes come from
the init type annotations, the component specs on modular pipelines, and
the video model definitions, so the real role function runs against real
classes with no weights loaded. The audit prints the role of every
component, lists which components turn resident under the shipped 22 GB
never-offload default, names the custom pipelines it cannot see, and
asserts that no component carries an undecorated entry point, that
opted-out classes route on-demand, and that every pipeline places a
per-step component.
Offline suite for the placement roles: the role table over the component
names sdnext loads, one dispatch arm per component with hooks landing on
text encoder wrappers, force sweeps scoped to stamped modules, a settings
change re-placing a resident component, on-demand contracts, enumeration
on both pipeline kinds, and the upstream markers the roles read.
The ref2va checkpoint partition conditions on reference images instead
of keyframes, so it gets its own registry row and reference card, and
the video core marshals PIL images into task_args as
MiniMaxH3ImageReference. Images are converted to RGB first, since the
reference encoder reads the array raw. The keyframe path is unchanged.
Validation runs before the model load in one funnel shared by the tab
and the API, so a rejected request costs nothing: references on a
non-reference model, a reference model with nothing to condition on,
more than nine images, non-images, and aspect outside 1:4 to 4:1 all
return 400. The image path rejects a reference pipe without references
instead of reaching a transformer that was never loaded.
Add POST /sdapi/v1/video plus GET /sdapi/v1/video/models and
GET /sdapi/v1/video/file. The generation body is extracted from the
gradio handler into a keyword-only core, video_run.run, which returns a
structured result and raises typed errors; the positional generate
signature is unchanged and now adapts to the core. Omitting engine and
model drives the currently loaded checkpoint when it is video-capable,
which covers models loaded from local folders without a registry entry.
- registry helpers in models_def (find, engines, pipeline_classes,
workflow_for_class); validate_pipeline reuses the shared class set
- modular pipes stamp their workflow so out-of-registry loads dispatch
onto the modular branch
- disk switches (mp4_*) and wire switches (send_*) are independent;
artifacts above the base64 cap fall back to path plus the file route,
which is jailed to the video output directory and serves video/mp4
with range support
- always-on video scripts get bootstrapped default args, matching the
txt2img handler; missing bootstrap raised a TypeError per frame
- checkpoint overrides are rejected with a pointer to the checkpoint
endpoint; unknown engine, model and sampler names return 404 with the
valid choices
- cli/api-video.py client, test/test-video-api.py suite and a
full-test.sh entry; video mimetypes registered; rate-limit cost set
- remove the unreferenced video_ui.run_video dispatcher
The Krea 2 transformer keeps checkpoint-style module names while the
official krea/Krea-2-LoRA releases are saved with upstream-diffusers
names, so all 264 modules failed to bind and the LoRAs silently did
nothing. Krea 2 is the only native-LoRA arch with an sdnext-owned
transformer, so its module names diverge from the diffusers ecosystem.
- native_adapter.resolve_group_targets consults the arch resolver first
for passthrough prefixes, falling back to verbatim binding; a no-op
for arches that load the diffusers class
- krea2_lora maps diffusers attn/ff/text_fusion/embedder names onto the
checkpoint module tree; checkpoint-named LoRAs still bind verbatim
- add test/test-krea2-native-adapters.py
A full-weight extraction on Z-Image bound 308 modules and applied 172 of
them, silently dropping the rest, and left 71 more unmapped.
assign_network_names_to_compvis_modules puts every transformer module in
network_layer_mapping but skips stamping network_layer_name on norms,
which is the attribute the apply pass keys off. try_load_full bound those
modules through the mapping and they then never applied; stamp them
loader-locally, as try_load_norm already does.
Z-Image also names three module groups differently from the diffusers
tree: the qk-norms (q_norm/k_norm vs norm_q/norm_k), and the patch
embedder and final layer, which live in ModuleDicts keyed by
"{patch_size}-{f_patch_size}" and so carry a key the checkpoint has no
notion of. Read that key from the live model rather than hardcoding it.
The counts close exactly: 68 qk-norms plus 3 non-block targets are the 71
that went unmapped.
zimage, chroma, ernie and krea2 chained only lora/lokr/loha/oft while
flux2 and anima ran all eight families, so ia3/glora/norm/full files
(e.g. full-diff extractions with diff/diff_b keys) reported not loaded
on the short-chain arches. The generic family loaders are arch-agnostic;
wire the missing four into each chain.
- add ia3/glora/norm/full wrappers and chain entries to the four arch
modules, with matching suffix/marker re-exports
- add a chain-level full-diff test per suite: zimage covers the legacy
attention.out alias and the fused-qkv skip, chroma the proj rename,
ernie the passthrough
LyCORIS extraction with use_sparse_bias saves bias_indices/bias_values/
bias_size per module: the sparse weight-shaped remainder of the SVD
extraction, named bias for historical reasons. The keys were dropped by
both loader paths, so extracted adapters applied without the residual
correction; the dense-bias branch in finalize_updown that consumes it
was unreachable.
- rebuild the COO tensor in NetworkModule.__init__ (int16 indices cast
to long), shared by the native and generic loaders; kept sparse so
the dense += sparse in finalize_updown materializes per module at
apply instead of near-model-size densification at load
- accept the triplet suffixes in LORA_SUFFIXES; fused targets skip
with the weight-shaped-bias warning
- cover an extraction-faithful numeric round-trip and the fused skip
in the offline suite
ai-toolkit DoRA saves lora_A/B plus a 1-D per-output magnitude key in
place of alpha; PEFT and diffusers name the same quantity
lora_magnitude_vector. Neither key was in the suffix table, so such
adapters loaded as plain LoRA with the magnitude renormalization
silently missing. The semantics match LyCORIS wd_on_out=True row norms,
so both keys convert onto the existing dora_scale path.
- accept .magnitude and .lora_magnitude_vector in LORA_SUFFIXES and
convert at group level in try_load_lora
- reshape 1-D vectors to (out, 1): on square layers the apply-time
orientation detection would otherwise renormalize the wrong axis
- cover square-layer numeric equality, fused-qkv slicing and the PEFT
key form in the offline suite
finalize_updown ran apply_weight_decompose on the unscaled delta and
multiplied the result by alpha/rank afterward. LyCORIS and ComfyUI both
bake alpha/rank into the diff before computing the row norms, so any
DoRA with alpha != rank renormalized against the wrong merged weight
(64% relative delta error for kohya-style alpha=1 rank=8; exact only
when alpha == rank, which full-matrix LoKR forces).
- scale updown by calc_scale() before apply_weight_decompose; apply
only the multiplier afterward
- multiplier lerps the full merged delta (0 disables, 1 equals the
trainer output); LyCORIS weight-mode ratio interpolation leaves the
diff applied at multiplier 0 and is not used
- add a numeric regression test mirroring the LyCORIS forward reference
A diff_b bias delta on a fused BFL target passed through whole and
failed at apply with a shape mismatch. diff_b stores one value per
output feature, so it partitions with the fused rows exactly like the
up-weight; slice it with the chunk in the LoRA loader. The legacy
weight-shaped bias key (LyCORIS sparse-residual heritage) has no
defined partition on a fused target and no known emitter pairs it with
chunk-capable families, so the group is skipped with a warning in the
LoRA, LoKR and LoHA loaders.
- add slice_bias_delta beside slice_dora_scale; warn and skip
non-per-output diff_b shapes
- cover sliced diff_b flowing out as ex_bias and the legacy-bias skip
in the offline suite
BFL-format chroma adapters targeting the embedders, final projection or
the distilled guidance layer MLPs resolved verbatim and unmapped: the
embedder/final-layer names differ from diffusers outright, and the
approximator MLP leaves are in_layer/out_layer in BFL but linear_1/
linear_2 in the diffusers PixArt projection. Only in_proj, out_proj and
norms.N shared names and bound.
- add CHROMA_EXTRA_MAP (kohya form derived) and GUIDANCE_LEAF_MAP to
both target resolvers
- route bare distilled_guidance_layer keys through the resolver instead
of the diffusers passthrough so both leaf namings resolve; add
img_in/txt_in/final_layer bare prefixes
- cover all key forms and end-to-end binding in the offline suite
A LoKR group whose Kronecker product does not fit the resolved module
previously bound anyway and failed at apply time as a caught per-module
error, leaving the adapter partially applied with only an error log.
Reject the group at load with a warning instead, matching the LoRA
path's shapes_match gate.
- lokr_kron_shape derives (out, in_flat) from full, rank-decomposed or
Tucker-rebuilt factors, folding conv kernel dims into in_flat
- lokr_shapes_match honors SDNQ original shapes and chunk partitions:
equal chunks need total * out rows, row-range slices an exact range;
the input dim is never chunked
- cover non-fused and fused rejection in the offline suite
LyCORIS wd=True saves a dora_scale companion for LoRA/LoHA/LoKR; on
fused BFL targets the chunk paths passed it through whole, so apply
failed with a shape mismatch and the module was dropped. Per-output
magnitudes (wd_on_out=True, the default) partition exactly with the
fused rows; per-input magnitudes couple the chunks through shared
column norms and have no exact split.
- slice per-output dora_scale rows with the chunk in the LoRA, LoKR
and LoHA loaders
- skip per-input DoRA on fused targets with a specific warning
- cover sliced and skipped orientations in the offline suite
The lycoris_ save format is arch-independent: LyCORIS standalone wraps
the loaded diffusers model and emits the wrapped module path with dots
as underscores, so verbatim passthrough is correct for any arch. Only
flux2 handled it; zimage, chroma, ernie and krea2 reported such files
as not loaded.
- add lycoris_ to KNOWN_PREFIXES_DEFAULT and PASSTHROUGH_PREFIXES_DEFAULT
- drop flux2's per-arch prefix append and resolve_targets branch
- add lycoris_ to ANIMA_PREFIXES (anima replaces the default tuple);
network_prefix_for already routes it to the transformer namespace
- cover the passthrough with a zimage loader test
BFL-format adapters targeting the embedders, timestep/guidance MLPs,
modulation layers and the final layer resolved to nothing and were
dropped as unmapped, for every adapter family on the f2 native path.
- add F2_EXTRA_MAP exact-match lookups in both target resolvers, with
the kohya underscore form derived from the BFL path
- add guidance_in. to BARE_FLUX_PREFIXES; groups targeting the guidance
embedder stay unmapped on models built without guidance_embeds
- extend the offline test mock with the non-block targets and cover all
three key forms plus full-matrix LoKR with placeholder alpha
Triton cannot compile e4m3 loads before sm_89, so fp8 weights fall back
to eager dequant there. The uint8-backed float8_e4m3fn_sdnq codec decodes
identically now that subnormals are handled (the two NaN codes become
+/-480), and its compiled dequant runs about 6x faster than eager native
fp8. Pre-quantized fp8 layers are viewed as uint8 at adoption when
compiled dequant is enabled on such hardware; the eager gate remains the
safety net for every other fp8 path.
Vladimir Mandic
CalamitousFelicitousness
Dity0