MiniMaxH3Scheduler counts the terminal sigma in num_inference_steps, so
Steps N ran N-1 evaluations while every other model runs N. The shim
hands the scheduler p.steps + 1, the slider starts at 1, and the PDD
pin records the evaluation count while passing the scheduler its grid
argument. Metadata written before this change counted grid points.
Shift is a property of the trained schedule, not of the step count, so
the tab sliders take absolute values, defaulting to the shipped 12 and
3. video_minimax resolves each request from the request value or the
scheduler config inside apply_overrides, which the tab, the api and
the still path all call, so a request without values lands on the
shipped schedule. The api maps sampler_shift onto the video schedule
and gains audio_shift. Applied values are recorded as Video shift and
Audio shift; the PDD pin records what it enforces.
The versions table showed only files[0], so precision variants of one
version could not be downloaded. Each file gets its own row under
rowspan version cells, sorted by precision then size, with a variant
column and a star on the primary; rows download by fileId URL and All
variants queues each version's primary from the fetched model. Companion
files pass their own type, and save names carry the precision, then
full/pruned and the file id only when needed to stay unique.
At 1.5 the ramp hands nearly every layer to the style network mid-generation,
which on few-step models overwrites the forming subject before identity sets.
Alpha 0 freezes the schedule into a static per-layer split: the subject keeps
its layers for the whole generation and style keeps the layers where it is
more salient. Nonzero values remain the scheduled handover toward the second
network.
- locale hints describe both regimes; the mode hint no longer implies the
shift is always on
EST-LoRA scores each layer by squared Frobenius energy, so a magnitude gap
between the two networks enters squared and the louder network wins nearly
every layer, starving the quieter one. The style side is now scaled by the
total-energy ratio (mirroring klora's gamma), making selection scale-invariant
so a network cannot take layers on magnitude alone. On the krea2 subject+style
pair this lifts the style network from 18% to 65% of the layer-step budget.
- lora_stack: accumulate per-mode energy totals, apply the balance in the est ramp
- test: content-louder est pair now hands over mid-schedule where raw scoring never would
- locale: note the est magnitude balance, and that a select mode gives each layer
to one network so both can be under-applied, while dense modes blend more fully
Text encoder networks now apply unconditionally in the native path.
- remove the option, the per-request parameter and the control threading
- collapse activate_filtered into plain activate at all call sites
- drop the toggle from the prompt embed cache key
- register the retired key so existing configs load without warnings
Restructure the seven hints to the settings pattern: lead definition,
tradeoff, scope, special values and stated default, with value and
cross-setting markup. Mode bullets and special values match loader
behavior: sub-8-bit hosting gate, rank 0 fallback, ramp 0 freezing
the balance.
Add lora_stack_mode with ties, dare_ties, dare_linear and
magnitude_prune combination of per-network deltas when several loaded
networks target one layer; sum stays the default and the exact factor
path. Combined deltas ride the existing tail: hosted svd on sub-8-bit
SDNQ, requantize at int8 and above, direct add elsewhere. Text-encoder
layers and single-network sets keep plain summation. DARE masks draw
from per-layer sha256 seeds so re-applies and cache entries stay
deterministic; the stack settings join the activation and factor-cache
signatures so settings changes re-apply without a reload.
Per-head selection is what breaks head-consistency sensitive models, measured on Anima across fourteen arms: the only clean sparse selections are the ones every head shares. The head sharing hint names itself as the first resort for banded output, and the exclusion hint carries why Cosmos stays listed.
sparse_attention_exclude is a comma separated denylist matched case insensitively against the architecture, the pipeline class and the denoiser class, so one entry works whichever name is to hand. It resolves once per model rather than per call and declines with a log line. Seeded with CosmosTransformer3DModel, the transformer Anima runs, which returns banded noise at every budget tested against a sound dense baseline; listing the class rather than the architecture covers the other models built on it, none of which have been checked.
The SDP overrides and SDNQ Attention hints describe the block mask capability
without naming what consumes it, so the two backends that can drive sparse
attention now say so where the reader is choosing between them.
Each hint now carries its default, the tradeoff in both directions, and the
setting it depends on. The two schedule settings do nothing unless both are
above zero, which nothing said before. The sequence floor is given in
resolution as well as tokens, since the setting is read in pixels.
The option defaulted to 0 and a property turned that into 8192, which hid
the real default behind a sentinel and left no way to sparsify every
sequence. The default is now 8192 in the settings registry, 0 means no
floor, and the slider tops out at 32768 rather than a length nothing
reaches. A negative value is left unclaimed for a derived floor.
The sdnq Triton kernel gains a block mask input (on the submodule's feat/block-mask branch; the pointer here is unchanged), so the backend advertises the block_mask cap and passes the router's selection as block_mask plus its block sizes. Prepare probes the installed kernel's signature and narrows the cap when the input is missing, warning when sparse attention is on, so an older submodule stays dense instead of failing. PlanEntry carries the narrowed caps, and the router, the sparse stage and the report read those rather than the declaration.
- benchmark: int8-sparse100/50/30/15 and int8-radial30 rows feed the same producer as the flex rows into the quantized kernel, skipped on builds without block_mask; a --configs filter selects attention rows the way --block-configs selects block rows; sparse rows of either backend are held out of the settings advice
- test/test-attention-sdnq-sparse.py: the kernel contract against the token-expanded mask, the nesting filter, the public entry on the quantized paths, the backward against the same selection and against fp32 autograd, and the flex consumer on one selection, pinned to one tile per run with a --tiles sweep
- locale: the sparse attention hint names both capable backends
Sparse attention is a stage over the chain rather than a member of it:
one switch, and the router hands the selection to whichever active
backend advertises that it consumes a block mask, currently flex. A
backend declares that through a capability set, so the quantized kernel
joins later without touching the router.
The stage gates on the component role, self attention, a minimum
sequence length defaulting to the measured 8192 token crossover, and the
absence of a token mask or causal flag, which flex cannot combine with a
block only mask. Budgets follow a precomputed per step schedule with at
most two distinct values. Enabling the feature with no capable backend
in the chain warns and leaves attention dense rather than doing nothing
quietly.
The modular pre-forward hook now receives kwargs and publishes whatever
token layout the pipeline passes by name, so a packed sequence gets its
conditioning pinned without any model specific code. Without a layout
the whole sequence is sparsified and that is logged once per length.
Hosting a non-factorable set costs one truncated svd per layer on every
fresh apply. The factors are deterministic in the checkpoint, the loaded
set, the host rank and the calibration statistics, so they persist under
data/lora-factor-cache keyed by that identity and replay bit-identically
on later applies. lora_sdnq_host_cache sets the disk budget in GB
(0 disables), least-recently-used entries are evicted past it. With the
cost paid once per configuration, svd subspace iterations rise from 2 to
4: about two thirds of the captured-energy gap to an exact decomposition
for +45% one-time compute; an exact svd measures 200-400x slower at
these shapes and is not viable.
- modules/lora/lora_factor_cache.py: signature, store, flush, eviction
- lora_sdnq.apply_hosted: fetch before computing, store after, hits in
the load summary
- cli/lora-quant-fidelity.py: matching niter for the hosted mirror
- test/test-sdnq-lora-factors.py: factor-cache category, 3 tests
New lora_sdnq_apply radio (exact, requantize) in the lora settings.
requantize keeps the previous behavior: every quantized layer takes the
dequantize-add-requantize path, with factor attach and svd hosting gated
off. A settings-only flip re-applies loaded networks: the mechanism
rides a per-module apply stamp and the network-changed signature, and
the activate fallthrough strips factors a closed gate leaves attached.
Requantize chosen by the setting logs as info instead of the
reduced-fidelity warning.
- locale hint covers fidelity and memory tradeoffs of both methods
- suite: gate, legacy routing and flip-transition tests
The lora block moves out of Networks into its own settings tab, with
header groups by what each option acts on: loading, prompt, application,
quantized models and metadata. Locale hints follow to the new section.
Plain svd truncation of hosted deltas is optimal in weight space but not
in output space: activations concentrate energy in a few input channels,
so scaling the delta by per-channel input RMS before the svd spends the
rank budget on output error instead. Statistics stream from the model's
own forwards on sub-8-bit SDNQ checkpoints and cache per checkpoint;
measured on real LoKR files this raises output-delta retention by ~0.05
at rank 256 and ~0.09 at rank 64, most on MLP down projections.
- modules/lora/lora_calib.py: capture hooks, per-checkpoint cache under
data/sdnq-calib, statistics land on layers as sdnq_calib_rms; gated by
lora_sdnq_host_calib, skipped when the model is compiled
- lora_sdnq.apply_hosted: weighted truncation when statistics exist,
calib count in the load summary
- cli/sdnq-calibrate.py: complete calibration now against a live server
- cli/lora-quant-fidelity.py --calib: hosted rho scored in the
activation-weighted norm
- test/test-sdnq-lora-factors.py: calibration category, 5 tests
Non-additive families (lokr, loha, oft, dora, full) merged into the
quantized weight and lost most of their delta on low-bit formats. On
sub-8-bit layers the set's calc_updown delta now rides the svd
side-channel as its top singular directions instead: factorable members
are subtracted out and appended exactly, so only the non-factorable
remainder is truncated. Truncation keeps the dominant part of the
effect and drops an orthogonal residual, where requantize keeps the
grid extrema and adds grid-shift noise of the delta's own magnitude;
on real lokr files retention rises from 0.04 to about 0.5 at the
default rank.
Hosted layers take no weight backup and unload bit-exactly. The svd
runs under a forked rng so generation seeds are unaffected. At 8 bits
and above requantize retains most of the delta and remains the path.
lora_sdnq_host_rank caps the hosted rank; 0 disables hosting.
dev added sdnq_attention_quantize_fp32 to modules/attention.py, which this
branch replaced with the modules/attention package. The option moves to the
sdnq backend in two places: the options dict the prepared call captures, and
the options tuple that rebuilds the chain when a captured option changes.
Reading it in only the first place would leave a setting that takes effect on
the next model load and not before.
The new option gets a hint alongside the rest of the section.
Everything under Cross Attention, SDNQ Attention and Attention Dispatcher
shipped with no hint, which left the layering invisible: the SDP kernel boxes
are candidates torch chooses from per call rather than a selection, and the
Flash box is torch's own build of the kernel rather than the flash-attn package
that the Flash attention override installs.
- cover attention method, sdp kernels, sdp overrides and attention slicing
- give the dynamic attention rates their unit and the estimate they compare against
- cover the sdnq attention kernel settings, including the head dimension clamp
that the hadamard group size resolves through
- cover the diffusers attention dispatcher and the backend names it takes
- record the constraints each override serves, leaving the throughput comparison
to the workload rather than naming a winner
Batch matrix-matrix and Dynamic Attention BMM applied a legacy Attention
processor to pipe.unet, which a diffusion transformer does not have, so
they served unet models alone and said nothing elsewhere. The choices, the
processor and its slice helper are removed, an unrecognized method now
warns rather than selecting nothing, and a stored value is rewritten to
Scaled-Dot-Product on load.
Vladimir Mandic
CalamitousFelicitousness
Yifan Chen