Anima-2.9B is a depth-expanded finetune of Anima 1.0 Base carrying 40
transformer blocks against the base repo's 28. The reference entry points at
the Diffusers conversion. Single-file releases load through the native loader:
TransformerSpec gains an infer_config hook, the Anima spec uses it to size
num_layers to the block indices in the file, and model_anima routes a
checkpoint-selected safetensors through the loader with the remaining
components from the base repo.
Triton cannot compile e4m3 loads before sm_89, so fp8 weights fall back
to eager dequant there. The uint8-backed float8_e4m3fn_sdnq codec decodes
identically now that subnormals are handled (the two NaN codes become
+/-480), and its compiled dequant runs about 6x faster than eager native
fp8. Pre-quantized fp8 layers are viewed as uint8 at adoption when
compiled dequant is enabled on such hardware; the eager gate remains the
safety net for every other fp8 path.
nvfp4 layers keep their packed 4-bit codes and land on SDNQ grouped
quantization as float4_e2m1fn: the nibble order is swapped once at load,
the e4m3 block scales are unswizzled from the cuBLAS tile layout, and
the fp32 global scale folds into them as per-group scales. Marker
orig_shape acts as a cross-check and alignment padding is sliced
against the model dimensions.
- reject nvfp4 markers with unknown group sizes or convrot flags
- accept uint8 storage in the krea2 real-file check
Newer quantized checkpoints record per-layer formats in the safetensors
header _quantization_metadata instead of marker tensors. The native loader
now reads the header map, re-keys it through the same prefix strip as the
tensors, and transcodes it into marker tensors so both container forms
share one detection path; header entries win over markers.
- drop optional input_scale sidecars for marked layers
- map full_precision_matrix_mult onto the sdnq per-layer matmul exclusion list
- log the detection source (markers, header, both)
ConvRot is the regular Hadamard rotation SDNQ implements: identical
construction, normalization, axis, and dequant order. Per-layer markers
map onto the dequantizer's use_hadamard and hadamard_group_size; group
sizes must be powers of 4 and divide in_features, else base-repo
fallback. Detection carries per-layer metadata since files mix plain
and rotated layers.
One UNET override cannot serve dual-transformer arches: ideogram4
conditional/unconditional and wan combined-stage experts need separate
files, and previously a single override landed on both experts.
- sd_unet_secondary option with per-slot tracking, consumed-state sync,
arch-change reset, and incompatible-override fallback
- dropdown renders beside the primary, follows it into quicksettings,
and is visible only for dual-transformer model types
- ideogram4 native single-file spec with a quant-aware fused-qkv
converter; such converters run before comfy_quant detection via
TransformerSpec.converter_handles_quant
- quicksettings render in configured order (sort keyed on the option
object and always fell back to alphabetical)
- post-load dtype warning skips quantized transformers
Parameterize the pre-quantized comfy_quant path by format so fp8
tensorwise checkpoints load alongside int8; both map onto SDNQ's
symmetric dequant for the matching weights dtype. Containers are
mislabeled in the wild, so the stored weight dtype is validated
against the marker, mixed-format files are rejected, and markers
carrying a convrot rotation flag fall back to the base repo since
they may require runtime inverse rotation.
Detect comfy_quant markers in native single-file transformer loads and
adopt the pre-quantized tensors as SDNQ int8 layers instead of rejecting
them. ComfyUI int8_tensorwise is a strict subset of SDNQ symmetric int8,
so weights and scales are taken bit-exact with no dequantize-requantize
round trip; quantized matmul and scale-dtype settings apply as usual.
- detect_comfy_quant/remap_comfy_quant helpers plus a prequantized
builder in native_transformer; file markers dictate the layer set
- unsupported formats and wrong-arch markers fall back to the base repo
- offline unit tests and an opt-in real-file krea2 test
All-in-one exports bundle the text encoder and VAE alongside the
transformer under LDM-style family prefixes (cond_stage_model.,
first_stage_model., text_encoders., vae.). The native loader treated
those keys as a mixed-prefix error and rejected the file. Drop known
companion families before prefix detection and log what was skipped;
keys matching neither a transformer prefix nor a known family still
raise. TE and VAE keep coming from the base repo or their own overrides.
strip_prefix returns the detected prefix instead of logging it, so the load
summary reports it alongside the key count and the redundant reading-state
marker is gone. Four near-identical cls/file lines become one.
Base ships last.up all-zeros so the branch is a no-op; pre-branch
finetunes omit both keys. A zero_init_missing spec field zero-fills
them on load instead of falling back to the base transformer.
A converter error or load_state_dict mismatch now raises OverrideArchMismatch, which load_transformer catches to drop the override and load the base transformer. No per-arch markers to maintain.
load_transformer threads **kwargs into its from_pretrained and from_single_file branches. The native dispatch branch bypasses both loaders and builds via cls.from_config, so those kwargs were dropped there. Thread them through load, build_component, and build_component_quantized into from_config so callers passing extra args alongside native_spec are honored instead of silently dropped. Siblings do not receive them.
The transformer load path now mirrors diffusers' load_model_dict_into_meta: build the component inside init_empty_weights(include_buffers=False) so parameter slots stay meta while computed buffers keep their __init__ values, then iterate the converted state_dict tensor by tensor. Each tensor is dispatched through SDNQQuantizer.check_if_quantized_param and create_quantized_param so Linear/Conv/Embed weights are packed to uint4 in flight, while non-quantizable parameters go through accelerate.set_module_tensor_to_device.
Peak memory during transformer load drops from the full bf16 model size to one layer's worth of bf16 plus the running uint4 result. The prior load_state_dict + .to(dtype) + post-walk sdnq_quantize_model sequence remains as the fallback path for siblings and non-SDNQ quant types.
generic.load_transformer already computes quant_args and quant_type at the top of the function, but native_transformer.load was redoing the same work and emitting a duplicate 'Quantization: module=Model ...' log line milliseconds later. Pass them through instead; fall back to computing them when called directly (model_anima still does so).
The native loader was emitting one line at dispatch, a single line after prefix detection, and the total time at the end. For an 18 GB transformer that meant ~60s of silence with no indication anything was happening.
Add DEBUG lines before each long phase: read_state_dict, optional converter, load_state_dict into the model, and dtype cast. Total-time log at the end is unchanged.
apply_quant called sdnq_quantize_model(transformer, op="transformer")
without the per-call skip lists, dropping modules_to_not_convert and
modules_dtype_dict that the caller passed into generic.load_transformer.
The from_pretrained path receives them via quantization_config; the
native path now threads them directly into the explicit quantizer call.
Plumbed load -> build_component -> apply_quant as keyword args. No
behavior change for callers that did not set the skip lists.
New native_spec=None kwarg. When set and the UNET dropdown points at a
.safetensors, dispatches to native_transformer.load (threading
allow_quant/dtype/modules_to_not_convert/modules_dtype_dict). Pipelines
without a spec stay on cls.from_single_file unchanged.
217-line bespoke loader collapses to a 40-line ANIMA_SPEC in
pipelines/anima/__init__.py (Cosmos converter + llm_adapter sibling +
Cosmos 1.0 forbidden marker).
Drop the class-keyed REGISTRY: Anima and raw Cosmos share
CosmosTransformer3DModel but need different specs. Specs pass via
explicit native_spec= kwarg; make_default_spec(cls) covers the
auto-converter case.
Vladimir Mandic
Ryan Meador
CalamitousFelicitousness
Dity0