The prompt-enhancer grafts an lm_head onto the shared Qwen3-VL body, so its
forward needs a device guard the stock forward lacks. Applying that patch
globally at load left it active for the whole process, including unrelated
Qwen3-VL users such as the VQA captioner.
Apply the hijack around the enhancer generate call and restore the stock
forward in a finally, so the patch affects only the ideogram enhancer.
Register the subclass with generic.set_pipeline and build it with
from_pretrained, passing the SDNQ transformers and shared text encoder
while the vae, scheduler, and tokenizer load from the repo.
The dynamic-prompt brace processor in apply_styles_to_prompts strips the
{} and [] out of a JSON caption, leaving non-JSON that trips the model's
weight-baked safety placeholder. Let a model opt out of style and wildcard
processing via keep_prompts and set it for Ideogram4, then normalize the
prompt in encode_prompt: valid JSON to the compact training form, plain
text wrapped into a minimal caption so basic prompts still generate.
Each denoise step runs both transformers, so balanced offload ping-pongs
them across PCIe every step. When both fit gpu_memory * max watermark, set
offload_never so the per-step pre-sweep skips them and they stay resident;
otherwise the normal offload path is kept, so smaller GPUs and bf16 fall
back to offloading unchanged.
diffusers shipped first-party Ideogram 4 (transformer + pipeline) in
9b0818cf, so drop the in-tree port and keep only SD.Next integration glue.
Bump the diffusers pin to 9b0818cf and build diffusers' Ideogram4Pipeline
from a thin loader with per-transformer SDNQ. A small subclass keeps the
text encoder resident for the Qwen3-VL tap under balanced offload, and the
step callback denormalizes the preview latent from vae.bn before unpatchify.
Deletes the ported transformer, pipeline, scheduler, text encoder, and
latent-norm constants.
Diffusers-native port of the 9.3B flow-matching DiT: dual-transformer asymmetric CFG, a 13-layer Qwen3-VL tap encoder deduped with VQA and prompt-enhance, the Flux.2 VAE, and a logit-normal schedule. Loads a published bf16 repo with SDNQ at load.
Vladimir Mandic
CalamitousFelicitousness