From e2a4ceb0a94e4de363df009c4b2f130df040ee31 Mon Sep 17 00:00:00 2001 From: Vladimir Mandic Date: Sun, 5 Jul 2026 20:04:48 +0200 Subject: [PATCH] update wiki Signed-off-by: Vladimir Mandic --- Model-Loading.md | 92 +++++++++++++++++++++++++++++++++++++++++++ Performance-Timers.md | 61 ++++++++++++++++++++++++++++ 2 files changed, 153 insertions(+) create mode 100644 Model-Loading.md create mode 100644 Performance-Timers.md diff --git a/Model-Loading.md b/Model-Loading.md new file mode 100644 index 0000000..2548410 --- /dev/null +++ b/Model-Loading.md @@ -0,0 +1,92 @@ +# Loading Models + +> [!NOTE] +> This guide is intended for ALL model types other than legacy **SD1.5/SD2.1/SD-XL** models that are loaded directly from a single file + +## Base Models + +> [!IMPORTANT] +> For all base models, start with the **Reference Models** listed in the **Networks** panel on the right sidebar +> SD.Next will automatically download and cache the required model files for you + +### Variants + +A single base model may have multiple available variants. In the Networks panel, variants are grouped by category + +- **Base** - the original model + This is the reference implementation for the family and should be used first +- **Distilled** - a smaller or faster version of the base model + Distilled variants often trade some quality for faster performance +- **Quantized** - a model version pre-quantized using [SDNQ-Quantization](SDNQ-Quantization) + These versions are much smaller (often ~4× smaller) and may run faster on limited memory +- **Community** - community-created variants + These can include fine-tuning, special-purpose training, or other custom modifications +- **Nunchaku** - optimized variants for NVIDIA GPUs + These may include specialized kernels or execution optimizations + +> [!TIP] +> When choosing a large model, prefer a *Quantized* variant first, unless you specifically need a *Base* model + +### Downloaded Cache + +SD.Next separates large shared model components from smaller model definition files: + +- `models/huggingface//`: Large model components + These are typically the heavy weights such as the *transformer*, and *text encoder* +- `models/Diffusers//`: Model definitions, configs + Also includes smaller supporting modules such as *VAE* + +This separation allows multiple models to share the same large components when possible +For example, different models may reuse the same text encoder, so the large encoder weights are downloaded once and reused + +## Finetunes & Custom Models + +### Downloading Safetensors + +When downloading safetensors from external sources such as [CivitAI](https://civitai.com/), put the files in the correct model folder so SD.Next can detect them properly + +- **SD15 / SD21 / SD-XL** → `models/Stable-diffusion/` +- **All other diffusion models** → `models/UNET/` + +This includes models such as Flux.1, Flux.2, Chroma, Z-Image, Anima, Krea.2, Qwen, and similar families + +Other model component types belong in these folders: + +- **VAE** → `models/VAE/` +- **LoRA** → `models/Lora/` +- **Text Encoder** → `models/Text-encoder/` +- **Embedding** → `models/Embeddings/` +- **ControlNet** → `models/control/` +- **Detailer** → `models/yolo/` +- **Upscaler** → `models/chainner/` + +> [!TIP] +> The root `models` folder is the base location for all model files +> You can change this path in **Settings → Paths** + +### Loading finetunes + +> [!IMPORTANT] +> Always load the reference base model first, then load the finetune on top of it + +A finetune is usually NOT a complete standalone model +It modifies a single component of the base model (typically, the *transformer* component), so the full base model and its standard components (for example, the *VAE* and *text encoder*) must be loaded first + +To load a finetune: + +1. Load the reference base model from the **Networks → Model** panel +2. Select the finetune model in **Networks → UNET** panel + +### Loading via Settings + +You can also configure base and finetune models in **Settings → Model Loading**. + +- **Base model**: usually found in the *Quicksettings* section +- **UNET model**: can also be added to *Quicksettings* for easier access + +### Common pitfalls + +- Custom models downloaded from external sources should be placed in the correct folder and may require restarting SD.Next or refreshing the model list +- If a finetune is loaded before its reference model, the model may fail to initialize +- If the file is in the wrong `models/` subfolder, SD.Next may not detect it automatically +- When a model uses separate encoder or VAE files, those files must be placed in the correct folder and loaded by the matching **Settings** option diff --git a/Performance-Timers.md b/Performance-Timers.md new file mode 100644 index 0000000..a321097 --- /dev/null +++ b/Performance-Timers.md @@ -0,0 +1,61 @@ +# Performance Timers + +After every generation, SD.Next prints a performance line below the image and in the server log that breaks down where the time went: + +> Time: 5.52s | total 8.57 pipeline 3.60 decode 1.80 onload 1.02 prompt 0.77 preview 0.76 move 0.46 | GPU 11238 MB 46% | RAM 15.73 GB 24% + +- First number is the wall-clock duration of the inference, including UI overhead of all stages +- The middle section lists per-stage timers, sorted by duration +- Stages taking less than 0.25 sec are hidden +- **GPU** and **RAM** show peak memory usage for the run + +Reading this line is the fastest way to find out *why* a generation is slow and which setting to tune + +## Stage reference + +### Model execution + +| Timer | Meaning | Note | +| ----- | ------- | ---- | +| `pipeline` | The main denoising loop: UNet or DiT/transformer forward passes for each step | This is the core compute of the generation | +| `hires` | Second-pass processing when HiRes fix is enabled | | +| `refine` | Refiner pass when a refiner model is enabled | | +| `decode` | The final decode stage: turning latents into the output image, including image post-processing | +| `vae` | Time spent inside actual VAE encode/decode calls, e.g. encoding the input image for img2img/inpaint | Overlaps with `decode` on the output side | +| `te` | Text encoder forward pass: turning the parsed prompt into conditioning tensors | +| `prompt` | Prompt parse and text-encode: attention syntax, prompt scheduling and building the embeddings | Overlaps with `te` | +| `lora` | Loading, applying and removing LoRA weights | See [LoRA](LoRA) for the performance impact of different LoRA modes | + +### Memory management + +| Timer | Meaning | Note | +| ----- | ------- | ---- | +| `sync` | GPU synchronization time: waiting for GPU to finish one task before moving to the next one | can be (unsafe) disabled in *Settings -> Backend Settings* | +| `onload` | Moving model components from RAM to VRAM when offloading is active | See [Offload](Offload) | +| `offload` | Moving model components from VRAM back to RAM | See [Offload](Offload) | +| `move` | Module component forced device moves | Used by legacy pipelines only, modern models report `onload`/`offload` instead | +| `gc` | Garbage collection: clearing unused RAM and VRAM between stages | + +### Everything else + +| Timer | Meaning | Note | +| ----- | ------- | ---- | +| `preview` | Generating live previews during sampling | Runs asynchronously on a separate thread, so it only partially adds to wall-clock time | +| `callback` | Per-step callbacks executed during sampling | +| `init` | Pipeline and sampler initialization | +| `prepare`, `pre` | Applying processing modifiers (IP-Adapter, HiDiffusion, PAG, etc.) before the run | +| `post` | Unapplying processing modifiers and post-processing after the run | +| `process` | Overall per-batch processing wrapper | overlaps with the stages above | +| `validate` | Validating decoded samples and converting them to image format | +| `proc` | Running Control input processors (canny, depth, pose, etc.) on the input image | + +> [!NOTE] +> Video models (LTX, FramePack) report their own stage timers such as `base`, `upsample`, `sample`, `encode` and `vision` which follow the same principle: each named stage is the time spent in that part of the video pipeline. + +## What to tune + +- **High `pipeline`**: this is the actual model compute. Fewer steps, a faster sampler, quantization, or model compile. See [Performance Tuning](Performance-Tuning) +- **High `onload`/`offload`/`move`**: model parts are being shuffled between RAM and VRAM every generation. If you have VRAM headroom, raise the offload low watermark or exempt specific model/module types from offloading in *Settings -> Models & Loading*. Note that offloading exists for a reason: workflows that spike VRAM usage, such as HiRes to a much higher resolution, need that headroom for decode. See [Offload](Offload) +- **High `decode`/`vae`**: consider a faster VAE, or check whether VAE tiling/slicing is enabled unnecessarily on a high-VRAM system. See [VAE](VAE) +- **High `lora`**: LoRA apply cost depends heavily on LoRA type and mode; see the measurements in [Offload](Offload#performance-notes) +- **High `te`/`prompt`**: large text encoders (T5, LLMs) are expensive; offloading them adds onload cost on every prompt change. Reusing the same prompt avoids re-encoding