Public/automatic
1
0
Fork 0
mirror of https://github.com/vladmandic/automatic synced 2026-09-18 00:34:33 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs(i18n): add hints for model offloading settings

Browse Source
This commit is contained in:
CalamitousFelicitousness
2026-06-07 01:50:47 +01:00
parent 5105e0ad87
commit 91a4e9cef4
1 changed files with 16 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ui/locale/locale_en.json
+16 -14
View File
@@ -621,6 +621,7 @@
{"id":"","label":"Generic","localized":"","hint":"","ui":"video"},
{"id":"","label":"Google GenAI","localized":"","hint":"","ui":"settings_model_options"},
{"id":"","label":"Group Offload","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Group offload type","localized":"","hint":"Granularity used by <b>group</b> offload.<br>- <b>leaf_level</b>: offloads at the smallest module level; maximum memory savings, slower<br>- <b>block_level</b>: offloads groups of transformer blocks (size set by <b><i>Offload blocks</i></b>); faster with less savings, and keeps the VAE resident<br><br>Applies only when <b><i>Model offload mode</i></b> is <b>group</b>.<br><br>Default is <b>leaf_level</b>.","ui":"settings_offload"},
{"id":"","label":"Grid Options","localized":"","hint":"","ui":"settings_saving-images"},
{"id":"","label":"Grids","localized":"","hint":"","ui":"settings_saving-paths"},
{"id":"","label":"Guider","localized":"","hint":"","ui":"txt2img"},
@@ -841,7 +842,7 @@
{"id":"","label":"Latent history size","localized":"","hint":"","ui":"settings_sd"},
{"id":"","label":"LLama repo","localized":"","hint":"","ui":"settings_model_options"},
{"id":"","label":"low noise","localized":"","hint":"","ui":"settings_model_options"},
{"id":"","label":"Load caption models direct to GPU","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Load caption models direct to GPU","localized":"","hint":"Loads captioning and interrogation models straight onto the GPU instead of loading into RAM first.<br>Faster to start captioning, but uses VRAM while the caption model is loaded.<br><br>Enabled by default.","ui":"settings_offload"},
{"id":"","label":"leaf_level","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"LLM","localized":"","hint":"","ui":"settings_quantization"},
{"id":"","label":"Layerwise casting storage","localized":"","hint":"","ui":"settings_quantization"},
@@ -939,9 +940,10 @@
{"id":"","label":"Model auto-download on demand","localized":"","hint":"","ui":"settings_sd"},
{"id":"","label":"Model load using streams","localized":"","hint":"When loading models attempt stream loading optimized for slow or network storage","ui":"settings_sd"},
{"id":"","label":"Model load model direct to GPU","localized":"","hint":"","ui":"settings_sd"},
{"id":"","label":"Model types not to offload","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Modules to always offload","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Modules to never offload","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Model offload mode","localized":"","hint":"Controls how model components move between VRAM and system RAM to fit larger models on less VRAM.<br>- <b>none</b>: keeps everything on the GPU; fastest, but only works if the whole model fits in VRAM<br>- <b>balanced</b>: the recommended default; offloads only when VRAM use crosses a threshold, so it suits almost any GPU (tuned by the watermarks below)<br>- <b>group</b>: offloads groups of layers via diffusers group offloading; an alternative middle ground with optional stream prefetch<br>- <b>model</b>: offloads whole components such as the VAE or text encoder when idle; a more compatible fallback when balanced or group are unsupported, with smaller savings<br>- <b>sequential</b>: offloads layer by layer; the most memory saving but slowest, for when even balanced runs out of memory<br><br>Command-line flags override the automatic choice:<br>- <code>--lowvram</code>: forces <b>sequential</b><br>- <code>--medvram</code>: forces <b>balanced</b> with low watermark <b>0</b><br><br>With no flag, <b>balanced</b> is the automatic default on any GPU, with watermarks set by GPU memory (low / high):<br>- 12 GB or less: <b>0</b> / <b>0.6</b><br>- 12-24 GB: <b>0.2</b> / <b>0.6</b><br>- 24 GB or more: <b>0.2</b> / <b>0.8</b><br>(or <b>none</b> if no GPU is detected)","ui":"settings_offload"},
{"id":"","label":"Model types not to offload","localized":"","hint":"Model architectures to skip when offloading, separated by spaces or commas.<br>Useful for model types that misbehave when offloaded.<br><br>Applies only to <b>balanced</b> offload.<br><br>Default is empty.","ui":"settings_offload"},
{"id":"","label":"Modules to always offload","localized":"","hint":"Module names that are always offloaded in <b>balanced</b> mode, separated by spaces, commas, or semicolons, regardless of the watermarks.<br><br>Applies only to <b>balanced</b> offload.<br><br>Default by GPU memory: the large text encoders (<i>T5</i>, <i>UMT5</i>) are added at roughly 4-12 GB and at 24 GB or more; empty otherwise.","ui":"settings_offload"},
{"id":"","label":"Modules to never offload","localized":"","hint":"Module names that are never offloaded in <b>balanced</b> mode, separated by spaces, commas, or semicolons, keeping them resident in VRAM.<br><br>Applies only to <b>balanced</b> offload.<br><br>Default by GPU memory: the CLIP text encoders and the VAE are kept resident at 24 GB or more; empty otherwise.","ui":"settings_offload"},
{"id":"","label":"Model types not to quantize","localized":"","hint":"","ui":"settings_quantization"},
{"id":"","label":"Modules to not convert","localized":"","hint":"Names of modules to leave unquantized (kept in original precision), separated by spaces, commas, or semicolons.<br>Useful for layers that are sensitive to quantization, such as gate or projection layers. Example: <code>proj_out, x_embedder</code>.<br><br>Some models already exclude sensitive modules by default; entries here extend that built-in list rather than replacing it.<br><br>Default is empty.","ui":"settings_quantization"},
{"id":"","label":"Modules dtype dict","localized":"","hint":"Advanced: JSON mapping a quantization type to a list of module names, to quantize specific modules at a different type than the global <b><i>Quantization type</i></b>. Example: <code>{\"uint4\": [\"proj_out\"]}</code>.<br><br>Some models already assign certain modules a specific type by default; entries here merge with those built-in mappings rather than replacing them.<br><br>Default is empty.","ui":"settings_quantization"},
@@ -995,7 +997,7 @@
{"id":"","label":"Near threshold","localized":"","hint":"","ui":"control"},
{"id":"","label":"Noise scale","localized":"","hint":"","ui":"video"},
{"id":"","label":"Note","localized":"","hint":"","ui":"component-8823"},
{"id":"","label":"Non-blocking move operations","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Non-blocking move operations","localized":"","hint":"Uses non-blocking transfers when moving weights between GPU and RAM, letting copies overlap with other work.<br>Can be faster, but may be unstable on some platforms.<br><br>Disabled by default.","ui":"settings_offload"},
{"id":"","label":"Nunchaku attention","localized":"","hint":"Replaces default attention with Nunchaku's custom FP16 attention kernel for faster inference on consumer NVIDIA GPUs.<br>Might provide performance improvement on GPUs which have higher FP16 tensor cores throughput than BF16.<br><br>Currently only affects <i>Flux</i>-based models (<i>Dev</i>, <i>Schnell</i>, <i>Kontext</i>, <i>Fill</i>, <i>Depth</i>, etc.). Has no effect on <i>Qwen</i>, <i>SDXL</i>, <i>Sana</i>, or other architectures.<br><br>Disabled by default.","ui":"settings_quantization"},
{"id":"","label":"Nunchaku offloading","localized":"","hint":"Enables Nunchaku's own per-block CPU offloading with asynchronous CUDA streams to reduce VRAM usage.<br>Uses a ping-pong buffer strategy: while one transformer block computes on GPU, the next block preloads from CPU in the background, hiding most of the transfer latency.<br><br>Can reduce VRAM usage at the cost of slower inference.<br>This replaces SD.Next's pipeline offloading for the transformer component.<br><br>Only useful on low-VRAM GPUs. If your GPU has enough memory to hold the quantized model (16+ GB), keep this disabled for maximum speed.<br>Supports <i>Flux</i> and <i>Qwen</i> models. Not supported for <i>SDXL</i> where this setting is ignored.<br>Disabled by default.","ui":"settings_quantization"},
{"id":"","label":"native","localized":"","hint":"","ui":"settings_text_encoder"},
@@ -1034,13 +1036,13 @@
{"id":"","label":"Offload processor","localized":"","hint":"","ui":"control"},
{"id":"","label":"Output directory","localized":"","hint":"Folder where the processed images should be saved to","ui":"extras"},
{"id":"","label":"original","localized":"","hint":"Original LDM backend","ui":"settings_sd"},
{"id":"","label":"Offload caption models","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Offload during pre-forward","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Offload using streams","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Offload low watermark","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Offload GPU high watermark","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Offload CPU high watermark","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Offload blocks","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Offload caption models","localized":"","hint":"Moves captioning and interrogation models out of VRAM when they are not in use, freeing memory for generation.<br><br>Enabled by default.","ui":"settings_offload"},
{"id":"","label":"Offload during pre-forward","localized":"","hint":"In <b>balanced</b> offload, rebalances VRAM just before each component runs rather than on demand.<br><br>Applies only to <b>balanced</b> offload.<br><br>Enabled by default.","ui":"settings_offload"},
{"id":"","label":"Offload using streams","localized":"","hint":"In <b>balanced</b> offload, uses CUDA streams to overlap weight transfers with computation, hiding transfer latency.<br>Faster, but uses more VRAM and needs a compatible GPU.<br><br>Applies only to <b>balanced</b> offload.<br><br>Disabled by default.","ui":"settings_offload"},
{"id":"","label":"Offload low watermark","localized":"","hint":"Lower VRAM threshold for <b>balanced</b> offload, as a fraction of total GPU memory. While VRAM use stays below this, nothing is offloaded; above it, idle components are moved back to RAM.<br><br>Applies only to <b>balanced</b> offload. <code>--lowvram</code> and <code>--medvram</code> set this to <b>0</b>.<br><br>Default by GPU memory: <b>0</b> at 12 GB or less, <b>0.2</b> above.","ui":"settings_offload"},
{"id":"","label":"Offload GPU high watermark","localized":"","hint":"Upper VRAM threshold for <b>balanced</b> offload, as a fraction of total GPU memory. Sets the most VRAM a single component may use before it is offloaded.<br><br>Applies only to <b>balanced</b> offload.<br><br>Default by GPU memory: <b>0.6</b>, rising to <b>0.8</b> at 24 GB or more.","ui":"settings_offload"},
{"id":"","label":"Offload CPU high watermark","localized":"","hint":"Upper system-RAM threshold for offloaded weights in <b>balanced</b> offload, as a fraction of total RAM.<br><br>Applies only to <b>balanced</b> offload.<br><br>Default is <b>0.9</b>.","ui":"settings_offload"},
{"id":"","label":"Offload blocks","localized":"","hint":"Number of transformer blocks per offload group when <b><i>Group offload type</i></b> is <b>block_level</b>. Larger groups offload less often (faster, more VRAM); smaller groups save more memory.<br><br>Applies only to <b>group</b> offload with <b>block_level</b>.<br><br>Default is <b>1</b>.","ui":"settings_offload"},
{"id":"","label":"OpenVINO activations mode","localized":"","hint":"","ui":"settings_quantization"},
{"id":"","label":"ONNX Execution Provider","localized":"","hint":"","ui":"settings_backends"},
{"id":"","label":"ONNX allow fallback to CPU","localized":"","hint":"Allow fallback to CPU when selected execution provider failed","ui":"settings_backends"},
@@ -1235,7 +1237,7 @@
{"id":"","label":"Rebase","localized":"","hint":"","ui":"tab_update"},
{"id":"","label":"Repos","localized":"","hint":"","ui":"component-8779"},
{"id":"","label":"Refiner model","localized":"","hint":"Refiner model used for second-pass operations","ui":"settings_sd"},
{"id":"","label":"Record torch streams","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Record torch streams","localized":"","hint":"Records CUDA stream usage during <b>group</b> offload so reused buffers stay correct when <b><i>Use torch streams</i></b> is enabled.<br><br>Applies only to <b>group</b> offload with streams.<br><br>Disabled by default.","ui":"settings_offload"},
{"id":"","label":"Remote VAE image type","localized":"","hint":"","ui":"settings_vae_encoder"},
{"id":"","label":"Remote VAE for encode","localized":"","hint":"","ui":"settings_vae_encoder"},
{"id":"","label":"RAS enabled","localized":"","hint":"","ui":"settings_advanced"},
@@ -1559,7 +1561,7 @@
{"id":"","label":"Use spaces","localized":"","hint":"Replace underscores with spaces in tag output.<br>Some prompt systems prefer spaces between words (e.g., 'long hair') while others use underscores (e.g., 'long_hair').","ui":"caption"},
{"id":"","label":"Username","localized":"","hint":"","ui":"component-8823"},
{"id":"","label":"UNET model","localized":"","hint":"","ui":"settings_sd"},
{"id":"","label":"Use torch streams","localized":"","hint":"","ui":"settings_offload"},
{"id":"","label":"Use torch streams","localized":"","hint":"In <b>group</b> offload, uses CUDA streams to prefetch the next group while the current one runs, hiding transfer latency.<br>Faster, but uses more VRAM.<br><br>Applies only to <b>group</b> offload.<br><br>Disabled by default.","ui":"settings_offload"},
{"id":"","label":"Use SVD quantization","localized":"","hint":"Adds a low-rank (SVDQuant) correction on top of SDNQ to recover accuracy lost at low bit widths, at the cost of extra size and compute.<br>Tuned by <b><i>SVD rank size</i></b> and <b><i>SVD steps</i></b>.<br><br>Disabled by default.","ui":"settings_quantization"},
{"id":"","label":"Use Dynamic quantization","localized":"","hint":"Picks a per-layer weight type automatically instead of one type everywhere. Each layer starts at the <b><i>Quantization type</i></b> (the minimum) and steps up to higher precision until its error meets the <b><i>Dynamic loss threshold</i></b>.<br>Protects error-sensitive layers at the cost of a larger model and slower load.<br><br>Disabled by default.","ui":"settings_quantization"},
{"id":"","label":"Use Hadamard rotations","localized":"","hint":"Applies a Hadamard rotation before quantizing to spread out weight outliers, which can improve accuracy at low bit widths.<br>Group size is set by <b><i>Hadamard group size</i></b>.<br><br>Disabled by default.","ui":"settings_quantization"},