Public/automatic
1
0
Fork 0
mirror of https://github.com/vladmandic/automatic synced 2026-09-19 09:14:35 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #4268 from vladmandic/dev

Browse Source 
merge dev
This commit is contained in:
Vladimir Mandic
2025-10-17 09:03:16 -04:00
committed by GitHub
parent 7c8f42fab9 e66d78f556
commit 5e796b928a
300 changed files with 30293 additions and 3106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.eslintrc.json
+2 -1
View File
@@ -116,7 +116,8 @@
"idbDel": "readonly",
"idbAdd": "readonly",
"initChangelog": "readonly",
"sendNotification": "readonly"
"sendNotification": "readonly",
"monitorConnection": "readonly"
},
"ignorePatterns": [
"node_modules",
.gitignore
-8
View File
@@ -46,14 +46,6 @@ tunableop_results*.csv
!package.json
!requirements.txt
# pyinstaller
*.spec
build/
dist/
# dynamically generated
/repositories/deepseek-vl2/
# all dynamic stuff
/extensions/**/*
/outputs/**/*
.gitmodules
-4
View File
@@ -14,10 +14,6 @@
path = extensions-builtin/stable-diffusion-webui-rembg
url = https://github.com/vladmandic/sd-extension-rembg
ignore = dirty
[submodule "extensions-builtin/sd-webui-agent-scheduler"]
path = extensions-builtin/sd-webui-agent-scheduler
url = https://github.com/ArtVentureX/sd-webui-agent-scheduler
ignore = dirty
[submodule "extensions-builtin/sdnext-modernui"]
path = extensions-builtin/sdnext-modernui
url = https://github.com/BinaryQuantumSoul/sdnext-modernui
.pylintrc
+3 -1
View File
@@ -29,6 +29,7 @@ ignore-paths=/usr/lib/.*$,
modules/postprocess/aurasr_arch.py,
modules/prompt_parser_xhinker.py,
modules/ras,
modules/seedvr,
modules/rife,
modules/schedulers,
modules/taesd,
@@ -42,8 +43,10 @@ ignore-paths=/usr/lib/.*$,
pipelines/meissonic,
pipelines/omnigen2,
pipelines/segmoe,
pipelines/xomni,
scripts/consistory,
scripts/ctrlx,
scripts/daam,
scripts/demofusion,
scripts/freescale,
scripts/infiniteyou,
@@ -73,7 +76,6 @@ persistent=yes
py-version=3.9
recursive=no
source-roots=
suggestion-mode=yes
unsafe-load-any-extension=no
[BASIC]
.ruff.toml
+3
View File
@@ -12,6 +12,7 @@ exclude = [
"modules/pag",
"modules/schedulers",
"modules/teacache",
"modules/seedvr",
"modules/control/proc",
"modules/control/units",
@@ -22,8 +23,10 @@ exclude = [
"pipelines/omnigen2",
"pipelines/hdm",
"pipelines/segmoe",
"pipelines/xomni",
"scripts/lbm",
"scripts/daam",
"scripts/xadapter",
"scripts/pulid",
"scripts/instantir",
CHANGELOG.md
+183 -9
View File
@@ -1,5 +1,174 @@
# Change Log for SD.Next
## Update for 2025-10-17
### Highlights for 2025-10-17
It's been a month since the last release and number of changes is yet again massive with over 300 commits!
Highlight are:
- **Torch**: ROCm on Windows for AMD GPUs
if you have a compatible GPU, performance gains are significant!
- **Models**:
a lot of new stuff with **Qwen-Image-Edit** including multi-image edits and distilled variants,
new **Flux**, **WAN**, **LTX**, **HiDream** variants, expanded **Nunchaku** support and new SOTA upscaler with **SeedVR2**
plus improved video support in general, including new methods of video encoding
- **Quantization**:
new **SVD**-style quantization using SDNQ offers almost zero-loss even with **4bit** quantization
and now you can also test your favorite quantization on-the-fly and then save/load model for future use
- Other: support for **Huggingface** mirrors, changes to installer to prevent unwanted `torch-cpu` operations, improved previews, etc.
### Details for 2025-10-17
- **Models**
- [WAN 2.2 14B VACE](https://huggingface.co/alibaba-pai/Wan2.2-VACE-Fun-A14B)
available for *text-to-image* and *text-to-video* and *image-to-video* workflows
- [Qwen Image Edit 2509](https://huggingface.co/Qwen/Qwen-Image-Edit-2509) and [Nunchaku Qwen Image Edit 2509](https://huggingface.co/nunchaku-tech/nunchaku-qwen-image-edit-2509)
updated version of Qwen Image Edit with improved image consistency
- [Qwen Image Pruning](https://huggingface.co/OPPOer/Qwen-Image-Pruning) and [Qwen Image Edit Pruning](https://huggingface.co/OPPOer/Qwen-Image-Edit-Pruning)
pruned versions of Qwen with 13B params instead of 20B, with some quality tradeoff
- [Tencent FLUX.1 Dev SRPO](https://huggingface.co/tencent/SRPO)
SRPO is trained by Tencent with specific technique: directly aligning the full diffusion trajectory with fine-grained human preference
- [Nunchaku SDXL](https://huggingface.co/nunchaku-tech/nunchaku-sdxl) and [Nunchaku SDXL Turbo](https://huggingface.co/nunchaku-tech/nunchaku-sdxl-turbo)
impact of nunchaku engine on unet-based model such as sdxl is much less than on a dit-based models, but its still significantly faster than baseline
note that nunchaku optimized and pre-quantized unet is replacement for base unet, so its only applicable to base models, not any of fine-tunes
*how to use*: enable nunchaku in settings -> quantization and then load either sdxl-base or sdxl-base-turbo reference models
- [HiDream E1.1](https://huggingface.co/HiDream-ai/HiDream-E1-1)
updated version of HiDream-E1 image editing model
- [LTXVideo 0.9.8](https://huggingface.co/Lightricks/LTX-Video-0.9.8-13B-distilled)
updated version of LTXVideo t2v/i2iv model
- [SeedVR2](https://iceclear.github.io/projects/seedvr/)
originally designed for video restoration, seedvr works great for image detailing and upscaling!
available in 3B, 7B and 7B-sharp variants, use as any other upscaler!
note: seedvr is a very large model (6.4GB and 16GB respectively) and not designed for lower-end hardware, quantization is highly recommended
note: seedvr is highly sensitive to its cfg scale, set in *settings -> postprocessing*
lower values will result in smoother output while higher values add details
- [X-Omni SFT](https://x-omni-team.github.io/)
*experimental*: X-omni is a transformer-only discrete auto-regressive image generative model trained with reinforcement learning
- **Features**
- **Model save**: ability to save currently loaded model as a new standalone model
why? SD.Next always prefers to start with full model and quantize on-demand during load
however, when you find your exact preferred quantization settings that work well for you,
saving such model as a new model allows for faster loads and reduced disk space usage
so its best of both worlds: you can experiment and test different quantization methods and once you find the one that works for you, save it as a new model
saved models appear in network tab as normal models and can be loaded as such
available in *models* tab
- [Qwen Image-Edit](https://huggingface.co/Qwen/Qwen-Image-Edit-2509) multi-image editing
requires qwen-image-edit-2509 or its variant as multi-image edits are not available in original qwen-image
in ui control tab: inputs -> separate init image
add image for *input media* and *control media*
can be
- [Cache-DiT](https://github.com/vipshop/cache-dit)
cache-dit is a unified, flexible and training-free cache acceleration framework
compatible with many dit-based models such as FLUX.1, Qwen, HunyuanImage, Wan2.2, Chroma, etc.
enable in *settings -> pipeline modifiers -> cache-dit*
- [Nunchaku Flux.1 PulID](https://nunchaku.tech/docs/nunchaku/python_api/nunchaku.pipeline.pipeline_flux_pulid.html)
automatically enabled if loaded model is FLUX.1 with Nunchaku engine enabled and when PulID script is enabled
- **Huggingface mirror** in *settings -> huggingface*
if you're working from location with limited access to huggingface, you can now specify a mirror site
for example enter, `https://hf-mirror.com`
- **Compute**
- **ROCm** for Windows
support for both official torch preview release of `torch-rocm` for windows and **TheRock** unofficial `torch-rocm` builds for windows
note that rocm for windows is still in preview and has limited gpu support, please check rocm docs for details
- **DirectML** warn as *end-of-life*
`torch-directml` received no updates in over 1 year and its currently superseded by `rocm` or `zluda`
- command line params `--use-zluda` and `--use-rocm` will attempt desired operation or fail if not possible
previously sdnext was performing a fallback to `torch-cpu` which is not desired
- **installer** if `--use-cuda` or `--use-rocm` are specified and `torch-cpu` is installed, installer will attempt to reinstall correct torch package
- **installer** warn if *cuda* or *rocm* are available and `torch-cpu` is installed
- support for `torch==2.10-nightly` with `cuda==13.0`
- **Extensions**
- [Agent-Scheduler](https://github.com/SipherAGI/sd-webui-agent-scheduler)
was a high-value built-in extension, but it has not been maintained for 1.5 years
it also does not work with control and video tabs which are the core of sdnext nowadays
so it has been removed from built-in extensions: manual installation is still possible
- [DAAM: Diffusion Attentive Attribution Maps](https://github.com/castorini/daam)
create heatmap visualizations of which parts of the prompt influenced which parts of the image
available in scripts for sdxl text-to-image workflows
- **Offloading**
- improve offloading for pipelines with multiple stages such as *wan-2.2-14b*
- add timers to measure onload/offload times during generate
- experimental offloading using `torch.streams`
enable in settings -> model offloading
- new feature to specify which models types not to offload
in *settings -> model offloading -> model types not to offload*
- **UI**
- **connection monitor**
main logo in top-left corner now indicates server connection status and hovering over it shows connection details
- separate guidance and detail sections
- networks ability to filter lora by base model version
- add interrogate button to input images
- disable spellchecks on all text inputs
- **SDNQ**
- add `SVDQuant` quantization method support
- make sdnq scales compatible with balanced offload
- add int8 `matmul` support for RDNA2 GPUs via triton
- improve int8 `matmul` performance on Intel GPUs
- **Other**
- server will note when restart is recommended due to package updates
- **interrupt** will now show last known preview image
*keep incomplete* setting is now *save interrupted*
- **logging** enable `debug`, `docs` and `api-docs` by default
- **logging** add detailed ram/vram utilization info to log
logging frequency can be specified using `--monitor x` command line param, where x is number of seconds
- **ipex** simplify internal implementation
- refactor to use new libraries
- styles and wildcards now use same seed as main generate for reproducible results
- **api** new endpoint POST `/sdapi/v1/civitai` to trigger civitai models metadata update
accepts optional `page` parameter to search specific networks page
- **reference models** additional example images, thanks @liutyi
- **reference models** add model size and release date, thanks @alerikaisattera
- **video** support for configurable multi-stage models such as WAN-2.2-14B
- **video** new LTX model selection
- replace `pynvml` with `nvidia-ml-py` for gpu monitoring
- update **loopback** script with radon seed option, thanks @rabanti
- **vae** slicing enable for *lowvram/medvram*, tiling for *lowvram*, both disabled otherwise
- **attention** remove split-attention and add explicitly attention slicing enable/disable option
enable in *settings -> compute settings*
can be combined with sdp, enabling may improve stability when used on iGPU or shared memory systems
- **nunchaku** update to `1.0.1` and enhance installer
- **xyz-grid** add guidance section
- **preview** implement configurable layers for WAN, Qwen, HV
- update swagger `/docs` endpoint style
- add `[epoch]` to filename template
- starting `[seq]` for filename template is now higher of largest previous sequence or number of files in folder
- **Video**
- use shared **T5** text encoder for video models when possible
- use shared **LLama** text encoder for video models when possible
- unified video save code across all video models
also avoids creation of temporary files for each frame unless user wants to save them
- unified prompt enhance code across all video models
- add job state tracking for video generation
- fix quantization not being applied on load for some models
- improve offloading for **ltx** and **wan**
- fix model selection in **ltx** tab
- **Experimental**
- `new` command line flag enables new `pydantic` and `albumentations` packages
- **modular pipelines**: enable in *settings -> model options*
only compatible with some pipelines, invalidates preview generation
- **modular guiders**: automatically used for compatible pipelines when *modular pipelines* is enabled
allows for using many different guidance methods:
*CFG, CFGZero, PAG, APG, SLG, SEG, TCFG, FDG*
- **Wiki**
- updates to *AMD-ROCm, ZLUDA, LoRA, DirectML, SDNQ, Quantization, Prompting, LoRA* pages
- new *Stability-Matrix* page
- **Fixes**
- **Microsoft Florence 2** both base and large variants
*note* this will trigger download of the new variant of the model, feel free to delete older variant in `huggingface` folder
- **MiaoshouAI PromptGen** 1.5/2.0 in both base and large variants
- fix prompt scheduling, thanks @nolbert82
- ui: fix image metadata display when switching selected image in control tab
- framepack: add explicit hf-login before framepack load
- framepack: patch solver for unsupported gpus
- benchmark: remove forced sampler from system info benchmark
- xyz-grid: fix xyz grid with random seeds
- reference: fix download for sd15/sdxl reference models
- fix checks in init/mask image decode
- fix hf token with extra chars
- image viewer refocus on gallery after returning from full screen mode
- fix attention guidance metadata save/restore
- vae preview add explicity cuda.sync
## Update for 2025-09-15
### Highlights for 2025-09-15
@@ -8,7 +177,7 @@
**StandardUI** is still available and can be selected in settings, but ModernUI is now the default for new installs
*What's else*? **Chroma** is in its final form, there are several new **Qwen-Image** variants and **Nunchaku** hit version 1.0!
Also, there are quite a few offloading improvements and many quality-of-life changes to UI and overal workflows
Also, there are quite a few offloading improvements and many quality-of-life changes to UI and overall workflows
And check out new **history** tab in the right panel, it now shows visualization of entire processing timeline!
![Screenshot](https://github.com/user-attachments/assets/d6119a63-6ee5-4597-95f6-29ed0701d3b5)
@@ -22,13 +191,13 @@ And check out new **history** tab in the right panel, it now shows visualization
- **Qwen-Image** [InstantX ControlNet Union](https://huggingface.co/InstantX/Qwen-Image-ControlNet-Union) support
*note* qwen-image is already a very large model and controlnet adds 3.5GB on top of that so quantization and offloading are highly recommended!
- [Qwen-Lightning-Edit](https://huggingface.co/vladmandic/Qwen-Lightning-Edit) and [Qwen-Image-Distill](https://huggingface.co/SahilCarterr/Qwen-Image-Distill-Full) variants
- **Nuchaku** variants of [Qwen-Image-Lightning](https://huggingface.co/nunchaku-tech/nunchaku-qwen-image), [Qwen-Image-Edit](https://huggingface.co/nunchaku-tech/nunchaku-qwen-image-edit), [Nunchaku-Qwen-Image-Edit-Lightning](https://huggingface.co/nunchaku-tech/nunchaku-qwen-image-edit)
- **Nunchaku** variants of [Qwen-Image-Lightning](https://huggingface.co/nunchaku-tech/nunchaku-qwen-image), [Qwen-Image-Edit](https://huggingface.co/nunchaku-tech/nunchaku-qwen-image-edit), [Nunchaku-Qwen-Image-Edit-Lightning](https://huggingface.co/nunchaku-tech/nunchaku-qwen-image-edit)
- **Nunchaku** variant of [Flux.1-Krea-Dev](https://huggingface.co/nunchaku-tech/nunchaku-flux.1-krea-dev)
if you have a compatible nVidia GPU, Nunchaku is the fastest quantization & inference engine
- [HunyuanDiT ControlNet](https://huggingface.co/Tencent-Hunyuan/HYDiT-ControlNet-v1.2) Canny, Depth, Pose
- [KBlueLeaf/HDM-xut-340M-anime](https://huggingface.co/KBlueLeaf/HDM-xut-340M-anime)
highly experimental: HDM *Home-made-Diffusion-Model* is a project to investigate specialized training recipe/scheme
for pretraining T2I model at home based on super-light architecture
for pre-training T2I model at home based on super-light architecture
*requires*: generator=cpu, dtype=float16, offload=none, both positive and negative prompts are required and must be long & detailed
- [Apple FastVLM](https://huggingface.co/apple/FastVLM-0.5B) in 0.5B, 1.5B and 7B variants
available in captioning tab
@@ -47,13 +216,13 @@ And check out new **history** tab in the right panel, it now shows visualization
- additional artwork for reference models in networks, thanks @liutyi
- improve ui hints display
- restyled all toolbuttons to be modernui native
- reodered system settings
- reordered system settings
- dynamic direction of dropdowns
- improve process tab layout
- improve detection of active tab
- configurable horizontal vs vertical panel layout
in settings -> user interface -> panel min width
*example*: if panel width is less than specified value, layout switches to verical
*example*: if panel width is less than specified value, layout switches to vertical
- configurable grid images size
in *settings -> user interface -> grid image size*
- gallery now includes reference model images
@@ -64,10 +233,10 @@ And check out new **history** tab in the right panel, it now shows visualization
- improve offloading of models with multiple dits
- improve offloading of models with impliciy vae processing
- improve offloading of models with controlnet
- more aggressive offloading of controlnets with lowvram flag
- more aggressive offloading of controlnet with lowvram flag
- **group**
- new offloading method, using *type=leaf* works on a similar level as sequential offloading
and can present siginificant savings on low-vram gpus, but comes at the higher performace cost
and can present significant savings on low-vram gpus, but comes at the higher performance cost
- **Quantization**
- option to specify models types not to quantize: *settings -> quantization*
allows for having quantization enabled, but skipping specific model types that do not need it
@@ -82,6 +251,13 @@ And check out new **history** tab in the right panel, it now shows visualization
*experimental*: requires new pydantic package which *may* break other things, to enable start sdnext with `--new` flag
*note*: this is model quantization only, no support for tensorRT inference yet
- **Other**
- **LoRA** allow specifying module to apply lora on
*example*: `<lora:mylora:1.0:module=unet>` would apply lora *only* on unet regardless of lora content
this is particularly useful when you have multiple loras and you want to apply them on different parts of the model
*example*: `<lora:firstlora:1.0:high>` and `<lora:secondlora:1.0:low>`
*note*: `low` is shorthand for `module=transformer_2` and `high` is shortcut for `module=transformer`
- **Detailer** allow manually setting processing resolution
*note*: this does not impact the actual image resolution, only the resolution at which detailer internally operates
- refactor reuse-seed and add functionality to all tabs
- refactor modernui js codebase
- move zluda flash attenion to *Triton Flash attention* option
@@ -96,8 +272,6 @@ And check out new **history** tab in the right panel, it now shows visualization
- add deprecation warning for `python==3.9`
- allow setting denoise strength to 0 in control/img2img
this allows to run workflows which only refine or detail existing image without changing it
- **Detailer** allow manually setting processing resolution
*note*: this does not impact the actual image resolution, only the resolution at which detailer internally operates
- **Fixes**
- normalize path hanlding when deleting images
- unified compile upscalers
README.md
+6 -5
View File
@@ -6,6 +6,7 @@
![Last update](https://img.shields.io/github/last-commit/vladmandic/sdnext?svg=true)
![License](https://img.shields.io/github/license/vladmandic/sdnext?svg=true)
[![Discord](https://img.shields.io/discord/1101998836328697867?logo=Discord&svg=true)](https://discord.gg/VjvR2tabEX)
[![DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/vladmandic/sdnext)
[![Sponsors](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&color=%23fe8e86)](https://github.com/sponsors/vladmandic)
[Docs](https://vladmandic.github.io/sdnext-docs/) | [Wiki](https://github.com/vladmandic/sdnext/wiki) | [Discord](https://discord.gg/VjvR2tabEX) | [Changelog](CHANGELOG.md)
@@ -30,12 +31,12 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
▹ **Standard | Modern**
- Multiple [diffusion models](https://vladmandic.github.io/sdnext-docs/Model-Support/)!
- Built-in Control for Text, Image, Batch and Video processing!
- Multiplatform!
- Multi-platform!
▹ **Windows | Linux | MacOS | nVidia CUDA | AMD ROCm | Intel Arc / IPEX XPU | DirectML | OpenVINO | ONNX+Olive | ZLUDA**
- Platform specific autodetection and tuning performed on install
- Platform specific auto-detection and tuning performed on install
- Optimized processing with latest `torch` developments with built-in support for model compile and quantize
Compile backends: *Triton | StableFast | DeepCache | OneDiff | TeaCache | etc.*
Quantization methods: *SDNQ | BitsAndBytes | Optimum-Quanto | TorchAO*
Quantization methods: *SDNQ | BitsAndBytes | Optimum-Quanto | TorchAO / LayerWise*
- **Interrogate/Captioning** with 150+ **OpenCLiP** models and 20+ built-in **VLMs**
- Built-in queue management
- Built in installer with automatic updates and dependency management
@@ -53,7 +54,7 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
<img src="https://github.com/user-attachments/assets/ced9fe0c-d2c2-46d1-94a7-8f9f2307ce38" alt="screenshot-modernui-mobile" width="35%">
</div>
For screenshots and informations on other available themes, see [Themes](https://vladmandic.github.io/sdnext-docs/Themes/)
For screenshots and information on other available themes, see [Themes](https://vladmandic.github.io/sdnext-docs/Themes/)
<br>
@@ -74,7 +75,7 @@ SD.Next supports broad range of models: [supported models](https://vladmandic.gi
- *ONNX/Olive*
- *AMD* GPUs on Windows using **ZLUDA** libraries
Plus Docker container receipes for: [CUDA, ROCm, Intel IPEX and OpenVINO](https://vladmandic.github.io/sdnext-docs/Docker/)
Plus Docker container recipes for: [CUDA, ROCm, Intel IPEX and OpenVINO](https://vladmandic.github.io/sdnext-docs/Docker/)
## Getting started
TODO.md
+7 -8
View File
@@ -13,21 +13,15 @@ Main ToDo list can be found at [GitHub projects](https://github.com/users/vladma
- Video Core: API
- Video LTX: TeaCache and others, API, Conditioning preprocess Video: LTX API
### Blocked items
- Upgrade: `albumentations`
- blocked by `insightface`
### Under Consideration
- [X-Omni](https://github.com/X-Omni-Team/X-Omni/blob/main/README.md)
- [Inf-DiT](https://github.com/zai-org/Inf-DiT)
- [DiffSynth Studio](https://github.com/modelscope/DiffSynth-Studio)
- [IPAdapter negative guidance](https://github.com/huggingface/diffusers/discussions/7167)
- [IPAdapter composition](https://huggingface.co/ostris/ip-composition-adapter)
- [STG](https://github.com/huggingface/diffusers/blob/main/examples/community/README.md#spatiotemporal-skip-guidance)
- [SmoothCache](https://github.com/huggingface/diffusers/issues/11135)
- [MagCache](https://github.com/lllyasviel/FramePack/pull/673/files)
- [Nunchaku PulID](https://github.com/mit-han-lab/nunchaku/pull/274)
- [Dream0 guidance](https://huggingface.co/ByteDance/DreamO)
- [SUPIR upscaler](https://github.com/Fanghua-Yu/SUPIR)
- [ByteDance OneReward](https://github.com/bytedance/OneReward)
@@ -40,14 +34,19 @@ Main ToDo list can be found at [GitHub projects](https://github.com/users/vladma
### New models
- [Ovi](https://github.com/character-ai/Ovi)
- [Kandinsky 5](https://github.com/ai-forever/Kandinsky-5)
- [Bytedance Lynx](https://github.com/bytedance/lynx)
- [HunyuanImage](https://huggingface.co/tencent/HunyuanImage-2.1)
- [Phantom HuMo](https://github.com/Phantom-video/Phantom)
- [Lumina-DiMOO](https://huggingface.co/Alpha-VLLM/Lumina-DiMOO)
- [Wan2.2 Fun](https://huggingface.co/collections/alibaba-pai/wan22-fun-68958eabec343b948f1225c5) (includes VACE, Control, etc)
- [Wan2.2-Animate-14B](https://huggingface.co/Wan-AI/Wan2.2-Animate-14B)
- [Magi](https://github.com/SandAI-org/MAGI-1)(https://github.com/huggingface/diffusers/pull/11713)
- [SEVA](https://github.com/huggingface/diffusers/pull/11440)
- [Ming](https://github.com/inclusionAI/Ming)
- [Liquid](https://github.com/FoundationVision/Liquid)
- [Step1X](https://github.com/stepfun-ai/Step1X-Edit)
- [LucyEdit](https://github.com/huggingface/diffusers/pull/12340)
- [SD3 UltraEdit](https://github.com/HaozheZhao/UltraEdit)
- [WAN2GP](https://github.com/deepbeepmeep/Wan2GP)
- [SelfForcing](https://github.com/guandeh17/Self-Forcing)
cli/localize.js
+4 -3
View File
@@ -7,9 +7,10 @@ const { GoogleGenerativeAI } = require('@google/generative-ai');
const api_key = process.env.GOOGLE_AI_API_KEY;
const model = 'gemini-2.5-flash';
const prompt = `
// eslint-disable-next-line max-len
Translate attached JSON from English to {language} using following rules: fields id, label and reload should be preserved from original, field localized should be a translated version of field label and field hint should be translated in-place. if field is less than 3 characters, do not translate it and keep it as is. Every JSON entry should have id, label, localized, reload and hint fields. Output should be pure JSON without any additional text. To better match translation, context of the text is related to Stable Diffusion and topic of Generative AI.`;
const prompt = `Translate attached JSON from English to {language} using following rules: fields id, label and reload should be preserved from original, field localized should be a translated version of field label and field hint should be translated in-place.
if field is less than 3 characters, do not translate it and keep it as is.
Every JSON entry should have id, label, localized, reload and hint fields.
Output should be pure JSON without any additional text. To better match translation, context of the text is related to Stable Diffusion and topic of Generative AI.`;
const languages = {
hr: 'Croatian',
de: 'German',
configs/Dockerfile.cuda
+2 -2
View File
@@ -41,11 +41,11 @@ ENV SD_DOCKER=true
# tcmalloc is not required but it is highly recommended
ENV LD_PRELOAD=libtcmalloc.so.4
# sdnext will run all necessary pip install ops and then exit
RUN ["python", "/app/launch.py", "--debug", "--uv", "--use-cuda", "--log", "sdnext.log", "--test", "--optional"]
RUN ["python", "/app/launch.py", "--uv", "--use-cuda", "--log", "sdnext.log", "--test", "--optional"]
# preinstall additional packages to avoid installation during runtime
# actually run sdnext
CMD ["python", "launch.py", "--debug", "--skip-all", "--listen", "--quick", "--api-log", "--log", "sdnext.log"]
CMD ["python", "launch.py", "--listen", "--quick", "--log", "sdnext.log"]
# expose port
EXPOSE 7860
extensions-builtin/sd-extension-chainner
+1 -1
Submodule extensions-builtin/sd-extension-chainner updated: 3e0108fedb...c6dc85eb28
extensions-builtin/sd-extension-system-info
+1 -1
Submodule extensions-builtin/sd-extension-system-info updated: 615d2f8103...19a1b1b722
extensions-builtin/sd-webui-agent-scheduler
-1
Submodule extensions-builtin/sd-webui-agent-scheduler deleted from a33753321b
extensions-builtin/sdnext-modernui
+1 -1
Submodule extensions-builtin/sdnext-modernui updated: 430c8140c0...db79b18f6f
html/locale_en.json
+1 -2
View File
@@ -32,7 +32,6 @@
{"id":"","label":"","localized":"","reload":"","hint":"Sort by time, descending"}
],
"main": [
{"id":"","label":"SD.Next","localized":"","reload":"","hint":"SD.Next<br>All-in-one WebUI for AI generative image and video creation"},
{"id":"","label":"Prompt","localized":"","reload":"","hint":"Describe image you want to generate"},
{"id":"","label":"Start","localized":"","reload":"","hint":"Start"},
{"id":"","label":"End","localized":"","reload":"","hint":"End"},
@@ -127,7 +126,7 @@
{"id":"","label":"➠ Sketch","localized":"","reload":"","hint":"Transfer image to sketch interface"},
{"id":"","label":"➠ Composite","localized":"","reload":"","hint":"Transfer image to inpaint sketch interface"},
{"id":"","label":"➠ Process","localized":"","reload":"","hint":"Transfer image to process interface"},
{"id":"","label":"➠ Control","localized":"","reload":"","hint":"Transfrer image to control interface"},
{"id":"","label":"➠ Control","localized":"","reload":"","hint":"Transfer image to control interface"},
{"id":"","label":"➠ Caption","localized":"","reload":"","hint":"Transfer image to caption interface"}
],
"generate": [
html/previews.json
+6 -5
View File
@@ -1,6 +1,6 @@
{
"sd-v21-512-ema": "models/Reference/stabilityai--stable-diffusion-2-1-base.jpg",
"stabilityai--stable-diffusion-xl-base-1.0": "models/Reference/sd_xl_base_1.0.jpg",
"stabilityai--stable-diffusion-xl-base-1.0": "models/Reference/stabilityai--stable-diffusion-xl-base-1.0.jpg",
"stabilityai--stable-diffusion-3-medium-diffusers": "models/Reference/stabilityai--stable-diffusion-3.jpg",
"stabilityai--stable-diffusion-3.5-medium": "models/Reference/stabilityai--stable-diffusion-3_5-medium.jpg",
"stabilityai--stable-diffusion-3.5-large": "models/Reference/stabilityai--stable-diffusion-3_5-large.jpg",
@@ -20,10 +20,11 @@
"HiDream-I1-Full": "models/Reference/HiDream-I1 Full",
"lodestones--Chroma1-Base": "models/Reference/lodestones--Chroma-Base.jpg",
"lodestones--Chroma1-HD": "models/Reference/lodestones--Chroma-HD.jpg",
"chroma-unlocked-v50": "models/Reference/lodestones Chroma Unlocked HD",
"chroma-unlocked-v50-annealed": "models/Reference/lodestones Chroma Unlocked HD",
"chroma-unlocked-v50": "models/Reference/lodestones--Chroma-detail.jpg",
"chroma-unlocked-v50-annealed": "models/Reference/lodestones--Chroma-annealed.jpg",
"vladmandic--Qwen-Lightning": "models/Reference/Qwen-Lightning.jpg",
"vladmandic--Qwen-Lightning-Edit": "models/Reference/Qwen-Lightning.jpg",
"Wan-AI--Wan2.2-T2V-A14B-Diffusers": "models/Reference/Wan2.2-T2V-A14B.jpg",
"Wan-AI--Wan2.1-T2V-14B-Diffusers": "models/Reference/Wan-AI--Wan2.1.jpg"
"Wan-AI--Wan2.2-T2V-A14B-Diffusers": "models/Reference/Wan-AI--Wan2.2-T2V-A14B-Diffusers.jpg",
"Wan-AI--Wan2.1-T2V-14B-Diffusers": "models/Reference/Wan-AI--Wan2.1-T2V-14B-Diffusers.jpg",
"linoyts--Wan2.2-VACE-Fun-14B-diffusers": "models/Reference/linoyts--Wan2.2-VACE-Fun-14B-diffusers.jpg"
}
html/reference.json
+320 -108
View File
@@ -1,26 +1,26 @@
{
"Tempest-by-Vlad XL": {
"path": "tempestByVlad_baseV01.safetensors@https://civitai.com/api/download/models/1301775",
"preview": "tempest-by-vlad-base.jpg",
"preview": "tempestByVlad_baseV01.jpg",
"desc": "Flexible SDXL model with custom encoder and finetuned for larger landscape resolutions with high details and high contrast.",
"extras": ""
},
"Tempest-by-Vlad XL Hyper": {
"path": "tempestByVlad_hyperV01.safetensors@https://civitai.com/api/download/models/1343512",
"preview": "tempest-by-vlad-hyper.jpg",
"preview": "tempestByVlad_hyperV01.jpg",
"desc": "Custom distilled variant with goal to get as-normal-as-possible model that works with low steps and guidance-free",
"extras": ""
},
"Juggernaut XL XI": {
"path": "juggernautXL_juggXIByRundiffusion.safetensors@https://civitai.com/api/download/models/782002",
"preview": "juggernautXL_v9Rundiffusionphoto2.jpg",
"preview": "juggernautXL_juggXIByRundiffusion.jpg",
"desc": "Showcase finetuned model based on Stable diffusion XL",
"extras": "sampler: DEIS, steps: 20, cfg_scale: 6.0"
},
"Juggernaut XL XI Lightning": {
"path": "juggernautXL_juggXILightningByRD.safetensors@https://civitai.com/api/download/models/920957",
"preview": "juggernautXL_v9Rdphoto2Lightning.jpg",
"preview": "juggernautXL_juggXILightningByRD.jpg",
"desc": "Showcase finetuned model based on Stable diffusion XL",
"extras": "sampler: DPM SDE, steps: 6, cfg_scale: 2.0"
},
@@ -37,7 +37,9 @@
"path": "v1-5-pruned-fp16-emaonly.safetensors@https://huggingface.co/Aptronym/SDNext/resolve/main/Reference/v1-5-pruned-fp16-emaonly.safetensors?download=true",
"preview": "v1-5-pruned-fp16-emaonly.jpg",
"desc": "Stable Diffusion 1.5 is the base model all other 1.5 checkpoint were trained from. It's a latent text-to-image diffusion model capable of generating photo-realistic images given any text input. The Stable-Diffusion-v1-5 checkpoint was initialized with the weights of the Stable-Diffusion-v1-2 checkpoint and subsequently fine-tuned on 595k steps at resolution 512x512.",
"extras": "width: 512, height: 512, sampler: DEIS, steps: 20, cfg_scale: 6.0"
"extras": "width: 512, height: 512, sampler: DEIS, steps: 20, cfg_scale: 6.0",
"size": 2.28,
"date": "2022 October"
},
"StabilityAI StableDiffusion 2.1": {
"path": "huggingface/stabilityai/stable-diffusion-2-1-base",
@@ -45,7 +47,9 @@
"skip": true,
"variant": "fp16",
"desc": "This stable-diffusion-2-1-base model fine-tunes stable-diffusion-2-base (512-base-ema.ckpt) with 220k extra steps taken",
"extras": "width: 512, height: 512, sampler: DEIS, steps: 20, cfg_scale: 6.0"
"extras": "width: 512, height: 512, sampler: DEIS, steps: 20, cfg_scale: 6.0",
"size": 2.58,
"date": "2022 December"
},
"StabilityAI StableDiffusion 2.1 V": {
"path": "huggingface/stabilityai/stable-diffusion-2-1",
@@ -55,11 +59,23 @@
"desc": "This stable-diffusion-2 model is resumed from stable-diffusion-2-base (512-base-ema.ckpt) and trained for 150k steps using a v-objective on the same dataset. Resumed for another 140k steps on 768x768 images",
"extras": "width: 768, height: 768, sampler: DEIS, steps: 20, cfg_scale: 6.0"
},
"StabilityAI StableDiffusion XL 1.0 Base": {
"path": "sd_xl_base_1.0.safetensors@https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors?download=true",
"preview": "sd_xl_base_1.0.jpg",
"desc": "Stable Diffusion XL (SDXL) is the latest AI image generation model that is tailored towards more photorealistic outputs with more detailed imagery and composition compared to previous SD models, including SD 2.1. It can make realistic faces, legible text within the images, and better image composition, all while using shorter and simpler prompts at a greatly increased base resolution of 1024x1024. Just like its predecessors, SDXL has the ability to generate image variations using image-to-image prompting, inpainting (reimagining of the selected parts of an image), and outpainting (creating new parts that lie outside the image borders).",
"extras": "sampler: DEIS, steps: 20, cfg_scale: 6.0"
"StabilityAI StableDiffusion XL": {
"path": "stabilityai/stable-diffusion-xl-base-1.0",
"preview": "stabilityai--stable-diffusion-xl-base-1.0.jpg",
"desc": "Stable Diffusion XL (SDXL) is AI image generation model that is tailored towards more photorealistic outputs with more detailed imagery and composition compared to previous SD models, including SD 2.1. It can make realistic faces and better image composition, all while using shorter and simpler prompts at a greatly increased base resolution of 1024x1024. Just like its predecessors, SDXL has the ability to generate image variations using image-to-image prompting, inpainting (reimagining of the selected parts of an image), and outpainting (creating new parts that lie outside the image borders).",
"skip": true,
"variant": "fp16",
"extras": "",
"size": 6.94,
"date": "2023 July"
},
"StabilityAI StableDiffusion XL Turbo": {
"path": "stabilityai/sdxl-turbo",
"preview": "stabilityai--sdxl-turbo.jpg",
"desc": "SDXL-Turbo is a fast generative text-to-image model that can synthesize photorealistic images from a text prompt in a 1-4 steps.",
"skip": true,
"variant": "fp16",
"extras": "steps: 4, cfg_scale: 0.0"
},
"StabilityAI Stable Cascade": {
"path": "huggingface/stabilityai/stable-cascade",
@@ -67,7 +83,9 @@
"variant": "bf16",
"desc": "Stable Cascade is a diffusion model built upon the Würstchen architecture and its main difference to other models like Stable Diffusion is that it is working at a much smaller latent space. Why is this important? The smaller the latent space, the faster you can run inference and the cheaper the training becomes. How small is the latent space? Stable Diffusion uses a compression factor of 8, resulting in a 1024x1024 image being encoded to 128x128. Stable Cascade achieves a compression factor of 42, meaning that it is possible to encode a 1024x1024 image to 24x24, while maintaining crisp reconstructions. The text-conditional model is then trained in the highly compressed latent space. Previous versions of this architecture, achieved a 16x cost reduction over Stable Diffusion 1.5",
"preview": "stabilityai--stable-cascade.jpg",
"extras": "sampler: Default, cfg_scale: 4.0, image_cfg_scale: 1.0"
"extras": "sampler: Default, cfg_scale: 4.0, image_cfg_scale: 1.0",
"size": 11.82,
"date": "2024 February"
},
"StabilityAI Stable Cascade Lite": {
"path": "huggingface/stabilityai/stable-cascade-lite",
@@ -75,7 +93,9 @@
"variant": "bf16",
"desc": "Stable Cascade is a diffusion model built upon the Würstchen architecture and its main difference to other models like Stable Diffusion is that it is working at a much smaller latent space. Why is this important? The smaller the latent space, the faster you can run inference and the cheaper the training becomes. How small is the latent space? Stable Diffusion uses a compression factor of 8, resulting in a 1024x1024 image being encoded to 128x128. Stable Cascade achieves a compression factor of 42, meaning that it is possible to encode a 1024x1024 image to 24x24, while maintaining crisp reconstructions. The text-conditional model is then trained in the highly compressed latent space. Previous versions of this architecture, achieved a 16x cost reduction over Stable Diffusion 1.5",
"preview": "stabilityai--stable-cascade-lite.jpg",
"extras": "sampler: Default, cfg_scale: 4.0, image_cfg_scale: 1.0"
"extras": "sampler: Default, cfg_scale: 4.0, image_cfg_scale: 1.0",
"size": 4.97,
"date": "2024 February"
},
"StabilityAI Stable Diffusion 3 Medium": {
"path": "stabilityai/stable-diffusion-3-medium-diffusers",
@@ -83,7 +103,9 @@
"variant": "fp16",
"desc": "Stable Diffusion 3 Medium is a Multimodal Diffusion Transformer (MMDiT) text-to-image model that features greatly improved performance in image quality, typography, complex prompt understanding, and resource-efficiency",
"preview": "stabilityai--stable-diffusion-3.jpg",
"extras": "sampler: Default, cfg_scale: 7.0"
"extras": "sampler: Default, cfg_scale: 7.0",
"size": 15.14,
"date": "2024 June"
},
"StabilityAI Stable Diffusion 3.5 Medium": {
"path": "stabilityai/stable-diffusion-3.5-medium",
@@ -91,7 +113,9 @@
"variant": "fp16",
"desc": "Stable Diffusion 3.5 Medium is a Multimodal Diffusion Transformer with improvements (MMDiT-X) text-to-image model that features improved performance in image quality, typography, complex prompt understanding, and resource-efficiency.",
"preview": "stabilityai--stable-diffusion-3_5-medium.jpg",
"extras": "sampler: Default, cfg_scale: 7.0"
"extras": "sampler: Default, cfg_scale: 7.0",
"size": 15.89,
"date": "2024 October"
},
"StabilityAI Stable Diffusion 3.5 Large": {
"path": "stabilityai/stable-diffusion-3.5-large",
@@ -99,7 +123,9 @@
"variant": "fp16",
"desc": "Stable Diffusion 3.5 Large is a Multimodal Diffusion Transformer (MMDiT) text-to-image model that features improved performance in image quality, typography, complex prompt understanding, and resource-efficiency.",
"preview": "stabilityai--stable-diffusion-3_5-large.jpg",
"extras": "sampler: Default, cfg_scale: 7.0"
"extras": "sampler: Default, cfg_scale: 7.0",
"size": 26.98,
"date": "2024 October"
},
"StabilityAI Stable Diffusion 3.5 Turbo": {
"path": "stabilityai/stable-diffusion-3.5-large-turbo",
@@ -115,107 +141,177 @@
"preview": "black-forest-labs--FLUX.1-dev.jpg",
"desc": "FLUX.1 models are based on a hybrid architecture of multimodal and parallel diffusion transformer blocks, scaled to 12B parameters and builing on flow matching",
"skip": true,
"extras": "sampler: Default, cfg_scale: 3.5"
"extras": "sampler: Default, cfg_scale: 3.5",
"size": 32.93,
"date": "2024 August"
},
"Black Forest Labs FLUX.1 Schnell": {
"path": "black-forest-labs/FLUX.1-schnell",
"preview": "black-forest-labs--FLUX.1-schnell.jpg",
"desc": "FLUX.1 models are based on a hybrid architecture of multimodal and parallel diffusion transformer blocks, scaled to 12B parameters and builing on flow matching. Trained using latent adversarial diffusion distillation, FLUX.1 [schnell] can generate high-quality images in only 1 to 4 steps",
"skip": true,
"extras": "sampler: Default, cfg_scale: 3.5"
"extras": "sampler: Default, cfg_scale: 3.5",
"size": 32.93,
"date": "2024 August"
},
"Black Forest Labs FLUX.1 Kontext Dev": {
"path": "black-forest-labs/FLUX.1-Kontext-dev",
"preview": "black-forest-labs--FLUX.1-Kontext-dev.jpg",
"desc": "FLUX.1 Kontext [dev] is a 12 billion parameter rectified flow transformer capable of editing images based on text instructions.",
"skip": true,
"extras": "sampler: Default, cfg_scale: 3.5"
"extras": "sampler: Default, cfg_scale: 3.5",
"size": 32.93,
"date": "2025 June"
},
"Black Forest Labs FLUX.1 Krea Dev": {
"path": "black-forest-labs/FLUX.1-Krea-dev",
"preview": "black-forest-labs--FLUX.1-Krea-dev.jpg",
"desc": "FLUX.1 Krea [dev] is a 12 billion parameter rectified flow transformer capable of generating images from text descriptions.",
"skip": true,
"extras": "sampler: Default, cfg_scale: 4.5",
"size": 32.93,
"date": "2025 July"
},
"Tencent FLUX.1 Dev SRPO": {
"path": "vladmandic/flux.1-dev-SRPO",
"preview": "vladmandic--flux.1-dev-SRPO.jpg",
"desc": "FLUX.1 Dev SRPO is Tencent trained with specific technique: Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference",
"skip": true,
"extras": "sampler: Default, cfg_scale: 4.5"
},
"Qwen-Image": {
"path": "Qwen/Qwen-Image",
"preview": "Qwen--Qwen-Image.jpg",
"desc": "Qwen-Image, an image generation foundation model in the Qwen series that achieves significant advances in complex text rendering and precise image editing.",
"skip": true,
"extras": "",
"size": 56.1,
"date": "2025 August"
},
"Qwen-Image-Edit": {
"path": "Qwen/Qwen-Image-Edit",
"preview": "Qwen--Qwen-Image-Edit.jpg",
"desc": "Qwen-Image-Edit, the image editing version of Qwen-Image. Built upon our 20B Qwen-Image model, Qwen-Image-Edit successfully extends Qwen-Image’s unique text rendering capabilities to image editing tasks, enabling precise text editing.",
"skip": true,
"extras": "",
"size": 56.1,
"date": "2025 August"
},
"Qwen-Image-Edit-2509": {
"path": "Qwen/Qwen-Image-Edit-2509",
"preview": "Qwen--Qwen-Image-Edit-2509.jpg",
"desc": "Qwen-Image-Edit, the image editing version of Qwen-Image. Built upon our 20B Qwen-Image model, Qwen-Image-Edit successfully extends Qwen-Image’s unique text rendering capabilities to image editing tasks, enabling precise text editing.",
"skip": true,
"extras": "",
"size": 56.1,
"date": "2025 September"
},
"Qwen-Image-Lightning": {
"path": "vladmandic/Qwen-Lightning",
"preview": "vladmandic--Qwen-Lightning.jpg",
"desc": "Qwen-Lightning is step-distilled from Qwen-Image to allow for generation in 8 steps.",
"skip": true,
"extras": "steps: 8",
"size": 56.1,
"date": "2025 August"
},
"Qwen-Image-Distill": {
"path": "SahilCarterr/Qwen-Image-Distill-Full",
"preview": "SahilCarterr--Qwen-Image-Distill-Full.jpg",
"desc": "Qwen-Image-Distill is a distilled and accelerated version of Qwen-Image by DiffSynth-Studio.",
"skip": true,
"extras": "steps: 15",
"size": 56.1,
"date": "2025 August"
},
"Qwen-Image-Lightning-Edit": {
"path": "vladmandic/Qwen-Lightning-Edit",
"preview": "vladmandic--Qwen-Lightning-Edit.jpg",
"desc": "Qwen-Lightning-Edit is step-distilled from Qwen-Image-Edit to allow for generation in 8 steps.",
"skip": true,
"extras": "steps: 8",
"size": 56.1,
"date": "2025 August"
},
"Qwen-Image Pruning-12B": {
"path": "OPPOer/Qwen-Image-Pruning",
"subfolder": "Qwen-Image-12B-8steps",
"preview": "OPPOer--Qwen-Image-Pruning.jpg",
"desc": "This open-source project is based on Qwen-Image and has attempted model pruning, removing 20 layers while retaining the weights of 40 layers, resulting in a model size of 12B parameters.",
"skip": true,
"date": "2025 Ocotober"
},
"Qwen-Image-Edit Pruning-13B": {
"path": "OPPOer/Qwen-Image-Edit-Pruning",
"subfolder": "Qwen-Image-Edit-13B-4steps",
"preview": "OPPOer--Qwen-Image-Edit-Pruning.jpg",
"desc": "This open-source project is based on Qwen-Image-Edit and has attempted model pruning, removing 20 layers while retaining the weights of 40 layers, resulting in a model size of 13.6B parameters.",
"skip": true,
"date": "2025 Ocotober"
},
"Qwen-Image-Edit-2509 Pruning-13B": {
"path": "OPPOer/Qwen-Image-Edit-2509-Pruning",
"subfolder": "Qwen-Image-Edit-2509-13B-4steps",
"preview": "OPPOer--Qwen-Image-Edit-2509-Pruning.jpg",
"desc": "This open-source project is based on Qwen-Image-Edit and has attempted model pruning, removing 20 layers while retaining the weights of 40 layers, resulting in a model size of 13.6B parameters.",
"skip": true,
"date": "2025 Ocotober"
},
"lodestones Chroma1 HD": {
"path": "lodestones/Chroma1-HD",
"preview": "lodestones--Chroma-HD.jpg",
"desc": "Chroma is a 8.9B parameter model based on FLUX.1-schnell. It’s fully Apache 2.0 licensed, ensuring that anyone can use, modify, and build on top of it—no corporate gatekeeping. This is the high-res fine-tune of the Chroma1-Base at a 1024x1024 resolution.",
"skip": true,
"extras": ""
"extras": "",
"size": 26.84,
"date": "2025 July"
},
"lodestones Chroma1 Base": {
"path": "lodestones/Chroma1-Base",
"preview": "lodestones--Chroma-Base.jpg",
"desc": "Chroma is a 8.9B parameter model based on FLUX.1-schnell. It’s fully Apache 2.0 licensed, ensuring that anyone can use, modify, and build on top of it—no corporate gatekeeping. This is the core 512x512 model. It's a solid, all-around foundation for pretty much any creative project.",
"skip": true,
"extras": ""
"extras": "",
"size": 26.84,
"date": "2025 July"
},
"lodestones Chroma1 Flash": {
"path": "lodestones/Chroma1-Flash",
"preview": "lodestones--Chroma-flash.jpg",
"desc": "Chroma is a 8.9B parameter model based on FLUX.1-schnell. It’s fully Apache 2.0 licensed, ensuring that anyone can use, modify, and build on top of it—no corporate gatekeeping. A fine-tuned version of the Chroma1-Base made to find the best way to make these flow matching models faster.",
"skip": true,
"extras": ""
"extras": "",
"size": 26.84,
"date": "2025 July"
},
"lodestones Chroma1 v50 Preview Annealed": {
"path": "vladmandic/chroma-unlocked-v50-annealed",
"preview": "lodestones--Chroma-annealed.jpg",
"desc": "Chroma is a 8.9B parameter model based on FLUX.1-schnell. It’s fully Apache 2.0 licensed, ensuring that anyone can use, modify, and build on top of it—no corporate gatekeeping. Re-tweaked variant with extra noise added.",
"skip": true,
"extras": ""
"extras": "",
"size": 26.84,
"date": "2025 July"
},
"lodestones Chroma1 v48 Preview": {
"path": "vladmandic/chroma-unlocked-v48",
"preview": "lodestones--Chroma.jpg",
"desc": "Chroma is a 8.9B parameter model based on FLUX.1-schnell. It’s fully Apache 2.0 licensed, ensuring that anyone can use, modify, and build on top of it—no corporate gatekeeping. Last raw version of Chroma before final finetuning.",
"skip": true,
"extras": ""
"extras": "",
"size": 26.84,
"date": "2025 July"
},
"lodestones Chroma1 v48 Preview Calibrated": {
"path": "vladmandic/chroma-unlocked-v48-detail-calibrated",
"preview": "lodestones--Chroma-detail.jpg",
"desc": "Chroma is a 8.9B parameter model based on FLUX.1-schnell. It’s fully Apache 2.0 licensed, ensuring that anyone can use, modify, and build on top of it—no corporate gatekeeping. Last raw version of Chroma before final finetuning but with some detail calibration.",
"skip": true,
"extras": ""
},
"Qwen-Image": {
"path": "Qwen/Qwen-Image",
"preview": "Qwen--Qwen-Image.jpg",
"desc": " Qwen-Image, an image generation foundation model in the Qwen series that achieves significant advances in complex text rendering and precise image editing.",
"skip": true,
"extras": ""
},
"Qwen-Image-Edit": {
"path": "Qwen/Qwen-Image-Edit",
"preview": "Qwen--Qwen-Image-Edit.jpg",
"desc": "Qwen-Image-Edit, the image editing version of Qwen-Image. Built upon our 20B Qwen-Image model, Qwen-Image-Edit successfully extends Qwen-Image’s unique text rendering capabilities to image editing tasks, enabling precise text editing.",
"skip": true,
"extras": ""
},
"Qwen-Image-Lightning": {
"path": "vladmandic/Qwen-Lightning",
"preview": "vladmandic--Qwen-Lightning.jpg",
"desc": "Qwen-Lightning is step-distilled from Qwen-Image to allow for generation in 8 steps.",
"skip": true,
"extras": "steps: 8"
},
"Qwen-Image-Distill": {
"path": "SahilCarterr/Qwen-Image-Distill-Full",
"preview": "SahilCarterr--Qwen-Image-Distill-Full.jpg",
"desc": "Qwen-Image-Distill is a distilled and accelerated version of Qwen-Image by DiffSynth-Studio.",
"skip": true,
"extras": "steps: 15"
},
"Qwen-Image-Lightning-Edit": {
"path": "vladmandic/Qwen-Lightning-Edit",
"preview": "vladmandic--Qwen-Lightning-Edit.jpg",
"desc": " Qwen-Lightning-Edit is step-distilled from Qwen-Image-Edit to allow for generation in 8 steps.",
"skip": true,
"extras": "steps: 8"
"extras": "",
"size": 26.84,
"date": "2025 July"
},
"Ostris Flex.2 Preview": {
@@ -223,14 +319,18 @@
"preview": "ostris--Flex.2-preview.jpg",
"desc": "Open Source 8B parameter Text to Image Diffusion Model with universal control and inpainting support built in. Early access preview release. The next version of Flex.1-alpha",
"skip": true,
"extras": "sampler: Default, cfg_scale: 3.5"
"extras": "sampler: Default, cfg_scale: 3.5",
"size": 25.65,
"date": "2025 April"
},
"Ostris Flex.1 Alpha": {
"path": "ostris/Flex.1-alpha",
"preview": "ostris--Flex.1-alpha.jpg",
"desc": "Flex.1 alpha is a pre-trained base 8 billion parameter rectified flow transformer capable of generating images from text descriptions. It has a similar architecture to FLUX.1-dev, but with fewer double transformer blocks (8 vs 19)",
"skip": true,
"extras": "sampler: Default, cfg_scale: 3.5"
"extras": "sampler: Default, cfg_scale: 3.5",
"size": 25.65,
"date": "2025 January"
},
"Wan-AI Wan2.1 1.3B": {
@@ -238,14 +338,18 @@
"preview": "Wan-AI--Wan2.1-T2V-1.3B-Diffusers.jpg",
"desc": "Wan is an advanced and powerful visual generation model developed by Tongyi Lab of Alibaba Group. It can generate videos based on text, images, and other control signals. The Wan2.1 series models are now fully open-source.",
"skip": true,
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 27.72,
"date": "2025 February"
},
"Wan-AI Wan2.1 14B": {
"path": "Wan-AI/Wan2.1-T2V-14B-Diffusers",
"preview": "Wan-AI--Wan2.1-T2V-14B-Diffusers.jpg",
"desc": "Wan is an advanced and powerful visual generation model developed by Tongyi Lab of Alibaba Group. It can generate videos based on text, images, and other control signals. The Wan2.1 series models are now fully open-source.",
"skip": true,
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 78.52,
"date": "2025 February"
},
"Wan-AI Wan2.2 5B": {
"path": "Wan-AI/Wan2.2-TI2V-5B-Diffusers",
@@ -268,27 +372,40 @@
"skip": true,
"extras": "sampler: Default"
},
"Wan-AI Wan2.2 14B VACE": {
"path": "linoyts/Wan2.2-VACE-Fun-14B-diffusers",
"preview": "linoyts--Wan2.2-VACE-Fun-14B-diffusers.jpg",
"desc": "Wan2.2, offering more powerful capabilities, better performance, and superior visual quality. With Wan2.2, we have focused on incorporating the following technical innovations: MoE Architecture, Data Scalling, Cinematic Aesthetics, Efficient High-Definition Hybrid",
"skip": true,
"extras": "sampler: Default"
},
"Freepik F-Lite": {
"path": "Freepik/F-Lite",
"preview": "Freepik--F-Lite.jpg",
"desc": "F Lite is a 10B parameter diffusion model created by Freepik and Fal, trained exclusively on copyright-safe and SFW content. The model was trained on Freepik's internal dataset comprising approximately 80 million copyright-safe images, making it the first publicly available model of this scale trained exclusively on legally compliant and SFW content.",
"skip": true,
"extras": "sampler: Default, cfg_scale: 3.5"
"extras": "sampler: Default, cfg_scale: 3.5",
"size": 19.81,
"date": "2025 May"
},
"Freepik F-Lite Texture": {
"path": "Freepik/F-Lite-Texture",
"preview": "Freepik--F-Lite-Texture.jpg",
"desc": "F Lite is a 10B parameter diffusion model created by Freepik and Fal, trained exclusively on copyright-safe and SFW content. The model was trained on Freepik's internal dataset comprising approximately 80 million copyright-safe images, making it the first publicly available model of this scale trained exclusively on legally compliant and SFW content.",
"skip": true,
"extras": "sampler: Default, cfg_scale: 3.5"
"extras": "sampler: Default, cfg_scale: 3.5",
"size": 19.81,
"date": "2025 May"
},
"Freepik F-Lite 7B": {
"path": "Freepik/F-Lite-7B",
"preview": "Freepik--F-Lite-7B.jpg",
"desc": "F Lite is a 10B parameter diffusion model created by Freepik and Fal, trained exclusively on copyright-safe and SFW content. The model was trained on Freepik's internal dataset comprising approximately 80 million copyright-safe images, making it the first publicly available model of this scale trained exclusively on legally compliant and SFW content.",
"skip": true,
"extras": "sampler: Default, cfg_scale: 3.5"
"extras": "sampler: Default, cfg_scale: 3.5",
"size": 13.89,
"date": "2025 May"
},
"SDXS DreamShaper 512": {
@@ -309,13 +426,17 @@
"path": "Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers",
"desc": "Sana is an efficient model with scaling of training-time and inference time techniques. SANA-1.5 delivers: efficient model growth from 1.6B Sana-1.0 model to 4.8B, achieving similar or better performance than training from scratch and saving 60% training cost; efficient model depth pruning, slimming any model size as you want; powerful VLM selection based inference scaling, smaller model+inference scaling > larger model.",
"preview": "Efficient-Large-Model--SANA1.5_1.6B_1024px_diffusers.jpg",
"skip": true
"skip": true,
"size": 9.49,
"date": "2025 March"
},
"NVLabs Sana 1.5 4.8B 1k": {
"path": "Efficient-Large-Model/SANA1.5_4.8B_1024px_diffusers",
"desc": "Sana is an efficient model with scaling of training-time and inference time techniques. SANA-1.5 delivers: efficient model growth from 1.6B Sana-1.0 model to 4.8B, achieving similar or better performance than training from scratch and saving 60% training cost; efficient model depth pruning, slimming any model size as you want; powerful VLM selection based inference scaling, smaller model+inference scaling > larger model.",
"preview": "Efficient-Large-Model--SANA1.5_4.8B_1024px_diffusers.jpg",
"skip": true
"skip": true,
"size": 15.58,
"date": "2025 March"
},
"NVLabs Sana 1.5 1.6B 1k Sprint": {
"path": "Efficient-Large-Model/Sana_Sprint_1.6B_1024px_diffusers",
@@ -327,63 +448,93 @@
"path": "Efficient-Large-Model/Sana_1600M_4Kpx_BF16_diffusers",
"desc": "Sana is a text-to-image framework that can efficiently generate images up to 4096 × 4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU.",
"preview": "Efficient-Large-Model--Sana_1600M_4Kpx_BF16_diffusers.jpg",
"skip": true
"skip": true,
"size": 12.63,
"date": "2024 November"
},
"NVLabs Sana 1.0 1.6B 2k": {
"path": "Efficient-Large-Model/Sana_1600M_2Kpx_BF16_diffusers",
"desc": "Sana is a text-to-image framework that can efficiently generate images up to 4096 × 4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU.",
"preview": "Efficient-Large-Model--Sana_1600M_2Kpx_BF16_diffusers.jpg",
"skip": true
"skip": true,
"size": 12.63,
"date": "2024 November"
},
"NVLabs Sana 1.0 1.6B 1k": {
"path": "Efficient-Large-Model/Sana_1600M_1024px_diffusers",
"desc": "Sana is a text-to-image framework that can efficiently generate images up to 4096 × 4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU.",
"preview": "Efficient-Large-Model--Sana_1600M_1024px_diffusers.jpg",
"skip": true
"skip": true,
"size": 12.63,
"date": "2024 November"
},
"NVLabs Sana 1.0 0.6B 0.5k": {
"path": "Efficient-Large-Model/Sana_600M_512px_diffusers",
"desc": "Sana is a text-to-image framework that can efficiently generate images up to 4096 × 4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU.",
"preview": "Efficient-Large-Model--Sana_600M_512px_diffusers.jpg",
"skip": true
"skip": true,
"size": 7.51,
"date": "2024 November"
},
"nVidia Cosmos-Predict2 T2I 2B": {
"path": "nvidia/Cosmos-Predict2-2B-Text2Image",
"desc": "Cosmos-Predict2: A family of highly performant pre-trained world foundation models purpose-built for generating physics-aware images, videos and world states for physical AI development.",
"preview": "nvidia--Cosmos-Predict2-2B-Text2Image.jpg",
"skip": true
"skip": true,
"size": 13.32,
"date": "2025 June"
},
"nVidia Cosmos-Predict2 T2I 14B": {
"path": "nvidia/Cosmos-Predict2-14B-Text2Image",
"desc": "Cosmos-Predict2: A family of highly performant pre-trained world foundation models purpose-built for generating physics-aware images, videos and world states for physical AI development.",
"preview": "nvidia--Cosmos-Predict2-14B-Text2Image.jpg",
"skip": true
"skip": true,
"size": 37.36,
"date": "2025 June"
},
"X-Omni SFT": {
"path": "X-Omni/X-Omni-SFT",
"desc": "X-Omni: Reinforcement learning makes discrete autoregressive image generative models great again",
"preview": "X-Omni--X-Omni-SFT.jpg",
"skip": true,
"size": 0,
"date": "2024 September",
"experimental": true
},
"VectorSpaceLab OmniGen v1": {
"path": "Shitao/OmniGen-v1-diffusers",
"desc": "OmniGen is a unified image generation model that can generate a wide range of images from multi-modal prompts. It is designed to be simple, flexible and easy to use.",
"preview": "Shitao--OmniGen-v1.jpg",
"skip": true
"skip": true,
"size": 15.47,
"date": "2024 October"
},
"VectorSpaceLab OmniGen v2": {
"path": "OmniGen2/OmniGen2",
"desc": "OmniGen2 is a powerful and efficient unified multimodal model. Unlike OmniGen v1, OmniGen2 features two distinct decoding pathways for text and image modalities, utilizing unshared parameters and a decoupled image tokenizer.",
"preview": "OmniGen2--OmniGen2.jpg",
"skip": true
"skip": true,
"size": 30.5,
"date": "2025 June"
},
"AuraFlow 0.3": {
"path": "fal/AuraFlow-v0.3",
"desc": "AuraFlow v0.3 is the fully open-sourced flow-based text-to-image generation model. The model was trained with more compute compared to the previous version, AuraFlow-v0.2. Compared to AuraFlow-v0.2, the model is fine-tuned on more aesthetic datasets and now supports various aspect ratio, (now width and height up to 1536 pixels).",
"preview": "fal--AuraFlow-v0.3.jpg",
"skip": true
"skip": true,
"size": 31.9,
"date": "2024 August"
},
"AuraFlow 0.2": {
"path": "fal/AuraFlow-v0.2",
"desc": "AuraFlow v0.2 is the fully open-sourced largest flow-based text-to-image generation model. The model was trained with more compute compared to the previous version, AuraFlow-v0.1",
"preview": "fal--AuraFlow-v0.2.jpg",
"skip": true
"skip": true,
"size": 31.9,
"date": "2024 July"
},
"Segmind Vega": {
@@ -392,7 +543,9 @@
"desc": "The Segmind-Vega Model is a distilled version of the Stable Diffusion XL (SDXL), offering a remarkable 70% reduction in size and an impressive 100% speedup while retaining high-quality text-to-image generation capabilities. Trained on diverse datasets, including Grit and Midjourney scrape data, it excels at creating a wide range of visual content based on textual prompts. Employing a knowledge distillation strategy, Segmind-Vega leverages the teachings of several expert models, including SDXL, ZavyChromaXL, and JuggernautXL, to combine their strengths and produce compelling visual outputs.",
"variant": "fp16",
"skip": true,
"extras": "sampler: Default, cfg_scale: 9.0"
"extras": "sampler: Default, cfg_scale: 9.0",
"size": 6.43,
"date": "2023 November"
},
"Segmind SSD-1B": {
"path": "huggingface/segmind/SSD-1B",
@@ -400,13 +553,17 @@
"desc": "The Segmind Stable Diffusion Model (SSD-1B) offers a compact, efficient, and distilled version of the SDXL model. At 50% smaller and 60% faster than Stable Diffusion XL (SDXL), it provides quick and seamless performance without sacrificing image quality.",
"variant": "fp16",
"skip": true,
"extras": "sampler: Default, cfg_scale: 9.0"
"extras": "sampler: Default, cfg_scale: 9.0",
"size": 8.72,
"date": "2023 October"
},
"Segmind Tiny": {
"path": "segmind/tiny-sd",
"preview": "segmind--tiny-sd.jpg",
"desc": "Segmind's Tiny-SD offers a compact, efficient, and distilled version of Realistic Vision 4.0 and is up to 80% faster than SD1.5",
"extras": "width: 512, height: 512, sampler: Default, cfg_scale: 9.0"
"extras": "width: 512, height: 512, sampler: Default, cfg_scale: 9.0",
"size": 1.03,
"date": "2023 July"
},
"Segmind SegMoE SD 4x2": {
"path": "segmind/SegMoE-SD-4x2-v0",
@@ -430,7 +587,9 @@
"path": "PixArt-alpha/PixArt-XL-2-1024-MS",
"desc": "PixArt-α is a Transformer-based T2I diffusion model whose image generation quality is competitive with state-of-the-art image generators (e.g., Imagen, SDXL, and even Midjourney), and the training speed markedly surpasses existing large-scale T2I models. Extensive experiments demonstrate that PIXART-α excels in image quality, artistry, and semantic control. It can directly generate 1024px images from text prompts within a single sampling process.",
"preview": "PixArt-alpha--PixArt-XL-2-1024-MS.jpg",
"extras": "sampler: Default, cfg_scale: 2.0"
"extras": "sampler: Default, cfg_scale: 2.0",
"size": 21.3,
"date": "2023 November"
},
"Pixart-Σ Small": {
"path": "huggingface/PixArt-alpha/PixArt-Sigma-XL-2-512-MS",
@@ -451,14 +610,18 @@
"desc": "PixArt-Σ, a Diffusion Transformer model (DiT) capable of directly generating images at 4K resolution. PixArt-Σ represents a significant advancement over its predecessor, PixArt-α, offering images of markedly higher fidelity and improved alignment with text prompts.",
"preview": "PixArt-alpha--PixArt-Sigma-XL-2-2K-MS.jpg",
"skip": true,
"extras": "sampler: Default, cfg_scale: 2.0"
"extras": "sampler: Default, cfg_scale: 2.0",
"size": 21.3,
"date": "2024 April"
},
"Tencent HunyuanDiT 1.2": {
"path": "Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers",
"desc": "Hunyuan-DiT : A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding.",
"preview": "Tencent-Hunyuan--HunyuanDiT-v1.2-Diffusers.jpg",
"extras": "sampler: Default, cfg_scale: 2.0"
"extras": "sampler: Default, cfg_scale: 2.0",
"size": 14.09,
"date": "2024 May"
},
"Tencent HunyuanDiT 1.2 Distilled": {
"path": "Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers-Distilled",
@@ -484,14 +647,18 @@
"desc": "The Lumina-Next-SFT is a Next-DiT model containing 2B parameters and utilizes Gemma-2B as the text encoder, enhanced through high-quality supervised fine-tuning (SFT).",
"preview": "Alpha-VLLM--Lumina-Next-SFT-diffusers.jpg",
"skip": true,
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 8.67,
"date": "2024 June"
},
"AlphaVLLM Lumina 2": {
"path": "Alpha-VLLM/Lumina-Image-2.0",
"desc": "A Unified and Efficient Image Generative Model. Lumina-Image-2.0 is a 2 billion parameter flow-based diffusion transformer capable of generating images from text descriptions.",
"preview": "Alpha-VLLM--Lumina-Image-2.0.jpg",
"skip": true,
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 20.75,
"date": "2025 January"
},
"HiDream-I1 Fast": {
@@ -499,21 +666,27 @@
"desc": "HiDream-I1 is a new open-source image generative foundation model with 17B parameters that achieves state-of-the-art image generation quality within seconds.",
"preview": "HiDream-ai--HiDream-I1-Fast.jpg",
"skip": true,
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 58.4,
"date": "2025 April"
},
"HiDream-I1 Dev": {
"path": "HiDream-ai/HiDream-I1-Dev",
"desc": "HiDream-I1 is a new open-source image generative foundation model with 17B parameters that achieves state-of-the-art image generation quality within seconds.",
"preview": "HiDream-ai--HiDream-I1-Dev.jpg",
"skip": true,
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 58.4,
"date": "2025 April"
},
"HiDream-I1 Full": {
"path": "HiDream-ai/HiDream-I1-Full",
"desc": "HiDream-I1 is a new open-source image generative foundation model with 17B parameters that achieves state-of-the-art image generation quality within seconds.",
"preview": "HiDream-ai--HiDream-I1-Full.jpg",
"skip": true,
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 58.4,
"date": "2025 April"
},
"HiDream-E1 Full": {
"path": "HiDream-ai/HiDream-E1-Full",
@@ -522,40 +695,57 @@
"skip": true,
"extras": "sampler: Default"
},
"HiDream-E1.1": {
"path": "HiDream-ai/HiDream-E1-1",
"desc": "HiDream-E1 is an image editing model built on HiDream-I1.",
"preview": "HiDream-ai--HiDream-E1-Full.jpg",
"skip": true,
"extras": "sampler: Default"
},
"Kwai Kolors": {
"path": "Kwai-Kolors/Kolors-diffusers",
"desc": "Kolors is a large-scale text-to-image generation model based on latent diffusion, developed by the Kuaishou Kolors team. Trained on billions of text-image pairs, Kolors exhibits significant advantages over both open-source and proprietary models in visual quality, complex semantic accuracy, and text rendering for both Chinese and English characters. Furthermore, Kolors supports both Chinese and English inputs",
"preview": "Kwai-Kolors--Kolors-diffusers.jpg",
"skip": true,
"extras": "width: 1024, height: 1024"
"extras": "width: 1024, height: 1024",
"size": 17.40,
"date": "2024 July"
},
"Kandinsky 2.1": {
"path": "kandinsky-community/kandinsky-2-1",
"desc": "Kandinsky 2.1 is a text-conditional diffusion model based on unCLIP and latent diffusion, composed of a transformer-based image prior model, a unet diffusion model, and a decoder. Kandinsky 2.1 inherits best practices from Dall-E 2 and Latent diffusion while introducing some new ideas. It uses the CLIP model as a text and image encoder, and diffusion image prior (mapping) between latent spaces of CLIP modalities. This approach increases the visual performance of the model and unveils new horizons in blending images and text-guided image manipulation.",
"preview": "kandinsky-community--kandinsky-2-1.jpg",
"extras": "width: 768, height: 768, sampler: Default"
"extras": "width: 768, height: 768, sampler: Default",
"size": 5.15,
"date": "2023 April"
},
"Kandinsky 2.2": {
"path": "kandinsky-community/kandinsky-2-2-decoder",
"desc": "Kandinsky 2.2 is a text-conditional diffusion model (+0.1!) based on unCLIP and latent diffusion, composed of a transformer-based image prior model, a unet diffusion model, and a decoder. Kandinsky 2.2 inherits best practices from Dall-E 2 and Latent diffusion while introducing some new ideas. It uses the CLIP model as a text and image encoder, and diffusion image prior (mapping) between latent spaces of CLIP modalities. This approach increases the visual performance of the model and unveils new horizons in blending images and text-guided image manipulation.",
"preview": "kandinsky-community--kandinsky-2-2-decoder.jpg",
"extras": "width: 768, height: 768, sampler: Default"
"extras": "width: 768, height: 768, sampler: Default",
"size": 5.15,
"date": "2023 July"
},
"Kandinsky 3": {
"path": "kandinsky-community/kandinsky-3",
"desc": "Kandinsky 3.0 is an open-source text-to-image diffusion model built upon the Kandinsky2-x model family. In comparison to its predecessors, Kandinsky 3.0 incorporates more data and specifically related to Russian culture, which allows to generate pictures related to Russin culture. Furthermore, enhancements have been made to the text understanding and visual quality of the model, achieved by increasing the size of the text encoder and Diffusion U-Net models, respectively.",
"preview": "kandinsky-community--kandinsky-3.jpg",
"variant": "fp16",
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 27.72,
"date": "2023 November"
},
"Playground v1": {
"path": "playgroundai/playground-v1",
"desc": "Playground v1 is a latent diffusion model that improves the overall HDR quality to get more stunning images.",
"preview": "playgroundai--playground-v1.jpg",
"extras": "width: 512, height: 512, sampler: Default"
"extras": "width: 512, height: 512, sampler: Default",
"size": 4.95,
"date": "2023 December"
},
"Playground v2 Small": {
"path": "playgroundai/playground-v2-256px-base",
@@ -578,22 +768,28 @@
"Playground v2.5": {
"path": "playgroundai/playground-v2.5-1024px-aesthetic",
"desc": "Playground v2.5 is a diffusion-based text-to-image generative model, and a successor to Playground v2. Playground v2.5 is the state-of-the-art open-source model in aesthetic quality.",
"preview": "playgroundai--playground-v2_5-1024px-aesthetic.jpg",
"preview": "playgroundai--playground-v2.5-1024px-aesthetic.jpg",
"variant": "fp16",
"extras": "sampler: DPM++ 2M EDM"
"extras": "sampler: DPM++ 2M EDM",
"size": 13.35,
"date": "2023 December"
},
"CogView 4": {
"path": "zai-org/CogView4-6B",
"desc": "An innovative cascaded framework that enhances the performance of text-to-image diffusion. CogView is the first model implementing relay diffusion in the realm of text-to-image generation, executing the task by first creating low-resolution images and subsequently applying relay-based super-resolution.",
"preview": "THUDM--CogView4-6B.jpg",
"skip": true
"skip": true,
"size": 30.39,
"date": "2025 March"
},
"CogView 3 Plus": {
"path": "zai-org/CogView3-Plus-3B",
"desc": "An innovative cascaded framework that enhances the performance of text-to-image diffusion. CogView is the first model implementing relay diffusion in the realm of text-to-image generation, executing the task by first creating low-resolution images and subsequently applying relay-based super-resolution.",
"preview": "THUDM--CogView3-Plus-3B.jpg",
"skip": true
"skip": true,
"size": 24.96,
"date": "2024 October"
},
"ShuttleAI Shuttle 3.0 Diffusion": {
@@ -619,14 +815,18 @@
"path": "briaai/BRIA-3.2",
"desc": "Bria 3.2 is the next-generation commercial-ready text-to-image model. With just 4 billion parameters, it provides exceptional aesthetics and text rendering, evaluated to provide on par results to leading open-source models, and outperforming other licensed models.",
"preview": "briaai--BRIA-3.2.jpg",
"skip": true
"skip": true,
"size": 18.66,
"date": "2025 June"
},
"Meissonic": {
"path": "MeissonFlow/Meissonic",
"desc": "Meissonic is a non-autoregressive mask image modeling text-to-image synthesis model that can generate high-resolution images. It is designed to run on consumer graphics cards.",
"preview": "MeissonFlow--Meissonic.jpg",
"skip": true
"skip": true,
"size": 3.64,
"date": "2024 October"
},
"aMUSEd 256": {
@@ -647,7 +847,9 @@
"path": "warp-ai/wuerstchen",
"desc": "Würstchen is a diffusion model whose text-conditional model works in a highly compressed latent space of images. Why is this important? Compressing data can reduce computational costs for both training and inference by magnitudes. Training on 1024x1024 images, is way more expensive than training at 32x32. Usually, other works make use of a relatively small compression, in the range of 4x - 8x spatial compression. Würstchen takes this to an extreme. Through its novel design, we achieve a 42x spatial compression. Würstchen employs a two-stage compression, what we call Stage A and Stage B. Stage A is a VQGAN, and Stage B is a Diffusion Autoencoder (more details can be found in the paper). A third model, Stage C, is learned in that highly compressed latent space. This training requires fractions of the compute used for current top-performing models, allowing also cheaper and faster inference.",
"preview": "warp-ai--wuerstchen.jpg",
"extras": "sampler: Default, cfg_scale: 4.0, image_cfg_scale: 0.0"
"extras": "sampler: Default, cfg_scale: 4.0, image_cfg_scale: 0.0",
"size": 12.16,
"date": "2023 August"
},
"KOALA 700M": {
@@ -656,7 +858,9 @@
"skip": true,
"desc": "Fast text-to-image model, called KOALA, by compressing SDXL's U-Net and distilling knowledge from SDXL into our model. KOALA-700M can generate a 1024x1024 image in less than 1.5 seconds on an NVIDIA 4090 GPU, which is more than 2x faster than SDXL.",
"preview": "etri-vilab--koala-700m-llava-cap.jpg",
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 6.58,
"date": "2024 January"
},
"HDM-XUT 340M Anime": {
@@ -671,13 +875,17 @@
"path": "thu-ml/unidiffuser-v1",
"desc": "UniDiffuser is a unified diffusion framework to fit all distributions relevant to a set of multi-modal data in one transformer. UniDiffuser is able to perform image, text, text-to-image, image-to-text, and image-text pair generation by setting proper timesteps without additional overhead.\nSpecifically, UniDiffuser employs a variation of transformer, called U-ViT, which parameterizes the joint noise prediction network. Other components perform as encoders and decoders of different modalities, including a pretrained image autoencoder from Stable Diffusion, a pretrained image ViT-B/32 CLIP encoder, a pretrained text ViT-L CLIP encoder, and a GPT-2 text decoder finetuned by ourselves.",
"preview": "thu-ml--unidiffuser-v1.jpg",
"extras": "width: 512, height: 512, sampler: Default"
"extras": "width: 512, height: 512, sampler: Default",
"size": 5.37,
"date": "2023 May"
},
"SalesForce BLIP-Diffusion": {
"path": "salesforce/blipdiffusion",
"desc": "BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation.",
"preview": "salesforce--blipdiffusion.jpg"
"preview": "salesforce--blipdiffusion.jpg",
"size": 7.23,
"date": "2023 July"
},
"InstaFlow 0.9B": {
@@ -690,13 +898,17 @@
"path": "DeepFloyd/IF-I-M-v1.0",
"desc": "DeepFloyd-IF is a pixel-based text-to-image triple-cascaded diffusion model, that can generate pictures with new state-of-the-art for photorealism and language understanding. The result is a highly efficient model that outperforms current state-of-the-art models, achieving a zero-shot FID-30K score of 6.66 on the COCO dataset. It is modular and composed of frozen text mode and three pixel cascaded diffusion modules, each designed to generate images of increasing resolution: 64x64, 256x256, and 1024x1024.",
"preview": "DeepFloyd--IF-I-M-v1.0.jpg",
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 12.79,
"date": "2023 April"
},
"DeepFloyd IF Large": {
"path": "DeepFloyd/IF-I-L-v1.0",
"desc": "DeepFloyd-IF is a pixel-based text-to-image triple-cascaded diffusion model, that can generate pictures with new state-of-the-art for photorealism and language understanding. The result is a highly efficient model that outperforms current state-of-the-art models, achieving a zero-shot FID-30K score of 6.66 on the COCO dataset. It is modular and composed of frozen text mode and three pixel cascaded diffusion modules, each designed to generate images of increasing resolution: 64x64, 256x256, and 1024x1024.",
"preview": "DeepFloyd--IF-I-L-v1.0.jpg",
"extras": "sampler: Default"
"extras": "sampler: Default",
"size": 15.48,
"date": "2023 April"
}
}
html/swagger.css
+10491 -860
View File
File diff suppressed because it is too large Load Diff
installer.py
+154 -150
View File
@@ -56,6 +56,7 @@ args = Dot({
})
git_commit = "unknown"
diffusers_commit = "unknown"
restart_required = False
extensions_commit = { # force specific commit for extensions
'sd-webui-controlnet': 'ecd33eb',
'adetailer': 'a89c01d'
@@ -345,6 +346,8 @@ def installed(package, friendly: str = None, reload = False, quiet = False): # p
log.warning(f'Install: package="{p[0]}" installed={pkg_version} required={p[1]} allowing experimental')
else:
log.warning(f'Install: package="{p[0]}" installed={pkg_version} required={p[1]} version mismatch')
global restart_required # pylint: disable=global-statement
restart_required = True
ok = ok and (exact or args.experimental)
else:
if not quiet:
@@ -605,7 +608,9 @@ def check_diffusers():
if args.skip_git:
install('diffusers')
return
sha = '5e181eddfe7e44c1444a2511b0d8e21d177850a0' # diffusers commit hash
sha = 'af769881d37fe916afef2c47279f66c79f5f2714' # diffusers commit hash
# if args.use_rocm or args.use_zluda or args.use_directml:
# sha = '043ab2520f6a19fce78e6e060a68dbc947edb9f9' # lock diffusers versions for now
pkg = pkg_resources.working_set.by_key.get('diffusers', None)
minor = int(pkg.version.split('.')[1] if pkg is not None else -1)
cur = opts.get('diffusers_version', '') if minor > -1 else ''
@@ -626,19 +631,19 @@ def check_transformers():
t_start = time.time()
if args.skip_all or args.skip_git or args.experimental:
return
pkg_transofmers = pkg_resources.working_set.by_key.get('transformers', None)
pkg_transformers = pkg_resources.working_set.by_key.get('transformers', None)
pkg_tokenizers = pkg_resources.working_set.by_key.get('tokenizers', None)
if args.use_directml:
target_transformers = '4.52.4'
target_tokenizers = '0.21.4'
else:
target_transformers = '4.56.1'
target_tokenizers = '0.22.0'
if (pkg_transofmers is None) or ((pkg_transofmers.version != target_transformers) or (pkg_tokenizers is None) or ((pkg_tokenizers.version != target_tokenizers) and (not args.experimental))):
if pkg_transofmers is None:
target_transformers = '4.57.1'
target_tokenizers = '0.22.1'
if (pkg_transformers is None) or ((pkg_transformers.version != target_transformers) or (pkg_tokenizers is None) or ((pkg_tokenizers.version != target_tokenizers) and (not args.experimental))):
if pkg_transformers is None:
log.info(f'Transformers install: version={target_transformers}')
else:
log.info(f'Transformers update: current={pkg_transofmers.version} target={target_transformers}')
log.info(f'Transformers update: current={pkg_transformers.version} target={target_transformers}')
pip('uninstall --yes transformers', ignore=True, quiet=True, uv=False)
pip(f'install --upgrade tokenizers=={target_tokenizers}', ignore=False, quiet=True, uv=False)
pip(f'install --upgrade transformers=={target_transformers}', ignore=False, quiet=True, uv=False)
@@ -674,82 +679,83 @@ def install_rocm_zluda():
if args.skip_all or args.skip_requirements:
return torch_command
from modules import rocm
if not rocm.is_installed:
log.warning('ROCm: could not find ROCm toolkit installed')
log.info('Using CPU-only torch')
return os.environ.get('TORCH_COMMAND', 'torch torchvision')
log.info('ROCm: AMD toolkit detected')
# if not is_windows:
# os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow-rocm')
device = None
amd_gpus = []
try:
amd_gpus = rocm.get_agents()
if len(amd_gpus) == 0:
log.warning('ROCm: no agent was found')
if sys.platform == "win32" and not rocm.is_installed:
amd_gpus = rocm.driver_get_agents()
else:
log.info(f'ROCm: agents={[gpu.name for gpu in amd_gpus]}')
if args.device_id is None:
index = 0
for idx, gpu in enumerate(amd_gpus):
index = idx
# if gpu.name.startswith('gfx11') and os.environ.get('TENSORFLOW_PACKAGE') == 'tensorflow-rocm': # do not use tensorflow-rocm for navi 3x
# os.environ['TENSORFLOW_PACKAGE'] = 'tensorflow==2.13.0'
if not gpu.is_apu:
# although apu was found, there can be a dedicated card. do not break loop.
# if no dedicated card was found, apu will be used.
break
os.environ.setdefault('HIP_VISIBLE_DEVICES', str(index))
device = amd_gpus[index]
else:
device_id = int(args.device_id)
if device_id < len(amd_gpus):
device = amd_gpus[device_id]
amd_gpus = rocm.get_agents()
log.info('ROCm: AMD toolkit detected')
except Exception as e:
log.warning(f'ROCm agent enumerator failed: {e}')
#os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow')
device = None
if len(amd_gpus) == 0:
log.warning('ROCm: no agent was found')
else:
log.info(f'ROCm: agents={[gpu.name for gpu in amd_gpus]}')
if args.device_id is None:
index = 0
for idx, gpu in enumerate(amd_gpus):
index = idx
if not gpu.is_apu:
# although apu was found, there can be a dedicated card. do not break loop.
# if no dedicated card was found, apu will be used.
break
os.environ.setdefault('HIP_VISIBLE_DEVICES', str(index))
device = amd_gpus[index]
else:
device_id = int(args.device_id)
if device_id < len(amd_gpus):
device = amd_gpus[device_id]
if sys.platform == "win32" and args.use_rocm and not rocm.is_installed:
check_python(supported_minors=[11, 12, 13], reason='ROCm backend requires a Python version between 3.11 and 3.13')
install(f"rocm rocm-sdk-core --index-url https://rocm.nightlies.amd.com/v2-staging/{device.therock}")
rocm.refresh()
msg = f'ROCm: version={rocm.version}'
if device is not None:
msg += f', using agent {device.name}'
log.info(msg)
if sys.platform == "win32": # TODO install: enable ROCm for windows when available
#check_python(supported_minors=[10, 11, 12, 13], reason='ZLUDA backend requires a Python version between 3.10 and 3.13')
if sys.platform == "win32":
if args.use_rocm: # TODO install: switch to pytorch source when it becomes available
if device is not None and isinstance(rocm.environment, rocm.PythonPackageEnvironment): # TheRock
check_python(supported_minors=[11, 12, 13], reason='ROCm backend requires a Python version between 3.11 and 3.13')
torch_command = os.environ.get('TORCH_COMMAND', f'torch torchvision --index-url https://rocm.nightlies.amd.com/v2-staging/{device.therock}')
else:
check_python(supported_minors=[12], reason='ROCm Windows preview requires Python version 3.12')
torch_command = os.environ.get('TORCH_COMMAND', '--no-cache-dir https://repo.radeon.com/rocm/windows/rocm-rel-6.4.4/torch-2.8.0a0%2Bgitfc14c65-cp312-cp312-win_amd64.whl https://repo.radeon.com/rocm/windows/rocm-rel-6.4.4/torchvision-0.24.0a0%2Bc85f008-cp312-cp312-win_amd64.whl')
else:
#check_python(supported_minors=[10, 11, 12, 13], reason='ZLUDA backend requires a Python version between 3.10 and 3.13')
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.7.1+cu118 torchvision==0.22.1+cu118 --index-url https://download.pytorch.org/whl/cu118')
if args.device_id is not None:
if os.environ.get('HIP_VISIBLE_DEVICES', None) is not None:
log.warning('Setting HIP_VISIBLE_DEVICES and --device-id at the same time may be mistake.')
os.environ['HIP_VISIBLE_DEVICES'] = args.device_id
del args.device_id
if args.device_id is not None:
if os.environ.get('HIP_VISIBLE_DEVICES', None) is not None:
log.warning('Setting HIP_VISIBLE_DEVICES and --device-id at the same time may be mistake.')
os.environ['HIP_VISIBLE_DEVICES'] = args.device_id
del args.device_id
error = None
from modules import zluda_installer
try:
if args.reinstall or zluda_installer.is_reinstall_needed():
zluda_installer.uninstall()
zluda_installer.install()
zluda_installer.set_default_agent(device)
except Exception as e:
error = e
log.warning(f'Failed to install ZLUDA: {e}')
from modules import zluda_installer
try:
if args.reinstall or zluda_installer.is_reinstall_needed():
zluda_installer.uninstall()
zluda_installer.install()
zluda_installer.set_default_agent(device)
except Exception as e:
log.warning(f'Failed to install ZLUDA: {e}')
if error is None:
try:
zluda_installer.load()
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.7.1+cu118 torchvision==0.22.1+cu118 --index-url https://download.pytorch.org/whl/cu118')
except Exception as e:
error = e
log.warning(f'Failed to load ZLUDA: {e}')
if error is not None:
log.info('Using CPU-only torch')
torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision')
else:
#check_python(supported_minors=[10, 11, 12, 13], reason='ROCm backend requires a Python version between 3.10 and 3.13')
if os.environ.get("TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL", None) is None:
os.environ.setdefault('TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL', '1')
if args.use_nightly:
if rocm.version is None or float(rocm.version) >= 6.4: # assume the latest if version check fails
torch_command = os.environ.get('TORCH_COMMAND', '--upgrade --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm6.4')
@@ -779,7 +785,7 @@ def install_rocm_zluda():
log.info(f'ROCm: HSA_OVERRIDE_GFX_VERSION auto config skipped: device={device.name if device is not None else None} version={os.environ.get("HSA_OVERRIDE_GFX_VERSION", None)}')
else:
gfx_ver = device.get_gfx_version()
if gfx_ver is not None:
if gfx_ver is not None and device.name.removeprefix("gfx") != gfx_ver.replace(".", ""):
os.environ.setdefault('HSA_OVERRIDE_GFX_VERSION', gfx_ver)
log.info(f'ROCm: HSA_OVERRIDE_GFX_VERSION config overridden: device={device.name} version={os.environ.get("HSA_OVERRIDE_GFX_VERSION", None)}')
@@ -793,27 +799,6 @@ def install_ipex():
args.use_ipex = True # pylint: disable=attribute-defined-outside-init
log.info('IPEX: Intel OneAPI toolkit detected')
if os.environ.get("NEOReadDebugKeys", None) is None:
os.environ.setdefault('NEOReadDebugKeys', '1')
if os.environ.get("ClDeviceGlobalMemSizeAvailablePercent", None) is None:
os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
if os.environ.get("SYCL_CACHE_PERSISTENT", None) is None:
os.environ.setdefault('SYCL_CACHE_PERSISTENT', '1') # Jit cache
if os.environ.get("PYTORCH_ENABLE_XPU_FALLBACK", None) is None:
os.environ.setdefault('PYTORCH_ENABLE_XPU_FALLBACK', '1') # CPU fallback for unsupported ops
if os.environ.get("UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS", None) is None:
os.environ.setdefault('UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS', '1') # Work around the 4G alloc limit on Alchemist
# FP64 emulation causes random UR Errors
#if os.environ.get("OverrideDefaultFP64Settings", None) is None:
# os.environ.setdefault('OverrideDefaultFP64Settings', '1')
#if os.environ.get("IGC_EnableDPEmulation", None) is None:
# os.environ.setdefault('IGC_EnableDPEmulation', '1') # FP64 Emulation
if args.use_nightly:
torch_command = os.environ.get('TORCH_COMMAND', '--upgrade --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/xpu')
else:
@@ -827,20 +812,17 @@ def install_ipex():
def install_openvino():
t_start = time.time()
log.info('OpenVINO: selected')
#check_python(supported_minors=[10, 11, 12, 13], reason='OpenVINO backend requires a Python version between 3.10 and 3.13')
os.environ.setdefault('PYTORCH_TRACING_MODE', 'TORCHFX')
#check_python(supported_minors=[10, 11, 12, 13], reason='OpenVINO backend requires a Python version between 3.10 and 3.13')
if sys.platform == 'darwin':
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.8.0 torchvision==0.23.0')
else:
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.8.0+cpu torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cpu')
install(os.environ.get('OPENVINO_COMMAND', 'openvino==2025.3.0'), 'openvino')
install(os.environ.get('NNCF_COMMAND', 'nncf==2.18.0'), 'nncf')
os.environ.setdefault('PYTORCH_TRACING_MODE', 'TORCHFX')
if os.environ.get("NEOReadDebugKeys", None) is None:
os.environ.setdefault('NEOReadDebugKeys', '1')
if os.environ.get("ClDeviceGlobalMemSizeAvailablePercent", None) is None:
os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
if not (args.skip_all or args.skip_requirements):
install(os.environ.get('OPENVINO_COMMAND', 'openvino==2025.3.0'), 'openvino')
install(os.environ.get('NNCF_COMMAND', 'nncf==2.18.0'), 'nncf')
ts('openvino', t_start)
return torch_command
@@ -874,6 +856,7 @@ def install_torch_addons():
install('pillow-jxl-plugin==1.3.4', 'pillow-jxl-plugin')
if not args.experimental:
uninstall('wandb', quiet=True)
uninstall('pynvml', quiet=True)
ts('addons', t_start)
@@ -921,9 +904,9 @@ def check_torch():
if torch_command != '':
pass
else:
is_cuda_available = allow_cuda and (shutil.which('nvidia-smi') is not None or args.use_xformers or os.path.exists(os.path.join(os.environ.get('SystemRoot') or r'C:\Windows', 'System32', 'nvidia-smi.exe')))
is_rocm_available = allow_rocm and rocm.is_installed
is_ipex_available = allow_ipex and (args.use_ipex or shutil.which('sycl-ls') is not None or shutil.which('sycl-ls.exe') is not None or os.environ.get('ONEAPI_ROOT') is not None or os.path.exists('/opt/intel/oneapi') or os.path.exists("C:/Program Files (x86)/Intel/oneAPI") or os.path.exists("C:/oneAPI"))
is_cuda_available = allow_cuda and (args.use_cuda or shutil.which('nvidia-smi') is not None or args.use_xformers or os.path.exists(os.path.join(os.environ.get('SystemRoot') or r'C:\Windows', 'System32', 'nvidia-smi.exe')))
is_rocm_available = allow_rocm and (args.use_rocm or args.use_zluda or rocm.is_installed)
is_ipex_available = allow_ipex and (args.use_ipex or shutil.which('sycl-ls') is not None or shutil.which('sycl-ls.exe') is not None or os.environ.get('ONEAPI_ROOT') is not None or os.path.exists('/opt/intel/oneapi') or os.path.exists("C:/Program Files (x86)/Intel/oneAPI") or os.path.exists("C:/oneAPI") or os.path.exists("C:/Program Files/Intel/Intel Graphics Software"))
if is_cuda_available and args.use_cuda: # prioritize cuda
torch_command = install_cuda()
@@ -950,63 +933,69 @@ def check_torch():
install(torch_command, 'torch torchvision')
install('onnxruntime-directml', 'onnxruntime-directml', ignore=True)
else:
if args.use_zluda:
log.warning("ZLUDA failed to initialize: no HIP SDK found")
log.warning('Torch: CPU-only version installed')
torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision')
if 'torch' in torch_command and not args.version:
if args.version:
return
if 'torch' in torch_command:
if not installed('torch'):
log.info(f'Torch: download and install in progress... cmd="{torch_command}"')
install('--upgrade pip', 'pip', reinstall=True) # pytorch rocm is too large for older pip
install(torch_command, 'torch torchvision', quiet=True)
else:
try:
import torch
try:
import torch
log.info(f'Torch {torch.__version__}')
if args.use_ipex and allow_ipex:
try:
import intel_extension_for_pytorch as ipex # pylint: disable=import-error, unused-import
log.info(f'Torch backend: Intel IPEX {ipex.__version__}')
except Exception:
log.warning('IPEX: not found')
if shutil.which('icpx') is not None:
log.info(f'{os.popen("icpx --version").read().rstrip()}')
for device in range(torch.xpu.device_count()):
log.info(f'Torch detected GPU: {torch.xpu.get_device_name(device)} VRAM {round(torch.xpu.get_device_properties(device).total_memory / 1024 / 1024)} Compute Units {torch.xpu.get_device_properties(device).max_compute_units}')
elif torch.cuda.is_available() and (allow_cuda or allow_rocm):
# log.debug(f'Torch allocator: {torch.cuda.get_allocator_backend()}')
if torch.version.cuda and allow_cuda:
log.info(f'Torch backend: nVidia CUDA {torch.version.cuda} cuDNN {torch.backends.cudnn.version() if torch.backends.cudnn.is_available() else "N/A"}')
elif torch.version.hip and allow_rocm:
log.info(f'Torch backend: AMD ROCm HIP {torch.version.hip}')
import intel_extension_for_pytorch as ipex # pylint: disable=import-error, unused-import
log.info(f'Torch backend: type=IPEX version={ipex.__version__}')
except Exception:
pass
if 'cpu' in torch.__version__:
if is_cuda_available:
if args.use_cuda:
log.warning(f'Torch: version="{torch.__version__}" CPU version installed and CUDA is selected - reinstalling')
install(torch_command, 'torch torchvision', quiet=True, reinstall=True, force=True) # foce reinstall
else:
log.warning('Unknown Torch backend')
for device in [torch.cuda.device(i) for i in range(torch.cuda.device_count())]:
log.info(f'Torch detected GPU: {torch.cuda.get_device_name(device)} VRAM {round(torch.cuda.get_device_properties(device).total_memory / 1024 / 1024)} Arch {torch.cuda.get_device_capability(device)} Cores {torch.cuda.get_device_properties(device).multi_processor_count}')
log.warning(f'Torch: version="{torch.__version__}" CPU version installed and CUDA is available - consider reinstalling')
elif is_rocm_available:
if args.use_rocm:
log.warning(f'Torch: version="{torch.__version__}" CPU version installed and ROCm is selected - reinstalling')
install(torch_command, 'torch torchvision', quiet=True, reinstall=True, force=True) # foce reinstall
else:
log.warning(f'Torch: version="{torch.__version__}" CPU version installed and ROCm is available - consider reinstalling')
if hasattr(torch, "xpu") and torch.xpu.is_available() and allow_ipex:
if shutil.which('icpx') is not None:
log.info(f'{os.popen("icpx --version").read().rstrip()}')
for device in range(torch.xpu.device_count()):
log.info(f'Torch detected: gpu="{torch.xpu.get_device_name(device)}" vram={round(torch.xpu.get_device_properties(device).total_memory / 1024 / 1024)} units={torch.xpu.get_device_properties(device).max_compute_units}')
elif torch.cuda.is_available() and (allow_cuda or allow_rocm):
if torch.version.cuda and allow_cuda:
log.info(f'Torch backend: version="{torch.__version__}" type=CUDA CUDA={torch.version.cuda} cuDNN={torch.backends.cudnn.version() if torch.backends.cudnn.is_available() else "N/A"}')
elif torch.version.hip and allow_rocm:
log.info(f'Torch backend: version="{torch.__version__}" type=ROCm HIP={torch.version.hip}')
else:
try:
if args.use_directml and allow_directml:
import torch_directml # pylint: disable=import-error
dml_ver = pkg_resources.get_distribution("torch-directml")
log.info(f'Torch backend: DirectML ({dml_ver})')
for i in range(0, torch_directml.device_count()):
log.info(f'Torch detected GPU: {torch_directml.device_name(i)}')
except Exception:
log.warning("Torch reports CUDA not available")
except Exception as e:
log.error(f'Torch cannot load: {e}')
if not args.ignore:
sys.exit(1)
if rocm.is_installed:
if sys.platform == "win32": # CPU, DirectML, ZLUDA
rocm.conceal()
elif rocm.is_wsl: # WSL ROCm
log.warning('Unknown Torch backend')
for device in [torch.cuda.device(i) for i in range(torch.cuda.device_count())]:
log.info(f'Torch detected: gpu="{torch.cuda.get_device_name(device)}" vram={round(torch.cuda.get_device_properties(device).total_memory / 1024 / 1024)} arch={torch.cuda.get_device_capability(device)} cores={torch.cuda.get_device_properties(device).multi_processor_count}')
else:
try:
rocm.load_hsa_runtime()
except OSError:
log.error("ROCm: failed to preload HSA runtime")
if args.version:
return
if args.use_directml and allow_directml:
import torch_directml # pylint: disable=import-error
dml_ver = pkg_resources.get_distribution("torch-directml")
log.warning(f'Torch backend: DirectML ({dml_ver})')
log.warning('DirectML: end-of-life')
for i in range(0, torch_directml.device_count()):
log.info(f'Torch detected GPU: {torch_directml.device_name(i)}')
except Exception:
log.warning("Torch reports CUDA not available")
except Exception as e:
log.error(f'Torch cannot load: {e}')
if not args.ignore:
sys.exit(1)
if rocm.is_installed:
rocm.postinstall()
if not args.skip_all:
install_torch_addons()
check_cudnn()
@@ -1264,6 +1253,17 @@ def install_pydantic():
reload('pydantic', '1.10.21')
def install_insightface():
install('git+https://github.com/deepinsight/insightface@29b6cd65aa0e9ae3b6602de3c52e9d8949c8ee86#subdirectory=python-package', 'insightface') # insightface==0.7.3 with patches
if args.new:
uninstall('albumentations')
install('albumentationsx')
else:
uninstall('albumentationsx')
install('albumentations==1.4.3', ignore=True, quiet=True)
install_pydantic()
def install_optional():
t_start = time.time()
log.info('Installing optional requirements...')
@@ -1277,8 +1277,6 @@ def install_optional():
install('nvidia-ml-py', ignore=True, quiet=True)
install('ultralytics==8.3.40', ignore=True, quiet=True)
install('Cython', ignore=True, quiet=True)
install('git+https://github.com/deepinsight/insightface@554a05561cb71cfebb4e012dfea48807f845a0c2#subdirectory=python-package', 'insightface') # insightface==0.7.3 with patches
install('albumentations==1.4.3', ignore=True, quiet=True)
install('av', ignore=True, quiet=True)
install('gguf', ignore=True)
try:
@@ -1355,11 +1353,18 @@ def set_environment():
allocator += ',backend:cudaMallocAsync'
if opts.get("torch_expandable_segments", False):
allocator += ',expandable_segments:True'
os.environ.setdefault('PYTORCH_ALLOC_CONF', allocator)
os.environ.setdefault('PYTORCH_CUDA_ALLOC_CONF', allocator)
os.environ.setdefault('PYTORCH_HIP_ALLOC_CONF', allocator)
log.debug(f'Torch allocator: "{allocator}"')
if sys.platform == 'darwin':
os.environ.setdefault('PYTORCH_ENABLE_MPS_FALLBACK', '1')
os.environ.setdefault('TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL', '1')
os.environ.setdefault('NEOReadDebugKeys', '1')
os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
os.environ.setdefault('SYCL_CACHE_PERSISTENT', '1')
os.environ.setdefault('UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS', '1')
os.environ.setdefault('PYTORCH_ENABLE_XPU_FALLBACK', '1')
os.environ.setdefault('PYTORCH_ENABLE_MPS_FALLBACK', '1')
os.environ.setdefault('TOKENIZERS_PARALLELISM', '0')
def check_extensions():
@@ -1634,7 +1639,6 @@ def add_args(parser):
group_http.add_argument("--cors-regex", type=str, default=os.environ.get("SD_CORSREGEX", None), help="Allowed CORS origins as regular expression, default: %(default)s")
group_http.add_argument('--subpath', type=str, default=os.environ.get("SD_SUBPATH", None), help='Customize the URL subpath for usage with reverse proxy')
group_http.add_argument("--autolaunch", default=os.environ.get("SD_AUTOLAUNCH", False), action='store_true', help="Open the UI URL in the system's default browser upon launch")
group_http.add_argument('--docs', default=os.environ.get("SD_DOCS", False), action='store_true', help = "Mount API docs, default: %(default)s")
group_http.add_argument("--auth", type=str, default=os.environ.get("SD_AUTH", None), help='Set access authentication like "user:pwd,user:pwd""')
group_http.add_argument("--auth-file", type=str, default=os.environ.get("SD_AUTHFILE", None), help='Set access authentication using file, default: %(default)s')
group_http.add_argument("--allowed-paths", nargs='+', default=[], type=str, required=False, help="add additional paths to paths allowed for web access")
@@ -1655,11 +1659,11 @@ def add_args(parser):
group_log = parser.add_argument_group('Logging')
group_log.add_argument("--log", type=str, default=os.environ.get("SD_LOG", None), help="Set log file, default: %(default)s")
group_log.add_argument('--debug', default=os.environ.get("SD_DEBUG",False), action='store_true', help="Run with debug logging, default: %(default)s")
group_log.add_argument('--debug', default=not os.environ.get("SD_NODEBUG",False), action='store_true', help="Run with debug logging, default: %(default)s")
group_log.add_argument("--trace", default=os.environ.get("SD_TRACE", False), action='store_true', help="Run with trace logging, default: %(default)s")
group_log.add_argument("--profile", default=os.environ.get("SD_PROFILE", False), action='store_true', help="Run profiler, default: %(default)s")
group_log.add_argument('--docs', default=os.environ.get("SD_DOCS", False), action='store_true', help="Mount API docs, default: %(default)s")
group_log.add_argument("--api-log", default=os.environ.get("SD_APILOG", False), action='store_true', help="Log all API requests")
group_log.add_argument('--docs', default=not os.environ.get("SD_NODOCS", False), action='store_true', help = "Mount API docs, default: %(default)s")
group_log.add_argument("--api-log", default=not os.environ.get("SD_NOAPILOG", False), action='store_true', help="Log all API requests")
group_nargs = parser.add_argument_group('Other')
group_nargs.add_argument('args', type=str, nargs='*', help=argparse.SUPPRESS)
javascript/control.js
+1 -6
View File
@@ -2,13 +2,8 @@ function controlInputMode(inputMode, ...args) {
const updateEl = gradioApp().getElementById('control_update');
if (updateEl) updateEl.click();
const tab = gradioApp().querySelector('#control-tab-input button.selected');
if (!tab) return ['Select', ...args];
if (!tab) return ['Image', ...args];
inputMode = tab.innerText;
if (inputMode === 'Image') {
if (!gradioApp().getElementById('control_input_select').classList.contains('hidden')) inputMode = 'Select';
else if (!gradioApp().getElementById('control_input_resize').classList.contains('hidden')) inputMode = 'Outpaint';
else if (!gradioApp().getElementById('control_input_inpaint').classList.contains('hidden')) inputMode = 'Inpaint';
}
return [inputMode, ...args];
}
javascript/extraNetworks.js
+3 -2
View File
@@ -293,10 +293,11 @@ function extraNetworksSearchButton(event) {
}
function extraNetworksFilterVersion(event) {
// log('extraNetworksFilterVersion', event);
log('extraNetworksFilterVersion', event);
const version = event.target.textContent.trim();
const activeTab = getENActiveTab();
const cardContainer = gradioApp().querySelector(`#${activeTab}_model_cards`);
const activePage = getENActivePage().toLowerCase();
const cardContainer = gradioApp().querySelector(`#${activeTab}_${activePage}_cards`);
if (!cardContainer) return;
if (cardContainer.dataset.activeVersion === version) {
cardContainer.dataset.activeVersion = '';
javascript/guidance.js
+20
View File
@@ -0,0 +1,20 @@
const guiders = {
None: '',
'LSC: LayerSkipConfig': 'https://github.com/huggingface/diffusers/blob/041501aea92919c9c7f36e189fc9cf7d865ebb96/src/diffusers/hooks/layer_skip.py#L41',
'CFG: ClassifierFreeGuidance': 'https://huggingface.co/docs/diffusers/v0.35.1/en/api/modular_diffusers/guiders#diffusers.ClassifierFreeGuidance',
'Auto: AutoGuidance': 'https://huggingface.co/docs/diffusers/v0.35.1/en/api/modular_diffusers/guiders#diffusers.AutoGuidance',
'Zero: ClassifierFreeZeroStar': 'https://huggingface.co/docs/diffusers/v0.35.1/en/api/modular_diffusers/guiders#diffusers.ClassifierFreeZeroStarGuidance',
'PAG: PerturbedAttentionGuidance': 'https://huggingface.co/docs/diffusers/v0.35.1/en/api/modular_diffusers/guiders#diffusers.PerturbedAttentionGuidance',
'APG: AdaptiveProjectedGuidance': 'https://huggingface.co/docs/diffusers/v0.35.1/en/api/modular_diffusers/guiders#diffusers.AdaptiveProjectedGuidance',
'SLG: SkipLayerGuidance': 'https://huggingface.co/docs/diffusers/v0.35.1/en/api/modular_diffusers/guiders#diffusers.SkipLayerGuidance',
'SEG: SmoothedEnergyGuidance': 'https://huggingface.co/docs/diffusers/v0.35.1/en/api/modular_diffusers/guiders#diffusers.SmoothedEnergyGuidance',
'TCFG: TangentialClassifierFreeGuidance': 'https://huggingface.co/docs/diffusers/v0.35.1/en/api/modular_diffusers/guiders#diffusers.TangentialClassifierFreeGuidance',
'FDG: FrequencyDecoupledGuidance': 'https://huggingface.co/docs/diffusers/v0.35.1/en/api/modular_diffusers/guiders#diffusers.FrequencyDecoupledGuidance',
};
function getGuidanceDocs(guider) {
if (guider.label) guider = guider.label;
const url = guiders[guider];
log('getGuidanceDocs', guider, url);
if (url) window.open(url, '_blank');
}
javascript/history.js
+1 -1
View File
@@ -45,7 +45,7 @@ function refreshHistory() {
if (inferenceTypes.some((type) => entry.job.toLowerCase().startsWith(type))) entry.type = 'inference';
else if (ioTypes.some((type) => entry.job.toLowerCase().startsWith(type))) entry.type = 'io';
else entry.type = 'default';
ts.push({ start, end: end.timestamp, label: entry.job, type: entry.type });
if (start && end.timestamp) ts.push({ start, end: end.timestamp, label: entry.job, type: entry.type });
}
}
if (!ts.length) return;
javascript/imageViewer.js
+9
View File
@@ -15,11 +15,20 @@ function cycleImageFit() {
log('cycleImageFit', current, next);
}
function isInViewport(element) {
const rect = element.getBoundingClientRect();
return rect.top >= 0 && rect.left >= 0 && rect.bottom <= (window.innerHeight || document.documentElement.clientHeight) && rect.right <= (window.innerWidth || document.documentElement.clientWidth);
}
function closeModal(evt, force = false) {
if (force) gradioApp().getElementById('lightboxModal').style.display = 'none';
if (previewDrag) return;
if (evt?.button !== 0) return;
gradioApp().getElementById('lightboxModal').style.display = 'none';
let thumbnails = Array.from(gradioApp().querySelectorAll('.thumbnails .thumbnail-item'));
thumbnails = thumbnails.filter((el) => el.checkVisibility());
if (thumbnails.length === 0) return;
thumbnails[0].focus();
}
function modalImageSwitch(offset) {
javascript/logMonitor.js
+1
View File
@@ -78,6 +78,7 @@ async function logMonitor() {
for (const line of lines) addLogLine(line);
if (!logConnected) {
logConnected = true;
monitorConnection();
xhrPost(`${window.api}/log`, { debug: 'connected' });
}
} else {
javascript/monitor.js
+35
View File
@@ -0,0 +1,35 @@
async function updateIndicator(online, data, msg) {
const el = document.getElementById('logo_nav');
if (!el || !data) return;
const status = online ? '<b style="color:lime">online</b>' : '<b style="color:darkred">offline</b>';
const date = new Date();
const template = `
Version: <b>${data.updated}</b><br>
Commit: <b>${data.hash}</b><br>
Branch: <b>${data.branch}</b><br>
Status: ${status}<br>
Since: ${date.toLocaleString()}<br>
`;
if (online) {
el.dataset.hint = template;
el.style.backgroundColor = 'var(--sd-main-accent-color)';
log('monitorConnection: online', data);
} else {
el.dataset.hint = template;
el.style.backgroundColor = 'var(--color-error)';
log('monitorConnection: offline', msg);
}
}
async function monitorConnection() {
try {
const res = await fetch(`${window.api}/version`);
const data = await res.json();
const url = res.url.split('/sdapi')[0].replace('http', 'ws'); // update global url as ws need fqdn
const ws = new WebSocket(`${url}/queue/join`);
ws.onopen = () => updateIndicator(true, data, '');
ws.onclose = () => updateIndicator(false, data, '');
ws.onerror = (e) => updateIndicator(false, data, e.message);
ws.onmessage = (evt) => log('monitorConnection: message', evt.data);
} catch { /**/ }
}
javascript/ui.js
+1
View File
@@ -601,4 +601,5 @@ async function reconnectUI() {
const sd_model_observer = new MutationObserver(sd_model_callback);
sd_model_observer.observe(sd_model, { attributes: true, childList: true, subtree: true });
log('reconnectUI');
monitorConnection();
}
launch.py
+4 -1
View File
@@ -133,7 +133,7 @@ def run_pip(pkg, desc=None): # compatbility function
desc = pkg
for f in forbidden:
if f in pkg:
debug_install('Blocked package installation: package={f}')
debug_install(f'Blocked package installation: package={f}')
return True
index_url_line = f' --index-url {index_url}' if index_url != '' else ''
return run(f'"{sys.executable}" -m pip {pkg} --prefer-binary{index_url_line}', desc=f"Installing {desc}", errdesc=f"Couldn't install {desc}")
@@ -297,6 +297,9 @@ def main():
installer.log.info(f'Installer time: {init_summary()}')
get_custom_args()
if installer.restart_required:
installer.log.warning('Restart is recommended due to packages updates...')
uv, instance = start_server(immediate=True, server=None)
t_server = time.time()
t_monitor = time.time()
models/Reference/OPPOer--Qwen-Image-Edit-2509-Pruning.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

models/Reference/OPPOer--Qwen-Image-Edit-Pruning.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 72 KiB

models/Reference/OPPOer--Qwen-Image-Pruning.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 70 KiB

models/Reference/Qwen--Qwen-Image-Edit-2509.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 85 KiB

models/Reference/Qwen-Image-13B.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 172 KiB

models/Reference/Qwen-Image-Distill.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 75 KiB

models/Reference/X-Omni--X-Omni-SFT.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 76 KiB

models/Reference/briaai--BRIA-3.2.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 48 KiB

models/Reference/fal--AuraFlow-v0.3.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 87 KiB

After

Width:  |  Height:  |  Size: 102 KiB

models/Reference/juggernautXL_v9Rundiffusionphoto2.jpg → models/Reference/juggernautXL_juggXIByRundiffusion.jpg
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 68 KiB

After

Width:  |  Height:  |  Size: 68 KiB

models/Reference/juggernautXL_v9Rdphoto2Lightning.jpg → models/Reference/juggernautXL_juggXILightningByRD.jpg
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 69 KiB

After

Width:  |  Height:  |  Size: 69 KiB

models/Reference/kandinsky-community--kandinsky-2-1.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 82 KiB

After

Width:  |  Height:  |  Size: 131 KiB

models/Reference/kandinsky-community--kandinsky-2-2-decoder.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 63 KiB

After

Width:  |  Height:  |  Size: 39 KiB

models/Reference/kandinsky-community--kandinsky-3.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 53 KiB

After

Width:  |  Height:  |  Size: 67 KiB

models/Reference/linoyts--Wan2.2-VACE-Fun-14B-diffusers.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

models/Reference/playgroundai--playground-v2_5-1024px-aesthetic.jpg → models/Reference/playgroundai--playground-v2.5-1024px-aesthetic.jpg
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 73 KiB

After

Width:  |  Height:  |  Size: 73 KiB

models/Reference/segmind--SSD-1B.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 49 KiB

After

Width:  |  Height:  |  Size: 84 KiB

models/Reference/segmind--SegMoE-4x2-v0.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 34 KiB

After

Width:  |  Height:  |  Size: 59 KiB

models/Reference/segmind--SegMoE-SD-4x2-v0.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 51 KiB

models/Reference/segmind--Segmind-Vega.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 76 KiB

models/Reference/segmind--tiny-sd.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 88 KiB

After

Width:  |  Height:  |  Size: 80 KiB

models/Reference/stabilityai--sdxl-turbo.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 36 KiB

models/Reference/stabilityai--stable-diffusion-2-1-base.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 57 KiB

After

Width:  |  Height:  |  Size: 48 KiB

models/Reference/stabilityai--stable-diffusion-2-1.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 62 KiB

After

Width:  |  Height:  |  Size: 81 KiB

models/Reference/sd_xl_base_1.0.jpg → models/Reference/stabilityai--stable-diffusion-xl-base-1.0.jpg
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 36 KiB

After

Width:  |  Height:  |  Size: 36 KiB

models/Reference/tempest-by-vlad-base.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 34 KiB

models/Reference/tempest-by-vlad-hyper.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 26 KiB

models/Reference/tempestByVlad_baseV01.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 74 KiB

models/Reference/tempestByVlad_hyperV01.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 83 KiB

models/Reference/v1-5-pruned-fp16-emaonly.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 92 KiB

After

Width:  |  Height:  |  Size: 83 KiB

models/Reference/vladmandic--flux.1-dev-SRPO.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

modules/api/docs.py
+12 -13
View File
@@ -41,20 +41,18 @@ def get_swagger_ui_html(*,
if oauth2_redirect_url:
html += f"oauth2RedirectUrl: window.location.origin + '{oauth2_redirect_url}',"
html += """
presets: [
SwaggerUIBundle.presets.apis,
SwaggerUIBundle.SwaggerUIStandalonePreset
],
})"""
presets: [
SwaggerUIBundle.presets.apis,
SwaggerUIBundle.SwaggerUIStandalonePreset
],
})"""
if init_oauth:
html += f"""
ui.initOAuth({json.dumps(jsonable_encoder(init_oauth))})
"""
html += f"ui.initOAuth({json.dumps(jsonable_encoder(init_oauth))})"
html += """
</script>
</body>
</html>
"""
</script>
</body>
</html>
"""
return HTMLResponse(html)
@@ -74,8 +72,9 @@ def create_docs(app: FastAPI):
title=f'{app.title}: Swagger UI',
openapi_url=app.openapi_url,
swagger_favicon_url='/file=html/favicon.svg',
swagger_css_url='/file=html/swagger.css',
swagger_ui_parameters=swagger_ui_parameters,
swagger_extra_css_url='file=html/swagger.css',
# swagger_extra_css_url='file=html/swagger.css',
)
# res = inject_css(html.content, 'html/swagger.css')
return res
modules/cachedit.py
+64
View File
@@ -0,0 +1,64 @@
import os
from installer import install
from modules import shared
def apply_cache_dit(pipe):
if not shared.opts.cache_dit_enabled:
return
install('git+https://github.com/vipshop/cache-dit', 'cache_dit')
os.environ.setdefault("CACHE_DIT_LOG_LEVEL", "error")
try:
import cache_dit
except Exception as e:
shared.log.error(f'Cache-DIT: {e}')
return
_, supported = cache_dit.supported_pipelines()
supported = [s.replace('*', '') for s in supported]
if not any(pipe.__class__.__name__.startswith(s) for s in supported):
shared.log.error(f'Cache-DiT: pipeline={pipe.__class__.__name__} unsupported')
return
if getattr(pipe, 'has_cache_dit', False):
unapply_cache_dir(pipe)
config_args = {}
if shared.opts.cache_dit_fcompute >= 0:
config_args['Fn_compute_blocks'] = int(shared.opts.cache_dit_fcompute)
if shared.opts.cache_dit_bcompute >= 0:
config_args['Bn_compute_blocks'] = int(shared.opts.cache_dit_bcompute)
if shared.opts.cache_dit_threshold >= 0:
config_args['residual_diff_threshold'] = float(shared.opts.cache_dit_threshold)
if shared.opts.cache_dit_warmup >= 0:
config_args['max_warmup_steps'] = int(shared.opts.cache_dit_warmup)
cache_config = cache_dit.BasicCacheConfig(**config_args)
if shared.opts.cache_dit_calibrator == "TaylorSeer":
calibrator_config = cache_dit.TaylorSeerCalibratorConfig(taylorseer_order=1)
elif shared.opts.cache_dit_calibrator == "FoCa":
calibrator_config = cache_dit.FoCaCalibratorConfig()
else:
calibrator_config = None
shared.log.info(f'Apply Cache-DiT: config="{cache_config.strify()}" calibrator="{calibrator_config.strify() if calibrator_config else "None"}"')
try:
cache_dit.enable_cache(
pipe,
cache_config=cache_config,
calibrator_config=calibrator_config,
)
shared.sd_model.has_cache_dit = True
except Exception as e:
shared.log.error(f'Cache-DiT: {e}')
return
def unapply_cache_dir(pipe):
if not shared.opts.cache_dit_enabled or not getattr(pipe, 'has_cache_dit', False):
return
try:
import cache_dit
# stats = cache_dit.summary(pipe)
# shared.log.critical(f'Unapply Cache-DiT: {stats}')
cache_dit.disable_cache(pipe)
pipe.has_cache_dit = False
except Exception:
return
modules/civitai/api_civitai.py
+9
View File
@@ -54,6 +54,15 @@ def get_civitai(
return JSONResponse(content=[], status_code=200)
def post_civitai(page:str=None):
from modules.civitai import metadata_civitai
result = []
for r in metadata_civitai.civit_search_metadata(title=page, raw=True):
result = r # get the last yielded result
return result
def register_api():
from modules.shared import api
api.add_api_route("/sdapi/v1/civitai", get_civitai, methods=["GET"], response_model=list)
api.add_api_route("/sdapi/v1/civitai", post_civitai, methods=["POST"], response_model=list)
modules/civitai/download_civitai.py
+5 -4
View File
@@ -103,10 +103,11 @@ def download_civit_model_thread(model_name: str, model_url: str, model_path: str
if os.path.isfile(temp_file):
starting_pos = os.path.getsize(temp_file)
headers['Range'] = f'bytes={starting_pos}-'
if token is None or len(token) == 0:
token = shared.opts.civitai_token
if token is not None and len(token) > 0:
headers['Authorization'] = f'Bearer {token}'
if 'civit' in model_url.lower(): # downloader can be used for other urls too
if token is None or len(token) == 0:
token = shared.opts.civitai_token
if (token is not None) and (len(token) > 0):
headers['Authorization'] = f'Bearer {token}'
r = shared.req(model_url, headers=headers, stream=True)
total_size = int(r.headers.get('content-length', 0))
modules/civitai/metadata_civitai.py
+7 -7
View File
@@ -26,7 +26,7 @@ class CivitModel:
self.status = 'Not found'
def civit_update_metadata():
def civit_update_metadata(raw:bool=False):
def create_update_metadata_table(rows: list[CivitModel]):
html = """
<table class="simple-table">
@@ -102,8 +102,8 @@ def civit_update_metadata():
model.status = 'Update available'
break
results.append(model)
yield create_update_metadata_table(results)
return create_update_metadata_table(results)
yield results if raw else create_update_metadata_table(results)
yield results if raw else create_update_metadata_table(results)
def civit_search_model(name, tag, model_type):
@@ -224,7 +224,7 @@ def atomic_civit_search_metadata(item, results):
results.append(result)
def civit_search_metadata(title: str = None):
def civit_search_metadata(title: str = None, raw: bool = False):
def create_search_metadata_table(rows):
html = """
<table class="simple-table">
@@ -258,7 +258,7 @@ def civit_search_metadata(title: str = None):
re_skip = [r.strip() for r in opts.extra_networks_scan_skip.split(',') if len(r.strip()) > 0]
for page in get_pages():
if type(title) == str:
if page.title != title:
if page.title.lower() != title.lower():
continue
if page.name == 'style' or page.name == 'wildcards':
continue
@@ -278,8 +278,8 @@ def civit_search_metadata(title: str = None):
future_items[executor.submit(atomic_civit_search_metadata, fn, results)] = fn
for future in concurrent.futures.as_completed(future_items):
future.result()
yield create_search_metadata_table(results)
yield results if raw else create_search_metadata_table(results)
t1 = time.time()
log.debug(f'CivitAI search metadata: scanned={scanned} skipped={skipped} time={t1-t0:.2f}')
yield create_search_metadata_table(results)
yield results if raw else create_search_metadata_table(results)
modules/control/run.py
+12 -3
View File
@@ -255,6 +255,7 @@ def control_run(state: str = '', # pylint: disable=keyword-arg-before-vararg
prompt: str = '', negative_prompt: str = '', styles: List[str] = [],
steps: int = 20, sampler_index: int = None,
seed: int = -1, subseed: int = -1, subseed_strength: float = 0, seed_resize_from_h: int = -1, seed_resize_from_w: int = -1,
guidance_name: str = 'Default', guidance_scale: float = 6.0, guidance_rescale: float = 0.0, guidance_start: float = 0.0, guidance_stop: float = 1.0,
cfg_scale: float = 6.0, clip_skip: float = 1.0, image_cfg_scale: float = 6.0, diffusers_guidance_rescale: float = 0.7, pag_scale: float = 0.0, pag_adaptive: float = 0.5, cfg_end: float = 1.0,
vae_type: str = 'Full', tiling: bool = False, hidiffusion: bool = False,
detailer_enabled: bool = True, detailer_prompt: str = '', detailer_negative: str = '', detailer_steps: int = 10, detailer_strength: float = 0.3, detailer_resolution: int = 1024,
@@ -306,7 +307,13 @@ def control_run(state: str = '', # pylint: disable=keyword-arg-before-vararg
seed_resize_from_h = seed_resize_from_h,
seed_resize_from_w = seed_resize_from_w,
denoising_strength = denoising_strength,
# advanced
# modular guidance
guidance_name = guidance_name,
guidance_scale = guidance_scale,
guidance_rescale = guidance_rescale,
guidance_start = guidance_start,
guidance_stop = guidance_stop,
# legacy guidance
cfg_scale = cfg_scale,
cfg_end = cfg_end,
clip_skip = clip_skip,
@@ -314,6 +321,7 @@ def control_run(state: str = '', # pylint: disable=keyword-arg-before-vararg
diffusers_guidance_rescale = diffusers_guidance_rescale,
pag_scale = pag_scale,
pag_adaptive = pag_adaptive,
# advanced
vae_type = vae_type,
tiling = tiling,
hidiffusion = hidiffusion,
@@ -380,6 +388,7 @@ def control_run(state: str = '', # pylint: disable=keyword-arg-before-vararg
)
p.state = state
p.is_tile = False
p.init_control = inits or []
p.orig_init_images = inputs
# TODO modernui: monkey-patch for missing tabs.select event
@@ -396,9 +405,9 @@ def control_run(state: str = '', # pylint: disable=keyword-arg-before-vararg
# hires/refine defined outside of main init
vae_scale_factor = sd_vae.get_vae_scale_factor()
if p.enable_hr and (p.hr_resize_x == 0 or p.hr_resize_y == 0):
p.hr_upscale_to_x, p.hr_upscale_to_y = vae_scale_factor * int(p.width_before * p.hr_scale / vae_scale_factor), vae_scale_factor * int(p.height_before * p.hr_scale / vae_scale_factor)
p.hr_upscale_to_x, p.hr_upscale_to_y = int(vae_scale_factor * int(p.width_before * p.hr_scale / vae_scale_factor)), int(vae_scale_factor * int(p.height_before * p.hr_scale / vae_scale_factor))
elif p.enable_hr and (p.hr_upscale_to_x == 0 or p.hr_upscale_to_y == 0):
p.hr_upscale_to_x, p.hr_upscale_to_y = 8 * int(p.hr_resize_x / vae_scale_factor), vae_scale_factor * int(p.hr_resize_y / vae_scale_factor)
p.hr_upscale_to_x, p.hr_upscale_to_y = 8 * int(p.hr_resize_x / vae_scale_factor), int(vae_scale_factor * int(p.hr_resize_y / vae_scale_factor))
global p_extra_args # pylint: disable=global-statement
for k, v in p_extra_args.items():
modules/devices.py
+4
View File
@@ -49,6 +49,10 @@ def has_xpu() -> bool:
return bool(hasattr(torch, 'xpu') and torch.xpu.is_available())
def has_rocm() -> bool:
return bool(torch.version.hip is not None and torch.cuda.is_available())
def has_zluda() -> bool:
if not cuda_ok:
return False
modules/extras.py
+7 -7
View File
@@ -39,24 +39,24 @@ def run_modelmerger(id_task, **kwargs): # pylint: disable=unused-argument
return [*[gr.update() for _ in range(4)], message]
kwargs["models"] = {
"model_a": sd_models.get_closet_checkpoint_match(kwargs.get("primary_model_name", None)).filename,
"model_b": sd_models.get_closet_checkpoint_match(kwargs.get("secondary_model_name", None)).filename,
"model_a": sd_models.get_closest_checkpoint_match(kwargs.get("primary_model_name", None)).filename,
"model_b": sd_models.get_closest_checkpoint_match(kwargs.get("secondary_model_name", None)).filename,
}
if kwargs.get("primary_model_name", None) in [None, 'None']:
return fail("Failed: Merging requires a primary model.")
primary_model_info = sd_models.get_closet_checkpoint_match(kwargs.get("primary_model_name", None))
primary_model_info = sd_models.get_closest_checkpoint_match(kwargs.get("primary_model_name", None))
if kwargs.get("secondary_model_name", None) in [None, 'None']:
return fail("Failed: Merging requires a secondary model.")
secondary_model_info = sd_models.get_closet_checkpoint_match(kwargs.get("secondary_model_name", None))
secondary_model_info = sd_models.get_closest_checkpoint_match(kwargs.get("secondary_model_name", None))
if kwargs.get("tertiary_model_name", None) in [None, 'None'] and kwargs.get("merge_mode", None) in merge_utils.TRIPLE_METHODS:
return fail(f"Failed: Interpolation method ({kwargs.get('merge_mode', None)}) requires a tertiary model.")
tertiary_model_info = sd_models.get_closet_checkpoint_match(kwargs.get("tertiary_model_name", None)) if kwargs.get("merge_mode", None) in merge_utils.TRIPLE_METHODS else None
tertiary_model_info = sd_models.get_closest_checkpoint_match(kwargs.get("tertiary_model_name", None)) if kwargs.get("merge_mode", None) in merge_utils.TRIPLE_METHODS else None
del kwargs["primary_model_name"]
del kwargs["secondary_model_name"]
if kwargs.get("tertiary_model_name", None) is not None:
kwargs["models"] |= {"model_c": sd_models.get_closet_checkpoint_match(kwargs.get("tertiary_model_name", None)).filename}
kwargs["models"] |= {"model_c": sd_models.get_closest_checkpoint_match(kwargs.get("tertiary_model_name", None)).filename}
del kwargs["tertiary_model_name"]
if kwargs.get("alpha_base", None) and kwargs.get("alpha_in_blocks", None) and kwargs.get("alpha_mid_block", None) and kwargs.get("alpha_out_blocks", None):
@@ -204,7 +204,7 @@ def run_model_modules(model_type:str, model_name:str, custom_name:str,
if len(custom_name) == 0:
yield msg("output name is required", err=True)
return
checkpoint_info = sd_models.get_closet_checkpoint_match(model_name)
checkpoint_info = sd_models.get_closest_checkpoint_match(model_name)
if checkpoint_info is None:
yield msg("input model not found", err=True)
return
modules/face/insightface.py
+2 -4
View File
@@ -10,11 +10,9 @@ instightface_mp = None
def get_app(mp_name, threshold=0.5, resolution=640):
global insightface_app, instightface_mp # pylint: disable=global-statement
from installer import install, installed, install_pydantic
from installer import install, installed, install_insightface
if not installed('insightface', reload=False, quiet=True):
install('git+https://github.com/deepinsight/insightface@554a05561cb71cfebb4e012dfea48807f845a0c2#subdirectory=python-package', 'insightface') # insightface==0.7.3 with patches
install('albumentations==1.4.3', ignore=False, reinstall=True)
install_pydantic()
install_insightface()
if not installed('ip_adapter', reload=False, quiet=True):
install('git+https://github.com/tencent-ailab/IP-Adapter.git', 'ip_adapter', ignore=False)
modules/framepack/framepack_load.py
+4
View File
@@ -124,6 +124,10 @@ def load_model(variant:str=None, pipeline:str=None, text_encoder:str=None, text_
sd_models.unload_model_weights()
t0 = time.time()
sd_models.hf_auth_check(model["transformer"]["repo"])
sd_models.hf_auth_check(model["text_encoder"]["repo"])
sd_models.hf_auth_check(model["text_encoder_2"]["repo"])
shared.log.debug(f'FramePack load: module=llm {model["text_encoder"]}')
load_args, quant_args = model_quant.get_dit_args({}, module='TE', device_map=True)
text_encoder = LlamaModel.from_pretrained(model["text_encoder"]["repo"], subfolder=model["text_encoder"]["subfolder"], cache_dir=shared.opts.hfcache_dir, **load_args, **quant_args)
modules/framepack/framepack_vae.py
+4 -1
View File
@@ -85,7 +85,10 @@ def vae_decode(latents, vae_type):
elif vae_type == 'Remote':
return vae_decode_remote(latents)
else: # vae_type == 'Full'
return vae_decode_full(latents)
jobid = shared.state.begin('VAE Decode')
result = vae_decode_full(latents)
shared.state.end(jobid)
return result
def vae_encode(image):
modules/framepack/framepack_worker.py
+10 -9
View File
@@ -54,7 +54,7 @@ def worker(
from modules.framepack.pipeline import hunyuan
from modules.framepack.pipeline import utils
from modules.framepack.pipeline.k_diffusion_hunyuan import sample_hunyuan
from modules.framepack.pipeline import k_diffusion_hunyuan
is_f1 = variant == 'forward-only'
total_generated_frames = 0
@@ -65,14 +65,13 @@ def worker(
metadata['title'] = 'sdnext framepack'
metadata['description'] = f'variant:{variant} seed:{seed} steps:{steps} scale:{cfg_scale} distilled:{cfg_distilled} rescale:{cfg_rescale} shift:{shift} start:{start_weight} end:{end_weight} vision:{vision_weight}'
shared.state.begin('Video')
videojob = shared.state.begin('Video')
shared.state.job_count = 1
text_encoder = shared.sd_model.text_encoder
text_encoder_2 = shared.sd_model.text_encoder_2
tokenizer = shared.sd_model.tokenizer
tokenizer_2 = shared.sd_model.tokenizer_2
vae = shared.sd_model.vae
feature_extractor = shared.sd_model.feature_extractor
image_encoder = shared.sd_model.image_processor
transformer = shared.sd_model.transformer
@@ -196,7 +195,6 @@ def worker(
image_encoder_last_hidden_state = vision_encode(input_image, end_image)
# Sample loop
shared.state.textinfo = 'Sample'
stream.output_queue.push(('progress', (None, 'Start sampling...')))
generator = torch.Generator("cpu").manual_seed(seed)
if is_f1:
@@ -213,6 +211,7 @@ def worker(
llama_vec, llama_vec_n, llama_attention_mask, llama_attention_mask_n, clip_l_pooler, clip_l_pooler_n = text_encode(current_prompt, i=lattent_padding_loop+1)
last_prompt = current_prompt
sammplejob = shared.state.begin('Sample')
lattent_padding_loop += 1
# shared.log.trace(f'FramePack: op=sample section={lattent_padding_loop}/{len(latent_paddings)} frames={total_generated_frames}/{num_frames*len(latent_paddings)} window={latent_window_size} size={num_frames}')
if is_f1:
@@ -244,7 +243,7 @@ def worker(
transformer.initialize_teacache(enable_teacache=use_teacache, num_steps=steps, rel_l1_thresh=shared.opts.teacache_thresh)
t_sample = time.time()
generated_latents = sample_hunyuan(
generated_latents = k_diffusion_hunyuan.sample_hunyuan(
transformer=transformer,
sampler='unipc',
width=width,
@@ -274,7 +273,6 @@ def worker(
dtype=devices.dtype,
callback=step_callback,
)
timer.process.add('sample', time.time()-t_sample)
if is_last_section:
generated_latents = torch.cat([start_latent.to(generated_latents), generated_latents], dim=2)
@@ -287,15 +285,18 @@ def worker(
history_latents = torch.cat([generated_latents.to(history_latents), history_latents], dim=2)
real_history_latents = history_latents[:, :, :total_generated_latent_frames, :, :]
t_vae = time.time()
sd_models.apply_balanced_offload(shared.sd_model)
timer.process.add('sample', time.time()-t_sample)
shared.state.end(sammplejob)
t_vae = time.time()
if history_pixels is None:
history_pixels = framepack_vae.vae_decode(real_history_latents, vae_type=vae_type).cpu()
else:
overlapped_frames = latent_window_size * 4 - 3
if is_f1:
section_latent_frames = latent_window_size * 2
current_pixels = framepack_vae.vae_decode(real_history_latents[:, :, -section_latent_frames:], vae).cpu()
current_pixels = framepack_vae.vae_decode(real_history_latents[:, :, -section_latent_frames:], vae_type=vae_type).cpu()
history_pixels = utils.soft_append_bcthw(history_pixels, current_pixels, overlapped_frames)
else:
section_latent_frames = (latent_window_size * 2 + 1) if is_last_section else (latent_window_size * 2)
@@ -321,4 +322,4 @@ def worker(
stream.output_queue.push(('end', None))
t1 = time.time()
shared.log.info(f'Processed: frames={total_generated_frames} fps={total_generated_frames/(t1-t0):.2f} its={(shared.state.sampling_step)/(t1-t0):.2f} time={t1-t0:.2f} timers={timer.process.dct()} memory={memstats.memory_stats()}')
shared.state.end()
shared.state.end(videojob)
modules/framepack/pipeline/uni_pc_fm.py
+33 -3
View File
@@ -5,7 +5,7 @@
import torch
import numpy as np
from tqdm.auto import trange
@@ -13,6 +13,36 @@ def expand_dims(v, dims):
return v[(...,) + (None,) * (dims - 1)]
torch_linalg_solve = None
def test_solver():
from modules import devices, shared
try:
a = torch.randn(50, 50).to(device=devices.device, dtype=torch.float32)
b = torch.randn(50, 2).to(device=devices.device, dtype=torch.float32)
_x = torch.linalg.solve(a, b)
return True
except Exception as e:
shared.log.debug(f'FramePack: solver=cpu {e}')
return False
def linalg_solve(A, B, device):
global torch_linalg_solve # pylint: disable=global-statement
if torch_linalg_solve is None:
torch_linalg_solve = test_solver()
if torch_linalg_solve:
X = torch.linalg.solve(A, B)
return X
else:
A_np = A.float().cpu().numpy()
B_np = B.float().cpu().numpy()
X_np = np.linalg.solve(A_np, B_np)
X = torch.from_numpy(X_np).to(device=device, dtype=A.dtype)
return X
class FlowMatchUniPC:
def __init__(self, model, extra_args, variant='bh1'):
self.model = model
@@ -78,7 +108,7 @@ class FlowMatchUniPC:
if order == 2:
rhos_p = torch.tensor([0.5], device=b.device)
else:
rhos_p = torch.linalg.solve(R[:-1, :-1], b[:-1])
rhos_p = linalg_solve(R[:-1, :-1], b[:-1], x.device)
else:
D1s = None
rhos_p = None
@@ -86,7 +116,7 @@ class FlowMatchUniPC:
if order == 1:
rhos_c = torch.tensor([0.5], device=b.device)
else:
rhos_c = torch.linalg.solve(R, b)
rhos_c = linalg_solve(R, b, x.device)
x_t_ = expand_dims(t / t_prev_0, dims) * x - expand_dims(h_phi_1, dims) * model_prev_0
modules/generation_parameters_copypaste.py
+5
View File
@@ -229,6 +229,11 @@ def connect_paste(button, local_paste_fields, input_comp, override_settings_comp
res.append(v)
applied[key] = v
else:
if isinstance(v, str) and v.strip() == '' and key in {'Prompt', 'Negative prompt'}:
debug(f'Paste skip empty: "{key}"')
res.append(gr.update())
skipped[key] = v
continue
if should_skip(key):
debug(f'Paste skip: "{key}"="{v}"')
res.append(gr.update())
modules/images_namegen.py
+16 -1
View File
@@ -1,5 +1,6 @@
import re
import os
import time
import uuid
import string
import hashlib
@@ -15,6 +16,7 @@ re_pattern_arg = re.compile(r"(.*)<([^>]*)>$")
re_attention = re.compile(r'[\(*\[*](\w+)(:\d+(\.\d+))?[\)*\]*]|')
re_network = re.compile(r'\<\w+:(\w+)(:\d+(\.\d+))?\>|')
re_brackets = re.compile(r'[\([{})\]]')
re_leading_seq = re.compile(r'^(0*\d+)(?=[-_.\s]|$)')
seq = 0
NOTHING = object()
@@ -33,6 +35,7 @@ class FilenameGenerator:
'hash': lambda self: self.image_hash(),
'image_hash': lambda self: self.image_hash(),
'timestamp': lambda self: getattr(self.p, "job_timestamp", shared.state.job_timestamp),
'epoch': lambda self: int(time.time()),
'job_timestamp': lambda self: getattr(self.p, "job_timestamp", shared.state.job_timestamp),
'model': lambda self: shared.sd_model.sd_checkpoint_info.title if shared.sd_loaded and getattr(shared.sd_model, 'sd_checkpoint_info', None) is not None else '',
@@ -183,12 +186,24 @@ class FilenameGenerator:
debug(f'Filename sanitize: input="{filename}" parts={parts} output="{fn}" ext={ext} max={max_length} len={len(fn)}')
return fn
def safe_int(self, s):
try:
return int(s)
except (ValueError, TypeError):
return 0
def sequence(self, fn):
global seq # pylint: disable=global-statement
x = fn
dirname = os.path.dirname(fn)
if seq == 0:
seq = len(os.listdir(dirname)) if os.path.exists(dirname) and os.path.isdir(dirname) else 0
files = os.listdir(dirname) if os.path.exists(dirname) and os.path.isdir(dirname) else []
files = [f for f in files if os.path.isfile(os.path.join(dirname, f))]
seq_files = len(files)
seq_nums = [re_leading_seq.match(f) for f in files]
seq_nums = [self.safe_int(m.group(1)) for m in seq_nums if m is not None]
seq_num = max(seq_nums) if len(seq_nums) > 0 else 0
seq = max(seq_files, seq_num)
if shared.opts.save_images_add_number or '[seq]' in fn:
if '[seq]' not in fn:
fn = os.path.join(os.path.dirname(fn), f"[seq]-{os.path.basename(fn)}")
modules/img2img.py
+7 -4
View File
@@ -161,10 +161,8 @@ def img2img(id_task: str, state: str, mode: int,
vae_type, tiling, hidiffusion,
detailer_enabled, detailer_prompt, detailer_negative, detailer_steps, detailer_strength, detailer_resolution,
n_iter, batch_size,
cfg_scale, image_cfg_scale,
diffusers_guidance_rescale,
pag_scale, pag_adaptive,
cfg_end,
guidance_name, guidance_scale, guidance_rescale, guidance_start, guidance_stop,
cfg_scale, image_cfg_scale, diffusers_guidance_rescale, pag_scale, pag_adaptive, cfg_end,
refiner_start,
clip_skip,
denoising_strength,
@@ -255,6 +253,11 @@ def img2img(id_task: str, state: str, mode: int,
batch_size=batch_size,
n_iter=n_iter,
steps=steps,
guidance_name=guidance_name,
guidance_scale=guidance_scale,
guidance_rescale=guidance_rescale,
guidance_start=guidance_start,
guidance_stop=guidance_stop,
cfg_scale=cfg_scale,
cfg_end=cfg_end,
clip_skip=clip_skip,
modules/intel/ipex/__init__.py
+2 -2
View File
@@ -152,7 +152,7 @@ def ipex_init(): # pylint: disable=too-many-statements
# torch._int_mm directly uses onednn quantized matmul
# onednn qlinear is a wrapper around onednn quantized matmul
if hasattr(torch.ops, "onednn") and hasattr(torch.ops.onednn, "qlinear_pointwise"):
def onednn_mm(x: torch.Tensor, y: torch.Tensor):
def onednn_mm(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
# supports int8, fp32, fp16, and bf16 matmul with accumulation using a different dtype
# int8 matmul with onednn is slower than 16 bit with dim_size < 4096
return torch.ops.onednn.qlinear_pointwise.default(x, 1.0, 0, y, torch.ones(1, device=y.device), torch.zeros(1, device=y.device), None, 1.0, 0, torch.float32, "none", [], "none")
@@ -165,7 +165,7 @@ def ipex_init(): # pylint: disable=too-many-statements
pass
# Memory:
if 'linux' in sys.platform and "WSL2" in os.popen("uname -a").read():
if "linux" in sys.platform and "WSL2" in os.popen("uname -a").read():
torch.xpu.empty_cache = lambda: None
torch.cuda.empty_cache = torch.xpu.empty_cache
modules/intel/ipex/attention.py
+2 -2
View File
@@ -8,8 +8,8 @@ from functools import cache, wraps
# ARC GPUs can't allocate more than 4GB to a single block so we slice the attention layers
dynamic_attention_slice_rate = float(os.environ.get('IPEX_SDPA_SLICE_TRIGGER_RATE', 1))
dynamic_attention_trigger_rate = float(os.environ.get('IPEX_ATTENTION_SLICE_RATE', 0.5))
dynamic_attention_slice_rate = float(os.environ.get("IPEX_SDPA_SLICE_TRIGGER_RATE", "1"))
dynamic_attention_trigger_rate = float(os.environ.get("IPEX_ATTENTION_SLICE_RATE", "0.5"))
# Find something divisible with the input_tokens
@cache
modules/intel/ipex/hijacks.py
+72 -157
View File
@@ -1,11 +1,9 @@
from typing import Optional
import os
from functools import wraps
from contextlib import nullcontext
import torch
import numpy as np
from modules import devices, errors
from modules import devices
torch_version = torch.__version__[:4]
@@ -16,38 +14,25 @@ torch_version[0], torch_version[1] = int(torch_version[0]), int(torch_version[1]
device_supports_fp64 = torch.xpu.has_fp64_dtype() if hasattr(torch.xpu, "has_fp64_dtype") else torch.xpu.get_device_properties(devices.device).has_fp64
if os.environ.get('IPEX_FORCE_ATTENTION_SLICE', '0') == '0':
if torch_version[0] > 2 or (torch_version[0] == 2 and torch_version[1] >= 7):
use_dynamic_attention = False # torch 2.7 has flash atten support
else:
use_dynamic_attention = True
else:
use_dynamic_attention = bool(os.environ.get('IPEX_FORCE_ATTENTION_SLICE', '0') == '1')
# pylint: disable=protected-access, missing-function-docstring, line-too-long, unnecessary-lambda, no-else-return
class DummyDataParallel(torch.nn.Module): # pylint: disable=missing-class-docstring, unused-argument, too-few-public-methods
def __new__(cls, module, device_ids=None, output_device=None, dim=0): # pylint: disable=unused-argument
if isinstance(device_ids, list) and len(device_ids) > 1:
errors.log.error("IPEX backend doesn't support DataParallel on multiple XPU devices")
return module.to(devices.device)
def return_null_context(*args, **kwargs): # pylint: disable=unused-argument
return nullcontext()
@property
def is_cuda(self):
return self.device.type == "xpu" or self.device.type == "cuda"
def check_device_type(device, device_type: str) -> bool:
if device is None or type(device) not in {str, int, torch.device}:
return False
else:
return bool(torch.device(device).type == device_type)
def check_cuda(device) -> bool:
return bool(isinstance(device, int) or check_device_type(device, "cuda"))
def return_xpu(device): # keep the device instance type, aka return string if the input is string
return devices.device if device is None else f"xpu:{device.split(':')[-1]}" if isinstance(device, str) and ":" in device else f"xpu:{device}" if isinstance(device, int) else torch.device(f"xpu:{device.index}" if device.index is not None else "xpu") if isinstance(device, torch.device) else "xpu"
@@ -95,8 +80,8 @@ def torch_get_autocast_dtype(device_type=None):
# IPEX 2.5 and above has partial support but doesn't really work most of the time.
original_interpolate = torch.nn.functional.interpolate
@wraps(torch.nn.functional.interpolate)
def interpolate(tensor, size=None, scale_factor=None, mode='nearest', align_corners=None, recompute_scale_factor=None, antialias=False): # pylint: disable=too-many-arguments
if mode in {'bicubic', 'bilinear'}:
def interpolate(tensor, size=None, scale_factor=None, mode="nearest", align_corners=None, recompute_scale_factor=None, antialias=False): # pylint: disable=too-many-arguments
if mode in {"bicubic", "bilinear"}:
return_device = tensor.device
return_dtype = tensor.dtype
return original_interpolate(tensor.to("cpu", dtype=torch.float32), size=size, scale_factor=scale_factor, mode=mode,
@@ -106,6 +91,32 @@ def interpolate(tensor, size=None, scale_factor=None, mode='nearest', align_corn
align_corners=align_corners, recompute_scale_factor=recompute_scale_factor, antialias=antialias)
# SwinIR BF16:
original_functional_pad = torch.nn.functional.pad
@wraps(torch.nn.functional.pad)
def functional_pad(input, pad, mode="constant", value=None):
if mode == "reflect" and input.dtype == torch.bfloat16:
return original_functional_pad(input.to(torch.float32), pad, mode=mode, value=value).to(dtype=torch.bfloat16)
else:
return original_functional_pad(input, pad, mode=mode, value=value)
# Diffusers FreeU
original_fft_fftn = torch.fft.fftn
@wraps(torch.fft.fftn)
def fft_fftn(input, s=None, dim=None, norm=None, *, out=None):
return_dtype = input.dtype
return original_fft_fftn(input.to(dtype=torch.float32), s=s, dim=dim, norm=norm, out=out).to(dtype=return_dtype)
# Diffusers FreeU
original_fft_ifftn = torch.fft.ifftn
@wraps(torch.fft.ifftn)
def fft_ifftn(input, s=None, dim=None, norm=None, *, out=None):
return_dtype = input.dtype
return original_fft_ifftn(input.to(dtype=torch.float32), s=s, dim=dim, norm=norm, out=out).to(dtype=return_dtype)
# Diffusers Float64 (Alchemist GPUs doesn't support 64 bit):
original_from_numpy = torch.from_numpy
@wraps(torch.from_numpy)
@@ -115,6 +126,7 @@ def from_numpy(ndarray):
else:
return original_from_numpy(ndarray)
original_as_tensor = torch.as_tensor
@wraps(torch.as_tensor)
def as_tensor(data, dtype=None, device=None):
@@ -126,120 +138,6 @@ def as_tensor(data, dtype=None, device=None):
return original_as_tensor(data, dtype=dtype, device=device)
if not use_dynamic_attention:
original_scaled_dot_product_attention = torch.nn.functional.scaled_dot_product_attention
else:
# 32 bit attention workarounds for Alchemist:
try:
from .attention import dynamic_scaled_dot_product_attention as original_scaled_dot_product_attention
except ImportError:
original_scaled_dot_product_attention = torch.nn.functional.scaled_dot_product_attention
@wraps(torch.nn.functional.scaled_dot_product_attention)
def scaled_dot_product_attention(query: torch.FloatTensor, key: torch.FloatTensor, value: torch.FloatTensor, attn_mask: Optional[torch.FloatTensor] = None, dropout_p: float = 0.0, is_causal: bool = False, scale: Optional[float] = None, enable_gqa: bool = False, **kwargs) -> torch.FloatTensor:
if query.dtype != key.dtype:
key = key.to(dtype=query.dtype)
if query.dtype != value.dtype:
value = value.to(dtype=query.dtype)
if attn_mask is not None and query.dtype != attn_mask.dtype:
attn_mask = attn_mask.to(dtype=query.dtype)
if enable_gqa:
kwargs["enable_gqa"] = enable_gqa
result = original_scaled_dot_product_attention(query, key, value, attn_mask=attn_mask, dropout_p=dropout_p, is_causal=is_causal, scale=scale, **kwargs)
if result.dtype != query.dtype:
result = result.to(dtype=query.dtype)
return result
# Data Type Errors:
original_torch_bmm = torch.bmm
@wraps(torch.bmm)
def torch_bmm(input, mat2, *, out=None):
if input.dtype != mat2.dtype:
mat2 = mat2.to(dtype=input.dtype)
return original_torch_bmm(input, mat2, out=out)
# Diffusers FreeU
original_fft_fftn = torch.fft.fftn
@wraps(torch.fft.fftn)
def fft_fftn(input, s=None, dim=None, norm=None, *, out=None):
return_dtype = input.dtype
return original_fft_fftn(input.to(dtype=torch.float32), s=s, dim=dim, norm=norm, out=out).to(dtype=return_dtype)
# Diffusers FreeU
original_fft_ifftn = torch.fft.ifftn
@wraps(torch.fft.ifftn)
def fft_ifftn(input, s=None, dim=None, norm=None, *, out=None):
return_dtype = input.dtype
return original_fft_ifftn(input.to(dtype=torch.float32), s=s, dim=dim, norm=norm, out=out).to(dtype=return_dtype)
# A1111 FP16
original_functional_group_norm = torch.nn.functional.group_norm
@wraps(torch.nn.functional.group_norm)
def functional_group_norm(input, num_groups, weight=None, bias=None, eps=1e-05):
if weight is not None and input.dtype != weight.data.dtype:
input = input.to(dtype=weight.data.dtype)
if bias is not None and weight is not None and bias.data.dtype != weight.data.dtype:
bias.data = bias.data.to(dtype=weight.data.dtype)
return original_functional_group_norm(input, num_groups, weight=weight, bias=bias, eps=eps)
# A1111 BF16
original_functional_layer_norm = torch.nn.functional.layer_norm
@wraps(torch.nn.functional.layer_norm)
def functional_layer_norm(input, normalized_shape, weight=None, bias=None, eps=1e-05):
if weight is not None and input.dtype != weight.data.dtype:
input = input.to(dtype=weight.data.dtype)
if bias is not None and weight is not None and bias.data.dtype != weight.data.dtype:
bias.data = bias.data.to(dtype=weight.data.dtype)
return original_functional_layer_norm(input, normalized_shape, weight=weight, bias=bias, eps=eps)
# Training
original_functional_linear = torch.nn.functional.linear
@wraps(torch.nn.functional.linear)
def functional_linear(input, weight, bias=None):
if input.dtype != weight.data.dtype:
input = input.to(dtype=weight.data.dtype)
if bias is not None and bias.data.dtype != weight.data.dtype:
bias.data = bias.data.to(dtype=weight.data.dtype)
return original_functional_linear(input, weight, bias=bias)
original_functional_conv1d = torch.nn.functional.conv1d
@wraps(torch.nn.functional.conv1d)
def functional_conv1d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
if input.dtype != weight.data.dtype:
input = input.to(dtype=weight.data.dtype)
if bias is not None and bias.data.dtype != weight.data.dtype:
bias.data = bias.data.to(dtype=weight.data.dtype)
return original_functional_conv1d(input, weight, bias=bias, stride=stride, padding=padding, dilation=dilation, groups=groups)
original_functional_conv2d = torch.nn.functional.conv2d
@wraps(torch.nn.functional.conv2d)
def functional_conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
if input.dtype != weight.data.dtype:
input = input.to(dtype=weight.data.dtype)
if bias is not None and bias.data.dtype != weight.data.dtype:
bias.data = bias.data.to(dtype=weight.data.dtype)
return original_functional_conv2d(input, weight, bias=bias, stride=stride, padding=padding, dilation=dilation, groups=groups)
# LTX Video
original_functional_conv3d = torch.nn.functional.conv3d
@wraps(torch.nn.functional.conv3d)
def functional_conv3d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
if input.dtype != weight.data.dtype:
input = input.to(dtype=weight.data.dtype)
if bias is not None and bias.data.dtype != weight.data.dtype:
bias.data = bias.data.to(dtype=weight.data.dtype)
return original_functional_conv3d(input, weight, bias=bias, stride=stride, padding=padding, dilation=dilation, groups=groups)
# SwinIR BF16:
original_functional_pad = torch.nn.functional.pad
@wraps(torch.nn.functional.pad)
def functional_pad(input, pad, mode='constant', value=None):
if mode == 'reflect' and input.dtype == torch.bfloat16:
return original_functional_pad(input.to(torch.float32), pad, mode=mode, value=value).to(dtype=torch.bfloat16)
else:
return original_functional_pad(input, pad, mode=mode, value=value)
original_torch_tensor = torch.tensor
@wraps(torch.tensor)
def torch_tensor(data, *args, dtype=None, device=None, **kwargs):
@@ -254,20 +152,20 @@ def torch_tensor(data, *args, dtype=None, device=None, **kwargs):
dtype = torch.float32
return original_torch_tensor(data, *args, dtype=dtype, device=device, **kwargs)
torch.Tensor.original_Tensor_to = torch.Tensor.to
@wraps(torch.Tensor.to)
def Tensor_to(self, device=None, *args, **kwargs):
global device_supports_fp64
if check_cuda(device):
if not device_supports_fp64 and kwargs.get("dtype", None) == torch.float64:
device = return_xpu(device)
if not device_supports_fp64:
if kwargs.get("dtype", None) == torch.float64 and ((device is None and self.device.type == "xpu") or (device is not None and torch.device(device).type == "xpu")):
kwargs["dtype"] = torch.float32
return self.original_Tensor_to(return_xpu(device), *args, **kwargs)
else:
if not device_supports_fp64:
if kwargs.get("dtype", None) == torch.float64 and ((device is None and self.device.type == "xpu") or (device is not None and torch.device(device).type == "xpu")):
kwargs["dtype"] = torch.float32
elif device == torch.float64 and self.device.type == "xpu":
device = torch.float32
return self.original_Tensor_to(device, *args, **kwargs)
elif device == torch.float64 and self.device.type == "xpu":
device = torch.float32
return self.original_Tensor_to(device, *args, **kwargs)
original_Tensor_cuda = torch.Tensor.cuda
@wraps(torch.Tensor.cuda)
@@ -277,6 +175,7 @@ def Tensor_cuda(self, device=None, *args, **kwargs):
else:
return original_Tensor_cuda(self, device, *args, **kwargs)
original_Tensor_pin_memory = torch.Tensor.pin_memory
@wraps(torch.Tensor.pin_memory)
def Tensor_pin_memory(self, device=None, *args, **kwargs):
@@ -285,6 +184,7 @@ def Tensor_pin_memory(self, device=None, *args, **kwargs):
else:
return original_Tensor_pin_memory(self, device, *args, **kwargs)
original_UntypedStorage_init = torch.UntypedStorage.__init__
@wraps(torch.UntypedStorage.__init__)
def UntypedStorage_init(*args, device=None, **kwargs):
@@ -293,6 +193,7 @@ def UntypedStorage_init(*args, device=None, **kwargs):
else:
return original_UntypedStorage_init(*args, device=device, **kwargs)
if torch_version[0] > 2 or (torch_version[0] == 2 and torch_version[1] >= 4):
original_UntypedStorage_to = torch.UntypedStorage.to
@wraps(torch.UntypedStorage.to)
@@ -310,6 +211,7 @@ if torch_version[0] > 2 or (torch_version[0] == 2 and torch_version[1] >= 4):
else:
return original_UntypedStorage_cuda(self, device=device, non_blocking=non_blocking, **kwargs)
original_torch_empty = torch.empty
@wraps(torch.empty)
def torch_empty(*args, device=None, **kwargs):
@@ -318,15 +220,17 @@ def torch_empty(*args, device=None, **kwargs):
else:
return original_torch_empty(*args, device=device, **kwargs)
original_torch_randn = torch.randn
@wraps(torch.randn)
def torch_randn(*args, device=None, dtype=None, **kwargs):
if dtype is bytes:
dtype = None
if check_cuda(device):
return original_torch_randn(*args, device=return_xpu(device), **kwargs)
return original_torch_randn(*args, device=return_xpu(device), dtype=dtype, **kwargs)
else:
return original_torch_randn(*args, device=device, **kwargs)
return original_torch_randn(*args, device=device, dtype=dtype, **kwargs)
original_torch_ones = torch.ones
@wraps(torch.ones)
@@ -336,6 +240,7 @@ def torch_ones(*args, device=None, **kwargs):
else:
return original_torch_ones(*args, device=device, **kwargs)
original_torch_zeros = torch.zeros
@wraps(torch.zeros)
def torch_zeros(*args, device=None, **kwargs):
@@ -344,6 +249,7 @@ def torch_zeros(*args, device=None, **kwargs):
else:
return original_torch_zeros(*args, device=device, **kwargs)
original_torch_full = torch.full
@wraps(torch.full)
def torch_full(*args, device=None, **kwargs):
@@ -352,6 +258,7 @@ def torch_full(*args, device=None, **kwargs):
else:
return original_torch_full(*args, device=device, **kwargs)
original_torch_linspace = torch.linspace
@wraps(torch.linspace)
def torch_linspace(*args, device=None, **kwargs):
@@ -360,6 +267,7 @@ def torch_linspace(*args, device=None, **kwargs):
else:
return original_torch_linspace(*args, device=device, **kwargs)
original_torch_eye = torch.eye
@wraps(torch.eye)
def torch_eye(*args, device=None, **kwargs):
@@ -368,6 +276,7 @@ def torch_eye(*args, device=None, **kwargs):
else:
return original_torch_eye(*args, device=device, **kwargs)
original_torch_load = torch.load
@wraps(torch.load)
def torch_load(f, map_location=None, *args, **kwargs):
@@ -376,6 +285,7 @@ def torch_load(f, map_location=None, *args, **kwargs):
else:
return original_torch_load(f, *args, map_location=map_location, **kwargs)
@wraps(torch.cuda.synchronize)
def torch_cuda_synchronize(device=None):
if check_cuda(device):
@@ -383,6 +293,7 @@ def torch_cuda_synchronize(device=None):
else:
return torch.xpu.synchronize(device)
@wraps(torch.cuda.device)
def torch_cuda_device(device):
if check_cuda(device):
@@ -390,6 +301,7 @@ def torch_cuda_device(device):
else:
return torch.xpu.device(device)
@wraps(torch.cuda.set_device)
def torch_cuda_set_device(device):
if check_cuda(device):
@@ -397,6 +309,7 @@ def torch_cuda_set_device(device):
else:
torch.xpu.set_device(device)
# torch.Generator has to be a class for isinstance checks
original_torch_Generator = torch.Generator
class torch_Generator(original_torch_Generator):
@@ -434,24 +347,14 @@ def ipex_hijacks():
torch.Generator = torch_Generator
torch._C.Generator = torch_Generator
torch.backends.cuda.sdp_kernel = return_null_context
torch.nn.DataParallel = DummyDataParallel
torch.UntypedStorage.is_cuda = is_cuda
torch.amp.autocast_mode.autocast.__init__ = autocast_init
torch.nn.functional.interpolate = interpolate
torch.nn.functional.scaled_dot_product_attention = scaled_dot_product_attention
torch.nn.functional.group_norm = functional_group_norm
torch.nn.functional.layer_norm = functional_layer_norm
torch.nn.functional.linear = functional_linear
torch.nn.functional.conv1d = functional_conv1d
torch.nn.functional.conv2d = functional_conv2d
torch.nn.functional.conv3d = functional_conv3d
torch.nn.functional.pad = functional_pad
torch.bmm = torch_bmm
torch.fft.fftn = fft_fftn
torch.fft.ifftn = fft_ifftn
if not device_supports_fp64:
torch.from_numpy = from_numpy
torch.as_tensor = as_tensor
@@ -462,6 +365,18 @@ def ipex_hijacks():
except Exception:
pass
if os.environ.get("IPEX_FORCE_ATTENTION_SLICE", "0") == "0":
if torch_version[0] > 2 or (torch_version[0] == 2 and torch_version[1] >= 7):
use_dynamic_attention = False # torch 2.7 has flash atten support
else:
use_dynamic_attention = True
else:
use_dynamic_attention = bool(os.environ.get("IPEX_FORCE_ATTENTION_SLICE", "0") == "1")
if use_dynamic_attention:
from .attention import dynamic_scaled_dot_product_attention
torch.nn.functional.scaled_dot_product_attention = dynamic_scaled_dot_product_attention
# AMP:
torch.amp.grad_scaler.GradScaler.__init__ = GradScaler_init
torch.is_autocast_enabled = torch_is_autocast_enabled
modules/interrogate/vqa.py
+48 -23
View File
@@ -13,7 +13,7 @@ from modules import shared, devices, errors, model_quant, sd_models, sd_models_c
processor = None
model = None
loaded: str = None
quant_args = {}
quant_args = None
vlm_default = "Alibaba Qwen 2.5 VL 4B"
vlm_models = {
"Google Gemma 3 4B": "google/gemma-3-4b-it",
@@ -27,12 +27,18 @@ vlm_models = {
"Apple FastVLM 0.5B": "apple/FastVLM-0.5B",
"Apple FastVLM 1.5B": "apple/FastVLM-1.5B",
"Apple FastVLM 7B": "apple/FastVLM-7B",
"Microsoft Florence 2 Base": "microsoft/Florence-2-base-ft", # 0.5GB
"Microsoft Florence 2 Large": "microsoft/Florence-2-large-ft", # 1.5GB
"MiaoshouAI PromptGen 1.5 Base": "MiaoshouAI/Florence-2-base-PromptGen-v1.5@c06a5f02cc6071a5d65ee5d294cf3732d3097540", # 1.1GB
"MiaoshouAI PromptGen 1.5 Large": "MiaoshouAI/Florence-2-large-PromptGen-v1.5@28a42440e39c9c32b83f7ae74ec2b3d1540404f0", # 3.3GB
"MiaoshouAI PromptGen 2.0 Base": "MiaoshouAI/Florence-2-base-PromptGen-v2.0", # 1.1GB
"MiaoshouAI PromptGen 2.0 Large": "MiaoshouAI/Florence-2-large-PromptGen-v2.0", # 3.3GB
# "Microsoft Florence 2 Base": "microsoft/Florence-2-base-ft", # 0.5GB
# "Microsoft Florence 2 Large": "microsoft/Florence-2-large-ft", # 1.5GB
"Microsoft Florence 2 Base": "florence-community/Florence-2-base-ft", # 0.5GB
"Microsoft Florence 2 Large": "florence-community/Florence-2-large-ft", # 1.5GB
#"MiaoshouAI PromptGen 1.5 Base": "MiaoshouAI/Florence-2-base-PromptGen-v1.5@c06a5f02cc6071a5d65ee5d294cf3732d3097540", # 1.1GB
#"MiaoshouAI PromptGen 1.5 Large": "MiaoshouAI/Florence-2-large-PromptGen-v1.5@28a42440e39c9c32b83f7ae74ec2b3d1540404f0", # 3.3GB
#"MiaoshouAI PromptGen 2.0 Base": "MiaoshouAI/Florence-2-base-PromptGen-v2.0", # 1.1GB
#"MiaoshouAI PromptGen 2.0 Large": "MiaoshouAI/Florence-2-large-PromptGen-v2.0", # 3.3GB
"MiaoshouAI PromptGen 1.5 Base": "Disty0/Florence-2-base-PromptGen-v1.5", # 0.5GB
"MiaoshouAI PromptGen 1.5 Large": "Disty0/Florence-2-large-PromptGen-v1.5", # 1.5GB
"MiaoshouAI PromptGen 2.0 Base": "Disty0/Florence-2-base-PromptGen-v2.0", # 0.5GB
"MiaoshouAI PromptGen 2.0 Large": "Disty0/Florence-2-large-PromptGen-v2.0", # 1.5GB
"CogFlorence 2.0 Large": "thwri/CogFlorence-2-Large-Freeze", # 1.6GB
"CogFlorence 2.2 Large": "thwri/CogFlorence-2.2-Large", # 1.6GB
"Moondream 2": "vikhyatk/moondream2", # 3.7GB
@@ -88,14 +94,19 @@ def b64(image):
def clean(response, question):
strip = ['---', '\r', '\t', '**', '"', '“', '”', 'Assistant:', 'Caption:', '<|im_end|>', '<pad>']
if isinstance(response, dict):
if 'task' in response:
response = response['task']
if isinstance(response, str):
response = response.strip()
elif isinstance(response, dict):
if 'answer' in response:
response = response['answer']
response = json.dumps(response)
if isinstance(response, list):
elif 'task' in response:
response = response['task']
else:
response = json.dumps(response)
elif isinstance(response, list):
response = response[0]
else:
response = str(response)
question = question.replace('<', '').replace('>', '').replace('_', ' ')
if question in response:
response = response.split(question, 1)[1]
@@ -354,7 +365,6 @@ def smol(question: str, image: Image.Image, repo: str = None, system_prompt: str
repo,
cache_dir=shared.opts.hfcache_dir,
torch_dtype=devices.dtype,
_attn_implementation="eager",
**quant_args,
)
processor = transformers.AutoProcessor.from_pretrained(repo, cache_dir=shared.opts.hfcache_dir)
@@ -524,11 +534,13 @@ def moondream(question: str, image: Image.Image, repo: str = None):
def florence(question: str, image: Image.Image, repo: str = None, revision: str = None):
global processor, model, loaded # pylint: disable=global-statement
_get_imports = transformers.dynamic_module_utils.get_imports
def get_imports(f):
R = _get_imports(f)
if "flash_attn" in R:
R.remove("flash_attn") # flash_attn is optional
return R
revision = None
if '@' in repo:
repo, revision = repo.split('@')
@@ -536,6 +548,7 @@ def florence(question: str, image: Image.Image, repo: str = None, revision: str
shared.log.debug(f'Interrogate load: vlm="{repo}" path="{shared.opts.hfcache_dir}"')
transformers.dynamic_module_utils.get_imports = get_imports
model = None
"""
model = transformers.AutoModelForCausalLM.from_pretrained(
repo,
trust_remote_code=True,
@@ -544,6 +557,15 @@ def florence(question: str, image: Image.Image, repo: str = None, revision: str
cache_dir=shared.opts.hfcache_dir,
**quant_args,
)
"""
model = transformers.Florence2ForConditionalGeneration.from_pretrained(
repo,
dtype=torch.bfloat16,
revision=revision,
torch_dtype=devices.dtype,
cache_dir=shared.opts.hfcache_dir,
**quant_args,
)
processor = transformers.AutoProcessor.from_pretrained(repo, trust_remote_code=True, revision=revision, cache_dir=shared.opts.hfcache_dir)
transformers.dynamic_module_utils.get_imports = _get_imports
loaded = repo
@@ -607,25 +629,28 @@ def interrogate(question:str='', system_prompt:str=None, prompt:str=None, image:
global quant_args # pylint: disable=global-statement
jobid = shared.state.begin('Interrogate LLM')
t0 = time.time()
quant_args = model_quant.create_config(module='LLM')
if quant_args is None:
quant_args = model_quant.create_config(module='LLM')
model_name = model_name or shared.opts.interrogate_vlm_model
if isinstance(image, list):
image = image[0] if len(image) > 0 else None
if isinstance(image, dict) and 'name' in image:
image = Image.open(image['name'])
if image is None:
return ''
if image.width > 768 or image.height > 768:
image.thumbnail((768, 768), Image.Resampling.LANCZOS)
if image.mode != 'RGB':
image = image.convert('RGB')
if isinstance(image, Image.Image):
if image.width > 768 or image.height > 768:
image.thumbnail((768, 768), Image.Resampling.LANCZOS)
if image.mode != 'RGB':
image = image.convert('RGB')
if prompt is not None and len(prompt) > 0:
question = prompt
if len(question) < 2:
question = "Describe the image."
"""
if shared.sd_loaded:
from modules.sd_models import apply_balanced_offload # prevent circular import
apply_balanced_offload(shared.sd_model)
"""
from modules import modelloader
modelloader.hf_login()
@@ -638,9 +663,9 @@ def interrogate(question:str='', system_prompt:str=None, prompt:str=None, image:
if vqa_model is None:
shared.log.error(f'Interrogate: type=vlm model="{model_name}" unknown')
return ''
if image is None:
shared.log.error(f'Interrogate: type=vlm model="{model_name}" no input image')
return ''
# if image is None:
# shared.log.error(f'Interrogate: type=vlm model="{model_name}" no input image')
# return ''
if 'git' in vqa_model.lower():
answer = git(question, image, vqa_model)
modules/ipadapter.py
+12
View File
@@ -354,6 +354,18 @@ def apply(pipe, p: processing.StableDiffusionProcessing, adapter_names=[], adapt
adapter_scales[i] = 0.00
pipe.set_ip_adapter_scale(adapter_scales if len(adapter_scales) > 1 else adapter_scales[0])
ip_str = [f'{os.path.splitext(adapter)[0]}:{scale}:{start}:{end}:{crop}' for adapter, scale, start, end, crop in zip(adapter_names, adapter_scales, adapter_starts, adapter_ends, adapter_crops)]
if hasattr(pipe, 'transformer') and 'Nunchaku' in pipe.transformer.__class__.__name__:
if isinstance(repos, str):
sd_models.clear_caches(full=True)
import accelerate
accelerate.hooks.remove_hook_from_module(pipe.transformer, recurse=True)
pipe.transformer = pipe.transformer.to(devices.device)
from nunchaku.models.ip_adapter.diffusers_adapters import apply_IPA_on_pipe
apply_IPA_on_pipe(pipe, ip_adapter_scale=adapter_scales[0], repo_id=repos)
pipe = sd_models.apply_balanced_offload(pipe)
shared.log.debug(f'IP adapter load: engine=nunchaku scale={adapter_scales[0]} repo="{repos}"')
else:
shared.log.error('IP adapter: Nunchaku only supports single adapter')
p.task_args['ip_adapter_image'] = crop_images(adapter_images, adapter_crops)
if len(adapter_masks) > 0:
p.cross_attention_kwargs = { 'ip_adapter_masks': adapter_masks }
modules/loader.py
+19 -9
View File
@@ -51,10 +51,15 @@ if torch.__version__.startswith('2.5.0'):
errors.log.warning(f'Disabling cuDNN for SDP on torch={torch.__version__}')
torch.backends.cuda.enable_cudnn_sdp(False)
try:
import intel_extension_for_pytorch as ipex # pylint: disable=import-error, unused-import
import intel_extension_for_pytorch as ipex # pylint: disable=import-error,unused-import
errors.log.debug(f'Load IPEX=={ipex.__version__}')
except Exception:
pass
try:
import torch.distributed.distributed_c10d as _c10d # pylint: disable=unused-import,ungrouped-imports
except Exception:
errors.log.warning('Loader: torch is not built with distributed support')
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
warnings.filterwarnings(action="ignore", category=UserWarning, module="torchvision")
@@ -123,15 +128,20 @@ timer.startup.record("pydantic")
import tqdm as tqdm_lib # pylint: disable=C0411
from tqdm.rich import tqdm # pylint: disable=W0611,C0411
import diffusers.utils.import_utils # pylint: disable=W0611,C0411
diffusers.utils.import_utils._k_diffusion_available = True # pylint: disable=protected-access # monkey-patch since we use k-diffusion from git
diffusers.utils.import_utils._k_diffusion_version = '0.0.12' # pylint: disable=protected-access
try:
import diffusers.utils.import_utils # pylint: disable=W0611,C0411
diffusers.utils.import_utils._k_diffusion_available = True # pylint: disable=protected-access # monkey-patch since we use k-diffusion from git
diffusers.utils.import_utils._k_diffusion_version = '0.0.12' # pylint: disable=protected-access
import diffusers # pylint: disable=W0611,C0411
import diffusers.loaders.single_file # pylint: disable=W0611,C0411
diffusers.loaders.single_file.logging.tqdm = partial(tqdm, unit='C')
logging.getLogger("diffusers.loaders.single_file").setLevel(logging.ERROR)
timer.startup.record("diffusers")
import diffusers # pylint: disable=W0611,C0411
import diffusers.loaders.single_file # pylint: disable=W0611,C0411
diffusers.loaders.single_file.logging.tqdm = partial(tqdm, unit='C')
logging.getLogger("diffusers.loaders.single_file").setLevel(logging.ERROR)
timer.startup.record("diffusers")
except Exception as e:
errors.log.error(f'Loader: diffusers=={diffusers.__version__ if "diffusers" in sys.modules else None} {e}')
errors.log.error('Please restart re-run the installer')
sys.exit(1)
import huggingface_hub # pylint: disable=W0611,C0411
timer.startup.record("hfhub")
modules/lora/extra_networks_lora.py
+36 -13
View File
@@ -2,7 +2,7 @@ from typing import List
import os
import re
import numpy as np
from modules.lora import networks, lora_overrides, lora_load
from modules.lora import networks, lora_overrides, lora_load, lora_diffusers
from modules.lora import lora_common as l
from modules import extra_networks, shared, sd_models
@@ -83,21 +83,34 @@ def infotext(p):
p.extra_generation_params["LoRA hashes"] = ", ".join(network_hashes)
def to_float(value):
try:
return float(value)
except (ValueError, TypeError):
return value
def parse(p, params_list, step=0):
names = []
te_multipliers = []
unet_multipliers = []
dyn_dims = []
lora_modules = []
for params in params_list:
assert params.items
names.append(params.positional[0])
te_multiplier = params.named.get("te", params.positional[1] if len(params.positional) > 1 else shared.opts.extra_networks_default_multiplier)
default_multiplier = params.positional[1] if len(params.positional) > 1 else shared.opts.extra_networks_default_multiplier
default_multiplier = to_float(default_multiplier)
if isinstance(default_multiplier, str) and "@" not in default_multiplier:
default_multiplier = shared.opts.extra_networks_default_multiplier
te_multiplier = params.named.get("te", default_multiplier)
if isinstance(te_multiplier, str) and "@" in te_multiplier:
te_multiplier = get_stepwise(te_multiplier, step, p.steps)
else:
te_multiplier = float(te_multiplier)
unet_multiplier = [params.positional[2] if len(params.positional) > 2 else te_multiplier] * 3
unet_multiplier = [params.named.get("unet", unet_multiplier[0])] * 3
te_multiplier = to_float(te_multiplier)
unet_multiplier = 3 * [params.named.get("unet", te_multiplier)] # fill all 3 with same value
unet_multiplier[0] = params.named.get("in", unet_multiplier[0])
unet_multiplier[1] = params.named.get("mid", unet_multiplier[1])
unet_multiplier[2] = params.named.get("out", unet_multiplier[2])
@@ -105,13 +118,23 @@ def parse(p, params_list, step=0):
if isinstance(unet_multiplier[i], str) and "@" in unet_multiplier[i]:
unet_multiplier[i] = get_stepwise(unet_multiplier[i], step, p.steps)
else:
unet_multiplier[i] = float(unet_multiplier[i])
dyn_dim = int(params.positional[3]) if len(params.positional) > 3 else None
dyn_dim = int(params.named["dyn"]) if "dyn" in params.named else dyn_dim
unet_multiplier[i] = to_float(unet_multiplier[i])
dyn_dim = int(params.named["dyn"]) if "dyn" in params.named else None
te_multipliers.append(te_multiplier)
unet_multipliers.append(unet_multiplier)
dyn_dims.append(dyn_dim)
return names, te_multipliers, unet_multipliers, dyn_dims
lora_module = []
if 'high' in params.positional or 'HIGH 14B' in params.positional[0]:
lora_module.append('transformer')
if 'low' in params.positional or 'LOW 14B' in params.positional[0]:
lora_module.append('transformer_2')
if params.named.get('module', None) is not None:
lora_module.append(params.named['module'].lower())
lora_modules.append(lora_module)
return names, te_multipliers, unet_multipliers, dyn_dims, lora_modules
def unload_diffusers():
@@ -168,7 +191,7 @@ class ExtraNetworkLora(extra_networks.ExtraNetwork):
if len(params_list) > 0 and not self.active: # activate patches once
self.active = True
self.model = shared.opts.sd_model_checkpoint
names, te_multipliers, unet_multipliers, dyn_dims = parse(p, params_list, step)
names, te_multipliers, unet_multipliers, dyn_dims, lora_modules = parse(p, params_list, step)
requested = self.signature(names, te_multipliers, unet_multipliers)
load_method = lora_overrides.get_method()
@@ -181,7 +204,7 @@ class ExtraNetworkLora(extra_networks.ExtraNetwork):
has_changed = False # diffusers handles its own loading
if len(exclude) == 0:
jobid = shared.state.begin('LoRA')
lora_load.network_load(names, te_multipliers, unet_multipliers, dyn_dims) # load only on first call
lora_load.network_load(names, te_multipliers, unet_multipliers, dyn_dims, lora_modules) # load only on first call
sd_models.set_diffuser_offload(shared.sd_model, op="model")
shared.state.end(jobid)
elif load_method == 'nunchaku':
@@ -208,7 +231,7 @@ class ExtraNetworkLora(extra_networks.ExtraNetwork):
shared.log.info(f'Network load: type=LoRA apply={[n.name for n in l.loaded_networks]} method={load_method} mode={"fuse" if shared.opts.lora_fuse_diffusers else "backup"} te={te_multipliers} unet={unet_multipliers} time={l.timer.summary}')
def deactivate(self, p):
if len(lora_load.diffuser_loaded) > 0:
if len(lora_diffusers.diffuser_loaded) > 0:
if not (shared.compiled_model_state is not None and shared.compiled_model_state.is_compiled is True):
unload_diffusers()
if self.active and l.debug:
modules/lora/lora_apply.py
+62 -8
View File
@@ -47,6 +47,17 @@ def network_backup_weights(self: Union[torch.nn.Conv2d, torch.nn.Linear, torch.n
self.network_weights_backup = weight.clone().to(devices.cpu)
if hasattr(self, "sdnq_dequantizer"):
self.sdnq_dequantizer_backup = self.sdnq_dequantizer.to(devices.cpu)
self.sdnq_scale_backup = self.scale.clone().to(devices.cpu)
if self.zero_point is not None:
self.sdnq_zero_point_backup = self.zero_point.clone().to(devices.cpu)
else:
self.sdnq_zero_point_backup = None
if self.svd_up is not None:
self.sdnq_svd_up_backup = self.svd_up.clone().to(devices.cpu)
self.sdnq_svd_down_backup = self.svd_down.clone().to(devices.cpu)
else:
self.sdnq_svd_up_backup = None
self.sdnq_svd_down_backup = None
if bias_backup is None:
if getattr(self, 'bias', None) is not None:
@@ -80,9 +91,23 @@ def network_calc_weights(self: Union[torch.nn.Conv2d, torch.nn.Linear, torch.nn.
try:
t0 = time.time()
if hasattr(self, "sdnq_dequantizer_backup"):
weight = self.sdnq_dequantizer_backup.to(devices.device)(self.weight.to(devices.device), skip_quantized_matmul=self.sdnq_dequantizer_backup.use_quantized_matmul)
weight = self.sdnq_dequantizer_backup.to(devices.device)(
self.weight.to(devices.device),
self.sdnq_scale_backup.to(devices.device),
self.sdnq_zero_point_backup.to(devices.device) if self.sdnq_zero_point_backup is not None else None,
self.sdnq_svd_up_backup.to(devices.device) if self.sdnq_svd_up_backup is not None else None,
self.sdnq_svd_down_backup.to(devices.device) if self.sdnq_svd_down_backup is not None else None,
skip_quantized_matmul=self.sdnq_dequantizer_backup.use_quantized_matmul
)
elif hasattr(self, "sdnq_dequantizer"):
weight = self.sdnq_dequantizer.to(devices.device)(self.weight.to(devices.device), skip_quantized_matmul=self.sdnq_dequantizer.use_quantized_matmul)
weight = self.sdnq_dequantizer.to(devices.device)(
self.weight.to(devices.device),
self.scale.to(devices.device),
self.zero_point.to(devices.device) if self.zero_point is not None else None,
self.svd_up.to(devices.device) if self.svd_up is not None else None,
self.svd_down.to(devices.device) if self.svd_down is not None else None,
skip_quantized_matmul=self.sdnq_dequantizer.use_quantized_matmul
)
else:
weight = self.weight.to(devices.device) # must perform calc on gpu due to performance
updown, ex_bias = module.calc_updown(weight)
@@ -142,18 +167,37 @@ def network_add_weights(self: Union[torch.nn.Conv2d, torch.nn.Linear, torch.nn.G
try:
from modules.sdnq import sdnq_quantize_layer
if hasattr(self, "sdnq_dequantizer_backup"):
sdnq_dequantizer = self.sdnq_dequantizer_backup.to(devices.device)
weights_dtype = self.sdnq_dequantizer_backup.weights_dtype
dequant_weight = self.sdnq_dequantizer_backup.to(devices.device)(
model_weights.to(devices.device),
self.sdnq_scale_backup.to(devices.device),
self.sdnq_zero_point_backup.to(devices.device) if self.sdnq_zero_point_backup is not None else None,
self.sdnq_svd_up_backup.to(devices.device) if self.sdnq_svd_up_backup is not None else None,
self.sdnq_svd_down_backup.to(devices.device) if self.sdnq_svd_down_backup is not None else None,
skip_quantized_matmul=self.sdnq_dequantizer_backup.use_quantized_matmul
)
else:
sdnq_dequantizer = self.sdnq_dequantizer.to(devices.device)
dequant_weight = sdnq_dequantizer(model_weights.to(devices.device), skip_quantized_matmul=sdnq_dequantizer.use_quantized_matmul)
weights_dtype = self.sdnq_dequantizer.weights_dtype
dequant_weight = self.sdnq_dequantizer.to(devices.device)(
model_weights.to(devices.device),
self.scale.to(devices.device),
self.zero_point.to(devices.device) if self.zero_point is not None else None,
self.svd_up.to(devices.device) if self.svd_up is not None else None,
self.svd_down.to(devices.device) if self.svd_down is not None else None,
skip_quantized_matmul=self.sdnq_dequantizer.use_quantized_matmul
)
new_weight = dequant_weight.to(devices.device, dtype=torch.float32) + lora_weights.to(devices.device, dtype=torch.float32)
self.weight = torch.nn.Parameter(new_weight, requires_grad=False)
self.sdnq_dequantizer = None
del self.sdnq_dequantizer, self.scale, self.zero_point, self.svd_up, self.svd_down
self = sdnq_quantize_layer(
self,
sdnq_dequantizer.weights_dtype,
weights_dtype=weights_dtype,
torch_dtype=devices.dtype,
group_size=shared.opts.sdnq_quantize_weights_group_size,
svd_rank=shared.opts.sdnq_svd_rank,
svd_steps=shared.opts.sdnq_svd_steps,
use_svd=shared.opts.sdnq_use_svd,
quant_conv=shared.opts.sdnq_quantize_conv_layers,
use_quantized_matmul=shared.opts.sdnq_use_quantized_matmul,
use_quantized_matmul_conv=shared.opts.sdnq_use_quantized_matmul_conv,
@@ -231,7 +275,17 @@ def network_apply_weights(self: Union[torch.nn.Conv2d, torch.nn.Linear, torch.nn
self.weight = torch.nn.Parameter(weights_backup.to(device), requires_grad=False)
if hasattr(self, "sdnq_dequantizer_backup"):
self.sdnq_dequantizer = self.sdnq_dequantizer_backup.to(device)
del self.sdnq_dequantizer_backup
self.scale = torch.nn.Parameter(self.sdnq_scale_backup.to(device), requires_grad=False)
if self.sdnq_zero_point_backup is not None:
self.zero_point = torch.nn.Parameter(self.sdnq_zero_point_backup.to(device), requires_grad=False)
else:
self.zero_point = None
if self.sdnq_svd_up_backup is not None:
self.svd_up = torch.nn.Parameter(self.sdnq_svd_up_backup.to(device), requires_grad=False)
self.svd_down = torch.nn.Parameter(self.sdnq_svd_down_backup.to(device), requires_grad=False)
else:
self.svd_up, self.svd_down = None, None
del self.sdnq_dequantizer_backup, self.sdnq_scale_backup, self.sdnq_zero_point_backup, self.sdnq_svd_up_backup, self.sdnq_svd_down_backup
if bias_backup is not None:
self.bias = None
modules/lora/lora_diffusers.py
+92
View File
@@ -0,0 +1,92 @@
from typing import Union
import os
import time
import diffusers
from modules import shared, errors
from modules.lora import network
from modules.lora import lora_common as l
diffuser_loaded = []
diffuser_scales = []
def load_per_module(sd_model: diffusers.DiffusionPipeline, filename: str, adapter_name: str, lora_modules: list[str]):
shared.log.debug(f'LoRA load: modules={lora_modules}')
try:
state_dict = sd_model.lora_state_dict(filename)
if isinstance(state_dict, tuple) and len(state_dict) == 2:
state_dict, network_alphas = state_dict
else:
network_alphas = {}
except Exception as e:
shared.log.error(f'LoRA load: {e}')
if l.debug:
errors.display(e, "LoRA")
return None
for lora_module in lora_modules:
if lora_module == 'transformer':
if hasattr(sd_model, 'transformer') and sd_model.transformer is not None:
sd_model.load_lora_into_transformer(state_dict, transformer=sd_model.transformer, adapter_name=adapter_name)
else:
shared.log.warning(f'LoRA load: requested={lora_module} missing')
elif lora_module == 'transformer_2':
if hasattr(sd_model, 'transformer_2') and sd_model.transformer_2 is not None:
sd_model.load_lora_into_transformer(state_dict, transformer=sd_model.transformer_2, adapter_name=adapter_name)
else:
shared.log.warning(f'LoRA load: requested={lora_module} missing')
elif lora_module == 'unet':
if hasattr(sd_model, 'unet') and sd_model.unet is not None:
sd_model.load_lora_into_unet(state_dict, network_alphas, unet=sd_model.unet, adapter_name=adapter_name)
else:
shared.log.warning(f'LoRA load: requested={lora_module} missing')
elif lora_module == 'text_encoder' or lora_module == 'te':
if hasattr(sd_model, 'text_encoder') and sd_model.text_encoder is not None:
sd_model.load_lora_into_text_encoder(state_dict, network_alphas, text_encoder=sd_model.text_encoder, adapter_name=adapter_name)
else:
shared.log.warning(f'LoRA load: requested={lora_module} missing')
else:
shared.log.warning(f'LoRA load: requested={lora_module} unknown')
return adapter_name
def load_diffusers(name: str, network_on_disk: network.NetworkOnDisk, lora_scale:float=shared.opts.extra_networks_default_multiplier, lora_module=None) -> Union[network.Network, None]:
t0 = time.time()
name = name.replace(".", "_")
sd_model: diffusers.DiffusionPipeline = getattr(shared.sd_model, "pipe", shared.sd_model)
shared.log.debug(f'Network load: type=LoRA name="{name}" file="{network_on_disk.filename}" detected={network_on_disk.sd_version} method=diffusers scale={lora_scale} fuse={shared.opts.lora_fuse_diffusers}')
if not hasattr(sd_model, 'load_lora_weights'):
shared.log.error(f'Network load: type=LoRA class={sd_model.__class__} does not implement load lora')
return None
try:
if lora_module is not None and isinstance(lora_module, list) and len(lora_module) > 0:
name = load_per_module(sd_model, network_on_disk.filename, adapter_name=name, lora_modules=lora_module)
else:
sd_model.load_lora_weights(network_on_disk.filename, adapter_name=name)
except Exception as e:
if 'already in use' in str(e):
pass
else:
if 'following keys have not been correctly renamed' in str(e):
shared.log.error(f'Network load: type=LoRA name="{name}" diffusers unsupported format')
elif 'object has no attribute' in str(e):
shared.log.error(f'Network load: type=LoRA name="{name}" diffusers empty module')
else:
shared.log.error(f'Network load: type=LoRA name="{name}" {e}')
if l.debug:
errors.display(e, "LoRA")
return None
if name is None:
return None
if name not in diffuser_loaded:
list_adapters = sd_model.get_list_adapters()
list_adapters = [adapter for adapters in list_adapters.values() for adapter in adapters]
if name not in list_adapters:
shared.log.error(f'Network load: type=LoRA name="{name}" adapters={list_adapters} not loaded')
else:
diffuser_loaded.append(name)
diffuser_scales.append(lora_scale)
net = network.Network(name, network_on_disk)
net.mtime = os.path.getmtime(network_on_disk.filename)
l.timer.activate += time.time() - t0
return net
modules/lora/lora_load.py
+13 -48
View File
@@ -3,12 +3,10 @@ import os
import time
import concurrent
from modules import shared, errors, sd_models, sd_models_compile, files_cache
from modules.lora import network, lora_overrides, lora_convert
from modules.lora import network, lora_overrides, lora_convert, lora_diffusers
from modules.lora import lora_common as l
diffuser_loaded = []
diffuser_scales = []
lora_cache = {}
available_networks = {}
available_network_aliases = {}
@@ -17,41 +15,6 @@ available_network_hash_lookup = {}
dump_lora_keys = os.environ.get('SD_LORA_DUMP', None) is not None
def load_diffusers(name, network_on_disk, lora_scale=shared.opts.extra_networks_default_multiplier) -> Union[network.Network, None]:
t0 = time.time()
name = name.replace(".", "_")
sd_model = getattr(shared.sd_model, "pipe", shared.sd_model)
shared.log.debug(f'Network load: type=LoRA name="{name}" file="{network_on_disk.filename}" detected={network_on_disk.sd_version} method=diffusers scale={lora_scale} fuse={shared.opts.lora_fuse_diffusers}')
if not hasattr(sd_model, 'load_lora_weights'):
shared.log.error(f'Network load: type=LoRA class={sd_model.__class__} does not implement load lora')
return None
try:
sd_model.load_lora_weights(network_on_disk.filename, adapter_name=name)
except Exception as e:
if 'already in use' in str(e):
pass
else:
if 'The following keys have not been correctly renamed' in str(e):
shared.log.error(f'Network load: type=LoRA name="{name}" diffusers unsupported format')
else:
shared.log.error(f'Network load: type=LoRA name="{name}" {e}')
if l.debug:
errors.display(e, "LoRA")
return None
if name not in diffuser_loaded:
list_adapters = sd_model.get_list_adapters()
list_adapters = [adapter for adapters in list_adapters.values() for adapter in adapters]
if name not in list_adapters:
shared.log.error(f'Network load: type=LoRA name="{name}" adapters={list_adapters} not loaded')
else:
diffuser_loaded.append(name)
diffuser_scales.append(lora_scale)
net = network.Network(name, network_on_disk)
net.mtime = os.path.getmtime(network_on_disk.filename)
l.timer.activate += time.time() - t0
return net
def lora_dump(lora, dct):
import tempfile
sd_model = getattr(shared.sd_model, "pipe", shared.sd_model)
@@ -73,7 +36,7 @@ def lora_dump(lora, dct):
f.write(line + "\n")
def load_safetensors(name, network_on_disk) -> Union[network.Network, None]:
def load_safetensors(name, network_on_disk: network.NetworkOnDisk) -> Union[network.Network, None]:
if not shared.sd_loaded:
return None
@@ -261,15 +224,15 @@ def gather_networks(names):
return networks_on_disk
def network_load(names, te_multipliers=None, unet_multipliers=None, dyn_dims=None):
def network_load(names, te_multipliers=None, unet_multipliers=None, dyn_dims=None, lora_modules=None):
networks_on_disk = gather_networks(names)
failed_to_load_networks = []
recompile_model, skip_lora_load = maybe_recompile_model(names, te_multipliers)
sd_model = getattr(shared.sd_model, "pipe", shared.sd_model)
l.loaded_networks.clear()
diffuser_loaded.clear()
diffuser_scales.clear()
lora_diffusers.diffuser_loaded.clear()
lora_diffusers.diffuser_scales.clear()
t0 = time.time()
for i, (network_on_disk, name) in enumerate(zip(networks_on_disk, names)):
@@ -279,11 +242,13 @@ def network_load(names, te_multipliers=None, unet_multipliers=None, dyn_dims=Non
if l.debug:
shared.log.debug(f'Network load: type=LoRA name="{name}" file="{network_on_disk.filename}" hash="{shorthash}"')
try:
lora_scale = te_multipliers[i] if te_multipliers else shared.opts.extra_networks_default_multiplier
lora_module = lora_modules[i] if lora_modules and len(lora_modules) > i else None
if recompile_model:
shared.compiled_model_state.lora_model.append(f"{name}:{te_multipliers[i] if te_multipliers else shared.opts.extra_networks_default_multiplier}")
shared.compiled_model_state.lora_model.append(f"{name}:{lora_scale}")
lora_method = lora_overrides.get_method(shorthash)
if lora_method == 'diffusers':
net = load_diffusers(name, network_on_disk, lora_scale=te_multipliers[i] if te_multipliers else shared.opts.extra_networks_default_multiplier)
net = lora_diffusers.load_diffusers(name, network_on_disk, lora_scale, lora_module)
elif lora_method == 'nunchaku':
pass # handled directly from extra_networks_lora.load_nunchaku
else:
@@ -311,21 +276,21 @@ def network_load(names, te_multipliers=None, unet_multipliers=None, dyn_dims=Non
name = next(iter(lora_cache))
lora_cache.pop(name, None)
if not skip_lora_load and len(diffuser_loaded) > 0:
shared.log.debug(f'Network load: type=LoRA loaded={diffuser_loaded} available={sd_model.get_list_adapters()} active={sd_model.get_active_adapters()} scales={diffuser_scales}')
if not skip_lora_load and len(lora_diffusers.diffuser_loaded) > 0:
shared.log.debug(f'Network load: type=LoRA loaded={lora_diffusers.diffuser_loaded} available={sd_model.get_list_adapters()} active={sd_model.get_active_adapters()} scales={lora_diffusers.diffuser_scales}')
try:
t1 = time.time()
if l.debug:
shared.log.trace(f'Network load: type=LoRA list={sd_model.get_list_adapters()}')
shared.log.trace(f'Network load: type=LoRA active={sd_model.get_active_adapters()}')
sd_model.set_adapters(adapter_names=diffuser_loaded, adapter_weights=diffuser_scales)
sd_model.set_adapters(adapter_names=lora_diffusers.diffuser_loaded, adapter_weights=lora_diffusers.diffuser_scales)
except Exception as e:
shared.log.error(f'Network load: type=LoRA action=set {e}')
if l.debug:
errors.display(e, 'LoRA')
try:
if shared.opts.lora_fuse_diffusers and not lora_overrides.disable_fuse():
sd_model.fuse_lora(adapter_names=diffuser_loaded, lora_scale=1.0, fuse_unet=True, fuse_text_encoder=True) # diffusers with fuse uses fixed scale since later apply does the scaling
sd_model.fuse_lora(adapter_names=lora_diffusers.diffuser_loaded, lora_scale=1.0, fuse_unet=True, fuse_text_encoder=True) # diffusers with fuse uses fixed scale since later apply does the scaling
sd_model.unload_lora_weights()
l.timer.activate += time.time() - t1
except Exception as e:
modules/lora/lora_nunchaku.py
+4 -1
View File
@@ -15,6 +15,9 @@ def load_nunchaku(names, strengths):
is_changed = loras != previously_loaded
if not is_changed:
return False
if not hasattr(shared.sd_model, 'transformer') or not hasattr(shared.sd_model.transformer, 'update_lora_params'):
shared.log.error(f'Network load: type=LoRA method=nunchaku model={shared.sd_model.__class__.__name__} unsupported')
return False
previously_loaded = loras
try:
@@ -27,7 +30,7 @@ def load_nunchaku(names, strengths):
lora_common.timer.load = t1 - t0
shared.log.debug(f"Network load: type=LoRA method=nunchaku loras={names} strength={strengths} time={t1-t0:.3f}")
except Exception as e:
shared.log.errors(f'Network load: type=LoRA method=nunchaku {e}')
shared.log.error(f'Network load: type=LoRA method=nunchaku {e}')
if lora_common.debug:
errors.display(e, 'LoRA')
return is_changed
modules/lora/lora_overrides.py
+3 -1
View File
@@ -61,7 +61,9 @@ def get_method(shorthash=''):
use_diffusers = use_diffusers or any(x.startswith(shorthash) for x in maybe_diffusers)
if shared.opts.lora_force_diffusers and len(shorthash) > 4:
use_diffusers = use_diffusers or any(x.startswith(shorthash) for x in force_diffusers)
use_nunchaku = hasattr(shared.sd_model, 'transformer') and 'Nunchaku' in shared.sd_model.transformer.__class__.__name__
nunchaku_dit = hasattr(shared.sd_model, 'transformer') and 'Nunchaku' in shared.sd_model.transformer.__class__.__name__
nunchaku_unet = hasattr(shared.sd_model, 'unet') and 'Nunchaku' in shared.sd_model.unet.__class__.__name__
use_nunchaku = nunchaku_dit or nunchaku_unet
if use_nunchaku:
return 'nunchaku'
elif use_diffusers:
modules/ltx/ltx_process.py
+20 -9
View File
@@ -13,7 +13,7 @@ from modules.ltx.ltx_util import get_bucket, get_frames, load_model, load_upsamp
debug = shared.log.trace if os.environ.get('SD_VIDEO_DEBUG', None) is not None else lambda *args, **kwargs: None
engine, model = 'LTX Video', 'LTXVideo 0.9.7 13B'
# engine, model = 'LTX Video', 'LTXVideo 0.9.7 13B'
upsample_repo_id = "a-r-r-o-w/LTX-Video-0.9.7-Latent-Spatial-Upsampler-diffusers"
upsample_pipe = None
queue_lock = threading.Lock()
@@ -21,6 +21,7 @@ queue_lock = threading.Lock()
def run_ltx(task_id,
_ui_state,
model:str,
prompt:str,
negative:str,
styles:list[str],
@@ -65,7 +66,10 @@ def run_ltx(task_id,
progress.finish_task(task_id)
yield None, f'LTX Error: {str(e)}'
from diffusers import LTXConditionPipeline # pylint: disable=unused-import
if model is None or len(model) == 0:
yield from abort('Video: no model selected', ok=True)
return
# from diffusers import LTXConditionPipeline # pylint: disable=unused-import
check_av()
progress.add_task_to_queue(task_id)
with queue_lock:
@@ -73,9 +77,14 @@ def run_ltx(task_id,
memstats.reset_stats()
timer.process.reset()
yield None, 'LTX: Loading...'
engine = 'LTX Video'
load_model(engine, model)
debug(f'Video: cls={shared.sd_model.__class__.__name__} op=init model="{model}"')
if not shared.sd_model.__class__.__name__.startswith("LTX"):
yield from abort(f'Video: cls={shared.sd_model.__class__.__name__} selected model is not LTX model', ok=True)
return
shared.state.begin('Video', task_id=task_id)
videojob = shared.state.begin('Video', task_id=task_id)
shared.state.job_count = 1
p = processing.StableDiffusionProcessingVideo(
@@ -105,7 +114,7 @@ def run_ltx(task_id,
prompt, negative, networks = get_prompts(prompt, negative, styles)
sampler_name = processing.get_sampler_name(sampler_index)
sd_samplers.create_sampler(sampler_name, shared.sd_model)
shared.log.debug(f'Video: cls={shared.sd_model.__class__.__name__} op=init prompt="{prompt}" negative="{negative}" styles={styles} networks={networks} sampler={shared.sd_model.scheduler.__class__.__name__}')
shared.log.debug(f'Video: cls={shared.sd_model.__class__.__name__} op=init styles={styles} networks={networks} sampler={shared.sd_model.scheduler.__class__.__name__}')
extra_networks.activate(p, networks)
t0 = time.time()
@@ -127,6 +136,7 @@ def run_ltx(task_id,
if len(conditions) > 0:
base_args["conditions"] = conditions
yield None, 'LTX: Generate in progress...'
samplejob = shared.state.begin('Sample')
try:
latents = shared.sd_model(**base_args).frames[0]
except AssertionError as e:
@@ -141,10 +151,11 @@ def run_ltx(task_id,
timer.process.add('offload', t1 - t0)
timer.process.add('base', t2 - t1)
timer.process.add('offload', t3 - t2)
shared.state.end(samplejob)
if upsample_enable:
t4 = time.time()
shared.state.begin('Upsample')
upsamplejob = shared.state.begin('Upsample')
global upsample_pipe # pylint: disable=global-statement
upsample_pipe = load_upsample(upsample_pipe, upsample_repo_id)
upsample_pipe = sd_models.apply_balanced_offload(upsample_pipe)
@@ -172,11 +183,11 @@ def run_ltx(task_id,
t6 = time.time()
timer.process.add('upsample', t5 - t4)
timer.process.add('offload', t6 - t5)
shared.state.end()
shared.state.end(upsamplejob)
if refine_enable:
t7 = time.time()
shared.state.begin('Refine')
refinejob = shared.state.begin('Refine')
shared.sd_model = sd_models.apply_balanced_offload(shared.sd_model)
refine_args = {
"prompt": prompt,
@@ -211,7 +222,7 @@ def run_ltx(task_id,
t9 = time.time()
timer.process.add('refine', t8 - t7)
timer.process.add('offload', t9 - t8)
shared.state.end()
shared.state.end(refinejob)
extra_networks.deactivate(p)
@@ -250,7 +261,7 @@ def run_ltx(task_id,
fps = f'{num_frames/(t_end-t0):.2f}'
its = f'{(steps)/(t_end-t0):.2f}'
shared.state.end()
shared.state.end(videojob)
progress.finish_task(task_id)
shared.log.info(f'Processed: fn="{video_file}" frames={num_frames} fps={fps} its={its} resolution={resolution} time={t_end-t0:.2f} timers={timer.process.dct()} memory={memstats.memory_stats()}')
modules/ltx/ltx_ui.py
+5
View File
@@ -3,6 +3,7 @@ import gradio as gr
from modules import shared, ui_sections, ui_symbols, ui_common
from modules.ui_components import ToolButton
from modules.video_models.video_utils import get_codecs
from modules.video_models.models_def import models
from modules.ltx import ltx_process
@@ -14,6 +15,9 @@ def create_ui(prompt, negative, styles, overrides):
with gr.Column(variant='compact', elem_id="ltx_settings", elem_classes=['settings-column'], scale=1):
with gr.Row():
generate = gr.Button('Generate', elem_id="ltx_generate_btn", variant='primary', visible=False)
with gr.Row():
ltx_models = [m.name for m in models['LTX Video']]
model = gr.Dropdown(label='LTX model', choices=ltx_models, value=ltx_models[0])
with gr.Accordion(open=True, label="LTX size", elem_id='ltx_generate_accordion'):
with gr.Row():
width, height = ui_sections.create_resolution_inputs('ltx', default_width=832, default_height=480)
@@ -73,6 +77,7 @@ def create_ui(prompt, negative, styles, overrides):
state_inputs = [task_id, ui_state]
video_inputs = [
model,
prompt, negative, styles,
width, height, frames,
steps, sampler_index, seed,
modules/ltx/ltx_util.py
+14 -1
View File
@@ -4,7 +4,12 @@ from PIL import Image
from modules import devices, shared, sd_models, timer, extra_networks
loaded_model: str = None
def get_bucket(size: int):
if not hasattr(shared.sd_model, 'vae_temporal_compression_ratio'):
return int(size) - (int(size) % 16)
return int(size) - (int(size) % shared.sd_model.vae_temporal_compression_ratio)
@@ -13,11 +18,19 @@ def get_frames(frames: int):
def load_model(engine: str, model: str):
global loaded_model # pylint: disable=global-statement
if loaded_model == model:
return
if model is None or model == '' or model=='None':
loaded_model = None
shared.sd_model = None
return
t0 = time.time()
from modules.video_models import models_def, video_load
selected: models_def.Model = [m for m in models_def.models[engine] if m.name == model][0]
shared.log.info(f'Video load: cls={selected.repo_cls.__name__} repo="{selected.repo}"')
shared.log.info(f'Video load: engine="{engine}" selected="{model}" {selected}')
video_load.load_model(selected)
loaded_model = model
t1 = time.time()
shared.sd_model = sd_models.apply_balanced_offload(shared.sd_model)
t2 = time.time()
modules/memstats.py
+19 -2
View File
@@ -54,11 +54,28 @@ def ram_stats():
ram_total = 100 * res.rss / process.memory_percent()
ram_total = min(ram_total, get_docker_limit(), get_runpod_limit())
ram['total'] = gb(ram_total)
ram['used'] = gb(res.rss)
ram['free'] = round(ram['total'] - ram['used'])
ram['rss'] = gb(res.rss)
except Exception as e:
ram['total'] = 0
ram['rss'] = 0
ram['error'] = str(e)
if not fail_once:
shared.log.error(f'RAM stats: {e}')
errors.display(e, 'RAM stats')
fail_once = True
try:
vmem = psutil.virtual_memory()
ram['used'] = gb(vmem.used) if hasattr(vmem, 'used') else 0
ram['free'] = gb(vmem.free) if hasattr(vmem, 'free') else 0
ram['avail'] = gb(vmem.available) if hasattr(vmem, 'available') else 0
ram['buffers'] = gb(vmem.buffers) if hasattr(vmem, 'buffers') else 0
ram['cached'] = gb(vmem.cached) if hasattr(vmem, 'cached') else 0
except Exception as e:
ram['used'] = 0
ram['free'] = 0
ram['avail'] = 0
ram['buffers'] = 0
ram['cached'] = 0
ram['error'] = str(e)
if not fail_once:
shared.log.error(f'RAM stats: {e}')
modules/mit_nunchaku.py
+6 -2
View File
@@ -4,7 +4,7 @@ from installer import log, pip
from modules import devices
ver = '1.0.0'
ver = '1.0.1'
ok = False
@@ -15,7 +15,11 @@ def check():
try:
import nunchaku
import nunchaku.utils
log.info(f'Nunchaku: path={nunchaku.__path__} precision={nunchaku.utils.get_precision()}')
from nunchaku import __version__
log.info(f'Nunchaku: path={nunchaku.__path__} version={__version__.__version__} precision={nunchaku.utils.get_precision()}')
if __version__.__version__ != ver:
ok = False
return False
ok = True
return True
except Exception as e:
modules/model_quant.py
+20 -29
View File
@@ -205,6 +205,9 @@ def create_sdnq_config(kwargs = None, allow: bool = True, module: str = 'Model',
sdnq_config = SDNQConfig(
weights_dtype=weights_dtype,
group_size=shared.opts.sdnq_quantize_weights_group_size,
svd_rank=shared.opts.sdnq_svd_rank,
svd_steps=shared.opts.sdnq_svd_steps,
use_svd=shared.opts.sdnq_use_svd,
quant_conv=shared.opts.sdnq_quantize_conv_layers,
use_quantized_matmul=shared.opts.sdnq_use_quantized_matmul,
use_quantized_matmul_conv=shared.opts.sdnq_use_quantized_matmul_conv,
@@ -215,7 +218,7 @@ def create_sdnq_config(kwargs = None, allow: bool = True, module: str = 'Model',
modules_to_not_convert=modules_to_not_convert,
modules_dtype_dict=modules_dtype_dict.copy(),
)
log.debug(f'Quantization: module="{module}" type=sdnq mode=pre dtype={weights_dtype} matmul={shared.opts.sdnq_use_quantized_matmul} group_size={shared.opts.sdnq_quantize_weights_group_size} quant_conv={shared.opts.sdnq_quantize_conv_layers} matmul_conv={shared.opts.sdnq_use_quantized_matmul_conv} dequantize_fp32={shared.opts.sdnq_dequantize_fp32} quantize_with_gpu={shared.opts.sdnq_quantize_with_gpu} quantization_device={quantization_device} return_device={return_device} device_map={shared.opts.device_map} offload_mode={shared.opts.diffusers_offload_mode} non_blocking={shared.opts.diffusers_offload_nonblocking} modules_to_not_convert={modules_to_not_convert} modules_dtype_dict={modules_dtype_dict}')
log.debug(f'Quantization: module="{module}" type=sdnq mode=pre dtype={weights_dtype} matmul={shared.opts.sdnq_use_quantized_matmul} group_size={shared.opts.sdnq_quantize_weights_group_size} svd_rank={shared.opts.sdnq_svd_rank} svd_steps={shared.opts.sdnq_svd_steps} use_svd={shared.opts.sdnq_use_svd} quant_conv={shared.opts.sdnq_quantize_conv_layers} matmul_conv={shared.opts.sdnq_use_quantized_matmul_conv} dequantize_fp32={shared.opts.sdnq_dequantize_fp32} quantize_with_gpu={shared.opts.sdnq_quantize_with_gpu} quantization_device={quantization_device} return_device={return_device} device_map={shared.opts.device_map} offload_mode={shared.opts.diffusers_offload_mode} non_blocking={shared.opts.diffusers_offload_nonblocking} modules_to_not_convert={modules_to_not_convert} modules_dtype_dict={modules_dtype_dict}')
if kwargs is None:
return sdnq_config
else:
@@ -477,10 +480,10 @@ def apply_layerwise(sd_model, quiet:bool=False):
def sdnq_quantize_model(model, op=None, sd_model=None, do_gc: bool = True, weights_dtype: str = None, modules_to_not_convert: list = None, modules_dtype_dict: dict = None):
global quant_last_model_name, quant_last_model_device # pylint: disable=global-statement
from modules import devices, shared, timer
from modules.sdnq import apply_sdnq_to_module
from modules.sdnq import sdnq_post_load_quant
if weights_dtype is None:
if op is not None and ("text_encoder" in op or op in {"TE", "LLM"}) and shared.opts.sdnq_quantize_weights_mode_te not in {"Same as model", "default"}:
if (op is not None) and ("text_encoder" in op or op in {"TE", "LLM"}) and (shared.opts.sdnq_quantize_weights_mode_te not in {"Same as model", "default"}):
weights_dtype = shared.opts.sdnq_quantize_weights_mode_te
else:
weights_dtype = shared.opts.sdnq_quantize_weights_mode
@@ -495,18 +498,6 @@ def sdnq_quantize_model(model, op=None, sd_model=None, do_gc: bool = True, weigh
if modules_dtype_dict is None:
modules_dtype_dict = {}
if getattr(model, "_keep_in_fp32_modules", None) is not None:
modules_to_not_convert.extend(model._keep_in_fp32_modules) # pylint: disable=protected-access
if getattr(model, "_skip_layerwise_casting_patterns", None) is not None:
modules_to_not_convert.extend(model._skip_layerwise_casting_patterns) # pylint: disable=protected-access
if model.__class__.__name__ == "ChromaTransformer2DModel":
modules_to_not_convert.append("distilled_guidance_layer")
if model.__class__.__name__ == "QwenImageTransformer2DModel":
if "minimum_6bit" not in modules_dtype_dict.keys():
modules_dtype_dict["minimum_6bit"] = ["img_mod", "pos_embed", "time_text_embed", "img_in", "txt_in", "norm_out"]
else:
modules_dtype_dict["minimum_6bit"].extend(["img_mod", "pos_embed", "time_text_embed", "img_in", "txt_in", "norm_out"])
sdnq_modules_to_not_convert = [m.strip() for m in re.split(';|,| ', shared.opts.sdnq_modules_to_not_convert) if len(m.strip()) > 1]
if len(sdnq_modules_to_not_convert) > 0:
modules_to_not_convert.extend(sdnq_modules_to_not_convert)
@@ -527,17 +518,16 @@ def sdnq_quantize_model(model, op=None, sd_model=None, do_gc: bool = True, weigh
except Exception as e:
log.warning(f'Quantization: SDNQ failed to parse sdnq_modules_dtype_dict: {e}')
model.eval()
backup_embeddings = None
if hasattr(model, "get_input_embeddings"):
backup_embeddings = copy.deepcopy(model.get_input_embeddings())
t0 = time.time()
model = apply_sdnq_to_module(
model = sdnq_post_load_quant(
model,
weights_dtype=weights_dtype,
torch_dtype=devices.dtype,
group_size=shared.opts.sdnq_quantize_weights_group_size,
svd_rank=shared.opts.sdnq_svd_rank,
svd_steps=shared.opts.sdnq_svd_steps,
use_svd=shared.opts.sdnq_use_svd,
quant_conv=shared.opts.sdnq_quantize_conv_layers,
use_quantized_matmul=shared.opts.sdnq_use_quantized_matmul,
use_quantized_matmul_conv=shared.opts.sdnq_use_quantized_matmul_conv,
@@ -549,12 +539,9 @@ def sdnq_quantize_model(model, op=None, sd_model=None, do_gc: bool = True, weigh
modules_dtype_dict=modules_dtype_dict.copy(),
op=op,
)
t1 = time.time()
timer.load.add('sdnq', t1 - t0)
model.quantization_method = 'SDNQ'
if hasattr(model, "set_input_embeddings") and backup_embeddings is not None:
model.set_input_embeddings(backup_embeddings)
if op is not None and shared.opts.sdnq_quantize_shuffle_weights:
if quant_last_model_name is not None:
@@ -577,7 +564,7 @@ def sdnq_quantize_model(model, op=None, sd_model=None, do_gc: bool = True, weigh
if do_gc:
devices.torch_gc(force=True, reason='sdnq')
log.debug(f'Quantization: module="{op if op is not None else model.__class__}" type=sdnq mode=post dtype={weights_dtype} matmul={shared.opts.sdnq_use_quantized_matmul} group_size={shared.opts.sdnq_quantize_weights_group_size} quant_conv={shared.opts.sdnq_quantize_conv_layers} matmul_conv={shared.opts.sdnq_use_quantized_matmul_conv} dequantize_fp32={shared.opts.sdnq_dequantize_fp32} quantize_with_gpu={shared.opts.sdnq_quantize_with_gpu} quantization_device={quantization_device} return_device={return_device} device_map={shared.opts.device_map} offload_mode={shared.opts.diffusers_offload_mode} non_blocking={shared.opts.diffusers_offload_nonblocking} modules_to_not_convert={modules_to_not_convert} modules_dtype_dict={modules_dtype_dict}')
log.debug(f'Quantization: module="{op if op is not None else model.__class__}" type=sdnq mode=post dtype={weights_dtype} matmul={shared.opts.sdnq_use_quantized_matmul} group_size={shared.opts.sdnq_quantize_weights_group_size} svd_rank={shared.opts.sdnq_svd_rank} svd_steps={shared.opts.sdnq_svd_steps} use_svd={shared.opts.sdnq_use_svd} quant_conv={shared.opts.sdnq_quantize_conv_layers} matmul_conv={shared.opts.sdnq_use_quantized_matmul_conv} dequantize_fp32={shared.opts.sdnq_dequantize_fp32} quantize_with_gpu={shared.opts.sdnq_quantize_with_gpu} quantization_device={quantization_device} return_device={return_device} device_map={shared.opts.device_map} offload_mode={shared.opts.diffusers_offload_mode} non_blocking={shared.opts.diffusers_offload_nonblocking} modules_to_not_convert={modules_to_not_convert} modules_dtype_dict={modules_dtype_dict}')
return model
@@ -585,7 +572,7 @@ def sdnq_quantize_weights(sd_model):
try:
t0 = time.time()
from modules import shared, devices, sd_models
log.debug(f"Quantization: type=SDNQ modules={shared.opts.sdnq_quantize_weights} dtype={shared.opts.sdnq_quantize_weights_mode} dtype_te={shared.opts.sdnq_quantize_weights_mode_te} matmul={shared.opts.sdnq_use_quantized_matmul} group_size={shared.opts.sdnq_quantize_weights_group_size} quant_conv={shared.opts.sdnq_quantize_conv_layers} matmul_conv={shared.opts.sdnq_use_quantized_matmul_conv} quantize_with_gpu={shared.opts.sdnq_quantize_with_gpu} dequantize_fp32={shared.opts.sdnq_dequantize_fp32} pre_forward={shared.opts.diffusers_offload_pre}")
log.debug(f"Quantization: type=SDNQ modules={shared.opts.sdnq_quantize_weights} dtype={shared.opts.sdnq_quantize_weights_mode} dtype_te={shared.opts.sdnq_quantize_weights_mode_te} matmul={shared.opts.sdnq_use_quantized_matmul} svd_rank={shared.opts.sdnq_svd_rank} svd_steps={shared.opts.sdnq_svd_steps} use_svd={shared.opts.sdnq_use_svd} group_size={shared.opts.sdnq_quantize_weights_group_size} quant_conv={shared.opts.sdnq_quantize_conv_layers} matmul_conv={shared.opts.sdnq_use_quantized_matmul_conv} quantize_with_gpu={shared.opts.sdnq_quantize_with_gpu} dequantize_fp32={shared.opts.sdnq_dequantize_fp32} pre_forward={shared.opts.diffusers_offload_pre}")
global quant_last_model_name, quant_last_model_device # pylint: disable=global-statement
sd_model = sd_models.apply_function_to_model(sd_model, sdnq_quantize_model, shared.opts.sdnq_quantize_weights, op="sdnq")
@@ -603,6 +590,8 @@ def sdnq_quantize_weights(sd_model):
log.info(f"Quantization: type=SDNQ time={t1-t0:.2f}")
except Exception as e:
log.warning(f"Quantization: type=SDNQ {e}")
from modules import errors
errors.display(e, 'Quantization')
return sd_model
@@ -610,12 +599,14 @@ def optimum_quanto_model(model, op=None, sd_model=None, weights=None, activation
from modules import devices, shared
quanto = load_quanto('Quantize model: type=Optimum Quanto')
global quant_last_model_name, quant_last_model_device # pylint: disable=global-statement
if sd_model is not None and ("Flux" in sd_model.__class__.__name__ or "Chroma" in sd_model.__class__.__name__): # LayerNorm is not supported
if model.__class__.__name__ in {"FluxTransformer2DModel", "ChromaTransformer2DModel"}: # LayerNorm is not supported
exclude_list = ["transformer_blocks.*.norm1.norm", "transformer_blocks.*.norm2", "transformer_blocks.*.norm1_context.norm", "transformer_blocks.*.norm2_context", "single_transformer_blocks.*.norm.norm", "norm_out.norm"]
if "Chroma" in sd_model.__class__.__name__:
if model.__class__.__name__ == "ChromaTransformer2DModel":
# we ignore the distilled guidance layer because it degrades quality too much
# see: https://github.com/huggingface/diffusers/pull/11698#issuecomment-2969717180 for more details
exclude_list.append("distilled_guidance_layer.*")
elif model.__class__.__name__ == "QwenImageTransformer2DModel":
exclude_list = ["transformer_blocks.0.img_mod.1.weight", "time_text_embed", "img_in", "txt_in", "proj_out", "norm_out", "pos_embed"]
else:
exclude_list = None
weights = getattr(quanto, weights) if weights is not None else getattr(quanto, shared.opts.optimum_quanto_weights_type)
modules/modeldata.py
+2
View File
@@ -62,6 +62,8 @@ def get_model_type(pipe):
model_type = 'qwen'
elif 'NextStep' in name:
model_type = 'nextstep'
elif 'X-Omni' in name:
model_type = 'x-omni'
# video models
elif "CogVideo" in name:
model_type = 'cogvideo'
modules/modelloader.py
+17 -5
View File
@@ -22,10 +22,13 @@ pbar = None
def hf_login(token=None):
global loggedin # pylint: disable=global-statement
token = token or shared.opts.huggingface_token
token = token.replace("\n", "").replace("\r", "").strip() if token is not None else None
install('hf_xet', quiet=True)
if token is None or len(token) <= 4:
log.debug('HF login: no token provided')
return False
if len(shared.opts.huggingface_mirror.strip()) > 0 and os.environ.get('HF_ENDPOINT', None) is None:
os.environ['HF_ENDPOINT'] = shared.opts.huggingface_mirror.strip()
if os.environ.get('HUGGING_FACE_HUB_TOKEN', None) is not None:
os.environ.pop('HUGGING_FACE_HUB_TOKEN', None)
os.unsetenv('HUGGING_FACE_HUB_TOKEN')
@@ -179,6 +182,13 @@ def find_diffuser(name: str, full=False):
if len(repo) > 0:
return [repo[0]['name']]
hf_api = hf.HfApi()
suffix = ''
if len(name) > 3 and name.count('/') > 1:
parts = name.split('/')
name = '/'.join(parts[:2]) # only user/model
suffix = '/'.join(parts[2:]) # subfolder
if len(suffix) > 0:
suffix = '/' + suffix
models = list(hf_api.list_models(model_name=name, library=['diffusers'], full=True, limit=20, sort="downloads", direction=-1))
if len(models) == 0:
models = list(hf_api.list_models(model_name=name, full=True, limit=20, sort="downloads", direction=-1)) # widen search
@@ -186,9 +196,9 @@ def find_diffuser(name: str, full=False):
shared.log.debug(f'Search model: repo="{name}" {len(models) > 0}')
if len(models) > 0:
if not full:
return models[0].id
return models[0].id + suffix
else:
return [m.id for m in models]
return [m.id + suffix for m in models]
return None
@@ -219,6 +229,8 @@ def get_reference_opts(name: str, quiet=False):
def load_reference(name: str, variant: str = None, revision: str = None, mirror: str = None, custom_pipeline: str = None):
if '+' in name:
name = name.split('+')[0]
found = [r for r in diffuser_repos if name == r['name'] or name == r['friendly'] or name == r['path']]
if len(found) > 0: # already downloaded
model_opts = get_reference_opts(found[0]['name'])
@@ -250,7 +262,7 @@ def load_reference(name: str, variant: str = None, revision: str = None, mirror:
def load_civitai(model: str, url: str):
from modules import sd_models
name, _ext = os.path.splitext(model)
info = sd_models.get_closet_checkpoint_match(name)
info = sd_models.get_closest_checkpoint_match(name)
if info is not None:
_model_opts = get_reference_opts(info.model_name)
return name # already downloaded
@@ -260,7 +272,7 @@ def load_civitai(model: str, url: str):
download_civit_model_thread(model_name=model, model_url=url, model_path='', model_type='safetensors', token=shared.opts.civitai_token)
shared.log.debug(f'Reference download complete: model="{name}"')
sd_models.list_models()
info = sd_models.get_closet_checkpoint_match(name)
info = sd_models.get_closest_checkpoint_match(name)
if info is not None:
shared.log.debug(f'Reference: model="{name}"')
return name # already downloaded
@@ -294,7 +306,7 @@ def download_url_to_file(url: str, dst: str):
continue
break
else:
shared.log.error('Error downloading: url={url} no usable temporary filename found')
shared.log.error(f'Error downloading: url={url} no usable temporary filename found')
return
try:
with Progress(TextColumn('[cyan]{task.description}'), BarColumn(), TaskProgressColumn(), TimeRemainingColumn(), TimeElapsedColumn(), console=shared.console) as progress:
modules/modelstats.py
+1 -1
View File
@@ -79,7 +79,7 @@ class Model():
return
self.cls = shared.sd_model.__class__.__name__
self.type = shared.sd_model_type
self.info = sd_models.get_closet_checkpoint_match(name)
self.info = sd_models.get_closest_checkpoint_match(name)
if self.info is not None:
self.name = self.info.name or self.name
self.hash = self.info.shorthash or ''
modules/modular.py
+64
View File
@@ -0,0 +1,64 @@
import time
import diffusers
from modules import shared
modular_map= {
'StableDiffusionXLPipeline': 'StableDiffusionXLAutoBlocks',
'StableDiffusionXLImg2ImgPipeline': 'StableDiffusionXLAutoBlocks',
'StableDiffusionXLInpaintPipeline': 'StableDiffusionXLAutoBlocks',
'FluxPipeline': 'FluxAutoBlocks',
'FluxImg2ImgPipeline': 'FluxAutoBlocks',
'FluxInpaintPipeline': 'FluxAutoBlocks',
'WanPipeline': 'WanAutoBlocks',
'WanImageToVideoPipeline': 'WanAutoBlocks',
'QwenImagePipeline': 'QwenImageAutoBlocks',
'QwenImageEditPipeline': 'QwenImageEditAutoBlocks',
}
def is_compatible(diffusion_pipeline: diffusers.DiffusionPipeline) -> bool:
if not shared.opts.model_modular_enable:
return False
compatible = diffusion_pipeline.__class__.__name__ in modular_map
if not compatible:
shared.log.debug(f'Modular: source={diffusion_pipeline.__class__.__name__} incompatible pipeline')
return compatible
def convert_to_modular(diffusion_pipeline: diffusers.DiffusionPipeline) -> diffusers.ModularPipeline:
modular_pipe = None
try:
t0 = time.time()
modular_cls = modular_map.get(diffusion_pipeline.__class__.__name__, None)
if modular_cls is None:
raise ValueError(f'unknown: cls={diffusion_pipeline.__class__.__name__}')
modular_cls = getattr(diffusers, modular_cls, None)
if modular_cls is None:
raise ValueError(f'invalid: cls={diffusion_pipeline.__class__.__name__}')
modular_blocks = modular_cls()
modular_pipe = modular_blocks.init_pipeline()
components_dct = {k: v for k, v in diffusion_pipeline.components.items() if v is not None}
modular_pipe.update_components(**components_dct, **diffusion_pipeline.parameters)
modular_pipe.original_pipe = diffusion_pipeline
t1 = time.time()
shared.log.debug(f'Modular: source={diffusion_pipeline.__class__.__name__} target={modular_pipe.__class__.__name__} time={t1 - t0:.2f}')
"""
for expected_input_param in modular_pipe.blocks.inputs:
name = expected_input_param.name
default = expected_input_param.default
kwargs_type = expected_input_param.kwargs_type
shared.log.trace(f'Modular input: name={name} type={kwargs_type} default={default}')
"""
except Exception as e:
shared.log.error(f'Modular: {e}')
raise e
return modular_pipe
def restore_standard(modular_pipe):
if hasattr(modular_pipe, 'original_pipe'):
shared.log.debug(f'Modular: source={modular_pipe.__class__.__name__} target={modular_pipe.original_pipe.__class__.__name__}')
return modular_pipe.original_pipe
return modular_pipe
modules/modular_guiders.py
+87
View File
@@ -0,0 +1,87 @@
import diffusers
from modules import shared, errors, processing
# ['Default', 'CFG', 'Zero', 'PAG', 'APG', 'SLG', 'SEG', 'TCFG', 'FDG']
guiders = {
# 'None': { 'cls': None, 'args': {}, },
'Default': { 'cls': None, 'args': {}, },
'CFG: ClassifierFreeGuidance': { 'cls': diffusers.ClassifierFreeGuidance, 'args': {} },
'Auto: AutoGuidance': { 'cls': diffusers.AutoGuidance, 'args': { 'dropout': 1.0, 'auto_guidance_layers': [7, 8, 9], 'auto_guidance_config': None } },
'Zero: ClassifierFreeZeroStar': { 'cls': diffusers.ClassifierFreeZeroStarGuidance, 'args': { 'zero_init_steps': 1 } },
'PAG: PerturbedAttentionGuidance': { 'cls': diffusers.PerturbedAttentionGuidance, 'args': { 'perturbed_guidance_scale': 2.8, 'perturbed_guidance_start': 0.01, 'perturbed_guidance_stop': 0.2, 'perturbed_guidance_layers': [7, 8, 9], 'perturbed_guidance_config': None } },
'APG: AdaptiveProjectedGuidance': { 'cls': diffusers.AdaptiveProjectedGuidance, 'args': { 'adaptive_projected_guidance_momentum': -1, 'adaptive_projected_guidance_rescale': 15.0 } },
'SLG: SkipLayerGuidance': { 'cls': diffusers.SkipLayerGuidance, 'args': { 'skip_layer_guidance_scale': 2.8, 'skip_layer_guidance_start': 0.01, 'skip_layer_guidance_stop': 0.2, 'skip_layer_guidance_layers': [7, 8, 9], 'skip_layer_config': None } },
'SEG: SmoothedEnergyGuidance': { 'cls': diffusers.SmoothedEnergyGuidance, 'args': { 'seg_guidance_scale': 3.0, 'seg_blur_sigma': 9999999.0, 'seg_blur_threshold_inf': 9999.0, 'seg_guidance_start': 0.0, 'seg_guidance_stop': 1.0, 'seg_guidance_layers': [7, 8, 9], 'seg_guidance_config': None } },
'TCFG: TangentialClassifierFreeGuidance': { 'cls': diffusers.TangentialClassifierFreeGuidance, 'args': {} },
'FDG: FrequencyDecoupledGuidance': { 'cls': diffusers.FrequencyDecoupledGuidance, 'args': { 'guidance_scales': [10.0, 5.0], 'parallel_weights': 1.0, 'guidance_rescale_space': "data" } },
}
base_args = {
'guidance_scale': 6.0,
'guidance_rescale': 0.0,
'start': 0.0,
'stop': 1.0,
}
def set_guider(p: processing.StableDiffusionProcessing):
guidance_name = p.guidance_name or 'Default'
if guidance_name not in guiders:
return
if guidance_name == 'Default':
if hasattr(shared.sd_model, 'default_guider'):
guider_info = shared.sd_model.default_guider
shared.sd_model.update_components(guider=guider_info)
else:
guider_info = shared.sd_model.get_component_spec("guider")
shared.sd_model.default_guider = guider_info
guider_cls = guider_info.type_hint
if guider_info is not None and guider_cls is not None and guider_info.config is not None:
guider_args = {k: v for k, v in guider_info.config.items() if not k.startswith('_') and v is not None}
else:
guider_args = {}
shared.log.info(f'Guider: name={guidance_name} cls={guider_cls.__name__} args={guider_args}')
return
if guidance_name == 'None':
shared.sd_model.update_components(guider=None) # breaks the pipeline
shared.log.info(f'Guider: name={guidance_name}')
return
guider_info = guiders[guidance_name]
guider_cls = guider_info['cls']
guider_args = {}
for k, v in base_args.items():
if v is not None and v >= 0.0:
guider_args[k] = v
shared.log.warning('Guiders: partially implemented') # TODO: guiders
for k, v in guider_info['args'].items():
try:
if k is None:
pass
elif k.endswith('_layers') and isinstance(v, str):
guider_args[k] = [int(x.strip()) for x in v.split(',') if x.strip().isdigit()]
elif k.endswith('_config'):
# if lsc_enabled
# guider_args[k] = diffusers.LayerSkipConfig(...)
pass
elif isinstance(v, list) and len(v) > 0:
guider_args[k] = v
elif isinstance(v, int) and (v >= 0):
guider_args[k] = int(v)
elif isinstance(v, float) and (v >= 0.0):
guider_args[k] = float(v)
elif isinstance(v, str) and (len(v) > 0):
guider_args[k] = v
except Exception as e:
shared.log.error(f'Guiders: arg={k} value={v} error={e}')
errors.display(e, 'Guiders')
# guider_args.update(guider_info['args'])
if guider_cls is not None:
try:
guider_instance = guider_cls(**guider_args)
shared.log.info(f'Guider: name={guidance_name} cls={guider_cls.__name__} args={guider_args}')
shared.sd_model.update_components(guider=guider_instance)
except Exception as e:
shared.log.error(f'Guider: name={guidance_name} cls={guider_cls.__name__} args={guider_args} {e}')
return
modules/onnx_impl/ui.py
+2 -2
View File
@@ -15,7 +15,7 @@ def create_ui():
from modules.ui_common import create_refresh_button
from modules.ui_components import DropdownMulti
from modules.shared import log, opts, cmd_opts, refresh_checkpoints
from modules.sd_models import checkpoint_titles, get_closet_checkpoint_match
from modules.sd_models import checkpoint_titles, get_closest_checkpoint_match
from modules.paths import sd_configs_path
from .execution_providers import ExecutionProvider, install_execution_provider
from .utils import check_diffusers_cache
@@ -74,7 +74,7 @@ def create_ui():
cache_remove_optimized.click(fn=remove_cache_optimized, inputs=[cache_state_dirname, cache_optimized_selected,])
def cache_update_menus(query: str):
checkpoint_info = get_closet_checkpoint_match(query)
checkpoint_info = get_closest_checkpoint_match(query)
if checkpoint_info is None:
log.error(f"Could not find checkpoint object for '{query}'.")
return
modules/pag/__init__.py
+2 -2
View File
@@ -42,8 +42,8 @@ def apply(p: processing.StableDiffusionProcessing): # pylint: disable=arguments-
pag_applied_layers_index = pag_applied_layers.split() if len(pag_applied_layers) > 0 else []
pag_applied_layers_index = [p.strip() for p in pag_applied_layers_index]
p.task_args['pag_applied_layers_index'] = pag_applied_layers_index if len(pag_applied_layers_index) > 0 else ['m0'] # Available layers: d[0-5], m[0], u[0-8]
p.extra_generation_params["PAG scale"] = p.pag_scale
p.extra_generation_params["PAG adaptive"] = p.pag_adaptive
p.extra_generation_params["CFG true"] = p.pag_scale
p.extra_generation_params["CFG adaptive"] = p.pag_adaptive
# shared.log.debug(f'{c}: args={p.task_args}')

Some files were not shown because too many files have changed in this diff Show More