Public/automatic
1
0
Fork 0
mirror of https://github.com/vladmandic/automatic synced 2026-09-19 17:24:32 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #3642 from vladmandic/dev

Browse Source 
Merge dev to master
This commit is contained in:
Vladimir Mandic
2024-12-24 08:51:08 -05:00
committed by GitHub
parent a2f5a39a59 56b29a3b9d
commit 123bf1fa95
204 changed files with 12008 additions and 1760 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.eslintrc.json
+7 -3
View File
@@ -37,14 +37,19 @@
"object-curly-newline":"off",
"prefer-rest-params":"off",
"prefer-destructuring":"off",
"radix":"off"
"radix":"off",
"node/shebang": "off"
},
"globals": {
// asssets
"panzoom": "readonly",
// script.js
// logger.js
"log": "readonly",
"debug": "readonly",
"error": "readonly",
"xhrGet": "readonly",
"xhrPost": "readonly",
// script.js
"gradioApp": "readonly",
"executeCallbacks": "readonly",
"onAfterUiUpdate": "readonly",
@@ -87,7 +92,6 @@
// settings.js
"registerDragDrop": "readonly",
// extraNetworks.js
"requestGet": "readonly",
"getENActiveTab": "readonly",
"quickApplyStyle": "readonly",
"quickSaveStyle": "readonly",
.github/ISSUE_TEMPLATE/bug_report.yml
+6 -2
View File
@@ -106,10 +106,14 @@ body:
- StableDiffusion 1.5
- StableDiffusion 2.1
- StableDiffusion XL
- StableDiffusion 3
- PixArt
- StableDiffusion 3.x
- StableCascade
- FLUX.1
- PixArt
- Kandinsky
- Playground
- AuraFlow
- Any Video Model
- Other
default: 0
validations:
.pylintrc
+1
View File
@@ -13,6 +13,7 @@ ignore-paths=/usr/lib/.*$,
modules/control/units,
modules/ctrlx,
modules/dml,
modules/freescale,
modules/ggml,
modules/hidiffusion,
modules/hijack,
.ruff.toml
+1
View File
@@ -7,6 +7,7 @@ exclude = [
"modules/consistory",
"modules/control/proc",
"modules/control/units",
"modules/freescale",
"modules/ggml",
"modules/hidiffusion",
"modules/hijack",
CHANGELOG.md
+222 -13
View File
@@ -1,21 +1,230 @@
# Change Log for SD.Next
## Update for 2024-11-22
## Update for 2024-12-24
- Model loader improvements:
### Highlights for 2024-12-24
### SD.Next Xmass edition: *What's new?*
While we have several new supported models, workflows and tools, this release is primarily about *quality-of-life improvements*:
- New memory management engine
list of changes that went into this one is long: changes to GPU offloading, brand new LoRA loader, system memory management, on-the-fly quantization, improved gguf loader, etc.
but main goal is enabling modern large models to run on standard consumer GPUs
without performance hits typically associated with aggressive memory swapping and needs for constant manual tweaks
- New [documentation website](https://vladmandic.github.io/sdnext-docs/)
with full search and tons of new documentation
- New settings panel with simplified and streamlined configuration
We've also added support for several new models such as highly anticipated [NVLabs Sana](https://huggingface.co/Efficient-Large-Model/Sana_1600M_1024px) (see [supported models](https://vladmandic.github.io/sdnext-docs/Model-Support/) for full list)
And several new SOTA video models: [Lightricks LTX-Video](https://huggingface.co/Lightricks/LTX-Video), [Hunyuan Video](https://huggingface.co/tencent/HunyuanVideo) and [Genmo Mochi.1 Preview](https://huggingface.co/genmo/mochi-1-preview)
And a lot of **Control** and **IPAdapter** goodies
- for **SDXL** there is new [ProMax](https://huggingface.co/xinsir/controlnet-union-sdxl-1.0), improved *Union* and *Tiling* models
- for **FLUX.1** there are [Flux Tools](https://blackforestlabs.ai/flux-1-tools/) as well as official *Canny* and *Depth* models,
a cool [Redux](https://huggingface.co/black-forest-labs/FLUX.1-Redux-dev) model as well as [XLabs](https://huggingface.co/XLabs-AI/flux-ip-adapter-v2) IP-adapter
- for **SD3.5** there are official *Canny*, *Blur* and *Depth* models in addition to existing 3rd party models
as well as [InstantX](https://huggingface.co/InstantX/SD3.5-Large-IP-Adapter) IP-adapter
Plus couple of new integrated workflows such as [FreeScale](https://github.com/ali-vilab/FreeScale) and [Style Aligned Image Generation](https://style-aligned-gen.github.io/)
And it wouldn't be a *Xmass edition* without couple of custom themes: *Snowflake* and *Elf-Green*!
All-in-all, we're around ~180 commits worth of updates, check the changelog for full list
[ReadMe](https://github.com/vladmandic/automatic/blob/master/README.md) | [ChangeLog](https://github.com/vladmandic/automatic/blob/master/CHANGELOG.md) | [Docs](https://vladmandic.github.io/sdnext-docs/) | [WiKi](https://github.com/vladmandic/automatic/wiki) | [Discord](https://discord.com/invite/sd-next-federal-batch-inspectors-1101998836328697867)
## Details for 2024-12-24
### New models and integrations
- [NVLabs Sana](https://huggingface.co/Efficient-Large-Model/Sana_1600M_1024px)
support for 1.6B 2048px, 1.6B 1024px and 0.6B 512px models
**Sana** can synthesize high-resolution images with strong text-image alignment by using **Gemma2** as text-encoder
and its *fast* - typically at least **2x** faster than sd-xl even for 1.6B variant and maintains performance regardless of resolution
e.g., rendering at 4k is possible in less than 8GB vram
to use, select from *networks -> models -> reference* and models will be auto-downloaded on first use
*reference values*: sampler: default (or any flow-match variant), steps: 20, width/height: 1024, guidance scale: 4.5
*note* like other LLM-based text-encoders, sana prefers long and descriptive prompts
any short prompt below 300 characters will be auto-expanded using built in Gemma LLM before encoding while long prompts will be passed as-is
- **ControlNet**
- improved support for **Union** controlnets with granular control mode type
- added support for latest [Xinsir ProMax](https://huggingface.co/xinsir/controlnet-union-sdxl-1.0) all-in-one controlnet
- added support for multiple **Tiling** controlnets, for example [Xinsir Tile](https://huggingface.co/xinsir/controlnet-tile-sdxl-1.0)
*note*: when selecting tiles in control settings, you can also specify non-square ratios
in which case it will use context-aware image resize to maintain overall composition
*note*: available tiling options can be set in settings -> control
- **IP-Adapter**
- FLUX.1 [XLabs](https://huggingface.co/XLabs-AI/flux-ip-adapter-v2) v1 and v2 IP-adapter
- FLUX.1 secondary guidance, enabled using *Attention guidance* in advanced menu
- SD 3.5 [InstantX](https://huggingface.co/InstantX/SD3.5-Large-IP-Adapter) IP-adapter
- [Flux Tools](https://blackforestlabs.ai/flux-1-tools/)
**Redux** is actually a tool, **Fill** is inpaint/outpaint optimized version of *Flux-dev*
**Canny** & **Depth** are optimized versions of *Flux-dev* for their respective tasks: they are *not* ControlNets that work on top of a model
to use, go to image or control interface and select *Flux Tools* in scripts
all models are auto-downloaded on first use
*note*: All models are [gated](https://github.com/vladmandic/automatic/wiki/Gated) and require acceptance of terms and conditions via web page
*recommended*: Enable on-the-fly [quantization](https://github.com/vladmandic/automatic/wiki/Quantization) or [compression](https://github.com/vladmandic/automatic/wiki/NNCF-Compression) to reduce resource usage
*todo*: support for Canny/Depth LoRAs
- [Redux](https://huggingface.co/black-forest-labs/FLUX.1-Redux-dev): ~0.1GB
works together with existing model and basically uses input image to analyze it and use that instead of prompt
*optional* can use prompt to combine guidance with input image
*recommended*: low denoise strength levels result in more variety
- [Fill](https://huggingface.co/black-forest-labs/FLUX.1-Fill-dev): ~23.8GB, replaces currently loaded model
*note*: can be used in inpaint/outpaint mode only
- [Canny](https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev): ~23.8GB, replaces currently loaded model
*recommended*: guidance scale 30
- [Depth](https://huggingface.co/black-forest-labs/FLUX.1-Depth-dev): ~23.8GB, replaces currently loaded model
*recommended*: guidance scale 10
- [Flux ControlNet LoRA](https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev-lora)
alternative to standard ControlNets, FLUX.1 also allows LoRA to help guide the generation process
both **Depth** and **Canny** LoRAs are available in standard control menus
- [StabilityAI SD35 ControlNets](https://huggingface.co/stabilityai/stable-diffusion-3.5-controlnets)
- In addition to previously released `InstantX` and `Alimama`, we now have *official* ones from StabilityAI
- [Style Aligned Image Generation](https://style-aligned-gen.github.io/)
enable in scripts, compatible with sd-xl
enter multiple prompts in prompt field separated by new line
style-aligned applies selected attention layers uniformly to all images to achive consistency
can be used with or without input image in which case first prompt is used to establish baseline
*note:* all prompts are processes as a single batch, so vram is limiting factor
- [FreeScale](https://github.com/ali-vilab/FreeScale)
enable in scripts, compatible with sd-xl for text and img2img
run iterative generation of images at different scales to achieve better results
can render 4k sdxl images
*note*: disable live preview to avoid memory issues when generating large images
### Video models
- [Lightricks LTX-Video](https://huggingface.co/Lightricks/LTX-Video)
model size: 27.75gb
support for 0.9.0, 0.9.1 and custom safetensor-based models with full quantization and offloading support
support for text-to-video and image-to-video, to use, select in *scripts -> ltx-video*
*refrence values*: steps 50, width 704, height 512, frames 161, guidance scale 3.0
- [Hunyuan Video](https://huggingface.co/tencent/HunyuanVideo)
model size: 40.92gb
support for text-to-video, to use, select in *scripts -> hunyuan video*
basic support only
*refrence values*: steps 50, width 1280, height 720, frames 129, guidance scale 6.0
- [Genmo Mochi.1 Preview](https://huggingface.co/genmo/mochi-1-preview)
support for text-to-video, to use, select in *scripts -> mochi.1 video*
basic support only
*refrence values*: steps 64, width 848, height 480, frames 19, guidance scale 4.5
*Notes*:
- all video models are very large and resource intensive!
any use on gpus below 16gb and systems below 48gb ram is experimental at best
- sdnext support for video models is relatively basic with further optimizations pending community interest
any future optimizations would likely have to go into partial loading and excecution instead of offloading inactive parts of the model
- new video models use generic llms for prompting and due to that requires very long and descriptive prompt
- you may need to enable sequential offload for maximum gpu memory savings
- optionally enable pre-quantization using bnb for additional memory savings
- reduce number of frames and/or resolution to reduce memory usage
### UI and workflow improvements
- **Docs**:
- New documentation site! <https://vladmandic.github.io/sdnext-docs/>
- Additional Wiki content: Styles, Wildcards, etc.
- **LoRA** handler rewrite:
- LoRA weights are no longer calculated on-the-fly during model execution, but are pre-calculated at the start
this results in perceived overhead on generate startup, but results in overall faster execution as LoRA does not need to be processed on each step
thanks @AI-Casanova
- LoRA weights can be applied/unapplied as on each generate or they can store weights backups for later use
this setting has large performance and resource implications, see [Offload](https://github.com/vladmandic/automatic/wiki/Offload) wiki for details
- LoRA name in prompt can now also be an absolute path to a LoRA file, even if LoRA is not indexed
example: `<lora:/test/folder/my-lora.safetensors:1.0>`
- LoRA name in prompt can now also be path to a LoRA file op `huggingface`
example: `<lora:/huggingface.co/vendor/repo/my-lora.safetensors:1.0>`
- **Model loader** improvements:
- detect model components on model load fail
- allow passing absolute path to model loader
- Flux, SD35: force unload model
- Flux: apply `bnb` quant when loading *unet/transformer*
- Flux: all-in-one safetensors
example: <https://civitai.com/models/646328?modelVersionId=1040235>
- Flux: do not recast quants
- Sampler improvements
- update DPM FlowMatch samplers
- Fixes:
- update `diffusers`
- fix README links
- fix sdxl controlnet single-file loader
- relax settings validator
- **Memory** improvements:
- faster and more compatible *balanced offload* mode
- balanced offload: units are now in percentage instead of bytes
- balanced offload: add both high and low watermark, defaults as below
`0.25` for low-watermark: skip offload if memory usage is below 25%
`0.70` high-watermark: must offload if memory usage is above 70%
- balanced offload will attempt to run offload as non-blocking and force gc at the end
- change-in-behavior:
low-end systems, triggered by either `lowvrwam` or by detection of <=4GB will use *sequential offload*
all other systems use *balanced offload* by default (can be changed in settings)
previous behavior was to use *model offload* on systems with <=8GB and `medvram` and no offload by default
- VAE upcase is now disabled by default on all systems
if you have issues with image decode, you'll need to enable it manually
- **UI**:
- improved stats on generate completion
- improved live preview display and performance
- improved accordion behavior
- auto-size networks height for sidebar
- control: hide preview column by default
- control: optionn to hide input column
- control: add stats
- settings: reorganized and simplified
- browser -> server logging framework
- add addtional themes: `black-reimagined`, thanks @Artheriax
- **Batch**
- image batch processing will use caption files if they exist instead of default prompt
### Updates
- **Quantization**
- Add `TorchAO` *pre* (during load) and *post* (during execution) quantization
**torchao** supports 4 different int-based and 3 float-based quantization schemes
This is in addition to existing support for:
- `BitsAndBytes` with 3 float-based quantization schemes
- `Optimium.Quanto` with 3 int-based and 2 float-based quantizations schemes
- `GGUF` with pre-quantized weights
- Switch `GGUF` loader from custom to diffuser native
- **IPEX**: update to IPEX 2.5.10+xpu
- **OpenVINO**:
- update to 2024.6.0
- disable model caching by default
- **Sampler** improvements
- UniPC, DEIS, SA, DPM-Multistep: allow FlowMatch sigma method and prediction type
- Euler FlowMatch: add sigma methods (*karras/exponential/betas*)
- Euler FlowMatch: allow using timestep presets to set sigmas
- DPM FlowMatch: update all and add sigma methods
- BDIA-DDIM: *experimental* new scheduler
- UFOGen: *experimental* new scheduler
### Fixes
- add `SD_NO_CACHE=true` env variable to disable file/folder caching
- add settings -> networks -> embeddings -> enable/disable
- update `diffusers`
- fix README links
- fix sdxl controlnet single-file loader
- relax settings validator
- improve js progress calls resiliency
- fix text-to-video pipeline
- avoid live-preview if vae-decode is running
- allow xyz-grid with multi-axis s&r
- fix xyz-grid with lora
- fix api script callbacks
- fix gpu memory monitoring
- simplify img2img/inpaint/sketch canvas handling
- fix prompt caching
- fix xyz grid skip final pass
- fix sd upscale script
- fix cogvideox-i2v
- lora auto-apply tags remove duplicates
- control load model on-demand if not already loaded
- taesd limit render to 2024px
- taesd downscale preview to 1024px max: configurable in settings -> live preview
- uninstall conflicting `wandb` package
- dont skip diffusers version check if quick is specified
- notify on torch install
- detect pipeline fro diffusers folder-style model
- do not recast flux quants
- fix xyz-grid with lora none
- fix svd image2video
- fix gallery display during generate
- fix wildcards replacement to be unique
- fix animatediff-xl
- fix pag with batch count
## Update for 2024-11-21
@@ -270,7 +479,7 @@ A month later and with nearly 300 commits, here is the latest [SD.Next](https://
#### New models for 2024-10-23
- New fine-tuned [CLiP-ViT-L]((https://huggingface.co/zer0int/CLIP-GmP-ViT-L-14)) 1st stage **text-encoders** used by most models (SD15/SDXL/SD3/Flux/etc.) brings additional details to your images
- New fine-tuned [CLiP-ViT-L](https://huggingface.co/zer0int/CLIP-GmP-ViT-L-14) 1st stage **text-encoders** used by most models (SD15/SDXL/SD3/Flux/etc.) brings additional details to your images
- New models:
[Stable Diffusion 3.5 Large](https://huggingface.co/stabilityai/stable-diffusion-3.5-large)
[OmniGen](https://arxiv.org/pdf/2409.11340)
@@ -370,7 +579,7 @@ And there are also other goodies like multiple *XYZ grid* improvements, addition
- xyz grid support for sampler options
- metadata updates for sampler options
- modernui updates for sampler options
- *note* sampler options defaults are not save in ui settings, they are saved in server settings
- *note* sampler options defaults are not saved in ui settings, they are saved in server settings
to apply your defaults, set ui values and apply via *system -> settings -> apply settings*
*sampler options*:
@@ -602,7 +811,7 @@ Examples:
- vae is list of manually downloaded safetensors
- text-encoder is list of predefined and manually downloaded text-encoders
- **controlnet** support:
support for **InstantX/Shakker-Labs** models including [Union-Pro](InstantX/FLUX.1-dev-Controlnet-Union)
support for **InstantX/Shakker-Labs** models including [Union-Pro](https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Union)
note that flux controlnet models are large, up to 6.6GB on top of already large base model!
as such, you may need to use offloading:sequential which is not as fast, but uses far less memory
when using union model, you must also select control mode in the control unit
@@ -2117,7 +2326,7 @@ Also new is support for **SDXL-Turbo** as well as new **Kandinsky 3** models and
- in *Advanced* params
- allows control of *latent clamping*, *color centering* and *range maximization*
- supported by *XYZ grid*
- [SD21 Turbo](https://huggingface.co/stabilityai/sd-turbo) and [SDXL Turbo](<https://huggingface.co/stabilityai/sdxl-turbo>) support
- [SD21 Turbo](https://huggingface.co/stabilityai/sd-turbo) and [SDXL Turbo](https://huggingface.co/stabilityai/sdxl-turbo) support
- just set CFG scale (0.0-1.0) and steps (1-3) to a very low value
- compatible with original StabilityAI SDXL-Turbo or any of the newer merges
- download safetensors or select from networks -> reference
README.md
+17 -65
View File
@@ -1,5 +1,5 @@
<div align="center">
<img src="https://github.com/vladmandic/automatic/blob/master/html/logo-transparent.png" width=200 alt="SD.Next">
<img src="https://github.com/vladmandic/automatic/raw/master/html/logo-transparent.png" width=200 alt="SD.Next">
**Image Diffusion implementation with advanced features**
@@ -8,15 +8,16 @@
[![Discord](https://img.shields.io/discord/1101998836328697867?logo=Discord&svg=true)](https://discord.gg/VjvR2tabEX)
[![Sponsors](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&color=%23fe8e86)](https://github.com/sponsors/vladmandic)
[Wiki](https://github.com/vladmandic/automatic/wiki) | [Discord](https://discord.gg/VjvR2tabEX) | [Changelog](CHANGELOG.md)
[Docs](https://vladmandic.github.io/sdnext-docs/) | [Wiki](https://github.com/vladmandic/automatic/wiki) | [Discord](https://discord.gg/VjvR2tabEX) | [Changelog](CHANGELOG.md)
</div>
</br>
## Table of contents
- [Documentation](https://vladmandic.github.io/sdnext-docs/)
- [SD.Next Features](#sdnext-features)
- [Model support](#model-support)
- [Model support](#model-support) and [Specifications]()
- [Platform support](#platform-support)
- [Getting started](#getting-started)
@@ -25,7 +26,7 @@
All individual features are not listed here, instead check [ChangeLog](CHANGELOG.md) for full list of changes
- Multiple UIs!
▹ **Standard | Modern**
- Multiple diffusion models!
- Multiple [diffusion models](https://vladmandic.github.io/sdnext-docs/Model-Support/)!
- Built-in Control for Text, Image, Batch and video processing!
- Multiplatform!
▹ **Windows | Linux | MacOS | nVidia | AMD | IntelArc/IPEX | DirectML | OpenVINO | ONNX+Olive | ZLUDA**
@@ -34,9 +35,7 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
- Platform specific autodetection and tuning performed on install
- Optimized processing with latest `torch` developments with built-in support for `torch.compile`
and multiple compile backends: *Triton, ZLUDA, StableFast, DeepCache, OpenVINO, NNCF, IPEX, OneDiff*
- Improved prompt parser
- Built-in queue management
- Enterprise level logging and hardened API
- Built in installer with automatic updates and dependency management
- Mobile compatible
@@ -49,42 +48,13 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
![screenshot-modernui](https://github.com/user-attachments/assets/39e3bc9a-a9f7-4cda-ba33-7da8def08032)
For screenshots and informations on other available themes, see [Themes Wiki](https://github.com/vladmandic/automatic/wiki/Themes)
For screenshots and informations on other available themes, see [Themes](https://vladmandic.github.io/sdnext-docs/Themes/)
<br>
## Model support
Additional models will be added as they become available and there is public interest in them
See [models overview](https://github.com/vladmandic/automatic/wiki/Models) for details on each model, including their architecture, complexity and other info
- [RunwayML Stable Diffusion](https://github.com/Stability-AI/stablediffusion/) 1.x and 2.x *(all variants)*
- [StabilityAI Stable Diffusion XL](https://github.com/Stability-AI/generative-models), [StabilityAI Stable Diffusion 3.0](https://stability.ai/news/stable-diffusion-3-medium) Medium, [StabilityAI Stable Diffusion 3.5](https://huggingface.co/stabilityai/stable-diffusion-3.5-large) Medium, Large, Large Turbo
- [StabilityAI Stable Video Diffusion](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid) Base, XT 1.0, XT 1.1
- [StabilityAI Stable Cascade](https://github.com/Stability-AI/StableCascade) *Full* and *Lite*
- [Black Forest Labs FLUX.1](https://blackforestlabs.ai/announcing-black-forest-labs/) Dev, Schnell
- [AuraFlow](https://huggingface.co/fal/AuraFlow)
- [AlphaVLLM Lumina-Next-SFT](https://huggingface.co/Alpha-VLLM/Lumina-Next-SFT-diffusers)
- [Playground AI](https://huggingface.co/playgroundai/playground-v2-256px-base) *v1, v2 256, v2 512, v2 1024 and latest v2.5*
- [Tencent HunyuanDiT](https://github.com/Tencent/HunyuanDiT)
- [OmniGen](https://arxiv.org/pdf/2409.11340)
- [Meissonic](https://github.com/viiika/Meissonic)
- [Kwai Kolors](https://huggingface.co/Kwai-Kolors/Kolors)
- [CogView 3+](https://huggingface.co/THUDM/CogView3-Plus-3B)
- [LCM: Latent Consistency Models](https://github.com/openai/consistency_models)
- [aMUSEd](https://huggingface.co/amused/amused-256) 256 and 512
- [Segmind Vega](https://huggingface.co/segmind/Segmind-Vega), [Segmind SSD-1B](https://huggingface.co/segmind/SSD-1B), [Segmind SegMoE](https://github.com/segmind/segmoe) *SD and SD-XL*, [Segmind SD Distilled](https://huggingface.co/blog/sd_distillation) *(all variants)*
- [Kandinsky](https://github.com/ai-forever/Kandinsky-2) *2.1 and 2.2 and latest 3.0*
- [PixArt-α XL 2](https://github.com/PixArt-alpha/PixArt-alpha) *Medium and Large*, [PixArt-Σ](https://github.com/PixArt-alpha/PixArt-sigma)
- [Warp Wuerstchen](https://huggingface.co/blog/wuertschen)
- [Tsinghua UniDiffusion](https://github.com/thu-ml/unidiffuser)
- [DeepFloyd IF](https://github.com/deep-floyd/IF) *Medium and Large*
- [ModelScope T2V](https://huggingface.co/damo-vilab/text-to-video-ms-1.7b)
- [BLIP-Diffusion](https://dxli94.github.io/BLIP-Diffusion-website/)
- [KOALA 700M](https://github.com/youngwanLEE/sdxl-koala)
- [VGen](https://huggingface.co/ali-vilab/i2vgen-xl)
- [SDXS](https://github.com/IDKiro/sdxs)
- [Hyper-SD](https://huggingface.co/ByteDance/Hyper-SD)
SD.Next supports broad range of models: [supported models](https://vladmandic.github.io/sdnext-docs/Model-Support/) and [model specs](https://vladmandic.github.io/sdnext-docs/Models/)
## Platform support
@@ -97,47 +67,29 @@ See [models overview](https://github.com/vladmandic/automatic/wiki/Models) for d
- Any GPU or device compatible with **OpenVINO** libraries on both *Windows and Linux*
- *Apple M1/M2* on *OSX* using built-in support in Torch with **MPS** optimizations
- *ONNX/Olive*
- *AMD* GPUs on Windows using **ZLUDA** libraries
- *AMD* GPUs on Windows using **ZLUDA** libraries
## Getting started
- Get started with **SD.Next** by following the [installation instructions](https://github.com/vladmandic/automatic/wiki/Installation)
- For more details, check out [advanced installation](https://github.com/vladmandic/automatic/wiki/Advanced-Install) guide
- List and explanation of [command line arguments](https://github.com/vladmandic/automatic/wiki/CLI-Arguments)
- Get started with **SD.Next** by following the [installation instructions](https://vladmandic.github.io/sdnext-docs/Installation/)
- For more details, check out [advanced installation](https://vladmandic.github.io/sdnext-docs/Advanced-Install/) guide
- List and explanation of [command line arguments](https://vladmandic.github.io/sdnext-docs/CLI-Arguments/)
- Install walkthrough [video](https://www.youtube.com/watch?v=nWTnTyFTuAs)
> [!TIP]
> And for platform specific information, check out
> [WSL](https://github.com/vladmandic/automatic/wiki/WSL) | [Intel Arc](https://github.com/vladmandic/automatic/wiki/Intel-ARC) | [DirectML](https://github.com/vladmandic/automatic/wiki/DirectML) | [OpenVINO](https://github.com/vladmandic/automatic/wiki/OpenVINO) | [ONNX & Olive](https://github.com/vladmandic/automatic/wiki/ONNX-Runtime) | [ZLUDA](https://github.com/vladmandic/automatic/wiki/ZLUDA) | [AMD ROCm](https://github.com/vladmandic/automatic/wiki/AMD-ROCm) | [MacOS](https://github.com/vladmandic/automatic/wiki/MacOS-Python.md) | [nVidia](https://github.com/vladmandic/automatic/wiki/nVidia)
> [WSL](https://vladmandic.github.io/sdnext-docs/WSL/) | [Intel Arc](https://vladmandic.github.io/sdnext-docs/Intel-ARC/) | [DirectML](https://vladmandic.github.io/sdnext-docs/DirectML/) | [OpenVINO](https://vladmandic.github.io/sdnext-docs/OpenVINO/) | [ONNX & Olive](https://vladmandic.github.io/sdnext-docs/ONNX-Runtime/) | [ZLUDA](https://vladmandic.github.io/sdnext-docs/ZLUDA/) | [AMD ROCm](https://vladmandic.github.io/sdnext-docs/AMD-ROCm/) | [MacOS](https://vladmandic.github.io/sdnext-docs/MacOS-Python/) | [nVidia](https://vladmandic.github.io/sdnext-docs/nVidia/) | [Docker](https://vladmandic.github.io/sdnext-docs/Docker/)
> [!WARNING]
> If you run into issues, check out [troubleshooting](https://github.com/vladmandic/automatic/wiki/Troubleshooting) and [debugging](https://github.com/vladmandic/automatic/wiki/Debug) guides
> If you run into issues, check out [troubleshooting](https://vladmandic.github.io/sdnext-docs/Troubleshooting/) and [debugging](https://vladmandic.github.io/sdnext-docs/Debug/) guides
> [!TIP]
> All command line options can also be set via env variable
> All command line options can also be set via env variable
> For example `--debug` is same as `set SD_DEBUG=true`
## Backend support
**SD.Next** supports two main backends: *Diffusers* and *Original*:
- **Diffusers**: Based on new [Huggingface Diffusers](https://huggingface.co/docs/diffusers/index) implementation
Supports *all* models listed below
This backend is set as default for new installations
- **Original**: Based on [LDM](https://github.com/Stability-AI/stablediffusion) reference implementation and significantly expanded on by [A1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui)
This backend and is fully compatible with most existing functionality and extensions written for *A1111 SDWebUI*
Supports **SD 1.x** and **SD 2.x** models
All other model types such as *SD-XL, LCM, Stable Cascade, PixArt, Playground, Segmind, Kandinsky, etc.* require backend **Diffusers**
### Collab
- We'd love to have additional maintainers (with comes with full repo rights). If you're interested, ping us!
- In addition to general cross-platform code, desire is to have a lead for each of the main platforms
This should be fully cross-platform, but we'd really love to have additional contributors and/or maintainers to join and help lead the efforts on different platforms
### Credits
- Main credit goes to [Automatic1111 WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) for original codebase
- Main credit goes to [Automatic1111 WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) for the original codebase
- Additional credits are listed in [Credits](https://github.com/AUTOMATIC1111/stable-diffusion-webui/#credits)
- Licenses for modules are listed in [Licenses](html/licenses.html)
@@ -154,8 +106,8 @@ This should be fully cross-platform, but we'd really love to have additional con
### Docs
If you're unsure how to use a feature, best place to start is [Wiki](https://github.com/vladmandic/automatic/wiki) and if its not there,
check [ChangeLog](CHANGELOG.md) for when feature was first introduced as it will always have a short note on how to use it
If you're unsure how to use a feature, best place to start is [Docs](https://vladmandic.github.io/sdnext-docs/) and if its not there,
check [ChangeLog](https://vladmandic.github.io/sdnext-docs/CHANGELOG/) for when feature was first introduced as it will always have a short note on how to use it
### Sponsors
TODO.md
+21 -12
View File
@@ -2,21 +2,30 @@
Main ToDo list can be found at [GitHub projects](https://github.com/users/vladmandic/projects)
## Pending
- LoRA direct with caching
- Previewer issues
- Redesign postprocessing
## Future Candidates
- SD35 IPAdapter: <https://github.com/huggingface/diffusers/issues/9966>
- SD35 LoRA: <https://github.com/huggingface/diffusers/issues/9950>
- Flux IPAdapter: <https://github.com/huggingface/diffusers/issues/9825>
- Flux Fill/ControlNet/Redux: <https://github.com/huggingface/diffusers/pull/9985>
- Flux NF4: <https://github.com/huggingface/diffusers/issues/9996>
- SANA: <https://github.com/huggingface/diffusers/pull/9982>
## Other
- Flux NF4 loader: <https://github.com/huggingface/diffusers/issues/9996>
- IPAdapter negative: <https://github.com/huggingface/diffusers/discussions/7167>
- Control API enhance scripts compatibility
- PixelSmith: <https://github.com/Thanos-DB/Pixelsmith>
## Workaround in place
## Code TODO
- GGUF <https://github.com/huggingface/diffusers/issues/9487>
- FlowMatch <https://github.com/huggingface/diffusers/issues/9607> <https://github.com/huggingface/diffusers/issues/9924>
- TODO install: python 3.12.4 or higher cause a mess with pydantic
- TODO install: enable ROCm for windows when available
- TODO resize image: enable full VAE mode for resize-latent
- TODO processing: remove duplicate mask params
- TODO flux: fix loader for civitai nf4 models
- TODO model loader: implement model in-memory caching
- TODO hypertile: vae breaks when using non-standard sizes
- TODO model load: force-reloading entire model as loading transformers only leads to massive memory usage
- TODO lora load: direct with bnb
- TODO lora make: support quantized flux
- TODO control: support scripts via api
- TODO modernui: monkey-patch for missing tabs.select event
cli/api-model.js
Executable
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env node
const sd_url = process.env.SDAPI_URL || 'http://127.0.0.1:7860';
const sd_username = process.env.SDAPI_USR;
const sd_password = process.env.SDAPI_PWD;
const models = [
'/mnt/models/stable-diffusion/sd15/lyriel_v16.safetensors',
'/mnt/models/stable-diffusion/flux/flux-finesse_v2-f1h-fp8.safetensors',
'/mnt/models/stable-diffusion/sdxl/TempestV0.1-Artistic.safetensors',
];
async function options(data) {
const method = 'POST';
const headers = new Headers();
const body = JSON.stringify(data);
headers.set('Content-Type', 'application/json');
if (sd_username && sd_password) headers.set({ Authorization: `Basic ${btoa('sd_username:sd_password')}` });
const res = await fetch(`${sd_url}/sdapi/v1/options`, { method, headers, body });
return res;
}
async function main() {
for (const model of models) {
console.log('model:', model);
const res = await options({ sd_model_checkpoint: model });
console.log('result:', res);
}
}
main();
cli/api-pulid.js
+17 -7
View File
@@ -10,12 +10,13 @@ const argparse = require('argparse');
const sd_url = process.env.SDAPI_URL || 'http://127.0.0.1:7860';
const sd_username = process.env.SDAPI_USR;
const sd_password = process.env.SDAPI_PWD;
let args = {};
function b64(file) {
const data = fs.readFileSync(file);
const b64 = Buffer.from(data).toString('base64');
const b64str = Buffer.from(data).toString('base64');
const ext = path.extname(file).replace('.', '');
str = `data:image/${ext};base64,${b64}`;
const str = `data:image/${ext};base64,${b64str}`;
// console.log('b64:', ext, b64.length);
return str;
}
@@ -39,7 +40,16 @@ function options() {
if (args.pulid) {
const b64image = b64(args.pulid);
opt.script_name = 'pulid';
opt.script_args = [b64image, 0.9];
opt.script_args = [
b64image, // b64 encoded image, required param
0.9, // strength, optional
20, // zero, optional
'dpmpp_sde', // sampler, optional
'v2', // ortho, optional
true, // restore (disable pulid after run), optional
true, // offload, optional
'v1.1', // version, optional
];
}
// console.log('options:', opt);
return opt;
@@ -53,8 +63,8 @@ function init() {
parser.add_argument('--height', { type: 'int', help: 'height' });
parser.add_argument('--pulid', { type: 'str', help: 'pulid init image' });
parser.add_argument('--output', { type: 'str', help: 'output path' });
const args = parser.parse_args();
return args
const parsed = parser.parse_args();
return parsed;
}
async function main() {
@@ -73,12 +83,12 @@ async function main() {
console.log('result:', json.info);
for (const i in json.images) { // eslint-disable-line guard-for-in
const file = args.output || `/tmp/test-${i}.jpg`;
const data = atob(json.images[i])
const data = atob(json.images[i]);
fs.writeFileSync(file, data, 'binary');
console.log('image saved:', file);
}
}
}
const args = init();
args = init();
main();
cli/full-test.sh
+3
View File
@@ -1,5 +1,8 @@
#!/usr/bin/env bash
node cli/api-txt2img.js
node cli/api-pulid.js
source venv/bin/activate
echo image-exif
python cli/api-info.py --input html/logo-bg-0.jpg
cli/load-unet.py
+2 -2
View File
@@ -33,13 +33,13 @@ def set_module_tensor(
stats.dtypes[value.dtype] = 0
stats.dtypes[value.dtype] += 1
if name in module._buffers: # pylint: disable=protected-access
module._buffers[name] = value.to(device=device, dtype=dtype, non_blocking=True) # pylint: disable=protected-access
module._buffers[name] = value.to(device=device, dtype=dtype) # pylint: disable=protected-access
if 'buffers' not in stats.weights:
stats.weights['buffers'] = 0
stats.weights['buffers'] += 1
elif value is not None:
param_cls = type(module._parameters[name]) # pylint: disable=protected-access
module._parameters[name] = param_cls(value, requires_grad=old_value.requires_grad).to(device, dtype=dtype, non_blocking=True) # pylint: disable=protected-access
module._parameters[name] = param_cls(value, requires_grad=old_value.requires_grad).to(device, dtype=dtype) # pylint: disable=protected-access
if 'parameters' not in stats.weights:
stats.weights['parameters'] = 0
stats.weights['parameters'] += 1
configs/flux/vae/config.json
+1 -1
View File
@@ -14,7 +14,7 @@
"DownEncoderBlock2D",
"DownEncoderBlock2D"
],
"force_upcast": true,
"force_upcast": false,
"in_channels": 3,
"latent_channels": 16,
"latents_mean": null,
configs/sd15/vae/config.json
+1
View File
@@ -14,6 +14,7 @@
"DownEncoderBlock2D",
"DownEncoderBlock2D"
],
"force_upcast": false,
"in_channels": 3,
"latent_channels": 4,
"layers_per_block": 2,
configs/sd3/vae/config.json
+1 -1
View File
@@ -15,7 +15,7 @@
"DownEncoderBlock2D",
"DownEncoderBlock2D"
],
"force_upcast": true,
"force_upcast": false,
"in_channels": 3,
"latent_channels": 16,
"latents_mean": null,
configs/sdxl/vae/config.json
+1 -1
View File
@@ -15,7 +15,7 @@
"DownEncoderBlock2D",
"DownEncoderBlock2D"
],
"force_upcast": true,
"force_upcast": false,
"in_channels": 3,
"latent_channels": 4,
"layers_per_block": 2,
extensions-builtin/Lora/lora_extract.py
-13
View File
@@ -182,19 +182,6 @@ def make_lora(fn, maxrank, auto_rank, rank_ratio, modules, overwrite):
progress.remove_task(task)
t3 = time.time()
# TODO: Handle quant for Flux
# if 'te' in modules and getattr(shared.sd_model, 'transformer', None) is not None:
# for name, module in shared.sd_model.transformer.named_modules():
# if "norm" in name and "linear" not in name:
# continue
# weights_backup = getattr(module, "network_weights_backup", None)
# if weights_backup is None:
# continue
# module.svdhandler = SVDHandler()
# module.svdhandler.network_name = "lora_transformer_" + name.replace(".", "_")
# module.svdhandler.decompose(module.weight, weights_backup)
# module.svdhandler.findrank(rank, rank_ratio)
lora_state_dict = {}
for sub in ['text_encoder', 'text_encoder_2', 'unet', 'transformer']:
submodel = getattr(shared.sd_model, sub, None)
extensions-builtin/Lora/network_lora.py
+2 -5
View File
@@ -22,7 +22,6 @@ class NetworkModuleLora(network.NetworkModule):
self.dim = weights.w["lora_down.weight"].shape[0]
def create_module(self, weights, key, none_ok=False):
from modules.shared import opts
weight = weights.get(key)
if weight is None and none_ok:
return None
@@ -32,7 +31,7 @@ class NetworkModuleLora(network.NetworkModule):
if is_linear:
weight = weight.reshape(weight.shape[0], -1)
module = torch.nn.Linear(weight.shape[1], weight.shape[0], bias=False)
elif is_conv and key == "lora_down.weight" or key == "dyn_up":
elif is_conv and (key == "lora_down.weight" or key == "dyn_up"):
if len(weight.shape) == 2:
weight = weight.reshape(weight.shape[0], -1, 1, 1)
if weight.shape[2] != 1 or weight.shape[3] != 1:
@@ -41,7 +40,7 @@ class NetworkModuleLora(network.NetworkModule):
module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], (1, 1), bias=False)
elif is_conv and key == "lora_mid.weight":
module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], self.sd_module.kernel_size, self.sd_module.stride, self.sd_module.padding, bias=False)
elif is_conv and key == "lora_up.weight" or key == "dyn_down":
elif is_conv and (key == "lora_up.weight" or key == "dyn_down"):
module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], (1, 1), bias=False)
else:
raise AssertionError(f'Lora unsupported: layer={self.network_key} type={type(self.sd_module).__name__}')
@@ -49,8 +48,6 @@ class NetworkModuleLora(network.NetworkModule):
if weight.shape != module.weight.shape:
weight = weight.reshape(module.weight.shape)
module.weight.copy_(weight)
if opts.lora_load_gpu:
module = module.to(device=devices.device, dtype=devices.dtype)
module.weight.requires_grad_(False)
return module
extensions-builtin/Lora/network_overrides.py
-1
View File
@@ -26,7 +26,6 @@ force_diffusers = [ # forced always
force_models = [ # forced always
'sc',
# 'sd3',
'kandinsky',
'hunyuandit',
'auraflow',
extensions-builtin/Lora/networks.py
+2 -2
View File
@@ -88,7 +88,7 @@ def assign_network_names_to_compvis_modules(sd_model):
network_name = name.replace(".", "_")
network_layer_mapping[network_name] = module
module.network_layer_name = network_name
shared.sd_model.network_layer_mapping = network_layer_mapping
sd_model.network_layer_mapping = network_layer_mapping
def load_diffusers(name, network_on_disk, lora_scale=shared.opts.extra_networks_default_multiplier) -> network.Network:
@@ -141,7 +141,7 @@ def load_network(name, network_on_disk) -> network.Network:
sd = sd_models.read_state_dict(network_on_disk.filename, what='network')
if shared.sd_model_type == 'f1': # if kohya flux lora, convert state_dict
sd = lora_convert._convert_kohya_flux_lora_to_diffusers(sd) or sd # pylint: disable=protected-access
assign_network_names_to_compvis_modules(shared.sd_model) # this should not be needed but is here as an emergency fix for an unknown error people are experiencing in 1.2.0
assign_network_names_to_compvis_modules(shared.sd_model)
keys_failed_to_match = {}
matched_networks = {}
bundle_embeddings = {}
extensions-builtin/Lora/scripts/lora_script.py
+7 -6
View File
@@ -5,7 +5,7 @@ from lora_extract import create_ui
from network import NetworkOnDisk
from ui_extra_networks_lora import ExtraNetworksPageLora
from extra_networks_lora import ExtraNetworkLora
from modules import script_callbacks, extra_networks, ui_extra_networks, ui_models # pylint: disable=unused-import
from modules import script_callbacks, extra_networks, ui_extra_networks, ui_models, shared # pylint: disable=unused-import
re_lora = re.compile("<lora:([^:]+):")
@@ -57,8 +57,9 @@ def infotext_pasted(infotext, d): # pylint: disable=unused-argument
d["Prompt"] = re.sub(re_lora, network_replacement, d["Prompt"])
script_callbacks.on_app_started(api_networks)
script_callbacks.on_before_ui(before_ui)
script_callbacks.on_model_loaded(networks.assign_network_names_to_compvis_modules)
script_callbacks.on_infotext_pasted(networks.infotext_pasted)
script_callbacks.on_infotext_pasted(infotext_pasted)
if not shared.native:
script_callbacks.on_app_started(api_networks)
script_callbacks.on_before_ui(before_ui)
script_callbacks.on_model_loaded(networks.assign_network_names_to_compvis_modules)
script_callbacks.on_infotext_pasted(networks.infotext_pasted)
script_callbacks.on_infotext_pasted(infotext_pasted)
extensions-builtin/sd-extension-system-info
+1 -1
Submodule extensions-builtin/sd-extension-system-info updated: 6a2a28a4f6...dfa01ce99a
extensions-builtin/sdnext-modernui
+1 -1
Submodule extensions-builtin/sdnext-modernui updated: 4647bd7f86...3f53ff719f
extensions-builtin/stable-diffusion-webui-rembg
+1 -1
Submodule extensions-builtin/stable-diffusion-webui-rembg updated: ff2bbd1682...50bc931ce6
html/locale_en.json
+1 -1
View File
@@ -223,7 +223,7 @@
{"id":"","label":"System Paths","localized":"","hint":"Settings related to location of various model directories"},
{"id":"","label":"Image Options","localized":"","hint":"Settings related to image format, metadata, and image grids"},
{"id":"","label":"image naming & paths","localized":"","hint":"Settings related to image filenames, and output directories"},
{"id":"","label":"User Interface Options","localized":"","hint":"Settings related to user interface themes, and Quicksettings list"},
{"id":"","label":"User Interface","localized":"","hint":"Settings related to user interface themes, and Quicksettings list"},
{"id":"","label":"Live Previews","localized":"","hint":"Settings related to live previews, audio notification, and log view"},
{"id":"","label":"Sampler Settings","localized":"","hint":"Settings related to sampler selection and configuration, and diffuser specific sampler configuration"},
{"id":"","label":"Postprocessing","localized":"","hint":"Settings related to post image generation processing, face restoration, and upscaling"},
html/reference.json
+19
View File
@@ -180,6 +180,25 @@
"extras": "sampler: Default, cfg_scale: 3.5"
},
"NVLabs Sana 1.6B 2048px": {
"path": "Efficient-Large-Model/Sana_1600M_2Kpx_BF16_diffusers",
"desc": "Sana is a text-to-image framework that can efficiently generate images up to 4096 × 4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU.",
"preview": "Efficient-Large-Model--Sana_1600M_1024px_diffusers.jpg",
"skip": true
},
"NVLabs Sana 1.6B 1024px": {
"path": "Efficient-Large-Model/Sana_1600M_1024px_diffusers",
"desc": "Sana is a text-to-image framework that can efficiently generate images up to 4096 × 4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU.",
"preview": "Efficient-Large-Model--Sana_1600M_1024px_diffusers.jpg",
"skip": true
},
"NVLabs Sana 0.6B 512px": {
"path": "Efficient-Large-Model/Sana_600M_512px_diffusers",
"desc": "Sana is a text-to-image framework that can efficiently generate images up to 4096 × 4096 resolution. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU.",
"preview": "Efficient-Large-Model--Sana_1600M_1024px_diffusers.jpg",
"skip": true
},
"VectorSpaceLab OmniGen v1": {
"path": "Shitao/OmniGen-v1",
"desc": "OmniGen is a unified image generation model that can generate a wide range of images from multi-modal prompts. It is designed to be simple, flexible and easy to use.",
installer.py
+37 -21
View File
@@ -212,7 +212,7 @@ def installed(package, friendly: str = None, reload = False, quiet = False):
if friendly:
pkgs = friendly.split()
else:
pkgs = [p for p in package.split() if not p.startswith('-') and not p.startswith('=')]
pkgs = [p for p in package.split() if not p.startswith('-') and not p.startswith('=') and not p.startswith('git+')]
pkgs = [p.split('/')[-1] for p in pkgs] # get only package name if installing from url
for pkg in pkgs:
if '!=' in pkg:
@@ -250,7 +250,7 @@ def uninstall(package, quiet = False):
if installed(p, p, quiet=True):
if not quiet:
log.warning(f'Package: {p} uninstall')
res += pip(f"uninstall {p} --yes --quiet", ignore=True, quiet=True)
res += pip(f"uninstall {p} --yes --quiet", ignore=True, quiet=True, uv=False)
return res
@@ -295,7 +295,7 @@ def install(package, friendly: str = None, ignore: bool = False, reinstall: bool
quick_allowed = False
if args.reinstall or reinstall or not installed(package, friendly, quiet=quiet):
deps = '' if not no_deps else '--no-deps '
res = pip(f"install{' --upgrade' if not args.uv else ''} {deps}{package}", ignore=ignore, uv=package != "uv")
res = pip(f"install{' --upgrade' if not args.uv else ''} {deps}{package}", ignore=ignore, uv=package != "uv" and not package.startswith('git+'))
try:
import importlib # pylint: disable=deprecated-module
importlib.reload(pkg_resources)
@@ -430,7 +430,7 @@ def check_python(supported_minors=[9, 10, 11, 12], reason=None):
if args.quick:
return
log.info(f'Python: version={platform.python_version()} platform={platform.system()} bin="{sys.executable}" venv="{sys.prefix}"')
if int(sys.version_info.major) == 3 and int(sys.version_info.minor) == 12 and int(sys.version_info.micro) > 3: # TODO python 3.12.4 or higher cause a mess with pydantic
if int(sys.version_info.major) == 3 and int(sys.version_info.minor) == 12 and int(sys.version_info.micro) > 3: # TODO install: python 3.12.4 or higher cause a mess with pydantic
log.error(f"Python version incompatible: {sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro} required 3.12.3 or lower")
if reason is not None:
log.error(reason)
@@ -457,9 +457,9 @@ def check_python(supported_minors=[9, 10, 11, 12], reason=None):
# check diffusers version
def check_diffusers():
if args.skip_all or args.skip_requirements:
if args.skip_all or args.skip_git:
return
sha = 'b5fd6f13f5434d69d919cc8cedf0b11db664cf06'
sha = '6dfaec348780c6153a4cfd03a01972a291d67f82' # diffusers commit hash
pkg = pkg_resources.working_set.by_key.get('diffusers', None)
minor = int(pkg.version.split('.')[1] if pkg is not None else 0)
cur = opts.get('diffusers_version', '') if minor > 0 else ''
@@ -483,6 +483,7 @@ def check_onnx():
def check_torchao():
"""
if args.skip_all or args.skip_requirements:
return
if installed('torchao', quiet=True):
@@ -492,6 +493,8 @@ def check_torchao():
pip('uninstall --yes torchao', ignore=True, quiet=True, uv=False)
for m in [m for m in sys.modules if m.startswith('torchao')]:
del sys.modules[m]
"""
return
def install_cuda():
@@ -549,7 +552,7 @@ def install_rocm_zluda():
log.info(msg)
torch_command = ''
if sys.platform == "win32":
# TODO after ROCm for Windows is released
# TODO install: enable ROCm for windows when available
if args.device_id is not None:
if os.environ.get('HIP_VISIBLE_DEVICES', None) is not None:
@@ -634,25 +637,30 @@ def install_ipex(torch_command):
os.environ.setdefault('NEOReadDebugKeys', '1')
if os.environ.get("ClDeviceGlobalMemSizeAvailablePercent", None) is None:
os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
if os.environ.get("PYTORCH_ENABLE_XPU_FALLBACK", None) is None:
os.environ.setdefault('PYTORCH_ENABLE_XPU_FALLBACK', '1')
if "linux" in sys.platform:
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.3.1+cxx11.abi torchvision==0.18.1+cxx11.abi intel-extension-for-pytorch==2.3.110+xpu oneccl_bind_pt==2.3.100+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/')
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.5.1+cxx11.abi torchvision==0.20.1+cxx11.abi intel-extension-for-pytorch==2.5.10+xpu oneccl_bind_pt==2.5.0+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/cn/')
# torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision --index-url https://download.pytorch.org/whl/test/xpu') # test wheels are stable previews, significantly slower than IPEX
# os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow==2.15.1 intel-extension-for-tensorflow[xpu]==2.15.0.1')
else:
torch_command = os.environ.get('TORCH_COMMAND', '--pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/xpu') # torchvision doesn't exist on test/stable branch for windows
install(os.environ.get('OPENVINO_PACKAGE', 'openvino==2024.3.0'), 'openvino', ignore=True)
install('nncf==2.7.0', 'nncf', ignore=True)
install(os.environ.get('OPENVINO_PACKAGE', 'openvino==2024.5.0'), 'openvino', ignore=True)
install('nncf==2.7.0', ignore=True, no_deps=True) # requires older pandas
install(os.environ.get('ONNXRUNTIME_PACKAGE', 'onnxruntime-openvino'), 'onnxruntime-openvino', ignore=True)
return torch_command
def install_openvino(torch_command):
check_python(supported_minors=[8, 9, 10, 11, 12], reason='OpenVINO backend requires Python 3.9, 3.10 or 3.11')
check_python(supported_minors=[9, 10, 11, 12], reason='OpenVINO backend requires Python 3.9, 3.10 or 3.11')
log.info('OpenVINO: selected')
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.3.1+cpu torchvision==0.18.1+cpu --index-url https://download.pytorch.org/whl/cpu')
install(os.environ.get('OPENVINO_PACKAGE', 'openvino==2024.3.0'), 'openvino')
if sys.platform == 'darwin':
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.3.1 torchvision==0.18.1')
else:
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.3.1+cpu torchvision==0.18.1+cpu --index-url https://download.pytorch.org/whl/cpu')
install(os.environ.get('OPENVINO_PACKAGE', 'openvino==2024.6.0'), 'openvino')
install(os.environ.get('ONNXRUNTIME_PACKAGE', 'onnxruntime-openvino'), 'onnxruntime-openvino', ignore=True)
install('nncf==2.12.0', 'nncf')
install('nncf==2.14.1', 'nncf')
os.environ.setdefault('PYTORCH_TRACING_MODE', 'TORCHFX')
if os.environ.get("NEOReadDebugKeys", None) is None:
os.environ.setdefault('NEOReadDebugKeys', '1')
@@ -682,7 +690,9 @@ def install_torch_addons():
if opts.get('nncf_compress_weights', False) and not args.use_openvino:
install('nncf==2.7.0', 'nncf')
if opts.get('optimum_quanto_weights', False):
install('optimum-quanto', 'optimum-quanto')
install('optimum-quanto==0.2.6', 'optimum-quanto')
if not args.experimental:
uninstall('wandb', quiet=True)
if triton_command is not None:
install(triton_command, 'triton', quiet=True)
@@ -727,8 +737,6 @@ def check_torch():
torch_command = install_rocm_zluda()
elif is_ipex_available:
torch_command = install_ipex(torch_command)
elif allow_openvino:
torch_command = install_openvino(torch_command)
else:
machine = platform.machine()
@@ -746,6 +754,8 @@ def check_torch():
log.warning('Torch: CPU-only version installed')
torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision')
if 'torch' in torch_command and not args.version:
if not installed('torch'):
log.info(f'Torch: download and install in progress... cmd="{torch_command}"')
install(torch_command, 'torch torchvision', quiet=True)
else:
try:
@@ -999,8 +1009,8 @@ def install_optional():
install('basicsr')
install('gfpgan')
install('clean-fid')
install('optimum-quanto', ignore=True)
install('bitsandbytes', ignore=True)
install('optimum-quanto=0.2.6', ignore=True)
install('bitsandbytes==0.45.0', ignore=True)
install('pynvml', ignore=True)
install('ultralytics==8.3.40', ignore=True)
install('Cython', ignore=True)
@@ -1174,9 +1184,15 @@ def check_ui(ver):
def check_venv():
def try_relpath(p):
try:
return os.path.relpath(p)
except ValueError:
return p
import site
pkg_path = [os.path.relpath(p) for p in site.getsitepackages() if os.path.exists(p)]
log.debug(f'Packages: venv={os.path.relpath(sys.prefix)} site={pkg_path}')
pkg_path = [try_relpath(p) for p in site.getsitepackages() if os.path.exists(p)]
log.debug(f'Packages: venv={try_relpath(sys.prefix)} site={pkg_path}')
for p in pkg_path:
invalid = []
for f in os.listdir(p):
javascript/base.css
+2 -3
View File
@@ -25,7 +25,6 @@
.progressDiv .progress { width: 0%; height: 20px; background: #0060df; color: white; font-weight: bold; line-height: 20px; padding: 0 8px 0 0; text-align: right; overflow: visible; white-space: nowrap; padding: 0 0.5em; }
.livePreview { position: absolute; z-index: 50; background-color: transparent; width: -moz-available; width: -webkit-fill-available; }
.livePreview img { position: absolute; object-fit: contain; width: 100%; height: 100%; }
.dark .livePreview { background-color: rgb(17 24 39 / var(--tw-bg-opacity)); }
.popup-metadata { color: white; background: #0000; display: inline-block; white-space: pre-wrap; font-size: 0.75em; }
/* fullpage image viewer */
@@ -80,7 +79,7 @@ table.settings-value-table td { padding: 0.4em; border: 1px solid #ccc; max-widt
/* extra networks */
.extra-networks > div { margin: 0; border-bottom: none !important; }
.extra-networks .second-line { display: flex; width: -moz-available; width: -webkit-fill-available; gap: 0.3em; box-shadow: var(--input-shadow); }
.extra-networks .second-line { display: flex; width: -moz-available; width: -webkit-fill-available; gap: 0.3em; box-shadow: var(--input-shadow); margin-bottom: 2px; }
.extra-networks .search { flex: 1; }
.extra-networks .description { flex: 3; }
.extra-networks .tab-nav > button { margin-right: 0; height: 24px; padding: 2px 4px 2px 4px; }
@@ -89,7 +88,7 @@ table.settings-value-table td { padding: 0.4em; border: 1px solid #ccc; max-widt
.extra-networks .custom-button { width: 120px; width: 100%; background: none; justify-content: left; text-align: left; padding: 2px 8px 2px 16px; text-indent: -8px; box-shadow: none; line-break: auto; }
.extra-networks .custom-button:hover { background: var(--button-primary-background-fill) }
.extra-networks-tab { padding: 0 !important; }
.extra-network-subdirs { background: var(--input-background-fill); overflow-x: hidden; overflow-y: auto; min-width: max(15%, 120px); padding-top: 0.5em; margin-top: -4px !important; }
.extra-network-subdirs { background: var(--input-background-fill); overflow-x: hidden; overflow-y: auto; min-width: max(15%, 120px); padding-top: 0.5em; }
.extra-networks-page { display: flex }
.extra-network-cards { display: flex; flex-wrap: wrap; overflow-y: auto; overflow-x: hidden; align-content: flex-start; width: -moz-available; width: -webkit-fill-available; }
.extra-network-cards .card { height: fit-content; margin: 0 0 0.5em 0.5em; position: relative; scroll-snap-align: start; scroll-margin-top: 0; }
javascript/black-teal-reimagined.css
+1193
View File
File diff suppressed because it is too large Load Diff
javascript/black-teal.css
+1 -1
View File
@@ -134,7 +134,7 @@ svg.feather.feather-image, .feather .feather-image { display: none }
.gallery-item { box-shadow: none !important; }
.performance { color: #888; }
.extra-networks { border-left: 2px solid var(--highlight-color) !important; padding-left: 4px; }
.image-buttons { gap: 10px !important; justify-content: center; }
.image-buttons { justify-content: center; gap: 0 !important; }
.image-buttons > button { max-width: 160px; }
.tooltip { background: var(--primary-300); color: black; border: none; border-radius: var(--radius-lg) }
#system_row > button, #settings_row > button, #config_row > button { max-width: 10em; }
javascript/extraNetworks.js
+17 -14
View File
@@ -3,19 +3,6 @@ let sortVal = -1;
// helpers
const requestGet = (url, data, handler) => {
const xhr = new XMLHttpRequest();
const args = Object.keys(data).map((k) => `${encodeURIComponent(k)}=${encodeURIComponent(data[k])}`).join('&');
xhr.open('GET', `${url}?${args}`, true);
xhr.onreadystatechange = () => {
if (xhr.readyState === 4) {
if (xhr.status === 200) handler(JSON.parse(xhr.responseText));
else console.error(`Request: url=${url} status=${xhr.status} err`);
}
};
xhr.send(JSON.stringify(data));
};
const getENActiveTab = () => {
let tabName = '';
if (gradioApp().getElementById('tab_txt2img').style.display === 'block') tabName = 'txt2img';
@@ -98,7 +85,7 @@ function readCardTags(el, tags) {
}
function readCardDescription(page, item) {
requestGet('/sd_extra_networks/description', { page, item }, (data) => {
xhrGet('/sd_extra_networks/description', { page, item }, (data) => {
const tabname = getENActiveTab();
const description = gradioApp().querySelector(`#${tabname}_description > label > textarea`);
description.value = data?.description?.trim() || '';
@@ -447,6 +434,22 @@ function setupExtraNetworksForTab(tabname) {
};
}
// auto-resize networks sidebar
const resizeObserver = new ResizeObserver((entries) => {
for (const entry of entries) {
for (const el of Array.from(gradioApp().getElementById(`${tabname}_extra_tabs`).querySelectorAll('.extra-networks-page'))) {
const h = Math.trunc(entry.contentRect.height);
if (h <= 0) return;
if (window.opts.extra_networks_card_cover === 'sidebar' && window.opts.theme_type === 'Standard') el.style.height = `max(55vh, ${h - 90}px)`;
// log(`${tabname} height: ${entry.target.id}=${h} ${el.id}=${el.clientHeight}`);
}
}
});
const settingsEl = gradioApp().getElementById(`${tabname}_settings`);
const interfaceEl = gradioApp().getElementById(`${tabname}_interface`);
if (settingsEl) resizeObserver.observe(settingsEl);
if (interfaceEl) resizeObserver.observe(interfaceEl);
// en style
if (!en) return;
let lastView;
javascript/gallery.js
+2 -2
View File
@@ -94,14 +94,14 @@ async function delayFetchThumb(fn) {
outstanding++;
const res = await fetch(`/sdapi/v1/browser/thumb?file=${encodeURI(fn)}`, { priority: 'low' });
if (!res.ok) {
console.error(res.statusText);
error(`fetchThumb: ${res.statusText}`);
outstanding--;
return undefined;
}
const json = await res.json();
outstanding--;
if (!res || !json || json.error || Object.keys(json).length === 0) {
if (json.error) console.error(json.error);
if (json.error) error(`fetchThumb: ${json.error}`);
return undefined;
}
return json;
javascript/imageMaskFix.js
-38
View File
@@ -1,38 +0,0 @@
/**
* temporary fix for https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/668
* @see https://github.com/gradio-app/gradio/issues/1721
*/
function imageMaskResize() {
const canvases = gradioApp().querySelectorAll('#img2maskimg .touch-none canvas');
if (!canvases.length) {
window.removeEventListener('resize', imageMaskResize);
return;
}
const wrapper = canvases[0].closest('.touch-none');
const previewImage = wrapper.previousElementSibling;
if (!previewImage.complete) {
previewImage.addEventListener('load', imageMaskResize);
return;
}
const w = previewImage.width;
const h = previewImage.height;
const nw = previewImage.naturalWidth;
const nh = previewImage.naturalHeight;
const portrait = nh > nw;
const wW = Math.min(w, portrait ? h / nh * nw : w / nw * nw);
const wH = Math.min(h, portrait ? h / nh * nh : w / nw * nh);
wrapper.style.width = `${wW}px`;
wrapper.style.height = `${wH}px`;
wrapper.style.left = '0px';
wrapper.style.top = '0px';
canvases.forEach((c) => {
c.style.width = '';
c.style.height = '';
c.style.maxWidth = '100%';
c.style.maxHeight = '100%';
c.style.objectFit = 'contain';
});
}
onAfterUiUpdate(imageMaskResize);
window.addEventListener('resize', imageMaskResize);
javascript/light-teal.css
+4 -4
View File
@@ -20,9 +20,9 @@
--body-text-color: var(--neutral-800);
--body-text-color-subdued: var(--neutral-600);
--background-color: #FFFFFF;
--background-fill-primary: var(--neutral-400);
--background-fill-primary: var(--neutral-300);
--input-padding: 4px;
--input-background-fill: var(--neutral-300);
--input-background-fill: var(--neutral-200);
--input-shadow: 2px 2px 2px 2px var(--neutral-500);
--button-secondary-text-color: black;
--button-secondary-background-fill: linear-gradient(to bottom right, var(--neutral-200), var(--neutral-500));
@@ -291,8 +291,8 @@ svg.feather.feather-image, .feather .feather-image { display: none }
--slider-color: ;
--stat-background-fill: linear-gradient(to right, var(--primary-400), var(--primary-600));
--table-border-color: var(--neutral-700);
--table-even-background-fill: #222222;
--table-odd-background-fill: #333333;
--table-even-background-fill: #FFFFFF;
--table-odd-background-fill: #CCCCCC;
--table-radius: var(--radius-lg);
--table-row-focus: var(--color-accent-soft);
}
javascript/loader.js
+5 -3
View File
@@ -20,7 +20,7 @@ async function preloadImages() {
try {
await Promise.all(imagePromises);
} catch (error) {
console.error('Error preloading images:', error);
error(`preloadImages: ${error}`);
}
}
@@ -43,14 +43,16 @@ async function createSplash() {
const motdEl = document.getElementById('motd');
if (motdEl) motdEl.innerHTML = text.replace(/["]+/g, '');
})
.catch((err) => console.error('getMOTD:', err));
.catch((err) => error(`getMOTD: ${err}`));
}
async function removeSplash() {
const splash = document.getElementById('splash');
if (splash) splash.remove();
log('removeSplash');
log('startupTime', Math.round(performance.now() - appStartTime) / 1000);
const t = Math.round(performance.now() - appStartTime) / 1000;
log('startupTime', t);
xhrPost('/sdapi/v1/log', { message: `ready time=${t}` });
}
window.onload = createSplash;
javascript/logMonitor.js
+11 -4
View File
@@ -2,6 +2,7 @@ let logMonitorEl = null;
let logMonitorStatus = true;
let logWarnings = 0;
let logErrors = 0;
let logConnected = false;
function dateToStr(ts) {
const dt = new Date(1000 * ts);
@@ -29,8 +30,7 @@ async function logMonitor() {
row.innerHTML = `<td>${dateToStr(l.created)}</td>${level}<td>${l.facility}</td>${module}<td>${l.msg}</td>`;
logMonitorEl.appendChild(row);
} catch (e) {
// console.log('logMonitor', e);
console.error('logMonitor line', line);
error(`logMonitor: ${line}`);
}
};
@@ -46,6 +46,7 @@ async function logMonitor() {
if (logMonitorStatus) setTimeout(logMonitor, opts.logmonitor_refresh_period);
else setTimeout(logMonitor, 10 * 1000); // on failure try to reconnect every 10sec
if (!opts.logmonitor_show) return;
logMonitorStatus = false;
if (!logMonitorEl) {
@@ -64,14 +65,20 @@ async function logMonitor() {
const lines = await res.json();
if (logMonitorEl && lines?.length > 0) logMonitorEl.parentElement.parentElement.style.display = opts.logmonitor_show ? 'block' : 'none';
for (const line of lines) addLogLine(line);
if (!logConnected) {
logConnected = true;
xhrPost('/sdapi/v1/log', { debug: 'connected' });
}
} else {
addLogLine(`{ "created": ${Date.now()}, "level":"ERROR", "module":"logMonitor", "facility":"ui", "msg":"Failed to fetch log: ${res?.status} ${res?.statusText}" }`);
logConnected = false;
logErrors++;
addLogLine(`{ "created": ${Date.now()}, "level":"ERROR", "module":"logMonitor", "facility":"ui", "msg":"Failed to fetch log: ${res?.status} ${res?.statusText}" }`);
}
cleanupLog(atBottom);
} catch (err) {
addLogLine(`{ "created": ${Date.now()}, "level":"ERROR", "module":"logMonitor", "facility":"ui", "msg":"Failed to fetch log: server unreachable" }`);
logConnected = false;
logErrors++;
addLogLine(`{ "created": ${Date.now()}, "level":"ERROR", "module":"logMonitor", "facility":"ui", "msg":"Failed to fetch log: server unreachable" }`);
cleanupLog(atBottom);
}
}
javascript/logger.js
+68
View File
@@ -0,0 +1,68 @@
const timeout = 10000;
const log = async (...msg) => {
const dt = new Date();
const ts = `${dt.getHours().toString().padStart(2, '0')}:${dt.getMinutes().toString().padStart(2, '0')}:${dt.getSeconds().toString().padStart(2, '0')}.${dt.getMilliseconds().toString().padStart(3, '0')}`;
if (window.logger) window.logger.innerHTML += window.logPrettyPrint(...msg);
console.log(ts, ...msg); // eslint-disable-line no-console
};
const debug = async (...msg) => {
const dt = new Date();
const ts = `${dt.getHours().toString().padStart(2, '0')}:${dt.getMinutes().toString().padStart(2, '0')}:${dt.getSeconds().toString().padStart(2, '0')}.${dt.getMilliseconds().toString().padStart(3, '0')}`;
if (window.logger) window.logger.innerHTML += window.logPrettyPrint(...msg);
console.debug(ts, ...msg); // eslint-disable-line no-console
};
const error = async (...msg) => {
const dt = new Date();
const ts = `${dt.getHours().toString().padStart(2, '0')}:${dt.getMinutes().toString().padStart(2, '0')}:${dt.getSeconds().toString().padStart(2, '0')}.${dt.getMilliseconds().toString().padStart(3, '0')}`;
if (window.logger) window.logger.innerHTML += window.logPrettyPrint(...msg);
console.error(ts, ...msg); // eslint-disable-line no-console
// const txt = msg.join(' ');
// if (!txt.includes('asctime') && !txt.includes('xhr.')) xhrPost('/sdapi/v1/log', { error: txt }); // eslint-disable-line no-use-before-define
};
const xhrInternal = (xhrObj, data, handler = undefined, errorHandler = undefined, ignore = false, serverTimeout = timeout) => {
const err = (msg) => {
if (!ignore) {
error(`${msg}: state=${xhrObj.readyState} status=${xhrObj.status} response=${xhrObj.responseText}`);
if (errorHandler) errorHandler(xhrObj);
}
};
xhrObj.setRequestHeader('Content-Type', 'application/json');
xhrObj.timeout = timeout;
xhrObj.ontimeout = () => err('xhr.ontimeout');
xhrObj.onerror = () => err('xhr.onerror');
xhrObj.onabort = () => err('xhr.onabort');
xhrObj.onreadystatechange = () => {
if (xhrObj.readyState === 4) {
if (xhrObj.status === 200) {
try {
const json = JSON.parse(xhrObj.responseText);
if (handler) handler(json);
} catch (e) {
error(`xhr.onreadystatechange: ${e}`);
}
} else {
err(`xhr.onreadystatechange: state=${xhrObj.readyState} status=${xhrObj.status} response=${xhrObj.responseText}`);
}
}
};
const req = JSON.stringify(data);
xhrObj.send(req);
};
const xhrGet = (url, data, handler = undefined, errorHandler = undefined, ignore = false, serverTimeout = timeout) => {
const xhr = new XMLHttpRequest();
const args = Object.keys(data).map((k) => `${encodeURIComponent(k)}=${encodeURIComponent(data[k])}`).join('&');
xhr.open('GET', `${url}?${args}`, true);
xhrInternal(xhr, data, handler, errorHandler, ignore, serverTimeout);
};
function xhrPost(url, data, handler = undefined, errorHandler = undefined, ignore = false, serverTimeout = timeout) {
const xhr = new XMLHttpRequest();
xhr.open('POST', url, true);
xhrInternal(xhr, data, handler, errorHandler, ignore, serverTimeout);
}
javascript/notification.js
+27 -23
View File
@@ -4,28 +4,32 @@ let lastHeadImg = null;
let notificationButton = null;
async function sendNotification() {
if (!notificationButton) {
notificationButton = gradioApp().getElementById('request_notifications');
if (notificationButton) notificationButton.addEventListener('click', (evt) => Notification.requestPermission(), true);
try {
if (!notificationButton) {
notificationButton = gradioApp().getElementById('request_notifications');
if (notificationButton) notificationButton.addEventListener('click', (evt) => Notification.requestPermission(), true);
}
if (document.hasFocus()) return; // window is in focus so don't send notifications
let galleryPreviews = gradioApp().querySelectorAll('div[id^="tab_"][style*="display: block"] div[id$="_results"] .thumbnail-item > img');
if (!galleryPreviews || galleryPreviews.length === 0) galleryPreviews = gradioApp().querySelectorAll('.thumbnail-item > img');
if (!galleryPreviews || galleryPreviews.length === 0) return;
const headImg = galleryPreviews[0]?.src;
if (!headImg || headImg === lastHeadImg || headImg.includes('logo-bg-')) return;
const audioNotification = gradioApp().querySelector('#audio_notification audio');
if (audioNotification) audioNotification.play();
lastHeadImg = headImg;
const imgs = new Set(Array.from(galleryPreviews).map((img) => img.src)); // Multiple copies of the images are in the DOM when one is selected
const notification = new Notification('SD.Next', {
body: `Generated ${imgs.size > 1 ? imgs.size - opts.return_grid : 1} image${imgs.size > 1 ? 's' : ''}`,
icon: headImg,
image: headImg,
});
notification.onclick = () => {
parent.focus();
this.close();
};
log('sendNotifications');
} catch (e) {
error(`sendNotification: ${e}`);
}
if (document.hasFocus()) return; // window is in focus so don't send notifications
let galleryPreviews = gradioApp().querySelectorAll('div[id^="tab_"][style*="display: block"] div[id$="_results"] .thumbnail-item > img');
if (!galleryPreviews || galleryPreviews.length === 0) galleryPreviews = gradioApp().querySelectorAll('.thumbnail-item > img');
if (!galleryPreviews || galleryPreviews.length === 0) return;
const headImg = galleryPreviews[0]?.src;
if (!headImg || headImg === lastHeadImg || headImg.includes('logo-bg-')) return;
const audioNotification = gradioApp().querySelector('#audio_notification audio');
if (audioNotification) audioNotification.play();
lastHeadImg = headImg;
const imgs = new Set(Array.from(galleryPreviews).map((img) => img.src)); // Multiple copies of the images are in the DOM when one is selected
const notification = new Notification('SD.Next', {
body: `Generated ${imgs.size > 1 ? imgs.size - opts.return_grid : 1} image${imgs.size > 1 ? 's' : ''}`,
icon: headImg,
image: headImg,
});
notification.onclick = () => {
parent.focus();
this.close();
};
log('sendNotifications');
}
javascript/nvml.js
+30
View File
@@ -1,6 +1,32 @@
let nvmlInterval = null; // eslint-disable-line prefer-const
let nvmlEl = null;
let nvmlTable = null;
const chartData = { mem: [], load: [] };
async function updateNVMLChart(mem, load) {
const maxLen = 120;
const colorRangeMap = $.range_map({ // eslint-disable-line no-undef
'0:5': '#fffafa',
'6:10': '#fff7ed',
'11:20': '#fed7aa',
'21:30': '#fdba74',
'31:40': '#fb923c',
'41:50': '#f97316',
'51:60': '#ea580c',
'61:70': '#c2410c',
'71:80': '#9a3412',
'81:90': '#7c2d12',
'91:100': '#6c2e12',
});
const sparklineConfigLOAD = { type: 'bar', height: '100px', barWidth: '2px', barSpacing: '1px', chartRangeMin: 0, chartRangeMax: 100, barColor: '#89007D' };
const sparklineConfigMEM = { type: 'bar', height: '100px', barWidth: '2px', barSpacing: '1px', chartRangeMin: 0, chartRangeMax: 100, colorMap: colorRangeMap, composite: true };
if (chartData.load.length > maxLen) chartData.load.shift();
chartData.load.push(load);
if (chartData.mem.length > maxLen) chartData.mem.shift();
chartData.mem.push(mem);
$('#nvmlChart').sparkline(chartData.load, sparklineConfigLOAD); // eslint-disable-line no-undef
$('#nvmlChart').sparkline(chartData.mem, sparklineConfigMEM); // eslint-disable-line no-undef
}
async function updateNVML() {
try {
@@ -35,6 +61,7 @@ async function updateNVML() {
<tr><td>State</td><td>${gpu.state}</td></tr>
`;
nvmlTbody.innerHTML = rows;
updateNVMLChart(gpu.load.memory, gpu.load.gpu);
}
nvmlEl.style.display = 'block';
} catch (e) {
@@ -56,7 +83,10 @@ async function initNVML() {
<thead><tr><th></th><th></th></tr></thead>
<tbody></tbody>
`;
const nvmlChart = document.createElement('div');
nvmlChart.id = 'nvmlChart';
nvmlEl.appendChild(nvmlTable);
nvmlEl.appendChild(nvmlChart);
gradioApp().appendChild(nvmlEl);
log('initNVML');
}
javascript/progressBar.js
+48 -43
View File
@@ -1,28 +1,5 @@
let lastState = {};
function request(url, data, handler, errorHandler) {
const xhr = new XMLHttpRequest();
xhr.open('POST', url, true);
xhr.setRequestHeader('Content-Type', 'application/json');
xhr.onreadystatechange = () => {
if (xhr.readyState === 4) {
if (xhr.status === 200) {
try {
const js = JSON.parse(xhr.responseText);
handler(js);
} catch (error) {
console.error(error);
errorHandler();
}
} else {
errorHandler();
}
}
};
const js = JSON.stringify(data);
xhr.send(js);
}
function pad2(x) {
return x < 10 ? `0${x}` : x;
}
@@ -35,8 +12,10 @@ function formatTime(secs) {
function checkPaused(state) {
lastState.paused = state ? !state : !lastState.paused;
document.getElementById('txt2img_pause').innerText = lastState.paused ? 'Resume' : 'Pause';
document.getElementById('img2img_pause').innerText = lastState.paused ? 'Resume' : 'Pause';
const t_el = document.getElementById('txt2img_pause');
const i_el = document.getElementById('img2img_pause');
if (t_el) t_el.innerText = lastState.paused ? 'Resume' : 'Pause';
if (i_el) i_el.innerText = lastState.paused ? 'Resume' : 'Pause';
}
function setProgress(res) {
@@ -89,28 +68,42 @@ function requestProgress(id_task, progressEl, galleryEl, atEnd = null, onProgres
let img;
const initLivePreview = () => {
img = new Image();
if (parentGallery) {
livePreview = document.createElement('div');
livePreview.className = 'livePreview';
parentGallery.insertBefore(livePreview, galleryEl);
const rect = galleryEl.getBoundingClientRect();
if (rect.width) {
livePreview.style.width = `${rect.width}px`;
livePreview.style.height = `${rect.height}px`;
}
img.onload = () => {
livePreview.appendChild(img);
if (livePreview.childElementCount > 2) livePreview.removeChild(livePreview.firstElementChild);
};
if (!parentGallery) return;
const footers = Array.from(gradioApp().querySelectorAll('.gallery_footer'));
for (const footer of footers) {
if (footer.id !== 'gallery_footer') footer.style.display = 'none'; // remove all footers
}
const galleries = Array.from(gradioApp().querySelectorAll('.gallery_main'));
for (const gallery of galleries) {
if (gallery.id !== 'gallery_gallery') gallery.style.display = 'none'; // remove all footers
}
livePreview = document.createElement('div');
livePreview.className = 'livePreview';
parentGallery.insertBefore(livePreview, galleryEl);
img = new Image();
img.id = 'livePreviewImage';
livePreview.appendChild(img);
img.onload = () => {
img.style.width = `min(100%, max(${img.naturalWidth}px, 512px))`;
parentGallery.style.minHeight = `${img.height}px`;
};
};
const done = () => {
debug('taskEnd:', id_task);
localStorage.removeItem('task');
setProgress();
if (parentGallery && livePreview) parentGallery.removeChild(livePreview);
const footers = Array.from(gradioApp().querySelectorAll('.gallery_footer'));
for (const footer of footers) footer.style.display = 'flex'; // restore all footers
const galleries = Array.from(gradioApp().querySelectorAll('.gallery_main'));
for (const gallery of galleries) gallery.style.display = 'flex'; // remove all galleries
try {
if (parentGallery && livePreview) {
parentGallery.removeChild(livePreview);
parentGallery.style.minHeight = 'unset';
}
} catch { /* ignore */ }
checkPaused(true);
sendNotification();
if (atEnd) atEnd();
@@ -118,20 +111,32 @@ function requestProgress(id_task, progressEl, galleryEl, atEnd = null, onProgres
const start = (id_task, id_live_preview) => { // eslint-disable-line no-shadow
if (!opts.live_previews_enable || opts.live_preview_refresh_period === 0 || opts.show_progress_every_n_steps === 0) return;
request('./internal/progress', { id_task, id_live_preview }, (res) => {
const onProgressHandler = (res) => {
// debug('onProgress', res);
lastState = res;
const elapsedFromStart = (new Date() - dateStart) / 1000;
hasStarted |= res.active;
if (res.completed || (!res.active && (hasStarted || once)) || (elapsedFromStart > 30 && !res.queued && res.progress === prevProgress)) {
debug('onProgressEnd', res);
done();
return;
}
setProgress(res);
if (res.live_preview && !livePreview) initLivePreview();
if (res.live_preview && galleryEl) img.src = res.live_preview;
if (res.live_preview && galleryEl) {
if (img.src !== res.live_preview) img.src = res.live_preview;
}
if (onProgress) onProgress(res);
setTimeout(() => start(id_task, id_live_preview), opts.live_preview_refresh_period || 500);
}, done);
};
const onProgressErrorHandler = (err) => {
error(`onProgressError: ${err}`);
done();
};
xhrPost('./internal/progress', { id_task, id_live_preview }, onProgressHandler, onProgressErrorHandler, false, 5000);
};
start(id_task, 0);
}
javascript/script.js
+4 -17
View File
@@ -1,17 +1,3 @@
const log = (...msg) => {
const dt = new Date();
const ts = `${dt.getHours().toString().padStart(2, '0')}:${dt.getMinutes().toString().padStart(2, '0')}:${dt.getSeconds().toString().padStart(2, '0')}.${dt.getMilliseconds().toString().padStart(3, '0')}`;
if (window.logger) window.logger.innerHTML += window.logPrettyPrint(...msg);
console.log(ts, ...msg); // eslint-disable-line no-console
};
const debug = (...msg) => {
const dt = new Date();
const ts = `${dt.getHours().toString().padStart(2, '0')}:${dt.getMinutes().toString().padStart(2, '0')}:${dt.getSeconds().toString().padStart(2, '0')}.${dt.getMilliseconds().toString().padStart(3, '0')}`;
if (window.logger) window.logger.innerHTML += window.logPrettyPrint(...msg);
console.debug(ts, ...msg); // eslint-disable-line no-console
};
async function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms)); // eslint-disable-line no-promise-executor-return
}
@@ -82,7 +68,7 @@ function executeCallbacks(queue, arg) {
try {
callback(arg);
} catch (e) {
console.error('error running callback', callback, ':', e);
error(`executeCallbacks: ${callback} ${e}`);
}
}
}
@@ -139,11 +125,12 @@ document.addEventListener('keydown', (e) => {
let elem;
if (e.key === 'Escape') elem = getUICurrentTabContent().querySelector('button[id$=_interrupt]');
if (e.key === 'Enter' && e.ctrlKey) elem = getUICurrentTabContent().querySelector('button[id$=_generate]');
if (e.key === 'Backspace' && e.ctrlKey) elem = getUICurrentTabContent().querySelector('button[id$=_reprocess]');
if (e.key === 'i' && e.ctrlKey) elem = getUICurrentTabContent().querySelector('button[id$=_reprocess]');
if (e.key === ' ' && e.ctrlKey) elem = getUICurrentTabContent().querySelector('button[id$=_extra_networks_btn]');
if (e.key === 'n' && e.ctrlKey) elem = getUICurrentTabContent().querySelector('button[id$=_extra_networks_btn]');
if (e.key === 's' && e.ctrlKey) elem = getUICurrentTabContent().querySelector('button[id^=save_]');
if (e.key === 'Insert' && e.ctrlKey) elem = getUICurrentTabContent().querySelector('button[id^=save_]');
if (e.key === 'Delete' && e.ctrlKey) elem = getUICurrentTabContent().querySelector('button[id^=delete_]');
if (e.key === 'd' && e.ctrlKey) elem = getUICurrentTabContent().querySelector('button[id^=delete_]');
// if (e.key === 'm' && e.ctrlKey) elem = gradioApp().getElementById('setting_sd_model_checkpoint');
if (elem) {
e.preventDefault();
javascript/sdnext.css
+27 -19
View File
@@ -14,9 +14,9 @@ table { overflow-x: auto !important; overflow-y: auto !important; }
td { border-bottom: none !important; padding: 0 0.5em !important; }
tr { border-bottom: none !important; padding: 0 0.5em !important; }
td > div > span { overflow-y: auto; max-height: 3em; overflow-x: hidden; }
textarea { overflow-y: auto !important; }
textarea { overflow-y: auto !important; border-radius: 4px !important; }
span { font-size: var(--text-md) !important; }
button { font-size: var(--text-lg) !important; }
button { font-size: var(--text-lg) !important; min-width: unset !important; }
input[type='color'] { width: 64px; height: 32px; }
input::-webkit-outer-spin-button, input::-webkit-inner-spin-button { margin-left: 4px; }
@@ -30,6 +30,19 @@ input::-webkit-outer-spin-button, input::-webkit-inner-spin-button { margin-left
.hidden { display: none; }
.tabitem { padding: 0 !important; }
/* gradio image/canvas elements */
.image-container { overflow: auto; }
/*
.gradio-image { min-height: fit-content; }
.gradio-image img { object-fit: contain; }
*/
/*
.gradio-image { min-height: 200px !important; }
.image-container { height: unset !important; }
.control-image { height: unset !important; }
#img2img_sketch, #img2maskimg, #inpaint_sketch { overflow: overlay !important; resize: auto; background: var(--panel-background-fill); z-index: 5; }
*/
/* color elements */
.gradio-dropdown, .block.gradio-slider, .block.gradio-checkbox, .block.gradio-textbox, .block.gradio-radio, .block.gradio-checkboxgroup, .block.gradio-number, .block.gradio-colorpicker { border-width: 0 !important; box-shadow: none !important;}
.gradio-accordion { padding-top: var(--spacing-md) !important; padding-right: 0 !important; padding-bottom: 0 !important; color: var(--body-text-color); }
@@ -83,13 +96,12 @@ button.custom-button { border-radius: var(--button-large-radius); padding: var(-
.block.token-counter div{ display: inline; }
.block.token-counter span{ padding: 0.1em 0.75em; }
.performance { font-size: var(--text-xs); color: #444; }
.performance p { display: inline-block; color: var(--body-text-color-subdued) !important }
.performance p { display: inline-block; color: var(--primary-500) !important }
.performance .time { margin-right: 0; }
.thumbnails { background: var(--body-background-fill); }
.control-image { height: calc(100vw/3) !important; }
.prompt textarea { resize: vertical; }
.grid-wrap { overflow-y: auto !important; }
#control_results { margin: 0; padding: 0; }
#control_gallery { height: calc(100vw/3 + 60px); }
#txt2img_gallery, #img2img_gallery { height: 50vh; }
#control-result { background: var(--button-secondary-background-fill); padding: 0.2em; }
#control-inputs { margin-top: 1em; }
@@ -105,7 +117,6 @@ button.custom-button { border-radius: var(--button-large-radius); padding: var(-
#txt2img_prompt, #txt2img_neg_prompt, #img2img_prompt, #img2img_neg_prompt, #control_prompt, #control_neg_prompt { display: contents; }
#txt2img_actions_column, #img2img_actions_column, #control_actions { flex-flow: wrap; justify-content: space-between; }
.interrogate-clip { position: absolute; right: 6em; top: 8px; max-width: fit-content; background: none !important; z-index: 50; }
.interrogate-blip { position: absolute; right: 4em; top: 8px; max-width: fit-content; background: none !important; z-index: 50; }
.interrogate-col { min-width: 0 !important; max-width: fit-content; margin-right: var(--spacing-xxl); }
@@ -118,11 +129,9 @@ div#extras_scale_to_tab div.form { flex-direction: row; }
#img2img_unused_scale_by_slider { visibility: hidden; width: 0.5em; max-width: 0.5em; min-width: 0.5em; }
.inactive{ opacity: 0.5; }
div#extras_scale_to_tab div.form { flex-direction: row; }
#mode_img2img .gradio-image>div.fixed-height, #mode_img2img .gradio-image>div.fixed-height img{ height: 480px !important; max-height: 480px !important; min-height: 480px !important; }
#img2img_sketch, #img2maskimg, #inpaint_sketch { overflow: overlay !important; resize: auto; background: var(--panel-background-fill); z-index: 5; }
.image-buttons button { min-width: auto; }
.infotext { overflow-wrap: break-word; line-height: 1.5em; font-size: 0.95em !important; }
.infotext > p { padding-left: 1em; text-indent: -1em; white-space: pre-wrap; color: var(--block-info-text-color) !important; }
.infotext > p { white-space: pre-wrap; color: var(--block-info-text-color) !important; }
.tooltip { display: block; position: fixed; top: 1em; right: 1em; padding: 0.5em; background: var(--input-background-fill); color: var(--body-text-color); border: 1pt solid var(--button-primary-border-color);
width: 22em; min-height: 1.3em; font-size: var(--text-xs); transition: opacity 0.2s ease-in; pointer-events: none; opacity: 0; z-index: 999; }
.tooltip-show { opacity: 0.9; }
@@ -140,29 +149,30 @@ div#extras_scale_to_tab div.form { flex-direction: row; }
#settings>div.tab-content { flex: 10 0 75%; display: grid; }
#settings>div.tab-content>div { border: none; padding: 0; }
#settings>div.tab-content>div>div>div>div>div { flex-direction: unset; }
#settings>div.tab-nav { display: grid; grid-template-columns: repeat(auto-fill, .5em minmax(10em, 1fr)); flex: 1 0 auto; width: 12em; align-self: flex-start; gap: var(--spacing-xxl); }
#settings>div.tab-nav { display: grid; grid-template-columns: repeat(auto-fill, .5em minmax(10em, 1fr)); flex: 1 0 auto; width: 12em; align-self: flex-start; gap: 8px; }
#settings>div.tab-nav button { display: block; border: none; text-align: left; white-space: initial; padding: 0; }
#settings>div.tab-nav>#settings_show_all_pages { padding: var(--size-2) var(--size-4); }
#settings .block.gradio-checkbox { margin: 0; width: auto; }
#settings .dirtyable { gap: .5em; }
#settings .dirtyable.hidden { display: none; }
#settings .modification-indicator { height: 1.2em; border-radius: 1em !important; padding: 0; width: 0; margin-right: 0.5em; }
#settings .modification-indicator { height: 1.2em; border-radius: 1em !important; padding: 0; width: 0; margin-right: 0.5em; border-left: inset; }
#settings .modification-indicator:disabled { visibility: hidden; }
#settings .modification-indicator.saved { background: var(--color-accent-soft); width: var(--spacing-sm); }
#settings .modification-indicator.changed { background: var(--color-accent); width: var(--spacing-sm); }
#settings .modification-indicator.changed.unsaved { background-image: linear-gradient(var(--color-accent) 25%, var(--color-accent-soft) 75%); width: var(--spacing-sm); }
#settings_result { margin: 0 1.2em; }
#tab_settings .gradio-slider, #tab_settings .gradio-dropdown { width: 300px !important; max-width: 300px; }
#tab_settings textarea { max-width: 500px; }
.licenses { display: block !important; }
/* live preview */
.progressDiv { position: relative; height: 20px; background: #b4c0cc; margin-bottom: -3px; }
.dark .progressDiv { background: #424c5b; }
.progressDiv .progress { width: 0%; height: 20px; background: #0060df; color: white; font-weight: bold; line-height: 20px; padding: 0 8px 0 0; text-align: right; overflow: visible; white-space: nowrap; padding: 0 0.5em; }
.livePreview { position: absolute; z-index: 50; background-color: transparent; width: -moz-available; width: -webkit-fill-available; }
.livePreview img { position: absolute; object-fit: contain; width: 100%; height: 100%; }
.dark .livePreview { background-color: rgb(17 24 39 / var(--tw-bg-opacity)); }
.livePreview { position: absolute; z-index: 50; width: -moz-available; width: -webkit-fill-available; height: 100%; background-color: var(--background-color); }
.livePreview img { object-fit: contain; width: 100%; justify-self: center; }
.popup-metadata { color: white; background: #0000; display: inline-block; white-space: pre-wrap; font-size: var(--text-xxs); }
.generating { animation: unset !important; border: unset !important; }
/* fullpage image viewer */
#lightboxModal { display: none; position: fixed; z-index: 1001; left: 0; top: 0; width: 100%; height: 100%; overflow: hidden; background-color: rgba(20, 20, 20, 0.75); backdrop-filter: blur(6px);
user-select: none; -webkit-user-select: none; flex-direction: row; font-family: 'NotoSans';}
@@ -207,7 +217,7 @@ table.settings-value-table td { padding: 0.4em; border: 1px solid #ccc; max-widt
.extra_networks_root { width: 0; position: absolute; height: auto; right: 0; top: 13em; z-index: 100; } /* default is sidebar view */
.extra-networks { background: var(--background-color); padding: var(--block-label-padding); }
.extra-networks > div { margin: 0; border-bottom: none !important; gap: 0.3em 0; }
.extra-networks .second-line { display: flex; width: -moz-available; width: -webkit-fill-available; gap: 0.3em; box-shadow: var(--input-shadow); }
.extra-networks .second-line { display: flex; width: -moz-available; width: -webkit-fill-available; gap: 0.3em; box-shadow: var(--input-shadow); margin-bottom: 2px; }
.extra-networks .search { flex: 1; height: 4em; }
.extra-networks .description { flex: 3; }
.extra-networks .tab-nav>button { margin-right: 0; height: 24px; padding: 2px 4px 2px 4px; }
@@ -216,7 +226,7 @@ table.settings-value-table td { padding: 0.4em; border: 1px solid #ccc; max-widt
.extra-networks .custom-button { width: 120px; width: 100%; background: none; justify-content: left; text-align: left; padding: 3px 3px 3px 12px; text-indent: -6px; box-shadow: none; line-break: auto; }
.extra-networks .custom-button:hover { background: var(--button-primary-background-fill) }
.extra-networks-tab { padding: 0 !important; }
.extra-network-subdirs { background: var(--input-background-fill); overflow-x: hidden; overflow-y: auto; min-width: max(15%, 120px); padding-top: 0.5em; margin-top: -4px !important; }
.extra-network-subdirs { background: var(--input-background-fill); overflow-x: hidden; overflow-y: auto; min-width: max(15%, 120px); padding-top: 0.5em; border-radius: 4px; }
.extra-networks-page { display: flex }
.extra-network-cards { display: flex; flex-wrap: wrap; overflow-y: auto; overflow-x: hidden; align-content: flex-start; width: -moz-available; width: -webkit-fill-available; }
.extra-network-cards .card { height: fit-content; margin: 0 0 0.5em 0.5em; position: relative; scroll-snap-align: start; scroll-margin-top: 0; }
@@ -380,8 +390,6 @@ div:has(>#tab-gallery-folders) { flex-grow: 0 !important; background-color: var(
#img2img_actions_column { display: flex; min-width: fit-content !important; flex-direction: row;justify-content: space-evenly; align-items: center;}
#txt2img_generate_box, #img2img_generate_box, #txt2img_enqueue_wrapper,#img2img_enqueue_wrapper {display: flex;flex-direction: column;height: 4em !important;align-items: stretch;justify-content: space-evenly;}
#img2img_interface, #img2img_results, #img2img_footer p { text-wrap: wrap; min-width: 100% !important; max-width: 100% !important;} /* maintain single column for from image operations on larger mobile devices */
#img2img_sketch, #img2maskimg, #inpaint_sketch {display: flex; overflow: auto !important; resize: none !important; } /* fix inpaint image display being too large for mobile displays */
#img2maskimg canvas { width: auto !important; max-height: 100% !important; height: auto !important; }
#txt2img_sampler, #txt2img_batch, #txt2img_seed_group, #txt2img_advanced, #txt2img_second_pass, #img2img_sampling_group, #img2img_resize_group, #img2img_batch_group, #img2img_seed_group, #img2img_denoise_group, #img2img_advanced_group { width: 100% !important; } /* fix from text/image UI elements to prevent them from moving around within the UI */
#img2img_resize_group .gradio-radio>div { display: flex; flex-direction: column; width: unset !important; }
#inpaint_controls div { display:flex;flex-direction: row;}
javascript/ui.js
+4 -4
View File
@@ -28,7 +28,7 @@ function clip_gallery_urls(gallery) {
const files = gallery.map((v) => v.data);
navigator.clipboard.writeText(JSON.stringify(files)).then(
() => log('clipboard:', files),
(err) => console.error('clipboard:', files, err),
(err) => error(`clipboard: ${files} ${err}`),
);
}
@@ -139,7 +139,7 @@ function switch_to_inpaint(...args) {
return Array.from(arguments);
}
function switch_to_inpaint_sketch(...args) {
function switch_to_composite(...args) {
switchToTab('Image');
switch_to_img2img_tab(3);
return Array.from(arguments);
@@ -493,9 +493,9 @@ function previewTheme() {
el.src = `/file=html/${name}.jpg`;
}
})
.catch((e) => console.error('previewTheme:', e));
.catch((e) => error(`previewTheme: ${e}`));
})
.catch((e) => console.error('previewTheme:', e));
.catch((e) => error(`previewTheme: ${e}`));
}
async function browseFolder() {
launch.py
+5 -3
View File
@@ -55,9 +55,11 @@ def get_custom_args():
if 'PS1' in env:
del env['PS1']
installer.log.trace(f'Environment: {installer.print_dict(env)}')
else:
env = [f'{k}={v}' for k, v in os.environ.items() if k.startswith('SD_')]
installer.log.debug(f'Env flags: {env}')
env = [f'{k}={v}' for k, v in os.environ.items() if k.startswith('SD_')]
installer.log.debug(f'Env flags: {env}')
ldd = os.environ.get('LD_PRELOAD', None)
if ldd is not None:
installer.log.debug(f'Linker flags: "{ldd}"')
@lru_cache()
models/Reference/Efficient-Large-Model--Sana_1600M_1024px_diffusers.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

modules/api/api.py
+7 -1
View File
@@ -35,7 +35,8 @@ class Api:
# server api
self.add_api_route("/sdapi/v1/motd", server.get_motd, methods=["GET"], response_model=str)
self.add_api_route("/sdapi/v1/log", server.get_log_buffer, methods=["GET"], response_model=List[str])
self.add_api_route("/sdapi/v1/log", server.get_log, methods=["GET"], response_model=List[str])
self.add_api_route("/sdapi/v1/log", server.post_log, methods=["POST"])
self.add_api_route("/sdapi/v1/start", self.get_session_start, methods=["GET"])
self.add_api_route("/sdapi/v1/version", server.get_version, methods=["GET"])
self.add_api_route("/sdapi/v1/status", server.get_status, methods=["GET"], response_model=models.ResStatus)
@@ -90,6 +91,11 @@ class Api:
self.add_api_route("/sdapi/v1/history", endpoints.get_history, methods=["GET"], response_model=List[str])
self.add_api_route("/sdapi/v1/history", endpoints.post_history, methods=["POST"], response_model=int)
# lora api
if shared.native:
self.add_api_route("/sdapi/v1/loras", endpoints.get_loras, methods=["GET"], response_model=List[dict])
self.add_api_route("/sdapi/v1/refresh-loras", endpoints.post_refresh_loras, methods=["POST"])
# gallery api
gallery.register_api(app)
modules/api/control.py
+2
View File
@@ -159,6 +159,8 @@ class APIControl():
output_images = []
output_processed = []
output_info = ''
# TODO control: support scripts via api
# init script args, call scripts.script_control.run, call scripts.script_control.after
run.control_set({ 'do_not_save_grid': not req.save_images, 'do_not_save_samples': not req.save_images, **self.prepare_ip_adapter(req) })
run.control_set(getattr(req, "extra", {}))
res = run.control_run(**args)
modules/api/endpoints.py
+10
View File
@@ -40,6 +40,12 @@ def get_embeddings():
return {"loaded": convert_embeddings(db.word_embeddings), "skipped": convert_embeddings(db.skipped_embeddings)}
def get_loras():
from modules.lora import network, networks
def create_lora_json(obj: network.NetworkOnDisk):
return { "name": obj.name, "alias": obj.alias, "path": obj.filename, "metadata": obj.metadata }
return [create_lora_json(obj) for obj in networks.available_networks.values()]
def get_extra_networks(page: Optional[str] = None, name: Optional[str] = None, filename: Optional[str] = None, title: Optional[str] = None, fullname: Optional[str] = None, hash: Optional[str] = None): # pylint: disable=redefined-builtin
res = []
for pg in shared.extra_networks:
@@ -126,6 +132,10 @@ def post_refresh_checkpoints():
def post_refresh_vae():
return shared.refresh_vaes()
def post_refresh_loras():
from modules.lora import networks
return networks.list_available_networks()
def get_extensions_list():
from modules import extensions
extensions.list_extensions()
modules/api/generate.py
+4
View File
@@ -116,6 +116,8 @@ class APIGenerate():
processed = scripts.scripts_txt2img.run(p, *script_args) # Need to pass args as list here
else:
processed = process_images(p)
processed = scripts.scripts_txt2img.after(p, processed, *script_args)
p.close()
shared.state.end(api=False)
if processed is None or processed.images is None or len(processed.images) == 0:
b64images = []
@@ -166,6 +168,8 @@ class APIGenerate():
processed = scripts.scripts_img2img.run(p, *script_args) # Need to pass args as list here
else:
processed = process_images(p)
processed = scripts.scripts_img2img.after(p, processed, *script_args)
p.close()
shared.state.end(api=False)
if processed is None or processed.images is None or len(processed.images) == 0:
b64images = []
modules/api/models.py
+7 -1
View File
@@ -286,10 +286,16 @@ class ResImageInfo(BaseModel):
items: dict = Field(title="Items", description="A dictionary containing all the other fields the image had")
parameters: dict = Field(title="Parameters", description="A dictionary with parsed generation info fields")
class ReqLog(BaseModel):
class ReqGetLog(BaseModel):
lines: int = Field(default=100, title="Lines", description="How many lines to return")
clear: bool = Field(default=False, title="Clear", description="Should the log be cleared after returning the lines?")
class ReqPostLog(BaseModel):
message: Optional[str] = Field(title="Message", description="The info message to log")
debug: Optional[str] = Field(title="Debug message", description="The debug message to log")
error: Optional[str] = Field(title="Error message", description="The error message to log")
class ReqProgress(BaseModel):
skip_current_image: bool = Field(default=False, title="Skip current image", description="Skip current image serialization")
modules/api/server.py
+11 -1
View File
@@ -37,12 +37,22 @@ def get_platform():
from modules.loader import get_packages as loader_get_packages
return { **installer_get_platform(), **loader_get_packages() }
def get_log_buffer(req: models.ReqLog = Depends()):
def get_log(req: models.ReqGetLog = Depends()):
lines = shared.log.buffer[:req.lines] if req.lines > 0 else shared.log.buffer.copy()
if req.clear:
shared.log.buffer.clear()
return lines
def post_log(req: models.ReqPostLog):
if req.message is not None:
shared.log.info(f'UI: {req.message}')
if req.debug is not None:
shared.log.debug(f'UI: {req.debug}')
if req.error is not None:
shared.log.error(f'UI: {req.error}')
return {}
def get_config():
options = {}
for k in shared.opts.data.keys():
modules/call_queue.py
+13 -8
View File
@@ -2,7 +2,7 @@ import html
import threading
import time
import cProfile
from modules import shared, progress, errors
from modules import shared, progress, errors, timer
queue_lock = threading.Lock()
@@ -73,15 +73,20 @@ def wrap_gradio_call(func, extra_outputs=None, add_stats=False, name=None):
elapsed_m = int(elapsed // 60)
elapsed_s = elapsed % 60
elapsed_text = f"{elapsed_m}m {elapsed_s:.2f}s" if elapsed_m > 0 else f"{elapsed_s:.2f}s"
vram_html = ''
summary = timer.process.summary(min_time=0.25, total=False).replace('=', ' ')
gpu = ''
cpu = ''
if not shared.mem_mon.disabled:
vram = {k: -(v//-(1024*1024)) for k, v in shared.mem_mon.read().items()}
if vram.get('active_peak', 0) > 0:
vram_html = " | <p class='vram'>"
vram_html += f"GPU active {max(vram['active_peak'], vram['reserved_peak'])} MB reserved {vram['reserved']} | used {vram['used']} MB free {vram['free']} MB total {vram['total']} MB"
vram_html += f" | retries {vram['retries']} oom {vram['oom']}" if vram.get('retries', 0) > 0 or vram.get('oom', 0) > 0 else ''
vram_html += "</p>"
peak = max(vram['active_peak'], vram['reserved_peak'], vram['used'])
used = round(100.0 * peak / vram['total']) if vram['total'] > 0 else 0
if used > 0:
gpu += f"| GPU {peak} MB {used}%"
gpu += f" | retries {vram['retries']} oom {vram['oom']}" if vram.get('retries', 0) > 0 or vram.get('oom', 0) > 0 else ''
ram = shared.ram_stats()
if ram['used'] > 0:
cpu += f"| RAM {ram['used']} GB {round(100.0 * ram['used'] / ram['total'])}%"
if isinstance(res, list):
res[-1] += f"<div class='performance'><p class='time'>Time: {elapsed_text}</p>{vram_html}</div>"
res[-1] += f"<div class='performance'><p>Time: {elapsed_text} | {summary} {gpu} {cpu}</p></div>"
return tuple(res)
return f
modules/consistory/consistory_unet_sdxl.py
-1
View File
@@ -916,7 +916,6 @@ class ConsistorySDXLUNet2DConditionModel(ModelMixin, ConfigMixin, UNet2DConditio
# 1. time
timesteps = timestep
if not torch.is_tensor(timesteps):
# TODO: this requires sync between CPU and GPU. So try to pass timesteps as tensors if you can
# This would be a good case for the `match` statement (Python 3.10+)
is_mps = sample.device.type == "mps"
if isinstance(timestep, float):
modules/control/run.py
+213 -173
View File
@@ -7,6 +7,7 @@ from PIL import Image
from modules.control import util # helper functions
from modules.control import unit # control units
from modules.control import processors # image preprocessors
from modules.control import tile # tiling module
from modules.control.units import controlnet # lllyasviel ControlNet
from modules.control.units import xs # VisLearn ControlNet-XS
from modules.control.units import lite # Kohya ControlLLLite
@@ -44,6 +45,167 @@ def terminate(msg):
return msg
def set_pipe(p, has_models, unit_type, selected_models, active_model, active_strength, control_conditioning, control_guidance_start, control_guidance_end, inits):
global pipe, instance # pylint: disable=global-statement
pipe = None
if has_models:
p.ops.append('control')
p.extra_generation_params["Control type"] = unit_type # overriden later with pretty-print
p.extra_generation_params["Control model"] = ';'.join([(m.model_id or '') for m in active_model if m.model is not None])
p.extra_generation_params["Control conditioning"] = control_conditioning if isinstance(control_conditioning, list) else [control_conditioning]
p.extra_generation_params['Control start'] = control_guidance_start if isinstance(control_guidance_start, list) else [control_guidance_start]
p.extra_generation_params['Control end'] = control_guidance_end if isinstance(control_guidance_end, list) else [control_guidance_end]
p.extra_generation_params["Control conditioning"] = ';'.join([str(c) for c in p.extra_generation_params["Control conditioning"]])
p.extra_generation_params['Control start'] = ';'.join([str(c) for c in p.extra_generation_params['Control start']])
p.extra_generation_params['Control end'] = ';'.join([str(c) for c in p.extra_generation_params['Control end']])
if unit_type == 't2i adapter' and has_models:
p.extra_generation_params["Control type"] = 'T2I-Adapter'
p.task_args['adapter_conditioning_scale'] = control_conditioning
instance = t2iadapter.AdapterPipeline(selected_models, shared.sd_model)
pipe = instance.pipeline
if inits is not None:
shared.log.warning('Control: T2I-Adapter does not support separate init image')
elif unit_type == 'controlnet' and has_models:
p.extra_generation_params["Control type"] = 'ControlNet'
p.task_args['controlnet_conditioning_scale'] = control_conditioning
p.task_args['control_guidance_start'] = control_guidance_start
p.task_args['control_guidance_end'] = control_guidance_end
p.task_args['guess_mode'] = p.guess_mode
instance = controlnet.ControlNetPipeline(selected_models, shared.sd_model, p=p)
pipe = instance.pipeline
elif unit_type == 'xs' and has_models:
p.extra_generation_params["Control type"] = 'ControlNet-XS'
p.controlnet_conditioning_scale = control_conditioning
p.control_guidance_start = control_guidance_start
p.control_guidance_end = control_guidance_end
instance = xs.ControlNetXSPipeline(selected_models, shared.sd_model)
pipe = instance.pipeline
if inits is not None:
shared.log.warning('Control: ControlNet-XS does not support separate init image')
elif unit_type == 'lite' and has_models:
p.extra_generation_params["Control type"] = 'ControlLLLite'
p.controlnet_conditioning_scale = control_conditioning
instance = lite.ControlLLitePipeline(shared.sd_model)
pipe = instance.pipeline
if inits is not None:
shared.log.warning('Control: ControlLLLite does not support separate init image')
elif unit_type == 'reference' and has_models:
p.extra_generation_params["Control type"] = 'Reference'
p.extra_generation_params["Control attention"] = p.attention
p.task_args['reference_attn'] = 'Attention' in p.attention
p.task_args['reference_adain'] = 'Adain' in p.attention
p.task_args['attention_auto_machine_weight'] = p.query_weight
p.task_args['gn_auto_machine_weight'] = p.adain_weight
p.task_args['style_fidelity'] = p.fidelity
instance = reference.ReferencePipeline(shared.sd_model)
pipe = instance.pipeline
if inits is not None:
shared.log.warning('Control: ControlNet-XS does not support separate init image')
else: # run in txt2img/img2img mode
if len(active_strength) > 0:
p.strength = active_strength[0]
pipe = shared.sd_model
instance = None
debug(f'Control: run type={unit_type} models={has_models} pipe={pipe.__class__.__name__ if pipe is not None else None}')
return pipe
def check_active(p, unit_type, units):
active_process: List[processors.Processor] = [] # all active preprocessors
active_model: List[Union[controlnet.ControlNet, xs.ControlNetXS, t2iadapter.Adapter]] = [] # all active models
active_strength: List[float] = [] # strength factors for all active models
active_start: List[float] = [] # start step for all active models
active_end: List[float] = [] # end step for all active models
num_units = 0
for u in units:
if u.type != unit_type:
continue
num_units += 1
debug(f'Control unit: i={num_units} type={u.type} enabled={u.enabled}')
if not u.enabled:
if u.controlnet is not None and u.controlnet.model is not None:
debug(f'Control unit offload: model="{u.controlnet.model_id}" device={devices.cpu}')
sd_models.move_model(u.controlnet.model, devices.cpu)
continue
if u.controlnet is not None and u.controlnet.model is not None:
debug(f'Control unit offload: model="{u.controlnet.model_id}" device={devices.device}')
sd_models.move_model(u.controlnet.model, devices.device)
if unit_type == 't2i adapter' and u.adapter.model is not None:
active_process.append(u.process)
active_model.append(u.adapter)
active_strength.append(float(u.strength))
p.adapter_conditioning_factor = u.factor
shared.log.debug(f'Control T2I-Adapter unit: i={num_units} process="{u.process.processor_id}" model="{u.adapter.model_id}" strength={u.strength} factor={u.factor}')
elif unit_type == 'controlnet' and u.controlnet.model is not None:
active_process.append(u.process)
active_model.append(u.controlnet)
active_strength.append(float(u.strength))
active_start.append(float(u.start))
active_end.append(float(u.end))
p.guess_mode = u.guess
if isinstance(u.mode, str):
p.control_mode = u.choices.index(u.mode) if u.mode in u.choices else 0
p.is_tile = p.is_tile or 'tile' in u.mode.lower()
p.control_tile = u.tile
p.extra_generation_params["Control mode"] = u.mode
shared.log.debug(f'Control ControlNet unit: i={num_units} process="{u.process.processor_id}" model="{u.controlnet.model_id}" strength={u.strength} guess={u.guess} start={u.start} end={u.end} mode={u.mode}')
elif unit_type == 'xs' and u.controlnet.model is not None:
active_process.append(u.process)
active_model.append(u.controlnet)
active_strength.append(float(u.strength))
active_start.append(float(u.start))
active_end.append(float(u.end))
shared.log.debug(f'Control ControlNet-XS unit: i={num_units} process={u.process.processor_id} model={u.controlnet.model_id} strength={u.strength} guess={u.guess} start={u.start} end={u.end}')
elif unit_type == 'lite' and u.controlnet.model is not None:
active_process.append(u.process)
active_model.append(u.controlnet)
active_strength.append(float(u.strength))
shared.log.debug(f'Control ControlLLite unit: i={num_units} process={u.process.processor_id} model={u.controlnet.model_id} strength={u.strength} guess={u.guess} start={u.start} end={u.end}')
elif unit_type == 'reference':
p.override = u.override
p.attention = u.attention
p.query_weight = float(u.query_weight)
p.adain_weight = float(u.adain_weight)
p.fidelity = u.fidelity
shared.log.debug('Control Reference unit')
else:
if u.process.processor_id is not None:
active_process.append(u.process)
shared.log.debug(f'Control process unit: i={num_units} process={u.process.processor_id}')
active_strength.append(float(u.strength))
debug(f'Control active: process={len(active_process)} model={len(active_model)}')
return active_process, active_model, active_strength, active_start, active_end
def check_enabled(p, unit_type, units, active_model, active_strength, active_start, active_end):
has_models = False
selected_models: List[Union[controlnet.ControlNetModel, xs.ControlNetXSModel, t2iadapter.AdapterModel]] = None
control_conditioning = None
control_guidance_start = None
control_guidance_end = None
if unit_type == 't2i adapter' or unit_type == 'controlnet' or unit_type == 'xs' or unit_type == 'lite':
if len(active_model) == 0:
selected_models = None
elif len(active_model) == 1:
selected_models = active_model[0].model if active_model[0].model is not None else None
p.is_tile = p.is_tile or 'tile' in active_model[0].model_id.lower()
has_models = selected_models is not None
control_conditioning = active_strength[0] if len(active_strength) > 0 else 1 # strength or list[strength]
control_guidance_start = active_start[0] if len(active_start) > 0 else 0
control_guidance_end = active_end[0] if len(active_end) > 0 else 1
else:
selected_models = [m.model for m in active_model if m.model is not None]
has_models = len(selected_models) > 0
control_conditioning = active_strength[0] if len(active_strength) == 1 else list(active_strength) # strength or list[strength]
control_guidance_start = active_start[0] if len(active_start) == 1 else list(active_start)
control_guidance_end = active_end[0] if len(active_end) == 1 else list(active_end)
elif unit_type == 'reference':
has_models = any(u.enabled for u in units if u.type == 'reference')
else:
pass
return has_models, selected_models, control_conditioning, control_guidance_start, control_guidance_end
def control_set(kwargs):
if kwargs:
global p_extra_args # pylint: disable=global-statement
@@ -83,20 +245,15 @@ def control_run(state: str = '',
u.adapter.load(u.model_name, force=False)
else:
u.controlnet.load(u.model_name, force=False)
u.update_choices(u.model_name)
if u.process is not None and u.process.override is None and u.override is not None:
u.process.override = u.override
global instance, pipe, original_pipeline # pylint: disable=global-statement
t_start = time.time()
global pipe, original_pipeline # pylint: disable=global-statement
debug(f'Control: type={unit_type} input={inputs} init={inits} type={input_type}')
if inputs is None or (type(inputs) is list and len(inputs) == 0):
inputs = [None]
output_images: List[Image.Image] = [] # output images
active_process: List[processors.Processor] = [] # all active preprocessors
active_model: List[Union[controlnet.ControlNet, xs.ControlNetXS, t2iadapter.Adapter]] = [] # all active models
active_strength: List[float] = [] # strength factors for all active models
active_start: List[float] = [] # start step for all active models
active_end: List[float] = [] # end step for all active models
processed_image: Image.Image = None # last processed image
if mask is not None and input_type == 0:
input_type = 1 # inpaint always requires control_image
@@ -150,10 +307,11 @@ def control_run(state: str = '',
outpath_grids=shared.opts.outdir_grids or shared.opts.outdir_control_grids,
)
p.state = state
p.is_tile = False
# processing.process_init(p)
resize_mode_before = resize_mode_before if resize_name_before != 'None' and inputs is not None and len(inputs) > 0 else 0
# TODO monkey-patch for modernui missing tabs.select event
# TODO modernui: monkey-patch for missing tabs.select event
if selected_scale_tab_before == 0 and resize_name_before != 'None' and scale_by_before != 1 and inputs is not None and len(inputs) > 0:
shared.log.debug('Control: override resize mode=before')
selected_scale_tab_before = 1
@@ -224,155 +382,17 @@ def control_run(state: str = '',
unit_type = unit_type.strip().lower() if unit_type is not None else ''
t0 = time.time()
num_units = 0
for u in units:
if u.type != unit_type:
continue
num_units += 1
debug(f'Control unit: i={num_units} type={u.type} enabled={u.enabled}')
if not u.enabled:
if u.controlnet is not None and u.controlnet.model is not None:
debug(f'Control unit offload: model="{u.controlnet.model_id}" device={devices.cpu}')
sd_models.move_model(u.controlnet.model, devices.cpu)
continue
if u.controlnet is not None and u.controlnet.model is not None:
debug(f'Control unit offload: model="{u.controlnet.model_id}" device={devices.device}')
sd_models.move_model(u.controlnet.model, devices.device)
if unit_type == 't2i adapter' and u.adapter.model is not None:
active_process.append(u.process)
active_model.append(u.adapter)
active_strength.append(float(u.strength))
p.adapter_conditioning_factor = u.factor
shared.log.debug(f'Control T2I-Adapter unit: i={num_units} process={u.process.processor_id} model={u.adapter.model_id} strength={u.strength} factor={u.factor}')
elif unit_type == 'controlnet' and u.controlnet.model is not None:
active_process.append(u.process)
active_model.append(u.controlnet)
active_strength.append(float(u.strength))
active_start.append(float(u.start))
active_end.append(float(u.end))
p.guess_mode = u.guess
p.control_mode = u.mode
shared.log.debug(f'Control ControlNet unit: i={num_units} process={u.process.processor_id} model={u.controlnet.model_id} strength={u.strength} guess={u.guess} start={u.start} end={u.end} mode={u.mode}')
elif unit_type == 'xs' and u.controlnet.model is not None:
active_process.append(u.process)
active_model.append(u.controlnet)
active_strength.append(float(u.strength))
active_start.append(float(u.start))
active_end.append(float(u.end))
shared.log.debug(f'Control ControlNet-XS unit: i={num_units} process={u.process.processor_id} model={u.controlnet.model_id} strength={u.strength} guess={u.guess} start={u.start} end={u.end}')
elif unit_type == 'lite' and u.controlnet.model is not None:
active_process.append(u.process)
active_model.append(u.controlnet)
active_strength.append(float(u.strength))
shared.log.debug(f'Control ControlLLite unit: i={num_units} process={u.process.processor_id} model={u.controlnet.model_id} strength={u.strength} guess={u.guess} start={u.start} end={u.end}')
elif unit_type == 'reference':
p.override = u.override
p.attention = u.attention
p.query_weight = float(u.query_weight)
p.adain_weight = float(u.adain_weight)
p.fidelity = u.fidelity
shared.log.debug('Control Reference unit')
else:
if u.process.processor_id is not None:
active_process.append(u.process)
shared.log.debug(f'Control process unit: i={num_units} process={u.process.processor_id}')
active_strength.append(float(u.strength))
debug(f'Control active: process={len(active_process)} model={len(active_model)}')
active_process, active_model, active_strength, active_start, active_end = check_active(p, unit_type, units)
has_models, selected_models, control_conditioning, control_guidance_start, control_guidance_end = check_enabled(p, unit_type, units, active_model, active_strength, active_start, active_end)
processed: processing.Processed = None
image_txt = ''
info_txt = []
has_models = False
selected_models: List[Union[controlnet.ControlNetModel, xs.ControlNetXSModel, t2iadapter.AdapterModel]] = None
control_conditioning = None
control_guidance_start = None
control_guidance_end = None
if unit_type == 't2i adapter' or unit_type == 'controlnet' or unit_type == 'xs' or unit_type == 'lite':
if len(active_model) == 0:
selected_models = None
elif len(active_model) == 1:
selected_models = active_model[0].model if active_model[0].model is not None else None
has_models = selected_models is not None
control_conditioning = active_strength[0] if len(active_strength) > 0 else 1 # strength or list[strength]
control_guidance_start = active_start[0] if len(active_start) > 0 else 0
control_guidance_end = active_end[0] if len(active_end) > 0 else 1
else:
selected_models = [m.model for m in active_model if m.model is not None]
has_models = len(selected_models) > 0
control_conditioning = active_strength[0] if len(active_strength) == 1 else list(active_strength) # strength or list[strength]
control_guidance_start = active_start[0] if len(active_start) == 1 else list(active_start)
control_guidance_end = active_end[0] if len(active_end) == 1 else list(active_end)
elif unit_type == 'reference':
has_models = any(u.enabled for u in units if u.type == 'reference')
else:
pass
def set_pipe():
global pipe, instance # pylint: disable=global-statement
pipe = None
if has_models:
p.ops.append('control')
p.extra_generation_params["Control mode"] = unit_type # overriden later with pretty-print
p.extra_generation_params["Control conditioning"] = control_conditioning if isinstance(control_conditioning, list) else [control_conditioning]
p.extra_generation_params['Control start'] = control_guidance_start if isinstance(control_guidance_start, list) else [control_guidance_start]
p.extra_generation_params['Control end'] = control_guidance_end if isinstance(control_guidance_end, list) else [control_guidance_end]
p.extra_generation_params["Control model"] = ';'.join([(m.model_id or '') for m in active_model if m.model is not None])
p.extra_generation_params["Control conditioning"] = ';'.join([str(c) for c in p.extra_generation_params["Control conditioning"]])
p.extra_generation_params['Control start'] = ';'.join([str(c) for c in p.extra_generation_params['Control start']])
p.extra_generation_params['Control end'] = ';'.join([str(c) for c in p.extra_generation_params['Control end']])
if unit_type == 't2i adapter' and has_models:
p.extra_generation_params["Control mode"] = 'T2I-Adapter'
p.task_args['adapter_conditioning_scale'] = control_conditioning
instance = t2iadapter.AdapterPipeline(selected_models, shared.sd_model)
pipe = instance.pipeline
if inits is not None:
shared.log.warning('Control: T2I-Adapter does not support separate init image')
elif unit_type == 'controlnet' and has_models:
p.extra_generation_params["Control mode"] = 'ControlNet'
p.task_args['controlnet_conditioning_scale'] = control_conditioning
p.task_args['control_guidance_start'] = control_guidance_start
p.task_args['control_guidance_end'] = control_guidance_end
p.task_args['guess_mode'] = p.guess_mode
instance = controlnet.ControlNetPipeline(selected_models, shared.sd_model)
pipe = instance.pipeline
elif unit_type == 'xs' and has_models:
p.extra_generation_params["Control mode"] = 'ControlNet-XS'
p.controlnet_conditioning_scale = control_conditioning
p.control_guidance_start = control_guidance_start
p.control_guidance_end = control_guidance_end
instance = xs.ControlNetXSPipeline(selected_models, shared.sd_model)
pipe = instance.pipeline
if inits is not None:
shared.log.warning('Control: ControlNet-XS does not support separate init image')
elif unit_type == 'lite' and has_models:
p.extra_generation_params["Control mode"] = 'ControlLLLite'
p.controlnet_conditioning_scale = control_conditioning
instance = lite.ControlLLitePipeline(shared.sd_model)
pipe = instance.pipeline
if inits is not None:
shared.log.warning('Control: ControlLLLite does not support separate init image')
elif unit_type == 'reference' and has_models:
p.extra_generation_params["Control mode"] = 'Reference'
p.extra_generation_params["Control attention"] = p.attention
p.task_args['reference_attn'] = 'Attention' in p.attention
p.task_args['reference_adain'] = 'Adain' in p.attention
p.task_args['attention_auto_machine_weight'] = p.query_weight
p.task_args['gn_auto_machine_weight'] = p.adain_weight
p.task_args['style_fidelity'] = p.fidelity
instance = reference.ReferencePipeline(shared.sd_model)
pipe = instance.pipeline
if inits is not None:
shared.log.warning('Control: ControlNet-XS does not support separate init image')
else: # run in txt2img/img2img mode
if len(active_strength) > 0:
p.strength = active_strength[0]
pipe = shared.sd_model
instance = None
debug(f'Control: run type={unit_type} models={has_models} pipe={pipe.__class__.__name__ if pipe is not None else None}')
return pipe
p.is_tile = p.is_tile and has_models
pipe = set_pipe()
pipe = set_pipe(p, has_models, unit_type, selected_models, active_model, active_strength, control_conditioning, control_guidance_start, control_guidance_end, inits)
debug(f'Control pipeline: class={pipe.__class__.__name__} args={vars(p)}')
t1, t2, t3 = time.time(), 0, 0
status = True
@@ -395,6 +415,8 @@ def control_run(state: str = '',
else:
original_pipeline = None
possible = sd_models.get_call(pipe).keys()
try:
with devices.inference_context():
if isinstance(inputs, str): # only video, the rest is a list
@@ -424,7 +446,7 @@ def control_run(state: str = '',
while status:
if pipe is None: # pipe may have been reset externally
pipe = set_pipe()
pipe = set_pipe(p, has_models, unit_type, selected_models, active_model, active_strength, control_conditioning, control_guidance_start, control_guidance_end, inits)
debug(f'Control pipeline reinit: class={pipe.__class__.__name__}')
processed_image = None
if frame is not None:
@@ -564,19 +586,29 @@ def control_run(state: str = '',
return [], '', '', 'Reference mode without image'
elif unit_type == 'controlnet' and has_models:
if input_type == 0: # Control only
if shared.sd_model_type in ['f1', 'sd3'] and 'control_image' not in p.task_args:
p.task_args['control_image'] = p.init_images # some controlnets mandate this
if 'control_image' in possible:
p.task_args['control_image'] = [p.init_images] if isinstance(p.init_images, Image.Image) else p.init_images
elif 'image' in possible:
p.task_args['image'] = [p.init_images] if isinstance(p.init_images, Image.Image) else p.init_images
if 'control_mode' in possible:
p.task_args['control_mode'] = getattr(p, 'control_mode', None)
if 'strength' in possible:
p.task_args['strength'] = p.denoising_strength
p.init_images = None
elif input_type == 1: # Init image same as control
p.task_args['control_image'] = p.init_images # switch image and control_image
p.task_args['strength'] = p.denoising_strength
if 'control_image' in possible:
p.task_args['control_image'] = p.init_images # switch image and control_image
if 'strength' in possible:
p.task_args['strength'] = p.denoising_strength
p.init_images = [p.override or input_image] * len(active_model)
elif input_type == 2: # Separate init image
if init_image is None:
shared.log.warning('Control: separate init image not provided')
init_image = input_image
p.task_args['control_image'] = p.init_images # switch image and control_image
p.task_args['strength'] = p.denoising_strength
if 'control_image' in possible:
p.task_args['control_image'] = p.init_images # switch image and control_image
if 'strength' in possible:
p.task_args['strength'] = p.denoising_strength
p.init_images = [init_image] * len(active_model)
if is_generator:
@@ -609,26 +641,31 @@ def control_run(state: str = '',
p.task_args['strength'] = denoising_strength
p.image_mask = mask
shared.sd_model = sd_models.set_diffuser_pipe(shared.sd_model, sd_models.DiffusersTaskType.INPAINTING) # only controlnet supports inpaint
elif 'control_image' in p.task_args:
if hasattr(p, 'init_images') and p.init_images is not None:
shared.sd_model = sd_models.set_diffuser_pipe(shared.sd_model, sd_models.DiffusersTaskType.IMAGE_2_IMAGE) # only controlnet supports img2img
else:
shared.sd_model = sd_models.set_diffuser_pipe(shared.sd_model, sd_models.DiffusersTaskType.TEXT_2_IMAGE)
if hasattr(p, 'init_images') and p.init_images is not None:
if hasattr(p, 'init_images') and p.init_images is not None and 'image' in possible:
p.task_args['image'] = p.init_images # need to set explicitly for txt2img
del p.init_images
if unit_type == 'lite':
p.init_image = [input_image]
instance.apply(selected_models, processed_image, control_conditioning)
if p.control_mode is not None:
p.task_args['control_mode'] = p.control_mode
if getattr(p, 'control_mode', None) is not None:
p.task_args['control_mode'] = getattr(p, 'control_mode', None)
if hasattr(p, 'init_images') and p.init_images is None: # delete empty
del p.init_images
# final check
if has_models:
if unit_type in ['controlnet', 't2i adapter', 'lite', 'xs'] and p.task_args.get('image', None) is None and getattr(p, 'init_images', None) is None:
if unit_type in ['controlnet', 't2i adapter', 'lite', 'xs'] \
and p.task_args.get('image', None) is None \
and p.task_args.get('control_image', None) is None \
and getattr(p, 'init_images', None) is None \
and getattr(p, 'image', None) is None:
if is_generator:
yield terminate(f'Mode={p.extra_generation_params.get("Control mode", None)} input image is none')
shared.log.debug(f'Control args: {p.task_args}')
yield terminate(f'Mode={p.extra_generation_params.get("Control type", None)} input image is none')
return [], '', '', 'Error: Input image is none'
# resize mask
@@ -658,12 +695,19 @@ def control_run(state: str = '',
script_runner.initialize_scripts(False)
p.script_args = script.init_default_script_args(script_runner)
processed = p.scripts.run(p, *p.script_args)
# actual processing
if p.is_tile:
processed: processing.Processed = tile.run_tiling(p, input_image)
if processed is None and p.scripts is not None:
processed = p.scripts.run(p, *p.script_args)
if processed is None:
processed: processing.Processed = processing.process_images(p) # run actual pipeline
else:
script_run = True
processed = p.scripts.after(p, processed, *p.script_args)
# postprocessing
if p.scripts is not None:
processed = p.scripts.after(p, processed, *p.script_args)
output = None
if processed is not None:
output = processed.images
@@ -717,14 +761,11 @@ def control_run(state: str = '',
shared.log.error(f'Control pipeline failed: type={unit_type} units={len(active_model)} error={e}')
errors.display(e, 'Control')
t_end = time.time()
if len(output_images) == 0:
output_images = None
image_txt = '| Images None'
else:
image_str = [f'{image.width}x{image.height}' for image in output_images]
image_txt = f'| Time {t_end-t_start:.2f}s | Images {len(output_images)} | Size {" ".join(image_str)}'
image_txt = ''
p.init_images = output_images # may be used for hires
if video_type != 'None' and isinstance(output_images, list):
@@ -738,10 +779,9 @@ def control_run(state: str = '',
restore_pipeline()
debug(f'Ready: {image_txt}')
html_txt = f'<p>Ready {image_txt}</p>'
html_txt = f'<p>Ready {image_txt}</p>' if image_txt != '' else ''
if len(info_txt) > 0:
html_txt = html_txt + infotext_to_html(info_txt[0])
if is_generator:
yield (output_images, blended_image, html_txt, output_filename)
else:
return (output_images, blended_image, html_txt, output_filename)
return (output_images, blended_image, html_txt, output_filename)
modules/control/tile.py
+73
View File
@@ -0,0 +1,73 @@
import time
from PIL import Image
from modules import shared, processing, images, sd_models
def get_tile(image: Image.Image, x: int, y: int, sx: int, sy: int) -> Image.Image:
return image.crop((
(x + 0) * image.width // sx,
(y + 0) * image.height // sy,
(x + 1) * image.width // sx,
(y + 1) * image.height // sy
))
def set_tile(image: Image.Image, x: int, y: int, tiled: Image.Image):
image.paste(tiled, (x * tiled.width, y * tiled.height))
return image
def run_tiling(p: processing.StableDiffusionProcessing, input_image: Image.Image) -> processing.Processed:
t0 = time.time()
# prepare images
sx, sy = p.control_tile.split('x')
sx = int(sx)
sy = int(sy)
if sx <= 0 or sy <= 0:
raise ValueError('Control Tile: invalid tile size')
control_image = p.task_args.get('control_image', None) or p.task_args.get('image', None)
control_upscaled = None
if isinstance(control_image, list) and len(control_image) > 0:
w, h = 8 * int(sx * control_image[0].width) // 8, 8 * int(sy * control_image[0].height) // 8
control_upscaled = images.resize_image(resize_mode=1 if sx==sy else 5, im=control_image[0], width=w, height=h, context='add with forward')
init_image = p.override or input_image
init_upscaled = None
if init_image is not None:
w, h = 8 * int(sx * init_image.width) // 8, 8 * int(sy * init_image.height) // 8
init_upscaled = images.resize_image(resize_mode=1 if sx==sy else 5, im=init_image, width=w, height=h, context='add with forward')
t1 = time.time()
shared.log.debug(f'Control Tile: scale={sx}x{sy} resize={"fixed" if sx==sy else "context"} control={control_upscaled} init={init_upscaled} time={t1-t0:.3f}')
# stop processing from restoring pipeline on each iteration
orig_restore_pipeline = getattr(shared.sd_model, 'restore_pipeline', None)
shared.sd_model.restore_pipeline = None
# run tiling
for x in range(sx):
for y in range(sy):
shared.log.info(f'Control Tile: tile={x+1}-{sx}/{y+1}-{sy} target={control_upscaled}')
shared.sd_model = sd_models.set_diffuser_pipe(shared.sd_model, sd_models.DiffusersTaskType.IMAGE_2_IMAGE)
p.init_images = None
p.task_args['control_mode'] = p.control_mode
p.task_args['strength'] = p.denoising_strength
if init_upscaled is not None:
p.task_args['image'] = [get_tile(init_upscaled, x, y, sx, sy)]
if control_upscaled is not None:
p.task_args['control_image'] = [get_tile(control_upscaled, x, y, sx, sy)]
processed: processing.Processed = processing.process_images(p) # run actual pipeline
if processed is None or len(processed.images) == 0:
continue
control_upscaled = set_tile(control_upscaled, x, y, processed.images[0])
# post-process
p.width = control_upscaled.width
p.height = control_upscaled.height
processed.images = [control_upscaled]
processed.info = processed.infotext(p, 0)
processed.infotexts = [processed.info]
shared.sd_model.restore_pipeline = orig_restore_pipeline
if hasattr(shared.sd_model, 'restore_pipeline') and shared.sd_model.restore_pipeline is not None:
shared.sd_model.restore_pipeline()
t2 = time.time()
shared.log.debug(f'Control Tile: image={control_upscaled} time={t2-t0:.3f}')
return processed
modules/control/unit.py
+34 -4
View File
@@ -16,6 +16,22 @@ unit_types = ['t2i adapter', 'controlnet', 'xs', 'lite', 'reference', 'ip']
class Unit(): # mashup of gradio controls and mapping to actual implementation classes
def update_choices(self, model_id=None):
name = model_id or self.model_name
if name == 'InstantX Union':
self.choices = ['canny', 'tile', 'depth', 'blur', 'pose', 'gray', 'lq']
elif name == 'Shakker-Labs Union':
self.choices = ['canny', 'tile', 'depth', 'blur', 'pose', 'gray', 'lq']
elif name == 'Xinsir Union XL':
self.choices = ['openpose', 'depth', 'scribble', 'canny', 'normal']
elif name == 'Xinsir ProMax XL':
self.choices = ['openpose', 'depth', 'scribble', 'canny', 'normal', 'segment', 'tile', 'repaint']
else:
self.choices = ['default']
def __str__(self):
return f'Unit: type={self.type} enabled={self.enabled} strength={self.strength} start={self.start} end={self.end} mode={self.mode} tile={self.tile}'
def __init__(self,
# values
index: int = None,
@@ -38,6 +54,7 @@ class Unit(): # mashup of gradio controls and mapping to actual implementation c
control_start = None,
control_end = None,
control_mode = None,
control_tile = None,
result_txt = None,
extra_controls: list = [],
):
@@ -70,6 +87,10 @@ class Unit(): # mashup of gradio controls and mapping to actual implementation c
self.fidelity = 0.5
self.query_weight = 1.0
self.adain_weight = 1.0
# control mode
self.choices = ['default']
# control tile
self.tile = '1x1'
def reset():
if self.process is not None:
@@ -92,10 +113,16 @@ class Unit(): # mashup of gradio controls and mapping to actual implementation c
self.end = max(start, end)
def control_mode_change(mode):
self.mode = mode - 1 if mode > 0 else None
self.mode = self.choices.index(mode) if mode is not None and mode in self.choices else 0
def control_mode_show(model_id):
return gr.update(visible='union' in model_id.lower())
def control_tile_change(tile):
self.tile = tile
def control_choices(model_id):
self.update_choices(model_id)
mode_visible = 'union' in model_id.lower() or 'promax' in model_id.lower()
tile_visible = 'union' in model_id.lower() or 'promax' in model_id.lower() or 'tile' in model_id.lower()
return [gr.update(visible=mode_visible, choices=self.choices), gr.update(visible=tile_visible)]
def adapter_extra(c1):
self.factor = c1
@@ -172,7 +199,7 @@ class Unit(): # mashup of gradio controls and mapping to actual implementation c
else:
self.controls.append(model_id)
model_id.change(fn=self.controlnet.load, inputs=[model_id], outputs=[result_txt], show_progress=True)
model_id.change(fn=control_mode_show, inputs=[model_id], outputs=[control_mode], show_progress=False)
model_id.change(fn=control_choices, inputs=[model_id], outputs=[control_mode, control_tile], show_progress=False)
if extra_controls is not None and len(extra_controls) > 0:
extra_controls[0].change(fn=controlnet_extra, inputs=extra_controls)
elif self.type == 'xs':
@@ -231,3 +258,6 @@ class Unit(): # mashup of gradio controls and mapping to actual implementation c
if control_mode is not None:
self.controls.append(control_mode)
control_mode.change(fn=control_mode_change, inputs=[control_mode])
if control_tile is not None:
self.controls.append(control_tile)
control_tile.change(fn=control_tile_change, inputs=[control_tile])
modules/control/units/controlnet.py
+78 -26
View File
@@ -5,6 +5,7 @@ from diffusers import StableDiffusionPipeline, StableDiffusionXLPipeline, FluxPi
from modules.control.units import detect
from modules.shared import log, opts, listdir
from modules import errors, sd_models, devices, model_quant
from modules.processing import StableDiffusionProcessingControl
what = 'ControlNet'
@@ -51,17 +52,20 @@ predefined_sdxl = {
'Depth Mid XL': 'diffusers/controlnet-depth-sdxl-1.0-mid',
'OpenPose XL': 'thibaud/controlnet-openpose-sdxl-1.0/bin',
'Xinsir Union XL': 'xinsir/controlnet-union-sdxl-1.0',
'Xinsir ProMax XL': 'brad-twinkl/controlnet-union-sdxl-1.0-promax',
'Xinsir OpenPose XL': 'xinsir/controlnet-openpose-sdxl-1.0',
'Xinsir Canny XL': 'xinsir/controlnet-canny-sdxl-1.0',
'Xinsir Depth XL': 'xinsir/controlnet-depth-sdxl-1.0',
'Xinsir Scribble XL': 'xinsir/controlnet-scribble-sdxl-1.0',
'Xinsir Anime Painter XL': 'xinsir/anime-painter',
'Xinsir Tile XL': 'xinsir/controlnet-tile-sdxl-1.0',
'NoobAI Canny XL': 'Eugeoter/noob-sdxl-controlnet-canny',
'NoobAI Lineart Anime XL': 'Eugeoter/noob-sdxl-controlnet-lineart_anime',
'NoobAI Depth XL': 'Eugeoter/noob-sdxl-controlnet-depth',
'NoobAI Normal XL': 'Eugeoter/noob-sdxl-controlnet-normal',
'NoobAI SoftEdge XL': 'Eugeoter/noob-sdxl-controlnet-softedge_hed',
'NoobAI OpenPose XL': 'einar77/noob-openpose',
'TTPlanet Tile Realistic XL': 'Yakonrus/SDXL_Controlnet_Tile_Realistic_v2',
# 'StabilityAI Canny R128': 'stabilityai/control-lora/control-LoRAs-rank128/control-lora-canny-rank128.safetensors',
# 'StabilityAI Depth R128': 'stabilityai/control-lora/control-LoRAs-rank128/control-lora-depth-rank128.safetensors',
# 'StabilityAI Recolor R128': 'stabilityai/control-lora/control-LoRAs-rank128/control-lora-recolor-rank128.safetensors',
@@ -75,6 +79,8 @@ predefined_f1 = {
"InstantX Union": 'InstantX/FLUX.1-dev-Controlnet-Union',
"InstantX Canny": 'InstantX/FLUX.1-dev-Controlnet-Canny',
"JasperAI Depth": 'jasperai/Flux.1-dev-Controlnet-Depth',
"BlackForrestLabs Canny LoRA": '/huggingface.co/black-forest-labs/FLUX.1-Canny-dev-lora/flux1-canny-dev-lora.safetensors',
"BlackForrestLabs Depth LoRA": '/huggingface.co/black-forest-labs/FLUX.1-Depth-dev-lora/flux1-depth-dev-lora.safetensors',
"JasperAI Surface Normals": 'jasperai/Flux.1-dev-Controlnet-Surface-Normals',
"JasperAI Upscaler": 'jasperai/Flux.1-dev-Controlnet-Upscaler',
"Shakker-Labs Union": 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro',
@@ -85,6 +91,9 @@ predefined_f1 = {
"XLabs-AI HED": 'XLabs-AI/flux-controlnet-hed-diffusers'
}
predefined_sd3 = {
"StabilityAI Canny": 'diffusers-internal-dev/sd35-controlnet-canny-8b',
"StabilityAI Depth": 'diffusers-internal-dev/sd35-controlnet-depth-8b',
"StabilityAI Blur": 'diffusers-internal-dev/sd35-controlnet-blur-8b',
"InstantX Canny": 'InstantX/SD3-Controlnet-Canny',
"InstantX Pose": 'InstantX/SD3-Controlnet-Pose',
"InstantX Depth": 'InstantX/SD3-Controlnet-Depth',
@@ -92,6 +101,14 @@ predefined_sd3 = {
"Alimama Inpainting": 'alimama-creative/SD3-Controlnet-Inpainting',
"Alimama SoftEdge": 'alimama-creative/SD3-Controlnet-Softedge',
}
variants = {
'NoobAI Canny XL': 'fp16',
'NoobAI Lineart Anime XL': 'fp16',
'NoobAI Depth XL': 'fp16',
'NoobAI Normal XL': 'fp16',
'NoobAI SoftEdge XL': 'fp16',
'TTPlanet Tile Realistic XL': 'fp16',
}
models = {}
all_models = {}
all_models.update(predefined_sd15)
@@ -159,26 +176,35 @@ class ControlNet():
self.model = None
self.model_id = None
def get_class(self):
import modules.shared
if modules.shared.sd_model_type == 'sd':
def get_class(self, model_id:str=''):
from modules import shared
if shared.sd_model_type == 'none':
_load = shared.sd_model # trigger a load
if shared.sd_model_type == 'sd':
from diffusers import ControlNetModel as cls # pylint: disable=reimported
config = 'lllyasviel/control_v11p_sd15_canny'
elif modules.shared.sd_model_type == 'sdxl':
from diffusers import ControlNetModel as cls # pylint: disable=reimported # sdxl shares same model class
config = 'Eugeoter/noob-sdxl-controlnet-canny'
elif modules.shared.sd_model_type == 'f1':
elif shared.sd_model_type == 'sdxl':
if 'union' in model_id.lower():
from diffusers import ControlNetUnionModel as cls
config = 'xinsir/controlnet-union-sdxl-1.0'
elif 'promax' in model_id.lower():
from diffusers import ControlNetUnionModel as cls
config = 'brad-twinkl/controlnet-union-sdxl-1.0-promax'
else:
from diffusers import ControlNetModel as cls # pylint: disable=reimported # sdxl shares same model class
config = 'Eugeoter/noob-sdxl-controlnet-canny'
elif shared.sd_model_type == 'f1':
from diffusers import FluxControlNetModel as cls
config = 'InstantX/FLUX.1-dev-Controlnet-Union'
elif modules.shared.sd_model_type == 'sd3':
elif shared.sd_model_type == 'sd3':
from diffusers import SD3ControlNetModel as cls
config = 'InstantX/SD3-Controlnet-Canny'
else:
log.error(f'Control {what}: type={modules.shared.sd_model_type} unsupported model')
log.error(f'Control {what}: type={shared.sd_model_type} unsupported model')
return None, None
return cls, config
def load_safetensors(self, model_path):
def load_safetensors(self, model_id, model_path):
name = os.path.splitext(model_path)[0]
config_path = None
if not os.path.exists(model_path):
@@ -203,7 +229,7 @@ class ControlNet():
config_path = f'{name}.json'
if config_path is not None:
self.load_config['original_config_file '] = config_path
cls, config = self.get_class()
cls, config = self.get_class(model_id)
if cls is None:
log.error(f'Control {what} model load failed: unknown base model')
else:
@@ -225,23 +251,26 @@ class ControlNet():
if model_path is None:
log.error(f'Control {what} model load failed: id="{model_id}" error=unknown model id')
return
if 'lora' in model_id.lower():
self.model = model_path
return
if model_id == self.model_id and not force:
log.debug(f'Control {what} model: id="{model_id}" path="{model_path}" already loaded')
return
log.debug(f'Control {what} model loading: id="{model_id}" path="{model_path}"')
cls, _config = self.get_class(model_id)
if model_path.endswith('.safetensors'):
self.load_safetensors(model_path)
self.load_safetensors(model_id, model_path)
else:
kwargs = {}
if '/bin' in model_path:
model_path = model_path.replace('/bin', '')
self.load_config['use_safetensors'] = False
cls, _config = self.get_class()
if cls is None:
log.error(f'Control {what} model load failed: id="{model_id}" unknown base model')
return
if 'Eugeoter' in model_path:
kwargs['variant'] = 'fp16'
if variants.get(model_id, None) is not None:
kwargs['variant'] = variants[model_id]
self.model = cls.from_pretrained(model_path, **self.load_config, **kwargs)
if self.model is None:
return
@@ -268,7 +297,7 @@ class ControlNet():
self.model.to(self.device)
t1 = time.time()
self.model_id = model_id
log.debug(f'Control {what} model loaded: id="{model_id}" path="{model_path}" time={t1-t0:.2f}')
log.debug(f'Control {what} model loaded: id="{model_id}" path="{model_path}" cls={cls.__name__} time={t1-t0:.2f}')
return f'{what} loaded model: {model_id}'
except Exception as e:
log.error(f'Control {what} model load failed: id="{model_id}" error={e}')
@@ -281,16 +310,27 @@ class ControlNetPipeline():
controlnet: Union[ControlNetModel, list[ControlNetModel]],
pipeline: Union[StableDiffusionXLPipeline, StableDiffusionPipeline, FluxPipeline, StableDiffusion3Pipeline],
dtype = None,
p: StableDiffusionProcessingControl = None, # pylint: disable=unused-argument
):
t0 = time.time()
self.orig_pipeline = pipeline
self.pipeline = None
controlnets = controlnet if isinstance(controlnet, list) else [controlnet]
loras = [cn for cn in controlnets if isinstance(cn, str)]
controlnets = [cn for cn in controlnets if not isinstance(cn, str)]
if pipeline is None:
log.error('Control model pipeline: model not loaded')
return
elif detect.is_sdxl(pipeline):
from diffusers import StableDiffusionXLControlNetPipeline
self.pipeline = StableDiffusionXLControlNetPipeline(
elif detect.is_sdxl(pipeline) and len(controlnets) > 0:
from diffusers import StableDiffusionXLControlNetPipeline, StableDiffusionXLControlNetUnionPipeline
if controlnet.__class__.__name__ == 'ControlNetUnionModel':
cls = StableDiffusionXLControlNetUnionPipeline
controlnets = controlnets[0] # using only first one
else:
cls = StableDiffusionXLControlNetPipeline
self.pipeline = cls(
vae=pipeline.vae,
text_encoder=pipeline.text_encoder,
text_encoder_2=pipeline.text_encoder_2,
@@ -299,9 +339,9 @@ class ControlNetPipeline():
unet=pipeline.unet,
scheduler=pipeline.scheduler,
feature_extractor=getattr(pipeline, 'feature_extractor', None),
controlnet=controlnet, # can be a list
controlnet=controlnets, # can be a list
)
elif detect.is_sd15(pipeline):
elif detect.is_sd15(pipeline) and len(controlnets) > 0:
from diffusers import StableDiffusionControlNetPipeline
self.pipeline = StableDiffusionControlNetPipeline(
vae=pipeline.vae,
@@ -312,10 +352,10 @@ class ControlNetPipeline():
feature_extractor=getattr(pipeline, 'feature_extractor', None),
requires_safety_checker=False,
safety_checker=None,
controlnet=controlnet, # can be a list
controlnet=controlnets, # can be a list
)
sd_models.move_model(self.pipeline, pipeline.device)
elif detect.is_f1(pipeline):
elif detect.is_f1(pipeline) and len(controlnets) > 0:
from diffusers import FluxControlNetPipeline
self.pipeline = FluxControlNetPipeline(
vae=pipeline.vae.to(devices.device),
@@ -325,9 +365,9 @@ class ControlNetPipeline():
tokenizer_2=pipeline.tokenizer_2,
transformer=pipeline.transformer,
scheduler=pipeline.scheduler,
controlnet=controlnet, # can be a list
controlnet=controlnets, # can be a list
)
elif detect.is_sd3(pipeline):
elif detect.is_sd3(pipeline) and len(controlnets) > 0:
from diffusers import StableDiffusion3ControlNetPipeline
self.pipeline = StableDiffusion3ControlNetPipeline(
vae=pipeline.vae,
@@ -339,8 +379,18 @@ class ControlNetPipeline():
tokenizer_3=pipeline.tokenizer_3,
transformer=pipeline.transformer,
scheduler=pipeline.scheduler,
controlnet=controlnet, # can be a list
controlnet=controlnets, # can be a list
)
elif len(loras) > 0:
self.pipeline = pipeline
for lora in loras:
log.debug(f'Control {what} pipeline: lora="{lora}"')
lora = lora.replace('/huggingface.co/', '')
self.pipeline.load_lora_weights(lora)
"""
if p is not None:
p.prompt += f'<lora:{lora}:1.0>'
"""
else:
log.error(f'Control {what} pipeline: class={pipeline.__class__.__name__} unsupported model type')
return
@@ -350,6 +400,7 @@ class ControlNetPipeline():
return
if dtype is not None:
self.pipeline = self.pipeline.to(dtype)
if opts.diffusers_offload_mode == 'none':
sd_models.move_model(self.pipeline, devices.device)
from modules.sd_models import set_diffuser_offload
@@ -359,5 +410,6 @@ class ControlNetPipeline():
log.debug(f'Control {what} pipeline: class={self.pipeline.__class__.__name__} time={t1-t0:.2f}')
def restore(self):
self.pipeline.unload_lora_weights()
self.pipeline = None
return self.orig_pipeline
modules/ctrlx/__init__.py
+4 -6
View File
@@ -136,7 +136,7 @@ class CtrlXStableDiffusionXLPipeline(StableDiffusionXLPipeline): # diffusers==0
@torch.no_grad()
def __call__(
self,
prompt: Union[str, List[str]] = None, # TODO: Support prompt_2 and negative_prompt_2
prompt: Union[str, List[str]] = None,
structure_prompt: Optional[Union[str, List[str]]] = None,
appearance_prompt: Optional[Union[str, List[str]]] = None,
structure_image: Optional[PipelineImageInput] = None,
@@ -180,7 +180,6 @@ class CtrlXStableDiffusionXLPipeline(StableDiffusionXLPipeline): # diffusers==0
callback_on_step_end_tensor_inputs: List[str] = ["latents"],
**kwargs,
):
# TODO: Add function argument documentation
callback = kwargs.pop("callback", None)
callback_steps = kwargs.pop("callback_steps", None)
@@ -205,7 +204,7 @@ class CtrlXStableDiffusionXLPipeline(StableDiffusionXLPipeline): # diffusers==0
target_size = target_size or (height, width)
# 1. Check inputs. Raise error if not correct
self.check_inputs( # TODO: Custom check_inputs for our method
self.check_inputs(
prompt,
None, # prompt_2
height,
@@ -425,7 +424,7 @@ class CtrlXStableDiffusionXLPipeline(StableDiffusionXLPipeline): # diffusers==0
# 7.2 Optionally get guidance scale embedding
timestep_cond = None
if self.unet.config.time_cond_proj_dim is not None: # TODO: Make guidance scale embedding work with batch_order
if self.unet.config.time_cond_proj_dim is not None:
guidance_scale_tensor = torch.tensor(self.guidance_scale - 1).repeat(batch_size * num_images_per_prompt)
timestep_cond = self.get_guidance_scale_embedding(
guidance_scale_tensor, embedding_dim=self.unet.config.time_cond_proj_dim
@@ -457,7 +456,6 @@ class CtrlXStableDiffusionXLPipeline(StableDiffusionXLPipeline): # diffusers==0
register_attr(self, t=t.item(), do_control=True, batch_order=batch_order)
# TODO: For now, assume we are doing classifier-free guidance, support no CF-guidance later
latent_model_input = self.scheduler.scale_model_input(latents, t)
structure_latent_model_input = self.scheduler.scale_model_input(structure_latents, t)
appearance_latent_model_input = self.scheduler.scale_model_input(appearance_latents, t)
@@ -563,7 +561,7 @@ class CtrlXStableDiffusionXLPipeline(StableDiffusionXLPipeline): # diffusers==0
# Self-recurrence
for _ in range(self_recurrence_schedule[i]):
if hasattr(self.scheduler, "_step_index"): # For fancier schedulers
self.scheduler._step_index -= 1 # TODO: Does this actually work?
self.scheduler._step_index -= 1
t_prev = 0 if i + 1 >= num_inference_steps else timesteps[i + 1]
latents = noise_t2t(self.scheduler, t_prev, t, latents)
modules/devices.py
+41 -26
View File
@@ -186,53 +186,68 @@ def get_device_for(task): # pylint: disable=unused-argument
return get_optimal_device()
def torch_gc(force=False, fast=False):
def torch_gc(force:bool=False, fast:bool=False, reason:str=None):
def get_stats():
mem_dict = memstats.memory_stats()
gpu_dict = mem_dict.get('gpu', {})
ram_dict = mem_dict.get('ram', {})
oom = gpu_dict.get('oom', 0)
ram = ram_dict.get('used', 0)
if backend == "directml":
gpu = torch.cuda.memory_allocated() / (1 << 30)
else:
gpu = gpu_dict.get('used', 0)
used_gpu = round(100 * gpu / gpu_dict.get('total', 1)) if gpu_dict.get('total', 1) > 1 else 0
used_ram = round(100 * ram / ram_dict.get('total', 1)) if ram_dict.get('total', 1) > 1 else 0
return gpu, used_gpu, ram, used_ram, oom
global previous_oom # pylint: disable=global-statement
import gc
from modules import timer, memstats
from modules.shared import cmd_opts
t0 = time.time()
mem = memstats.memory_stats()
gpu = mem.get('gpu', {})
ram = mem.get('ram', {})
oom = gpu.get('oom', 0)
if backend == "directml":
used_gpu = round(100 * torch.cuda.memory_allocated() / (1 << 30) / gpu.get('total', 1)) if gpu.get('total', 1) > 1 else 0
else:
used_gpu = round(100 * gpu.get('used', 0) / gpu.get('total', 1)) if gpu.get('total', 1) > 1 else 0
used_ram = round(100 * ram.get('used', 0) / ram.get('total', 1)) if ram.get('total', 1) > 1 else 0
global previous_oom # pylint: disable=global-statement
gpu, used_gpu, ram, _used_ram, oom = get_stats()
threshold = 0 if (cmd_opts.lowvram and not cmd_opts.use_zluda) else opts.torch_gc_threshold
collected = 0
if force or threshold == 0 or used_gpu >= threshold or used_ram >= threshold:
if reason is None and force:
reason='force'
if threshold == 0 or used_gpu >= threshold:
force = True
if reason is None:
reason = 'threshold'
if oom > previous_oom:
previous_oom = oom
log.warning(f'Torch GPU out-of-memory error: {mem}')
log.warning(f'Torch GPU out-of-memory error: {memstats.memory_stats()}')
force = True
if reason is None:
reason = 'oom'
if force:
# actual gc
collected = gc.collect() if not fast else 0 # python gc
if cuda_ok:
try:
with torch.cuda.device(get_cuda_device_string()):
torch.cuda.synchronize()
torch.cuda.empty_cache() # cuda gc
torch.cuda.ipc_collect()
except Exception:
pass
else:
return gpu, ram
t1 = time.time()
if 'gc' not in timer.process.records:
timer.process.records['gc'] = 0
timer.process.records['gc'] += t1 - t0
if not force or collected == 0:
return
mem = memstats.memory_stats()
saved = round(gpu.get('used', 0) - mem.get('gpu', {}).get('used', 0), 2)
before = { 'gpu': gpu.get('used', 0), 'ram': ram.get('used', 0) }
after = { 'gpu': mem.get('gpu', {}).get('used', 0), 'ram': mem.get('ram', {}).get('used', 0), 'retries': mem.get('retries', 0), 'oom': mem.get('oom', 0) }
utilization = { 'gpu': used_gpu, 'ram': used_ram, 'threshold': threshold }
results = { 'collected': collected, 'saved': saved }
timer.process.add('gc', t1 - t0)
if fast:
return gpu, ram
new_gpu, new_used_gpu, new_ram, new_used_ram, oom = get_stats()
before = { 'gpu': gpu, 'ram': ram }
after = { 'gpu': new_gpu, 'ram': new_ram, 'oom': oom }
utilization = { 'gpu': new_used_gpu, 'ram': new_used_ram }
results = { 'gpu': round(gpu - new_gpu, 2), 'py': collected }
fn = f'{sys._getframe(2).f_code.co_name}:{sys._getframe(1).f_code.co_name}' # pylint: disable=protected-access
log.debug(f'GC: utilization={utilization} gc={results} before={before} after={after} device={torch.device(get_optimal_device_name())} fn={fn} time={round(t1 - t0, 2)}') # pylint: disable=protected-access
log.debug(f'GC: current={after} prev={before} load={utilization} gc={results} fn={fn} why={reason} time={t1-t0:.2f}')
return new_gpu, new_ram
def set_cuda_sync_mode(mode):
@@ -471,7 +486,7 @@ def set_cuda_params():
device_name = get_raw_openvino_device()
else:
device_name = torch.device(get_optimal_device_name())
log.info(f'Torch parameters: backend={backend} device={device_name} config={opts.cuda_dtype} dtype={dtype} vae={dtype_vae} unet={dtype_unet} context={inference_context.__name__} nohalf={opts.no_half} nohalfvae={opts.no_half_vae} upscast={opts.upcast_sampling} deterministic={opts.cudnn_deterministic} test-fp16={fp16_ok} test-bf16={bf16_ok} optimization="{opts.cross_attention_optimization}"')
log.info(f'Torch parameters: backend={backend} device={device_name} config={opts.cuda_dtype} dtype={dtype} vae={dtype_vae} unet={dtype_unet} context={inference_context.__name__} nohalf={opts.no_half} nohalfvae={opts.no_half_vae} upcast={opts.upcast_sampling} deterministic={opts.cudnn_deterministic} test-fp16={fp16_ok} test-bf16={bf16_ok} optimization="{opts.cross_attention_optimization}"')
def cond_cast_unet(tensor):
modules/devices_mac.py
+2 -2
View File
@@ -24,7 +24,7 @@ def cumsum_fix(input, cumsum_func, *args, **kwargs): # pylint: disable=redefined
output_dtype = kwargs.get('dtype', input.dtype)
if output_dtype == torch.int64:
return cumsum_func(input.cpu(), *args, **kwargs).to(input.device)
elif output_dtype == torch.bool or cumsum_needs_int_fix and (output_dtype == torch.int8 or output_dtype == torch.int16):
elif output_dtype == torch.bool or (cumsum_needs_int_fix and (output_dtype == torch.int8 or output_dtype == torch.int16)):
return cumsum_func(input.to(torch.int32), *args, **kwargs).to(torch.int64)
return cumsum_func(input, *args, **kwargs)
@@ -42,7 +42,7 @@ if has_mps:
# MPS workaround for https://github.com/pytorch/pytorch/issues/79383
CondFunc('torch.Tensor.to', lambda orig_func, self, *args, **kwargs: orig_func(self.contiguous(), *args, **kwargs),
lambda _, self, *args, **kwargs: self.device.type != 'mps' and (args and isinstance(args[0], torch.device) and args[0].type == 'mps' or isinstance(kwargs.get('device'), torch.device) and kwargs['device'].type == 'mps'))
lambda _, self, *args, **kwargs: self.device.type != 'mps' and ((args and isinstance(args[0], torch.device) and args[0].type == 'mps') or (isinstance(kwargs.get('device'), torch.device) and kwargs['device'].type == 'mps')))
# MPS workaround for https://github.com/pytorch/pytorch/issues/80800
CondFunc('torch.nn.functional.layer_norm', lambda orig_func, *args, **kwargs: orig_func(*([args[0].contiguous()] + list(args[1:])), **kwargs),
lambda _, *args, **kwargs: args and isinstance(args[0], torch.Tensor) and args[0].device.type == 'mps')
modules/errors.py
+5 -4
View File
@@ -36,7 +36,7 @@ def print_error_explanation(message):
log.error(line)
def display(e: Exception, task, suppress=[]):
def display(e: Exception, task: str, suppress=[]):
log.error(f"{task or 'error'}: {type(e).__name__}")
console.print_exception(show_locals=False, max_frames=16, extra_lines=1, suppress=suppress, theme="ansi_dark", word_wrap=False, width=console.width)
@@ -48,7 +48,7 @@ def display_once(e: Exception, task):
already_displayed[task] = 1
def run(code, task):
def run(code, task: str):
try:
code()
except Exception as e:
@@ -59,14 +59,14 @@ def exception(suppress=[]):
console.print_exception(show_locals=False, max_frames=16, extra_lines=2, suppress=suppress, theme="ansi_dark", word_wrap=False, width=min([console.width, 200]))
def profile(profiler, msg: str, n: int = 5):
def profile(profiler, msg: str, n: int = 16):
profiler.disable()
import io
import pstats
stream = io.StringIO() # pylint: disable=abstract-class-instantiated
p = pstats.Stats(profiler, stream=stream)
p.sort_stats(pstats.SortKey.CUMULATIVE)
p.print_stats(100)
p.print_stats(200)
# p.print_title()
# p.print_call_heading(10, 'time')
# p.print_callees(10)
@@ -81,6 +81,7 @@ def profile(profiler, msg: str, n: int = 5):
and '_lsprof' not in x
and '/profiler' not in x
and 'rich' not in x
and 'profile_torch' not in x
and x.strip() != ''
]
txt = '\n'.join(lines[:min(n, len(lines))])
modules/extensions.py
+1 -1
View File
@@ -154,4 +154,4 @@ def list_extensions():
for dirname, path, is_builtin in extension_paths:
extension = Extension(name=dirname, path=path, enabled=dirname not in disabled_extensions, is_builtin=is_builtin)
extensions.append(extension)
shared.log.info(f'Disabled extensions: {[e.name for e in extensions if not e.enabled]}')
shared.log.debug(f'Disabled extensions: {[e.name for e in extensions if not e.enabled]}')
modules/extra_networks.py
+40 -25
View File
@@ -1,6 +1,7 @@
import re
import inspect
from collections import defaultdict
from modules import errors, shared, devices
from modules import errors, shared
extra_network_registry = {}
@@ -15,10 +16,14 @@ def register_extra_network(extra_network):
def register_default_extra_networks():
from modules.ui_extra_networks_hypernet import ExtraNetworkHypernet
register_extra_network(ExtraNetworkHypernet())
from modules.ui_extra_networks_styles import ExtraNetworkStyles
register_extra_network(ExtraNetworkStyles())
if shared.native:
from modules.lora.networks import extra_network_lora
register_extra_network(extra_network_lora)
if shared.opts.hypernetwork_enabled:
from modules.ui_extra_networks_hypernet import ExtraNetworkHypernet
register_extra_network(ExtraNetworkHypernet())
class ExtraNetworkParams:
@@ -70,9 +75,12 @@ def is_stepwise(en_obj):
return any([len(str(x).split("@")) > 1 for x in all_args]) # noqa C419 # pylint: disable=use-a-generator
def activate(p, extra_network_data, step=0):
def activate(p, extra_network_data=None, step=0, include=[], exclude=[]):
"""call activate for extra networks in extra_network_data in specified order, then call activate for all remaining registered networks with an empty argument list"""
if extra_network_data is None:
if p.disable_extra_networks:
return
extra_network_data = extra_network_data or p.network_data
if extra_network_data is None or len(extra_network_data) == 0:
return
stepwise = False
for extra_network_args in extra_network_data.values():
@@ -82,35 +90,42 @@ def activate(p, extra_network_data, step=0):
shared.log.warning("Composable LoRA not compatible with 'lora_force_diffusers'")
stepwise = False
shared.opts.data['lora_functional'] = stepwise or functional
with devices.autocast():
for extra_network_name, extra_network_args in extra_network_data.items():
extra_network = extra_network_registry.get(extra_network_name, None)
if extra_network is None:
errors.log.warning(f"Skipping unknown extra network: {extra_network_name}")
continue
try:
for extra_network_name, extra_network_args in extra_network_data.items():
extra_network = extra_network_registry.get(extra_network_name, None)
if extra_network is None:
errors.log.warning(f"Skipping unknown extra network: {extra_network_name}")
continue
try:
signature = list(inspect.signature(extra_network.activate).parameters)
if 'include' in signature and 'exclude' in signature:
extra_network.activate(p, extra_network_args, step=step, include=include, exclude=exclude)
else:
extra_network.activate(p, extra_network_args, step=step)
except Exception as e:
errors.display(e, f"Activating network: type={extra_network_name} args:{extra_network_args}")
except Exception as e:
errors.display(e, f"Activating network: type={extra_network_name} args:{extra_network_args}")
for extra_network_name, extra_network in extra_network_registry.items():
args = extra_network_data.get(extra_network_name, None)
if args is not None:
continue
try:
extra_network.activate(p, [])
except Exception as e:
errors.display(e, f"Activating network: type={extra_network_name}")
for extra_network_name, extra_network in extra_network_registry.items():
args = extra_network_data.get(extra_network_name, None)
if args is not None:
continue
try:
extra_network.activate(p, [])
except Exception as e:
errors.display(e, f"Activating network: type={extra_network_name}")
p.extra_network_data = extra_network_data
p.network_data = extra_network_data
if stepwise:
p.stepwise_lora = True
shared.opts.data['lora_functional'] = functional
def deactivate(p, extra_network_data):
def deactivate(p, extra_network_data=None):
"""call deactivate for extra networks in extra_network_data in specified order, then call deactivate for all remaining registered networks"""
if extra_network_data is None:
if p.disable_extra_networks:
return
extra_network_data = extra_network_data or p.network_data
if extra_network_data is None or len(extra_network_data) == 0:
return
for extra_network_name in extra_network_data:
extra_network = extra_network_registry.get(extra_network_name, None)
modules/face/faceid.py
+3 -7
View File
@@ -204,7 +204,6 @@ def face_id(
ip_model_dict["face_image"] = face_images
ip_model_dict["faceid_embeds"] = face_embeds # overwrite placeholder
faceid_model.set_scale(scale)
extra_network_data = None
if p.all_prompts is None or len(p.all_prompts) == 0:
processing.process_init(p)
@@ -215,11 +214,9 @@ def face_id(
p.negative_prompts = p.all_negative_prompts[n * p.batch_size:(n+1) * p.batch_size]
p.seeds = p.all_seeds[n * p.batch_size:(n+1) * p.batch_size]
p.subseeds = p.all_subseeds[n * p.batch_size:(n+1) * p.batch_size]
p.prompts, extra_network_data = extra_networks.parse_prompts(p.prompts)
p.prompts, p.network_data = extra_networks.parse_prompts(p.prompts)
if not p.disable_extra_networks:
with devices.autocast():
extra_networks.activate(p, extra_network_data)
extra_networks.activate(p, p.network_data)
ip_model_dict.update({
"prompt": p.prompts[0],
"negative_prompt": p.negative_prompts[0],
@@ -239,8 +236,7 @@ def face_id(
devices.torch_gc()
ipadapter.unapply(p.sd_model)
if not p.disable_extra_networks:
extra_networks.deactivate(p, extra_network_data)
extra_networks.deactivate(p, p.network_data)
p.extra_generation_params["IP Adapter"] = f"{basename}:{scale}"
finally:
modules/face/instantid_model.py
-1
View File
@@ -344,7 +344,6 @@ class IPAttnProcessor(nn.Module):
return hidden_states
def _memory_efficient_attention_xformers(self, query, key, value, attention_mask):
# TODO attention_mask
query = query.contiguous()
key = key.contiguous()
value = value.contiguous()
modules/face/photomaker_model.py
+5 -5
View File
@@ -244,7 +244,7 @@ class PhotoMakerStableDiffusionXLPipeline(StableDiffusionXLPipeline):
prompt_embeds_list = []
prompts = [prompt, prompt_2]
for prompt, tokenizer, text_encoder in zip(prompts, tokenizers, text_encoders):
input_ids = tokenizer.encode(prompt) # TODO: batch encode
input_ids = tokenizer.encode(prompt)
clean_index = 0
clean_input_ids = []
class_token_index = []
@@ -296,7 +296,7 @@ class PhotoMakerStableDiffusionXLPipeline(StableDiffusionXLPipeline):
prompt_embeds = torch.concat(prompt_embeds_list, dim=-1)
prompt_embeds = prompt_embeds.to(dtype=self.text_encoder_2.dtype, device=device)
class_tokens_mask = class_tokens_mask.to(device=device) # TODO: ignoring two-prompt case
class_tokens_mask = class_tokens_mask.to(device=device)
return prompt_embeds, pooled_prompt_embeds, class_tokens_mask
@@ -332,7 +332,7 @@ class PhotoMakerStableDiffusionXLPipeline(StableDiffusionXLPipeline):
callback_steps: int = 1,
# Added parameters (for PhotoMaker)
input_id_images: PipelineImageInput = None,
start_merge_step: int = 0, # TODO: change to `style_strength_ratio` in the future
start_merge_step: int = 0,
class_tokens_mask: Optional[torch.LongTensor] = None,
prompt_embeds_text_only: Optional[torch.FloatTensor] = None,
pooled_prompt_embeds_text_only: Optional[torch.FloatTensor] = None,
@@ -410,7 +410,7 @@ class PhotoMakerStableDiffusionXLPipeline(StableDiffusionXLPipeline):
(
prompt_embeds_text_only,
negative_prompt_embeds,
pooled_prompt_embeds_text_only, # TODO: replace the pooled_prompt_embeds with text only prompt
pooled_prompt_embeds_text_only,
negative_pooled_prompt_embeds,
) = self.encode_prompt(
prompt=prompt_text_only,
@@ -431,7 +431,7 @@ class PhotoMakerStableDiffusionXLPipeline(StableDiffusionXLPipeline):
if not isinstance(input_id_images[0], torch.Tensor):
id_pixel_values = self.id_image_processor(input_id_images, return_tensors="pt").pixel_values
id_pixel_values = id_pixel_values.unsqueeze(0).to(device=device, dtype=dtype) # TODO: multiple prompts
id_pixel_values = id_pixel_values.unsqueeze(0).to(device=device, dtype=dtype)
# 6. Get the update text embedding with the stacked ID embedding
prompt_embeds = self.id_encoder(id_pixel_values, prompt_embeds, class_tokens_mask)
modules/files_cache.py
+4 -4
View File
@@ -6,6 +6,7 @@ from typing import Callable, Dict, Iterator, List, Optional, Union
from installer import log
do_cache_folders = os.environ.get('SD_NO_CACHE', None) is None
class Directory: # forward declaration
...
@@ -87,8 +88,6 @@ class Directory(Directory): # pylint: disable=E0102
return not self.is_directory or self.mtime != self.live_mtime
class DirectoryCache(UserDict, DirectoryCollection):
def __delattr__(self, directory_path: str) -> None:
directory: Directory = get_directory(directory_path, fetch=False)
@@ -126,7 +125,7 @@ def clean_directory(directory: Directory, /, recursive: RecursiveType=False) ->
return is_clean
def get_directory(directory_or_path: str, /, fetch:bool=True) -> Union[Directory, None]:
def get_directory(directory_or_path: str, /, fetch: bool=True) -> Union[Directory, None]:
if isinstance(directory_or_path, Directory):
if directory_or_path.is_directory:
return directory_or_path
@@ -136,8 +135,9 @@ def get_directory(directory_or_path: str, /, fetch:bool=True) -> Union[Directory
if not cache_folders.get(directory_or_path, None):
if fetch:
directory = fetch_directory(directory_path=directory_or_path)
if directory:
if directory and do_cache_folders:
cache_folders[directory_or_path] = directory
return directory
else:
clean_directory(cache_folders[directory_or_path])
return cache_folders[directory_or_path] if directory_or_path in cache_folders else None
modules/freescale/__init__.py
+4
View File
@@ -0,0 +1,4 @@
# Credits: https://github.com/ali-vilab/FreeScale
from .freescale_pipeline import StableDiffusionXLFreeScale
from .freescale_pipeline_img2img import StableDiffusionXLFreeScaleImg2Img
modules/freescale/free_lunch_utils.py
+305
View File
@@ -0,0 +1,305 @@
from typing import Any, Dict, Optional, Tuple
import torch
import torch.fft as fft
from diffusers.utils import is_torch_version
""" Borrowed from https://github.com/ChenyangSi/FreeU/blob/main/demo/free_lunch_utils.py
"""
def isinstance_str(x: object, cls_name: str):
"""
Checks whether x has any class *named* cls_name in its ancestry.
Doesn't require access to the class's implementation.
Useful for patching!
"""
for _cls in x.__class__.__mro__:
if _cls.__name__ == cls_name:
return True
return False
def Fourier_filter(x, threshold, scale):
dtype = x.dtype
x = x.type(torch.float32)
# FFT
x_freq = fft.fftn(x, dim=(-2, -1))
x_freq = fft.fftshift(x_freq, dim=(-2, -1))
B, C, H, W = x_freq.shape
mask = torch.ones((B, C, H, W)).cuda()
crow, ccol = H // 2, W //2
mask[..., crow - threshold:crow + threshold, ccol - threshold:ccol + threshold] = scale
x_freq = x_freq * mask
# IFFT
x_freq = fft.ifftshift(x_freq, dim=(-2, -1))
x_filtered = fft.ifftn(x_freq, dim=(-2, -1)).real
x_filtered = x_filtered.type(dtype)
return x_filtered
def register_upblock2d(model):
def up_forward(self):
def forward(hidden_states, res_hidden_states_tuple, temb=None, upsample_size=None):
for resnet in self.resnets:
# pop res hidden states
res_hidden_states = res_hidden_states_tuple[-1]
res_hidden_states_tuple = res_hidden_states_tuple[:-1]
#print(f"in upblock2d, hidden states shape: {hidden_states.shape}")
hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1)
if self.training and self.gradient_checkpointing:
def create_custom_forward(module):
def custom_forward(*inputs):
return module(*inputs)
return custom_forward
if is_torch_version(">=", "1.11.0"):
hidden_states = torch.utils.checkpoint.checkpoint(
create_custom_forward(resnet), hidden_states, temb, use_reentrant=False
)
else:
hidden_states = torch.utils.checkpoint.checkpoint(
create_custom_forward(resnet), hidden_states, temb
)
else:
hidden_states = resnet(hidden_states, temb)
if self.upsamplers is not None:
for upsampler in self.upsamplers:
hidden_states = upsampler(hidden_states, upsample_size)
return hidden_states
return forward
for i, upsample_block in enumerate(model.unet.up_blocks):
if isinstance_str(upsample_block, "UpBlock2D"):
upsample_block.forward = up_forward(upsample_block)
def register_free_upblock2d(model, b1=1.2, b2=1.4, s1=0.9, s2=0.2):
def up_forward(self):
def forward(hidden_states, res_hidden_states_tuple, temb=None, upsample_size=None):
for resnet in self.resnets:
# pop res hidden states
res_hidden_states = res_hidden_states_tuple[-1]
res_hidden_states_tuple = res_hidden_states_tuple[:-1]
#print(f"in free upblock2d, hidden states shape: {hidden_states.shape}")
# --------------- FreeU code -----------------------
# Only operate on the first two stages
if hidden_states.shape[1] == 1280:
hidden_states[:,:640] = hidden_states[:,:640] * self.b1
res_hidden_states = Fourier_filter(res_hidden_states, threshold=1, scale=self.s1)
if hidden_states.shape[1] == 640:
hidden_states[:,:320] = hidden_states[:,:320] * self.b2
res_hidden_states = Fourier_filter(res_hidden_states, threshold=1, scale=self.s2)
# ---------------------------------------------------------
hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1)
if self.training and self.gradient_checkpointing:
def create_custom_forward(module):
def custom_forward(*inputs):
return module(*inputs)
return custom_forward
if is_torch_version(">=", "1.11.0"):
hidden_states = torch.utils.checkpoint.checkpoint(
create_custom_forward(resnet), hidden_states, temb, use_reentrant=False
)
else:
hidden_states = torch.utils.checkpoint.checkpoint(
create_custom_forward(resnet), hidden_states, temb
)
else:
hidden_states = resnet(hidden_states, temb)
if self.upsamplers is not None:
for upsampler in self.upsamplers:
hidden_states = upsampler(hidden_states, upsample_size)
return hidden_states
return forward
for i, upsample_block in enumerate(model.unet.up_blocks):
if isinstance_str(upsample_block, "UpBlock2D"):
upsample_block.forward = up_forward(upsample_block)
setattr(upsample_block, 'b1', b1)
setattr(upsample_block, 'b2', b2)
setattr(upsample_block, 's1', s1)
setattr(upsample_block, 's2', s2)
def register_crossattn_upblock2d(model):
def up_forward(self):
def forward(
hidden_states: torch.FloatTensor,
res_hidden_states_tuple: Tuple[torch.FloatTensor, ...],
temb: Optional[torch.FloatTensor] = None,
encoder_hidden_states: Optional[torch.FloatTensor] = None,
cross_attention_kwargs: Optional[Dict[str, Any]] = None,
upsample_size: Optional[int] = None,
attention_mask: Optional[torch.FloatTensor] = None,
encoder_attention_mask: Optional[torch.FloatTensor] = None,
):
for resnet, attn in zip(self.resnets, self.attentions):
# pop res hidden states
#print(f"in crossatten upblock2d, hidden states shape: {hidden_states.shape}")
res_hidden_states = res_hidden_states_tuple[-1]
res_hidden_states_tuple = res_hidden_states_tuple[:-1]
hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1)
if self.training and self.gradient_checkpointing:
def create_custom_forward(module, return_dict=None):
def custom_forward(*inputs):
if return_dict is not None:
return module(*inputs, return_dict=return_dict)
else:
return module(*inputs)
return custom_forward
ckpt_kwargs: Dict[str, Any] = {"use_reentrant": False} if is_torch_version(">=", "1.11.0") else {}
hidden_states = torch.utils.checkpoint.checkpoint(
create_custom_forward(resnet),
hidden_states,
temb,
**ckpt_kwargs,
)
hidden_states = torch.utils.checkpoint.checkpoint(
create_custom_forward(attn, return_dict=False),
hidden_states,
encoder_hidden_states,
None, # timestep
None, # class_labels
cross_attention_kwargs,
attention_mask,
encoder_attention_mask,
**ckpt_kwargs,
)[0]
else:
hidden_states = resnet(hidden_states, temb)
hidden_states = attn(
hidden_states,
encoder_hidden_states=encoder_hidden_states,
cross_attention_kwargs=cross_attention_kwargs,
attention_mask=attention_mask,
encoder_attention_mask=encoder_attention_mask,
return_dict=False,
)[0]
if self.upsamplers is not None:
for upsampler in self.upsamplers:
hidden_states = upsampler(hidden_states, upsample_size)
return hidden_states
return forward
for i, upsample_block in enumerate(model.unet.up_blocks):
if isinstance_str(upsample_block, "CrossAttnUpBlock2D"):
upsample_block.forward = up_forward(upsample_block)
def register_free_crossattn_upblock2d(model, b1=1.2, b2=1.4, s1=0.9, s2=0.2):
def up_forward(self):
def forward(
hidden_states: torch.FloatTensor,
res_hidden_states_tuple: Tuple[torch.FloatTensor, ...],
temb: Optional[torch.FloatTensor] = None,
encoder_hidden_states: Optional[torch.FloatTensor] = None,
cross_attention_kwargs: Optional[Dict[str, Any]] = None,
upsample_size: Optional[int] = None,
attention_mask: Optional[torch.FloatTensor] = None,
encoder_attention_mask: Optional[torch.FloatTensor] = None,
):
for resnet, attn in zip(self.resnets, self.attentions):
# pop res hidden states
#print(f"in free crossatten upblock2d, hidden states shape: {hidden_states.shape}")
res_hidden_states = res_hidden_states_tuple[-1]
res_hidden_states_tuple = res_hidden_states_tuple[:-1]
# --------------- FreeU code -----------------------
# Only operate on the first two stages
if hidden_states.shape[1] == 1280:
hidden_states[:,:640] = hidden_states[:,:640] * self.b1
res_hidden_states = Fourier_filter(res_hidden_states, threshold=1, scale=self.s1)
if hidden_states.shape[1] == 640:
hidden_states[:,:320] = hidden_states[:,:320] * self.b2
res_hidden_states = Fourier_filter(res_hidden_states, threshold=1, scale=self.s2)
# ---------------------------------------------------------
hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1)
if self.training and self.gradient_checkpointing:
def create_custom_forward(module, return_dict=None):
def custom_forward(*inputs):
if return_dict is not None:
return module(*inputs, return_dict=return_dict)
else:
return module(*inputs)
return custom_forward
ckpt_kwargs: Dict[str, Any] = {"use_reentrant": False} if is_torch_version(">=", "1.11.0") else {}
hidden_states = torch.utils.checkpoint.checkpoint(
create_custom_forward(resnet),
hidden_states,
temb,
**ckpt_kwargs,
)
hidden_states = torch.utils.checkpoint.checkpoint(
create_custom_forward(attn, return_dict=False),
hidden_states,
encoder_hidden_states,
None, # timestep
None, # class_labels
cross_attention_kwargs,
attention_mask,
encoder_attention_mask,
**ckpt_kwargs,
)[0]
else:
hidden_states = resnet(hidden_states, temb)
# hidden_states = attn(
# hidden_states,
# encoder_hidden_states=encoder_hidden_states,
# cross_attention_kwargs=cross_attention_kwargs,
# encoder_attention_mask=encoder_attention_mask,
# return_dict=False,
# )[0]
hidden_states = attn(
hidden_states,
encoder_hidden_states=encoder_hidden_states,
cross_attention_kwargs=cross_attention_kwargs,
)[0]
if self.upsamplers is not None:
for upsampler in self.upsamplers:
hidden_states = upsampler(hidden_states, upsample_size)
return hidden_states
return forward
for i, upsample_block in enumerate(model.unet.up_blocks):
if isinstance_str(upsample_block, "CrossAttnUpBlock2D"):
upsample_block.forward = up_forward(upsample_block)
setattr(upsample_block, 'b1', b1)
setattr(upsample_block, 'b2', b2)
setattr(upsample_block, 's1', s1)
setattr(upsample_block, 's2', s2)
modules/freescale/freescale_pipeline.py
+1189
View File
File diff suppressed because it is too large Load Diff
modules/freescale/freescale_pipeline_img2img.py
+1245
View File
File diff suppressed because it is too large Load Diff
modules/freescale/scale_attention.py
+367
View File
@@ -0,0 +1,367 @@
from typing import Any, Dict, Optional
import random
import torch
import torch.nn.functional as F
from einops import rearrange
def gaussian_kernel(kernel_size=3, sigma=1.0, channels=3):
x_coord = torch.arange(kernel_size)
gaussian_1d = torch.exp(-(x_coord - (kernel_size - 1) / 2) ** 2 / (2 * sigma ** 2))
gaussian_1d = gaussian_1d / gaussian_1d.sum()
gaussian_2d = gaussian_1d[:, None] * gaussian_1d[None, :]
kernel = gaussian_2d[None, None, :, :].repeat(channels, 1, 1, 1)
return kernel
def gaussian_filter(latents, kernel_size=3, sigma=1.0):
channels = latents.shape[1]
kernel = gaussian_kernel(kernel_size, sigma, channels).to(latents.device, latents.dtype)
blurred_latents = F.conv2d(latents, kernel, padding=kernel_size//2, groups=channels)
return blurred_latents
def get_views(height, width, h_window_size=128, w_window_size=128, scale_factor=8):
height = int(height)
width = int(width)
h_window_stride = h_window_size // 2
w_window_stride = w_window_size // 2
h_window_size = int(h_window_size / scale_factor)
w_window_size = int(w_window_size / scale_factor)
h_window_stride = int(h_window_stride / scale_factor)
w_window_stride = int(w_window_stride / scale_factor)
num_blocks_height = int((height - h_window_size) / h_window_stride - 1e-6) + 2 if height > h_window_size else 1
num_blocks_width = int((width - w_window_size) / w_window_stride - 1e-6) + 2 if width > w_window_size else 1
total_num_blocks = int(num_blocks_height * num_blocks_width)
views = []
for i in range(total_num_blocks):
h_start = int((i // num_blocks_width) * h_window_stride)
h_end = h_start + h_window_size
w_start = int((i % num_blocks_width) * w_window_stride)
w_end = w_start + w_window_size
if h_end > height:
h_start = int(h_start + height - h_end)
h_end = int(height)
if w_end > width:
w_start = int(w_start + width - w_end)
w_end = int(width)
if h_start < 0:
h_end = int(h_end - h_start)
h_start = 0
if w_start < 0:
w_end = int(w_end - w_start)
w_start = 0
random_jitter = True
if random_jitter:
h_jitter_range = h_window_size // 8
w_jitter_range = w_window_size // 8
h_jitter = 0
w_jitter = 0
if (w_start != 0) and (w_end != width):
w_jitter = random.randint(-w_jitter_range, w_jitter_range)
elif (w_start == 0) and (w_end != width):
w_jitter = random.randint(-w_jitter_range, 0)
elif (w_start != 0) and (w_end == width):
w_jitter = random.randint(0, w_jitter_range)
if (h_start != 0) and (h_end != height):
h_jitter = random.randint(-h_jitter_range, h_jitter_range)
elif (h_start == 0) and (h_end != height):
h_jitter = random.randint(-h_jitter_range, 0)
elif (h_start != 0) and (h_end == height):
h_jitter = random.randint(0, h_jitter_range)
h_start += (h_jitter + h_jitter_range)
h_end += (h_jitter + h_jitter_range)
w_start += (w_jitter + w_jitter_range)
w_end += (w_jitter + w_jitter_range)
views.append((h_start, h_end, w_start, w_end))
return views
def scale_forward(
self,
hidden_states: torch.FloatTensor,
attention_mask: Optional[torch.FloatTensor] = None,
encoder_hidden_states: Optional[torch.FloatTensor] = None,
encoder_attention_mask: Optional[torch.FloatTensor] = None,
timestep: Optional[torch.LongTensor] = None,
cross_attention_kwargs: Dict[str, Any] = None,
class_labels: Optional[torch.LongTensor] = None,
):
# Notice that normalization is always applied before the real computation in the following blocks.
if self.current_hw:
current_scale_num_h, current_scale_num_w = max(self.current_hw[0] // 1024, 1), max(self.current_hw[1] // 1024, 1)
else:
current_scale_num_h, current_scale_num_w = 1, 1
# 0. Self-Attention
if self.use_ada_layer_norm:
norm_hidden_states = self.norm1(hidden_states, timestep)
elif self.use_ada_layer_norm_zero:
norm_hidden_states, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.norm1(
hidden_states, timestep, class_labels, hidden_dtype=hidden_states.dtype
)
else:
norm_hidden_states = self.norm1(hidden_states)
# 2. Prepare GLIGEN inputs
cross_attention_kwargs = cross_attention_kwargs.copy() if cross_attention_kwargs is not None else {}
gligen_kwargs = cross_attention_kwargs.pop("gligen", None)
ratio_hw = current_scale_num_h / current_scale_num_w
latent_h = int((norm_hidden_states.shape[1] * ratio_hw) ** 0.5)
latent_w = int(latent_h / ratio_hw)
scale_factor = 128 * current_scale_num_h / latent_h
if ratio_hw > 1:
sub_h = 128
sub_w = int(128 / ratio_hw)
else:
sub_h = int(128 * ratio_hw)
sub_w = 128
h_jitter_range = int(sub_h / scale_factor // 8)
w_jitter_range = int(sub_w / scale_factor // 8)
views = get_views(latent_h, latent_w, sub_h, sub_w, scale_factor = scale_factor)
current_scale_num = max(current_scale_num_h, current_scale_num_w)
global_views = [[h, w] for h in range(current_scale_num_h) for w in range(current_scale_num_w)]
four_window = True
fourg_window = False
if four_window:
norm_hidden_states_ = rearrange(norm_hidden_states, 'bh (h w) d -> bh h w d', h = latent_h)
norm_hidden_states_ = F.pad(norm_hidden_states_, (0, 0, w_jitter_range, w_jitter_range, h_jitter_range, h_jitter_range), 'constant', 0)
value = torch.zeros_like(norm_hidden_states_)
count = torch.zeros_like(norm_hidden_states_)
for index, view in enumerate(views):
h_start, h_end, w_start, w_end = view
local_states = norm_hidden_states_[:, h_start:h_end, w_start:w_end, :]
local_states = rearrange(local_states, 'bh h w d -> bh (h w) d')
local_output = self.attn1(
local_states,
encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,
attention_mask=attention_mask,
**cross_attention_kwargs,
)
local_output = rearrange(local_output, 'bh (h w) d -> bh h w d', h = int(sub_h / scale_factor))
value[:, h_start:h_end, w_start:w_end, :] += local_output * 1
count[:, h_start:h_end, w_start:w_end, :] += 1
value = value[:, h_jitter_range:-h_jitter_range, w_jitter_range:-w_jitter_range, :]
count = count[:, h_jitter_range:-h_jitter_range, w_jitter_range:-w_jitter_range, :]
attn_output = torch.where(count>0, value/count, value)
gaussian_local = gaussian_filter(attn_output, kernel_size=(2*current_scale_num-1), sigma=1.0)
attn_output_global = self.attn1(
norm_hidden_states,
encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,
attention_mask=attention_mask,
**cross_attention_kwargs,
)
attn_output_global = rearrange(attn_output_global, 'bh (h w) d -> bh h w d', h = latent_h)
gaussian_global = gaussian_filter(attn_output_global, kernel_size=(2*current_scale_num-1), sigma=1.0)
attn_output = gaussian_local + (attn_output_global - gaussian_global)
attn_output = rearrange(attn_output, 'bh h w d -> bh (h w) d')
elif fourg_window:
norm_hidden_states = rearrange(norm_hidden_states, 'bh (h w) d -> bh h w d', h = latent_h)
norm_hidden_states_ = F.pad(norm_hidden_states, (0, 0, w_jitter_range, w_jitter_range, h_jitter_range, h_jitter_range), 'constant', 0)
value = torch.zeros_like(norm_hidden_states_)
count = torch.zeros_like(norm_hidden_states_)
for index, view in enumerate(views):
h_start, h_end, w_start, w_end = view
local_states = norm_hidden_states_[:, h_start:h_end, w_start:w_end, :]
local_states = rearrange(local_states, 'bh h w d -> bh (h w) d')
local_output = self.attn1(
local_states,
encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,
attention_mask=attention_mask,
**cross_attention_kwargs,
)
local_output = rearrange(local_output, 'bh (h w) d -> bh h w d', h = int(sub_h / scale_factor))
value[:, h_start:h_end, w_start:w_end, :] += local_output * 1
count[:, h_start:h_end, w_start:w_end, :] += 1
value = value[:, h_jitter_range:-h_jitter_range, w_jitter_range:-w_jitter_range, :]
count = count[:, h_jitter_range:-h_jitter_range, w_jitter_range:-w_jitter_range, :]
attn_output = torch.where(count>0, value/count, value)
gaussian_local = gaussian_filter(attn_output, kernel_size=(2*current_scale_num-1), sigma=1.0)
value = torch.zeros_like(norm_hidden_states)
count = torch.zeros_like(norm_hidden_states)
for index, global_view in enumerate(global_views):
h, w = global_view
global_states = norm_hidden_states[:, h::current_scale_num_h, w::current_scale_num_w, :]
global_states = rearrange(global_states, 'bh h w d -> bh (h w) d')
global_output = self.attn1(
global_states,
encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,
attention_mask=attention_mask,
**cross_attention_kwargs,
)
global_output = rearrange(global_output, 'bh (h w) d -> bh h w d', h = int(global_output.shape[1] ** 0.5))
value[:, h::current_scale_num_h, w::current_scale_num_w, :] += global_output * 1
count[:, h::current_scale_num_h, w::current_scale_num_w, :] += 1
attn_output_global = torch.where(count>0, value/count, value)
gaussian_global = gaussian_filter(attn_output_global, kernel_size=(2*current_scale_num-1), sigma=1.0)
attn_output = gaussian_local + (attn_output_global - gaussian_global)
attn_output = rearrange(attn_output, 'bh h w d -> bh (h w) d')
else:
attn_output = self.attn1(
norm_hidden_states,
encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,
attention_mask=attention_mask,
**cross_attention_kwargs,
)
if self.use_ada_layer_norm_zero:
attn_output = gate_msa.unsqueeze(1) * attn_output
hidden_states = attn_output + hidden_states
# 2.5 GLIGEN Control
if gligen_kwargs is not None:
hidden_states = self.fuser(hidden_states, gligen_kwargs["objs"])
# 2.5 ends
# 3. Cross-Attention
if self.attn2 is not None:
norm_hidden_states = (
self.norm2(hidden_states, timestep) if self.use_ada_layer_norm else self.norm2(hidden_states)
)
attn_output = self.attn2(
norm_hidden_states,
encoder_hidden_states=encoder_hidden_states,
attention_mask=encoder_attention_mask,
**cross_attention_kwargs,
)
hidden_states = attn_output + hidden_states
# 4. Feed-forward
norm_hidden_states = self.norm3(hidden_states)
if self.use_ada_layer_norm_zero:
norm_hidden_states = norm_hidden_states * (1 + scale_mlp[:, None]) + shift_mlp[:, None]
if self._chunk_size is not None:
# "feed_forward_chunk_size" can be used to save memory
if norm_hidden_states.shape[self._chunk_dim] % self._chunk_size != 0:
raise ValueError(
f"`hidden_states` dimension to be chunked: {norm_hidden_states.shape[self._chunk_dim]} has to be divisible by chunk size: {self._chunk_size}. Make sure to set an appropriate `chunk_size` when calling `unet.enable_forward_chunking`."
)
num_chunks = norm_hidden_states.shape[self._chunk_dim] // self._chunk_size
ff_output = torch.cat(
[
self.ff(hid_slice)
for hid_slice in norm_hidden_states.chunk(num_chunks, dim=self._chunk_dim)
],
dim=self._chunk_dim,
)
else:
ff_output = self.ff(norm_hidden_states)
if self.use_ada_layer_norm_zero:
ff_output = gate_mlp.unsqueeze(1) * ff_output
hidden_states = ff_output + hidden_states
return hidden_states
def ori_forward(
self,
hidden_states: torch.FloatTensor,
attention_mask: Optional[torch.FloatTensor] = None,
encoder_hidden_states: Optional[torch.FloatTensor] = None,
encoder_attention_mask: Optional[torch.FloatTensor] = None,
timestep: Optional[torch.LongTensor] = None,
cross_attention_kwargs: Dict[str, Any] = None,
class_labels: Optional[torch.LongTensor] = None,
):
# Notice that normalization is always applied before the real computation in the following blocks.
# 0. Self-Attention
if self.use_ada_layer_norm:
norm_hidden_states = self.norm1(hidden_states, timestep)
elif self.use_ada_layer_norm_zero:
norm_hidden_states, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.norm1(
hidden_states, timestep, class_labels, hidden_dtype=hidden_states.dtype
)
else:
norm_hidden_states = self.norm1(hidden_states)
# 2. Prepare GLIGEN inputs
cross_attention_kwargs = cross_attention_kwargs.copy() if cross_attention_kwargs is not None else {}
gligen_kwargs = cross_attention_kwargs.pop("gligen", None)
attn_output = self.attn1(
norm_hidden_states,
encoder_hidden_states=encoder_hidden_states if self.only_cross_attention else None,
attention_mask=attention_mask,
**cross_attention_kwargs,
)
if self.use_ada_layer_norm_zero:
attn_output = gate_msa.unsqueeze(1) * attn_output
hidden_states = attn_output + hidden_states
# 2.5 GLIGEN Control
if gligen_kwargs is not None:
hidden_states = self.fuser(hidden_states, gligen_kwargs["objs"])
# 2.5 ends
# 3. Cross-Attention
if self.attn2 is not None:
norm_hidden_states = (
self.norm2(hidden_states, timestep) if self.use_ada_layer_norm else self.norm2(hidden_states)
)
attn_output = self.attn2(
norm_hidden_states,
encoder_hidden_states=encoder_hidden_states,
attention_mask=encoder_attention_mask,
**cross_attention_kwargs,
)
hidden_states = attn_output + hidden_states
# 4. Feed-forward
norm_hidden_states = self.norm3(hidden_states)
if self.use_ada_layer_norm_zero:
norm_hidden_states = norm_hidden_states * (1 + scale_mlp[:, None]) + shift_mlp[:, None]
if self._chunk_size is not None:
# "feed_forward_chunk_size" can be used to save memory
if norm_hidden_states.shape[self._chunk_dim] % self._chunk_size != 0:
raise ValueError(
f"`hidden_states` dimension to be chunked: {norm_hidden_states.shape[self._chunk_dim]} has to be divisible by chunk size: {self._chunk_size}. Make sure to set an appropriate `chunk_size` when calling `unet.enable_forward_chunking`."
)
num_chunks = norm_hidden_states.shape[self._chunk_dim] // self._chunk_size
ff_output = torch.cat(
[
self.ff(hid_slice)
for hid_slice in norm_hidden_states.chunk(num_chunks, dim=self._chunk_dim)
],
dim=self._chunk_dim,
)
else:
ff_output = self.ff(norm_hidden_states)
if self.use_ada_layer_norm_zero:
ff_output = gate_mlp.unsqueeze(1) * ff_output
hidden_states = ff_output + hidden_states
return hidden_states
modules/ggml/__init__.py
+40 -5
View File
@@ -1,11 +1,35 @@
from pathlib import Path
import os
import time
import torch
import gguf
from .gguf_utils import TORCH_COMPATIBLE_QTYPES
from .gguf_tensor import GGMLTensor
import diffusers
import transformers
def load_gguf_state_dict(path: str, compute_dtype: torch.dtype) -> dict[str, GGMLTensor]:
def install_gguf():
# pip install git+https://github.com/junejae/transformers@feature/t5-gguf
# https://github.com/ggerganov/llama.cpp/issues/9566
from installer import install
install('gguf', quiet=True)
import importlib
import gguf
from modules import shared
scripts_dir = os.path.join(os.path.dirname(gguf.__file__), '..', 'scripts')
if os.path.exists(scripts_dir):
os.rename(scripts_dir, scripts_dir + str(time.time()))
# monkey patch transformers/diffusers so they detect newly installed gguf pacakge correctly
ver = importlib.metadata.version('gguf')
transformers.utils.import_utils._is_gguf_available = True # pylint: disable=protected-access
transformers.utils.import_utils._gguf_version = ver # pylint: disable=protected-access
diffusers.utils.import_utils._is_gguf_available = True # pylint: disable=protected-access
diffusers.utils.import_utils._gguf_version = ver # pylint: disable=protected-access
shared.log.debug(f'Load GGUF: version={ver}')
return gguf
def load_gguf_state_dict(path: str, compute_dtype: torch.dtype) -> dict:
gguf = install_gguf()
from .gguf_utils import TORCH_COMPATIBLE_QTYPES
from .gguf_tensor import GGMLTensor
sd: dict[str, GGMLTensor] = {}
stats = {}
reader = gguf.GGUFReader(path)
@@ -19,3 +43,14 @@ def load_gguf_state_dict(path: str, compute_dtype: torch.dtype) -> dict[str, GGM
stats[tensor.tensor_type.name] = 0
stats[tensor.tensor_type.name] += 1
return sd, stats
def load_gguf(path, cls, compute_dtype: torch.dtype):
_gguf = install_gguf()
module = cls.from_single_file(
path,
quantization_config = diffusers.GGUFQuantizationConfig(compute_dtype=compute_dtype),
torch_dtype=compute_dtype,
)
module.gguf = 'gguf'
return module
modules/ggml/gguf_tensor.py
-1
View File
@@ -131,7 +131,6 @@ class GGMLTensor(torch.Tensor):
if self._ggml_quantization_type in TORCH_COMPATIBLE_QTYPES:
return self.quantized_data.to(self.compute_dtype)
elif self._ggml_quantization_type in DEQUANTIZE_FUNCTIONS:
# TODO(ryand): Look into how the dtype param is intended to be used.
return dequantize(
data=self.quantized_data, qtype=self._ggml_quantization_type, oshape=self.tensor_shape, dtype=None
).to(self.compute_dtype)
modules/hashes.py
+7
View File
@@ -9,6 +9,13 @@ cache_filename = os.path.join(data_path, "cache.json")
cache_data = None
progress_ok = True
def init_cache():
global cache_data # pylint: disable=global-statement
if cache_data is None:
cache_data = {} if not os.path.isfile(cache_filename) else shared.readfile(cache_filename, lock=True)
def dump_cache():
shared.writefile(cache_data, cache_filename)
modules/hidiffusion/hidiffusion.py
+3 -3
View File
@@ -234,7 +234,7 @@ def make_diffusers_transformer_block(block_class: Type[torch.nn.Module]) -> Type
norm_hidden_states = self.norm2(hidden_states)
norm_hidden_states = norm_hidden_states * (1 + scale_mlp) + shift_mlp
if self._chunk_size is not None:
ff_output = _chunked_feed_forward(self.ff, norm_hidden_states, self._chunk_dim, self._chunk_size) # pylint: disable=undefined-variable # TODO hidiffusion undefined
ff_output = _chunked_feed_forward(self.ff, norm_hidden_states, self._chunk_dim, self._chunk_size) # pylint: disable=undefined-variable
else:
ff_output = self.ff(norm_hidden_states)
if self.use_ada_layer_norm_zero:
@@ -308,7 +308,7 @@ def make_diffusers_cross_attn_down_block(block_class: Type[torch.nn.Module]) ->
self.T1 = int(self.max_timestep * self.T1_ratio)
output_states = ()
_scale = cross_attention_kwargs.get("scale", 1.0) if cross_attention_kwargs is not None else 1.0 # TODO hidiffusion unused
_scale = cross_attention_kwargs.get("scale", 1.0) if cross_attention_kwargs is not None else 1.0
blocks = list(zip(self.resnets, self.attentions))
@@ -407,7 +407,7 @@ def make_diffusers_cross_attn_up_block(block_class: Type[torch.nn.Module]) -> Ty
encoder_attention_mask: Optional[torch.FloatTensor] = None,
) -> torch.FloatTensor:
def fix_scale(first, second): # TODO hidiffusion breaks hidden_scale.shape on 3rd generate with sdxl
def fix_scale(first, second):
if (first.shape[-1] != second.shape[-1] or first.shape[-2] != second.shape[-2]):
rescale = min(second.shape[-2] / first.shape[-2], second.shape[-1] / first.shape[-1])
# log.debug(f"HiDiffusion rescale: {hidden_states.shape} => {res_hidden_states_tuple[0].shape} scale={rescale}")
modules/images.py
+2
View File
@@ -267,6 +267,8 @@ def safe_decode_string(s: bytes):
def read_info_from_image(image: Image, watermark: bool = False):
if image is None:
return '', {}
items = image.info or {}
geninfo = items.pop('parameters', None) or items.pop('UserComment', None)
if geninfo is not None and len(geninfo) > 0:
modules/images_namegen.py
+2 -2
View File
@@ -45,12 +45,12 @@ class FilenameGenerator:
'prompt_hash': lambda self: hashlib.sha256(self.prompt.encode()).hexdigest()[0:8],
'sampler': lambda self: self.p and self.p.sampler_name,
'seed': lambda self: self.seed and str(self.seed) or '',
'seed': lambda self: (self.seed and str(self.seed)) or '',
'steps': lambda self: self.p and getattr(self.p, 'steps', 0),
'cfg': lambda self: self.p and getattr(self.p, 'cfg_scale', 0),
'clip_skip': lambda self: self.p and getattr(self.p, 'clip_skip', 0),
'denoising': lambda self: self.p and getattr(self.p, 'denoising_strength', 0),
'styles': lambda self: self.p and ", ".join([style for style in self.p.styles if not style == "None"]) or "None",
'styles': lambda self: (self.p and ", ".join([style for style in self.p.styles if not style == "None"])) or "None",
'uuid': lambda self: str(uuid.uuid4()),
}
default_time_format = '%Y%m%d%H%M%S'
modules/images_resize.py
+7 -7
View File
@@ -5,13 +5,13 @@ from PIL import Image
from modules import shared
def resize_image(resize_mode, im, width, height, upscaler_name=None, output_type='image', context=None):
def resize_image(resize_mode: int, im: Image.Image, width: int, height: int, upscaler_name: str=None, output_type: str='image', context: str=None):
upscaler_name = upscaler_name or shared.opts.upscaler_for_img2img
def latent(im, w, h, upscaler):
from modules.processing_vae import vae_encode, vae_decode
import torch
latents = vae_encode(im, shared.sd_model, full_quality=False) # TODO enable full VAE mode for resize-latent
latents = vae_encode(im, shared.sd_model, full_quality=False) # TODO resize image: enable full VAE mode for resize-latent
latents = torch.nn.functional.interpolate(latents, size=(int(h // 8), int(w // 8)), mode=upscaler["mode"], antialias=upscaler["antialias"])
im = vae_decode(latents, shared.sd_model, output_type='pil', full_quality=False)[0]
return im
@@ -79,18 +79,18 @@ def resize_image(resize_mode, im, width, height, upscaler_name=None, output_type
def context_aware(im, width, height, context):
import seam_carving # https://github.com/li-plus/seam-carving
if 'forward' in context:
if 'forward' in context.lower():
energy_mode = "forward"
elif 'backward' in context:
elif 'backward' in context.lower():
energy_mode = "backward"
else:
return im
if 'Add' in context:
if 'add' in context.lower():
src_ratio = min(width / im.width, height / im.height)
src_w = int(im.width * src_ratio)
src_h = int(im.height * src_ratio)
src_image = resize(im, src_w, src_h)
elif 'Remove' in context:
elif 'remove' in context.lower():
ratio = width / height
src_ratio = im.width / im.height
src_w = width if ratio > src_ratio else im.width * height // im.height
@@ -122,7 +122,7 @@ def resize_image(resize_mode, im, width, height, upscaler_name=None, output_type
from modules import masking
res = fill(im, color=0)
res, _mask = masking.outpaint(res)
elif resize_mode == 5: # context-aware
elif resize_mode == 5: # context-aware
res = context_aware(im, width, height, context)
else:
res = im.copy()
modules/img2img.py
+58 -43
View File
@@ -1,6 +1,7 @@
import os
import itertools # SBM Batch frames
import numpy as np
import filetype
from PIL import Image, ImageOps, ImageFilter, ImageEnhance, ImageChops, UnidentifiedImageError
import modules.scripts
from modules import shared, processing, images
@@ -8,7 +9,6 @@ from modules.generation_parameters_copypaste import create_override_settings_dic
from modules.ui import plaintext_to_html
from modules.memstats import memory_stats
debug = shared.log.trace if os.environ.get('SD_PROCESS_DEBUG', None) is not None else lambda *args, **kwargs: None
debug('Trace: PROCESS')
@@ -16,24 +16,25 @@ debug('Trace: PROCESS')
def process_batch(p, input_files, input_dir, output_dir, inpaint_mask_dir, args):
shared.log.debug(f'batch: {input_files}|{input_dir}|{output_dir}|{inpaint_mask_dir}')
processing.fix_seed(p)
image_files = []
if input_files is not None and len(input_files) > 0:
image_files = [f.name for f in input_files]
else:
if not os.path.isdir(input_dir):
shared.log.error(f"Process batch: directory not found: {input_dir}")
return
image_files = os.listdir(input_dir)
image_files = [os.path.join(input_dir, f) for f in image_files]
image_files = [f for f in image_files if filetype.is_image(f)]
shared.log.info(f'Process batch: input images={len(image_files)}')
elif os.path.isdir(input_dir):
image_files = [os.path.join(input_dir, f) for f in os.listdir(input_dir)]
image_files = [f for f in image_files if filetype.is_image(f)]
shared.log.info(f'Process batch: input folder="{input_dir}" images={len(image_files)}')
is_inpaint_batch = False
if inpaint_mask_dir:
inpaint_masks = os.listdir(inpaint_mask_dir)
inpaint_masks = [os.path.join(inpaint_mask_dir, f) for f in inpaint_masks]
if inpaint_mask_dir and os.path.isdir(inpaint_mask_dir):
inpaint_masks = [os.path.join(inpaint_mask_dir, f) for f in os.listdir(inpaint_mask_dir)]
inpaint_masks = [f for f in inpaint_masks if filetype.is_image(f)]
is_inpaint_batch = len(inpaint_masks) > 0
if is_inpaint_batch:
shared.log.info(f"Process batch: inpaint batch masks={len(inpaint_masks)}")
shared.log.info(f'Process batch: mask folder="{input_dir}" images={len(inpaint_masks)}')
save_normally = output_dir == ''
p.do_not_save_grid = True
p.do_not_save_samples = not save_normally
p.default_prompt = p.prompt
shared.state.job_count = len(image_files) * p.n_iter
if shared.opts.batch_frame_mode: # SBM Frame mode is on, process each image in batch with same seed
window_size = p.batch_size
@@ -55,14 +56,29 @@ def process_batch(p, input_files, input_dir, output_dir, inpaint_mask_dir, args)
for image_file in batch_image_files:
try:
img = Image.open(image_file)
if p.scale_by != 1:
p.width = int(img.width * p.scale_by)
p.height = int(img.height * p.scale_by)
img = ImageOps.exif_transpose(img)
batch_images.append(img)
# p.init()
p.width = int(img.width * p.scale_by)
p.height = int(img.height * p.scale_by)
caption_file = os.path.splitext(image_file)[0] + '.txt'
prompt_type='default'
if os.path.exists(caption_file):
with open(caption_file, 'r', encoding='utf8') as f:
p.prompt = f.read()
prompt_type='file'
else:
p.prompt = p.default_prompt
p.all_prompts = None
p.all_negative_prompts = None
p.all_seeds = None
p.all_subseeds = None
shared.log.debug(f'Process batch: image="{image_file}" prompt={prompt_type} i={i+1}/{len(image_files)}')
except UnidentifiedImageError as e:
shared.log.error(f"Image error: {e}")
continue
img = ImageOps.exif_transpose(img)
batch_images.append(img)
shared.log.error(f'Process batch: image="{image_file}" {e}')
if len(batch_images) == 0:
shared.log.warning("Process batch: no images found in batch")
continue
batch_images = batch_images * btcrept # Standard mode sends the same image per batchsize.
p.init_images = batch_images
@@ -81,17 +97,20 @@ def process_batch(p, input_files, input_dir, output_dir, inpaint_mask_dir, args)
batch_image_files = batch_image_files * btcrept # List used for naming later.
proc = modules.scripts.scripts_img2img.run(p, *args)
if proc is None:
proc = processing.process_images(p)
for n, (image, image_file) in enumerate(itertools.zip_longest(proc.images,batch_image_files)):
processed = modules.scripts.scripts_img2img.run(p, *args)
if processed is None:
processed = processing.process_images(p)
for n, (image, image_file) in enumerate(itertools.zip_longest(processed.images, batch_image_files)):
if image is None:
continue
basename = ''
if shared.opts.use_original_name_batch:
forced_filename, ext = os.path.splitext(os.path.basename(image_file))
else:
forced_filename = None
ext = shared.opts.samples_format
if len(proc.images) > 1:
if len(processed.images) > 1:
basename = f'{n + i}' if shared.opts.batch_frame_mode else f'{n}'
else:
basename = ''
@@ -103,7 +122,7 @@ def process_batch(p, input_files, input_dir, output_dir, inpaint_mask_dir, args)
for k, v in items.items():
image.info[k] = v
images.save_image(image, path=output_dir, basename=basename, seed=None, prompt=None, extension=ext, info=geninfo, short_filename=True, no_prompt=True, grid=False, pnginfo_section_name="extras", existing_info=image.info, forced_filename=forced_filename)
proc = modules.scripts.scripts_img2img.after(p, proc, *args)
processed = modules.scripts.scripts_img2img.after(p, processed, *args)
shared.log.debug(f'Processed: images={len(batch_image_files)} memory={memory_stats()} batch')
@@ -147,29 +166,20 @@ def img2img(id_task: str, state: str, mode: int,
debug(f'img2img: id_task={id_task}|mode={mode}|prompt={prompt}|negative_prompt={negative_prompt}|prompt_styles={prompt_styles}|init_img={init_img}|sketch={sketch}|init_img_with_mask={init_img_with_mask}|inpaint_color_sketch={inpaint_color_sketch}|inpaint_color_sketch_orig={inpaint_color_sketch_orig}|init_img_inpaint={init_img_inpaint}|init_mask_inpaint={init_mask_inpaint}|steps={steps}|sampler_index={sampler_index}||mask_blur={mask_blur}|mask_alpha={mask_alpha}|inpainting_fill={inpainting_fill}|full_quality={full_quality}|detailer={detailer}|tiling={tiling}|hidiffusion={hidiffusion}|n_iter={n_iter}|batch_size={batch_size}|cfg_scale={cfg_scale}|image_cfg_scale={image_cfg_scale}|clip_skip={clip_skip}|denoising_strength={denoising_strength}|seed={seed}|subseed{subseed}|subseed_strength={subseed_strength}|seed_resize_from_h={seed_resize_from_h}|seed_resize_from_w={seed_resize_from_w}|selected_scale_tab={selected_scale_tab}|height={height}|width={width}|scale_by={scale_by}|resize_mode={resize_mode}|resize_name={resize_name}|resize_context={resize_context}|inpaint_full_res={inpaint_full_res}|inpaint_full_res_padding={inpaint_full_res_padding}|inpainting_mask_invert={inpainting_mask_invert}|img2img_batch_files={img2img_batch_files}|img2img_batch_input_dir={img2img_batch_input_dir}|img2img_batch_output_dir={img2img_batch_output_dir}|img2img_batch_inpaint_mask_dir={img2img_batch_inpaint_mask_dir}|override_settings_texts={override_settings_texts}')
if mode == 5:
if img2img_batch_files is None or len(img2img_batch_files) == 0:
shared.log.debug('Init bactch images not set')
elif init_img:
shared.log.debug('Init image not set')
if sampler_index is None:
shared.log.warning('Sampler: invalid')
sampler_index = 0
mode = int(mode)
image = None
mask = None
override_settings = create_override_settings_dict(override_settings_texts)
if mode == 0: # img2img
if mode == 0: # img2img
if init_img is None:
return [], '', '', 'Error: init image not provided'
image = init_img.convert("RGB")
mask = None
elif mode == 1: # img2img sketch
if sketch is None:
return [], '', '', 'Error: sketch image not provided'
image = sketch.convert("RGB")
mask = None
elif mode == 2: # inpaint
elif mode == 1: # inpaint
if init_img_with_mask is None:
return [], '', '', 'Error: init image with mask not provided'
image = init_img_with_mask["image"]
@@ -177,7 +187,11 @@ def img2img(id_task: str, state: str, mode: int,
alpha_mask = ImageOps.invert(image.split()[-1]).convert('L').point(lambda x: 255 if x > 0 else 0, mode='1')
mask = ImageChops.lighter(alpha_mask, mask.convert('L')).convert('L')
image = image.convert("RGB")
elif mode == 3: # inpaint sketch
elif mode == 2: # sketch
if sketch is None:
return [], '', '', 'Error: sketch image not provided'
image = sketch.convert("RGB")
elif mode == 3: # composite
if inpaint_color_sketch is None:
return [], '', '', 'Error: color sketch image not provided'
image = inpaint_color_sketch
@@ -188,15 +202,16 @@ def img2img(id_task: str, state: str, mode: int,
blur = ImageFilter.GaussianBlur(mask_blur)
image = Image.composite(image.filter(blur), orig, mask.filter(blur))
image = image.convert("RGB")
elif mode == 4: # inpaint upload mask
elif mode == 4: # inpaint upload mask
if init_img_inpaint is None:
return [], '', '', 'Error: inpaint image not provided'
image = init_img_inpaint
mask = init_mask_inpaint
elif mode == 5: # process batch
pass # handled later
else:
shared.log.error(f'Image processing unknown mode: {mode}')
image = None
mask = None
if image is not None:
image = ImageOps.exif_transpose(image)
if selected_scale_tab == 1 and resize_mode != 0:
modules/infotext.py
+36
View File
@@ -10,6 +10,7 @@ else:
debug = lambda *args, **kwargs: None # pylint: disable=unnecessary-lambda-assignment
re_size = re.compile(r"^(\d+)x(\d+)$") # int x int
re_param = re.compile(r'\s*([\w ]+):\s*("(?:\\"[^,]|\\"|\\|[^\"])+"|[^,]*)(?:,|$)') # multi-word: value
re_lora = re.compile("<lora:([^:]+):")
def quote(text):
@@ -27,6 +28,40 @@ def unquote(text):
return text
# disabled by default can be enabled if needed
def check_lora(params):
try:
import modules.lora.networks as networks
from modules.errors import log # pylint: disable=redefined-outer-name
except Exception:
return
loras = [s.strip() for s in params.get('LoRA hashes', '').split(',')]
found = []
missing = []
for l in loras:
lora = networks.available_network_hash_lookup.get(l, None)
if lora is not None:
found.append(lora.name)
else:
missing.append(l)
loras = [s.strip() for s in params.get('LoRA networks', '').split(',')]
for l in loras:
lora = networks.available_network_aliases.get(l, None)
if lora is not None:
found.append(lora.name)
else:
missing.append(l)
# networks.available_network_aliases.get(name, None)
loras = re_lora.findall(params.get('Prompt', ''))
for l in loras:
lora = networks.available_network_aliases.get(l, None)
if lora is not None:
found.append(lora.name)
else:
missing.append(l)
log.debug(f'LoRA: found={list(set(found))} missing={list(set(missing))}')
def parse(infotext):
if not isinstance(infotext, str):
return {}
@@ -75,6 +110,7 @@ def parse(infotext):
params[key] = val
debug(f'Param parsed: type={type(params[key])} {key}={params[key]} raw="{val}"')
# check_lora(params)
return params
modules/instantir/aggregator.py
-1
View File
@@ -823,7 +823,6 @@ class Aggregator(ModelMixin, ConfigMixin, FromOriginalModelMixin):
# 1. time
timesteps = timestep
if not torch.is_tensor(timesteps):
# TODO: this requires sync between CPU and GPU. So try to pass timesteps as tensors if you can
# This would be a good case for the `match` statement (Python 3.10+)
is_mps = sample.device.type == "mps"
if isinstance(timestep, float):
modules/instantir/ip_adapter/attention_processor.py
-11
View File
@@ -390,7 +390,6 @@ class AttnProcessor2_0(torch.nn.Module):
value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
# the output of sdp = (batch, num_heads, seq_len, head_dim)
# TODO: add support for attn.scale when we move to Torch 2.1
hidden_states = F.scaled_dot_product_attention(
query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
)
@@ -496,7 +495,6 @@ class split_AttnProcessor2_0(torch.nn.Module):
value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
# the output of sdp = (batch, num_heads, seq_len, head_dim)
# TODO: add support for attn.scale when we move to Torch 2.1
hidden_states = F.scaled_dot_product_attention(
query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
)
@@ -640,7 +638,6 @@ class sep_split_AttnProcessor2_0(torch.nn.Module):
value_1 = value_1.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
# the output of sdp = (batch, num_heads, seq_len, head_dim)
# TODO: add support for attn.scale when we move to Torch 2.1
hidden_states_0 = F.scaled_dot_product_attention(
query_0, key_0, value_0, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
)
@@ -654,7 +651,6 @@ class sep_split_AttnProcessor2_0(torch.nn.Module):
)
hidden_states_0 = hidden_states_0 + ref_scale * _hidden_states_0 * 10
# TODO: drop this cross-attn
_hidden_states_1 = F.scaled_dot_product_attention(
query_1, key_0, value_0, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
)
@@ -771,7 +767,6 @@ class AdditiveKV_AttnProcessor2_0(torch.nn.Module):
value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
# the output of sdp = (batch, num_heads, seq_len, head_dim)
# TODO: add support for attn.scale when we move to Torch 2.1
hidden_states = F.scaled_dot_product_attention(
query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
)
@@ -883,7 +878,6 @@ class TA_AdditiveKV_AttnProcessor2_0(torch.nn.Module):
value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
# the output of sdp = (batch, num_heads, seq_len, head_dim)
# TODO: add support for attn.scale when we move to Torch 2.1
hidden_states = F.scaled_dot_product_attention(
query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
)
@@ -1018,7 +1012,6 @@ class IPAttnProcessor2_0(torch.nn.Module):
value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
# the output of sdp = (batch, num_heads, seq_len, head_dim)
# TODO: add support for attn.scale when we move to Torch 2.1
hidden_states = F.scaled_dot_product_attention(
query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
)
@@ -1034,7 +1027,6 @@ class IPAttnProcessor2_0(torch.nn.Module):
ip_value = ip_value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
# the output of sdp = (batch, num_heads, seq_len, head_dim)
# TODO: add support for attn.scale when we move to Torch 2.1
ip_hidden_states = F.scaled_dot_product_attention(
query, ip_key, ip_value, attn_mask=None, dropout_p=0.0, is_causal=False
)
@@ -1161,7 +1153,6 @@ class TA_IPAttnProcessor2_0(torch.nn.Module):
value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
# the output of sdp = (batch, num_heads, seq_len, head_dim)
# TODO: add support for attn.scale when we move to Torch 2.1
hidden_states = F.scaled_dot_product_attention(
query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
)
@@ -1181,7 +1172,6 @@ class TA_IPAttnProcessor2_0(torch.nn.Module):
ip_value = ip_value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
# the output of sdp = (batch, num_heads, seq_len, head_dim)
# TODO: add support for attn.scale when we move to Torch 2.1
ip_hidden_states = F.scaled_dot_product_attention(
query, ip_key, ip_value, attn_mask=None, dropout_p=0.0, is_causal=False
)
@@ -1337,7 +1327,6 @@ class CNAttnProcessor2_0:
value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
# the output of sdp = (batch, num_heads, seq_len, head_dim)
# TODO: add support for attn.scale when we move to Torch 2.1
hidden_states = F.scaled_dot_product_attention(
query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
)
modules/intel/ipex/__init__.py
+5 -5
View File
@@ -77,7 +77,7 @@ def ipex_init(): # pylint: disable=too-many-statements
torch.cuda.warnings = torch.xpu.warnings
torch.cuda.classproperty = torch.xpu.classproperty
torch.UntypedStorage.cuda = torch.UntypedStorage.xpu
if not ipex.__version__.startswith("2.3"):
if float(ipex.__version__[:3]) < 2.3:
torch.cuda._initialization_lock = torch.xpu.lazy_init._initialization_lock
torch.cuda._initialized = torch.xpu.lazy_init._initialized
torch.cuda._is_in_bad_fork = torch.xpu.lazy_init._is_in_bad_fork
@@ -111,7 +111,7 @@ def ipex_init(): # pylint: disable=too-many-statements
torch.cuda.ComplexFloatStorage = torch.xpu.ComplexFloatStorage
torch.cuda.ComplexDoubleStorage = torch.xpu.ComplexDoubleStorage
if not legacy or ipex.__version__.startswith("2.3"):
if not legacy or float(ipex.__version__[:3]) >= 2.3:
torch.cuda._initialization_lock = torch.xpu._initialization_lock
torch.cuda._initialized = torch.xpu._initialized
torch.cuda._is_in_bad_fork = torch.xpu._is_in_bad_fork
@@ -122,7 +122,7 @@ def ipex_init(): # pylint: disable=too-many-statements
torch.cuda.traceback = torch.xpu.traceback
# Memory:
if legacy and 'linux' in sys.platform and "WSL2" in os.popen("uname -a").read():
if 'linux' in sys.platform and "WSL2" in os.popen("uname -a").read():
torch.xpu.empty_cache = lambda: None
torch.cuda.empty_cache = torch.xpu.empty_cache
@@ -159,7 +159,7 @@ def ipex_init(): # pylint: disable=too-many-statements
torch.xpu.amp.custom_fwd = torch.cuda.amp.custom_fwd
torch.xpu.amp.custom_bwd = torch.cuda.amp.custom_bwd
torch.cuda.amp = torch.xpu.amp
if not ipex.__version__.startswith("2.3"):
if float(ipex.__version__[:3]) < 2.3:
torch.is_autocast_enabled = torch.xpu.is_autocast_xpu_enabled
torch.get_autocast_gpu_dtype = torch.xpu.get_autocast_xpu_dtype
@@ -178,7 +178,7 @@ def ipex_init(): # pylint: disable=too-many-statements
torch.cuda.amp.GradScaler = ipex.cpu.autocast._grad_scaler.GradScaler
# C
if legacy and not ipex.__version__.startswith("2.3"):
if legacy and float(ipex.__version__[:3]) < 2.3:
torch._C._cuda_getCurrentRawStream = ipex._C._getCurrentRawStream
ipex._C._DeviceProperties.multi_processor_count = ipex._C._DeviceProperties.gpu_subslice_count
ipex._C._DeviceProperties.major = 12
modules/intel/ipex/diffusers.py
+27 -15
View File
@@ -1,7 +1,7 @@
import os
from functools import wraps, cache
import torch
import diffusers #0.29.1 # pylint: disable=import-error
import diffusers # pylint: disable=import-error
from diffusers.models.attention_processor import Attention
# pylint: disable=protected-access, missing-function-docstring, line-too-long
@@ -20,20 +20,31 @@ def fourier_filter(x_in, threshold, scale):
# fp64 error
def rope(pos: torch.Tensor, dim: int, theta: int) -> torch.Tensor:
assert dim % 2 == 0, "The dimension must be even."
class FluxPosEmbed(torch.nn.Module):
def __init__(self, theta: int, axes_dim):
super().__init__()
self.theta = theta
self.axes_dim = axes_dim
scale = torch.arange(0, dim, 2, dtype=torch.float32, device=pos.device) / dim # force fp32 instead of fp64
omega = 1.0 / (theta**scale)
batch_size, seq_length = pos.shape
out = torch.einsum("...n,d->...nd", pos, omega)
cos_out = torch.cos(out)
sin_out = torch.sin(out)
stacked_out = torch.stack([cos_out, -sin_out, sin_out, cos_out], dim=-1)
out = stacked_out.view(batch_size, -1, dim // 2, 2, 2)
return out.float()
def forward(self, ids: torch.Tensor) -> torch.Tensor:
n_axes = ids.shape[-1]
cos_out = []
sin_out = []
pos = ids.float()
for i in range(n_axes):
cos, sin = diffusers.models.embeddings.get_1d_rotary_pos_embed(
self.axes_dim[i],
pos[:, i],
theta=self.theta,
repeat_interleave_real=True,
use_real=True,
freqs_dtype=torch.float32,
)
cos_out.append(cos)
sin_out.append(sin)
freqs_cos = torch.cat(cos_out, dim=-1).to(ids.device)
freqs_sin = torch.cat(sin_out, dim=-1).to(ids.device)
return freqs_cos, freqs_sin
@cache
@@ -337,4 +348,5 @@ def ipex_diffusers():
if not device_supports_fp64 or os.environ.get('IPEX_FORCE_ATTENTION_SLICE', None) is not None:
diffusers.models.attention_processor.SlicedAttnProcessor = SlicedAttnProcessor
diffusers.models.attention_processor.AttnProcessor = AttnProcessor
diffusers.models.transformers.transformer_flux.rope = rope
if not device_supports_fp64:
diffusers.models.embeddings.FluxPosEmbed = FluxPosEmbed
modules/intel/ipex/hijacks.py
+22 -1
View File
@@ -149,6 +149,15 @@ def functional_linear(input, weight, bias=None):
bias.data = bias.data.to(dtype=weight.data.dtype)
return original_functional_linear(input, weight, bias=bias)
original_functional_conv1d = torch.nn.functional.conv1d
@wraps(torch.nn.functional.conv1d)
def functional_conv1d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
if input.dtype != weight.data.dtype:
input = input.to(dtype=weight.data.dtype)
if bias is not None and bias.data.dtype != weight.data.dtype:
bias.data = bias.data.to(dtype=weight.data.dtype)
return original_functional_conv1d(input, weight, bias=bias, stride=stride, padding=padding, dilation=dilation, groups=groups)
original_functional_conv2d = torch.nn.functional.conv2d
@wraps(torch.nn.functional.conv2d)
def functional_conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
@@ -158,6 +167,16 @@ def functional_conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1,
bias.data = bias.data.to(dtype=weight.data.dtype)
return original_functional_conv2d(input, weight, bias=bias, stride=stride, padding=padding, dilation=dilation, groups=groups)
# LTX Video
original_functional_conv3d = torch.nn.functional.conv3d
@wraps(torch.nn.functional.conv3d)
def functional_conv3d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1):
if input.dtype != weight.data.dtype:
input = input.to(dtype=weight.data.dtype)
if bias is not None and bias.data.dtype != weight.data.dtype:
bias.data = bias.data.to(dtype=weight.data.dtype)
return original_functional_conv3d(input, weight, bias=bias, stride=stride, padding=padding, dilation=dilation, groups=groups)
# SwinIR BF16:
original_functional_pad = torch.nn.functional.pad
@wraps(torch.nn.functional.pad)
@@ -294,7 +313,7 @@ def torch_load(f, map_location=None, *args, **kwargs):
# Hijack Functions:
def ipex_hijacks(legacy=True):
if legacy:
if legacy and float(torch.__version__[:3]) < 2.5:
torch.nn.functional.interpolate = interpolate
torch.tensor = torch_tensor
torch.Tensor.to = Tensor_to
@@ -320,7 +339,9 @@ def ipex_hijacks(legacy=True):
torch.nn.functional.group_norm = functional_group_norm
torch.nn.functional.layer_norm = functional_layer_norm
torch.nn.functional.linear = functional_linear
torch.nn.functional.conv1d = functional_conv1d
torch.nn.functional.conv2d = functional_conv2d
torch.nn.functional.conv3d = functional_conv3d
torch.nn.functional.pad = functional_pad
torch.bmm = torch_bmm
modules/ipadapter.py
+148 -80
View File
@@ -9,11 +9,16 @@ import os
import time
import json
from PIL import Image
from modules import processing, shared, devices, sd_models
import diffusers
import transformers
from modules import processing, shared, devices, sd_models, errors
clip_repo = "h94/IP-Adapter"
clip_loaded = None
adapters_loaded = []
CLIP_ID = "h94/IP-Adapter"
OPEN_ID = "openai/clip-vit-large-patch14"
SIGLIP_ID = 'google/siglip-so400m-patch14-384'
ADAPTERS_NONE = {
'None': { 'name': 'none', 'repo': 'none', 'subfolder': 'none' },
}
@@ -36,11 +41,13 @@ ADAPTERS_SDXL = {
'Ostris Composition ViT-H SDXL': { 'name': 'ip_plus_composition_sdxl.safetensors', 'repo': 'ostris/ip-composition-adapter', 'subfolder': '' },
}
ADAPTERS_SD3 = {
'InstantX Large': { 'name': 'ip-adapter.bin', 'repo': 'InstantX/SD3.5-Large-IP-Adapter' },
'None': { 'name': 'none', 'repo': 'none', 'subfolder': 'none' },
'InstantX Large': { 'name': 'none', 'repo': 'InstantX/SD3.5-Large-IP-Adapter', 'subfolder': 'none', 'revision': 'refs/pr/10' },
}
ADAPTERS_F1 = {
'XLabs AI v1': { 'name': 'ip_adapter.safetensors', 'repo': 'XLabs-AI/flux-ip-adapter' },
'XLabs AI v2': { 'name': 'ip_adapter.safetensors', 'repo': 'XLabs-AI/flux-ip-adapter-v2' },
'None': { 'name': 'none', 'repo': 'none', 'subfolder': 'none' },
'XLabs AI v1': { 'name': 'ip_adapter.safetensors', 'repo': 'XLabs-AI/flux-ip-adapter', 'subfolder': 'none' },
'XLabs AI v2': { 'name': 'ip_adapter.safetensors', 'repo': 'XLabs-AI/flux-ip-adapter-v2', 'subfolder': 'none' },
}
ADAPTERS = { **ADAPTERS_SD15, **ADAPTERS_SDXL, **ADAPTERS_SD3, **ADAPTERS_F1 }
ADAPTERS_ALL = { **ADAPTERS_SD15, **ADAPTERS_SDXL, **ADAPTERS_SD3, **ADAPTERS_F1 }
@@ -125,14 +132,27 @@ def crop_images(images, crops):
shared.log.error(f'IP adapter: failed to crop image: source={len(images[i])} faces={len(cropped)}')
except Exception as e:
shared.log.error(f'IP adapter: failed to crop image: {e}')
if shared.sd_model_type == 'sd3' and len(images) == 1:
return images[0]
return images
def unapply(pipe): # pylint: disable=arguments-differ
def unapply(pipe, unload: bool = False): # pylint: disable=arguments-differ
if len(adapters_loaded) == 0:
return
try:
if hasattr(pipe, 'set_ip_adapter_scale'):
pipe.set_ip_adapter_scale(0)
if hasattr(pipe, 'unet') and hasattr(pipe.unet, 'config') and pipe.unet.config.encoder_hid_dim_type == 'ip_image_proj':
if unload:
shared.log.debug('IP adapter unload')
pipe.unload_ip_adapter()
if hasattr(pipe, 'unet'):
module = pipe.unet
elif hasattr(pipe, 'transformer'):
module = pipe.transformer
else:
module = None
if module is not None and hasattr(module, 'config') and module.config.encoder_hid_dim_type == 'ip_image_proj':
pipe.unet.encoder_hid_proj = None
pipe.config.encoder_hid_dim_type = None
pipe.unet.set_default_attn_processor()
@@ -140,27 +160,78 @@ def unapply(pipe): # pylint: disable=arguments-differ
pass
def apply(pipe, p: processing.StableDiffusionProcessing, adapter_names=[], adapter_scales=[1.0], adapter_crops=[False], adapter_starts=[0.0], adapter_ends=[1.0], adapter_images=[]):
def load_image_encoder(pipe: diffusers.DiffusionPipeline, adapter_names: list[str]):
global clip_loaded # pylint: disable=global-statement
# overrides
if hasattr(p, 'ip_adapter_names'):
if isinstance(p.ip_adapter_names, str):
p.ip_adapter_names = [p.ip_adapter_names]
adapters = [ADAPTERS_ALL.get(adapter_name, None) for adapter_name in p.ip_adapter_names if adapter_name is not None and adapter_name.lower() != 'none']
adapter_names = p.ip_adapter_names
else:
if isinstance(adapter_names, str):
adapter_names = [adapter_names]
adapters = [ADAPTERS.get(adapter, None) for adapter in adapter_names]
adapters = [adapter for adapter in adapters if adapter is not None and adapter['name'].lower() != 'none']
if len(adapters) == 0:
unapply(pipe)
if hasattr(p, 'ip_adapter_images'):
del p.ip_adapter_images
return False
if shared.sd_model_type not in ['sd', 'sdxl', 'sd3', 'f1']:
shared.log.error(f'IP adapter: model={shared.sd_model_type} class={pipe.__class__.__name__} not supported')
return False
for adapter_name in adapter_names:
# which clip to use
clip_repo = CLIP_ID
if 'ViT' not in adapter_name: # defaults per model
clip_subfolder = 'models/image_encoder' if shared.sd_model_type == 'sd' else 'sdxl_models/image_encoder'
if 'ViT-H' in adapter_name:
clip_subfolder = 'models/image_encoder' # this is vit-h
elif 'ViT-G' in adapter_name:
clip_subfolder = 'sdxl_models/image_encoder' # this is vit-g
else:
if shared.sd_model_type == 'sd':
clip_subfolder = 'models/image_encoder'
elif shared.sd_model_type == 'sdxl':
clip_subfolder = 'sdxl_models/image_encoder'
elif shared.sd_model_type == 'sd3':
clip_repo = SIGLIP_ID
clip_subfolder = None
elif shared.sd_model_type == 'f1':
clip_repo = OPEN_ID
clip_subfolder = None
else:
shared.log.error(f'IP adapter: unknown model type: {adapter_name}')
return False
# load image encoder used by ip adapter
if pipe.image_encoder is None or clip_loaded != f'{clip_repo}/{clip_subfolder}':
try:
if shared.sd_model_type == 'sd3':
image_encoder = transformers.SiglipVisionModel.from_pretrained(clip_repo, torch_dtype=devices.dtype, cache_dir=shared.opts.hfcache_dir)
else:
if clip_subfolder is None:
image_encoder = transformers.CLIPVisionModelWithProjection.from_pretrained(clip_repo, torch_dtype=devices.dtype, cache_dir=shared.opts.hfcache_dir, use_safetensors=True)
shared.log.debug(f'IP adapter load: encoder="{clip_repo}" cls={pipe.image_encoder.__class__.__name__}')
else:
image_encoder = transformers.CLIPVisionModelWithProjection.from_pretrained(clip_repo, subfolder=clip_subfolder, torch_dtype=devices.dtype, cache_dir=shared.opts.hfcache_dir, use_safetensors=True)
shared.log.debug(f'IP adapter load: encoder="{clip_repo}/{clip_subfolder}" cls={pipe.image_encoder.__class__.__name__}')
if hasattr(pipe, 'register_modules'):
pipe.register_modules(image_encoder=image_encoder)
else:
pipe.image_encoder = image_encoder
clip_loaded = f'{clip_repo}/{clip_subfolder}'
except Exception as e:
shared.log.error(f'IP adapter load: encoder="{clip_repo}/{clip_subfolder}" {e}')
errors.display(e, 'IP adapter: type=encoder')
return False
sd_models.move_model(pipe.image_encoder, devices.device)
return True
def load_feature_extractor(pipe):
# load feature extractor used by ip adapter
if pipe.feature_extractor is None:
try:
if shared.sd_model_type == 'sd3':
feature_extractor = transformers.SiglipImageProcessor.from_pretrained(SIGLIP_ID, torch_dtype=devices.dtype, cache_dir=shared.opts.hfcache_dir)
else:
feature_extractor = transformers.CLIPImageProcessor()
if hasattr(pipe, 'register_modules'):
pipe.register_modules(feature_extractor=feature_extractor)
else:
pipe.feature_extractor = feature_extractor
shared.log.debug(f'IP adapter load: extractor={pipe.feature_extractor.__class__.__name__}')
except Exception as e:
shared.log.error(f'IP adapter load: extractor {e}')
errors.display(e, 'IP adapter: type=extractor')
return False
return True
def parse_params(p: processing.StableDiffusionProcessing, adapters: list, adapter_scales: list[float], adapter_crops: list[bool], adapter_starts: list[float], adapter_ends: list[float], adapter_images: list):
if hasattr(p, 'ip_adapter_scales'):
adapter_scales = p.ip_adapter_scales
if hasattr(p, 'ip_adapter_crops'):
@@ -201,6 +272,33 @@ def apply(pipe, p: processing.StableDiffusionProcessing, adapter_names=[], adapt
p.ip_adapter_starts = adapter_starts.copy()
adapter_ends = get_scales(adapter_ends, adapter_images)
p.ip_adapter_ends = adapter_ends.copy()
return adapter_images, adapter_masks, adapter_scales, adapter_crops, adapter_starts, adapter_ends
def apply(pipe, p: processing.StableDiffusionProcessing, adapter_names=[], adapter_scales=[1.0], adapter_crops=[False], adapter_starts=[0.0], adapter_ends=[1.0], adapter_images=[]):
global adapters_loaded # pylint: disable=global-statement
# overrides
if hasattr(p, 'ip_adapter_names'):
if isinstance(p.ip_adapter_names, str):
p.ip_adapter_names = [p.ip_adapter_names]
adapters = [ADAPTERS_ALL.get(adapter_name, None) for adapter_name in p.ip_adapter_names if adapter_name is not None and adapter_name.lower() != 'none']
adapter_names = p.ip_adapter_names
else:
if isinstance(adapter_names, str):
adapter_names = [adapter_names]
adapters = [ADAPTERS.get(adapter_name, None) for adapter_name in adapter_names if adapter_name.lower() != 'none']
if len(adapters) == 0:
unapply(pipe, getattr(p, 'ip_adapter_unload', False))
if hasattr(p, 'ip_adapter_images'):
del p.ip_adapter_images
return False
if shared.sd_model_type not in ['sd', 'sdxl', 'sd3', 'f1']:
shared.log.error(f'IP adapter: model={shared.sd_model_type} class={pipe.__class__.__name__} not supported')
return False
adapter_images, adapter_masks, adapter_scales, adapter_crops, adapter_starts, adapter_ends = parse_params(p, adapters, adapter_scales, adapter_crops, adapter_starts, adapter_ends, adapter_images)
# init code
if pipe is None:
return False
@@ -211,7 +309,7 @@ def apply(pipe, p: processing.StableDiffusionProcessing, adapter_names=[], adapt
shared.log.error('IP adapter: no image provided')
adapters = [] # unload adapter if previously loaded as it will cause runtime errors
if len(adapters) == 0:
unapply(pipe)
unapply(pipe, getattr(p, 'ip_adapter_unload', False))
if hasattr(p, 'ip_adapter_images'):
del p.ip_adapter_images
return False
@@ -219,61 +317,30 @@ def apply(pipe, p: processing.StableDiffusionProcessing, adapter_names=[], adapt
shared.log.error(f'IP adapter: pipeline not supported: {pipe.__class__.__name__}')
return False
for adapter_name in adapter_names:
# which clip to use
if 'ViT' not in adapter_name: # defaults per model
if shared.sd_model_type == 'sd':
clip_subfolder = 'models/image_encoder'
else:
clip_subfolder = 'sdxl_models/image_encoder'
if 'ViT-H' in adapter_name:
clip_subfolder = 'models/image_encoder' # this is vit-h
elif 'ViT-G' in adapter_name:
clip_subfolder = 'sdxl_models/image_encoder' # this is vit-g
else:
if shared.sd_model_type == 'sd':
clip_subfolder = 'models/image_encoder'
elif shared.sd_model_type == 'sdxl':
clip_subfolder = 'sdxl_models/image_encoder'
elif shared.sd_model_type == 'sd3':
shared.log.error(f'IP adapter: adapter={adapter_name} type={shared.sd_model_type} cls={shared.sd_model.__class__.__name__}: unsupported base model')
return False
elif shared.sd_model_type == 'f1':
shared.log.error(f'IP adapter: adapter={adapter_name} type={shared.sd_model_type} cls={shared.sd_model.__class__.__name__}: unsupported base model')
return False
else:
shared.log.error(f'IP adapter: unknown model type: {adapter_name}')
return False
if not load_image_encoder(pipe, adapter_names):
return False
# load feature extractor used by ip adapter
if pipe.feature_extractor is None:
try:
from transformers import CLIPImageProcessor
shared.log.debug('IP adapter load: feature extractor')
pipe.feature_extractor = CLIPImageProcessor()
except Exception as e:
shared.log.error(f'IP adapter load: feature extractor {e}')
return False
# load image encoder used by ip adapter
if pipe.image_encoder is None or clip_loaded != f'{clip_repo}/{clip_subfolder}':
try:
from transformers import CLIPVisionModelWithProjection
shared.log.debug(f'IP adapter load: image encoder="{clip_repo}/{clip_subfolder}"')
pipe.image_encoder = CLIPVisionModelWithProjection.from_pretrained(clip_repo, subfolder=clip_subfolder, torch_dtype=devices.dtype, cache_dir=shared.opts.diffusers_dir, use_safetensors=True)
clip_loaded = f'{clip_repo}/{clip_subfolder}'
except Exception as e:
shared.log.error(f'IP adapter load: image encoder="{clip_repo}/{clip_subfolder}" {e}')
return False
sd_models.move_model(pipe.image_encoder, devices.device)
if not load_feature_extractor(pipe):
return False
# main code
try:
t0 = time.time()
repos = [adapter['repo'] for adapter in adapters]
subfolders = [adapter['subfolder'] for adapter in adapters]
names = [adapter['name'] for adapter in adapters]
pipe.load_ip_adapter(repos, subfolder=subfolders, weight_name=names)
repos = [adapter.get('repo', None) for adapter in adapters if adapter.get('repo', 'none') != 'none']
subfolders = [adapter.get('subfolder', None) for adapter in adapters if adapter.get('subfolder', 'none') != 'none']
names = [adapter.get('name', None) for adapter in adapters if adapter.get('name', 'none') != 'none']
revisions = [adapter.get('revision', None) for adapter in adapters if adapter.get('revision', 'none') != 'none']
kwargs = {}
if len(repos) == 1:
repos = repos[0]
if len(subfolders) > 0:
kwargs['subfolder'] = subfolders if len(subfolders) > 1 else subfolders[0]
if len(names) > 0:
kwargs['weight_name'] = names if len(names) > 1 else names[0]
if len(revisions) > 0:
kwargs['revision'] = revisions[0]
pipe.load_ip_adapter(repos, **kwargs)
adapters_loaded = names
if hasattr(p, 'ip_adapter_layers'):
pipe.set_ip_adapter_scale(p.ip_adapter_layers)
ip_str = ';'.join(adapter_names) + ':' + json.dumps(p.ip_adapter_layers)
@@ -281,8 +348,8 @@ def apply(pipe, p: processing.StableDiffusionProcessing, adapter_names=[], adapt
for i in range(len(adapter_scales)):
if adapter_starts[i] > 0:
adapter_scales[i] = 0.00
pipe.set_ip_adapter_scale(adapter_scales)
ip_str = [f'{os.path.splitext(adapter)[0]}:{scale}:{start}:{end}' for adapter, scale, start, end in zip(adapter_names, adapter_scales, adapter_starts, adapter_ends)]
pipe.set_ip_adapter_scale(adapter_scales if len(adapter_scales) > 1 else adapter_scales[0])
ip_str = [f'{os.path.splitext(adapter)[0]}:{scale}:{start}:{end}:{crop}' for adapter, scale, start, end, crop in zip(adapter_names, adapter_scales, adapter_starts, adapter_ends, adapter_crops)]
p.task_args['ip_adapter_image'] = crop_images(adapter_images, adapter_crops)
if len(adapter_masks) > 0:
p.cross_attention_kwargs = { 'ip_adapter_masks': adapter_masks }
@@ -291,4 +358,5 @@ def apply(pipe, p: processing.StableDiffusionProcessing, adapter_names=[], adapt
shared.log.info(f'IP adapter: {ip_str} image={adapter_images} mask={adapter_masks is not None} time={t1-t0:.2f}')
except Exception as e:
shared.log.error(f'IP adapter load: adapters={adapter_names} repo={repos} folders={subfolders} names={names} {e}')
errors.display(e, 'IP adapter: type=adapter')
return True
modules/loader.py
+1 -1
View File
@@ -14,7 +14,7 @@ errors.install()
logging.getLogger("DeepSpeed").disabled = True
os.environ.setdefault('TORCH_LOGS', '-all')
# os.environ.setdefault('TORCH_LOGS', '-all')
import torch # pylint: disable=C0411
if torch.__version__.startswith('2.5.0'):
errors.log.warning(f'Disabling cuDNN for SDP on torch={torch.__version__}')
modules/lora/extra_networks_lora.py
+152
View File
@@ -0,0 +1,152 @@
import re
import numpy as np
import modules.lora.networks as networks
from modules import extra_networks, shared
# from https://github.com/cheald/sd-webui-loractl/blob/master/loractl/lib/utils.py
def get_stepwise(param, step, steps):
def sorted_positions(raw_steps):
steps = [[float(s.strip()) for s in re.split("[@~]", x)]
for x in re.split("[,;]", str(raw_steps))]
if len(steps[0]) == 1: # If we just got a single number, just return it
return steps[0][0]
steps = [[s[0], s[1] if len(s) == 2 else 1] for s in steps] # Add implicit 1s to any steps which don't have a weight
steps.sort(key=lambda k: k[1]) # Sort by index
steps = [list(v) for v in zip(*steps)]
return steps
def calculate_weight(m, step, max_steps, step_offset=2):
if isinstance(m, list):
if m[1][-1] <= 1.0:
step = step / (max_steps - step_offset) if max_steps > 0 else 1.0
v = np.interp(step, m[1], m[0])
return v
else:
return m
stepwise = calculate_weight(sorted_positions(param), step, steps)
return stepwise
def prompt(p):
if shared.opts.lora_apply_tags == 0:
return
all_tags = []
for loaded in networks.loaded_networks:
page = [en for en in shared.extra_networks if en.name == 'lora'][0]
item = page.create_item(loaded.name)
tags = (item or {}).get("tags", {})
loaded.tags = list(tags)
if len(loaded.tags) == 0:
loaded.tags.append(loaded.name)
if shared.opts.lora_apply_tags > 0:
loaded.tags = loaded.tags[:shared.opts.lora_apply_tags]
all_tags.extend(loaded.tags)
if len(all_tags) > 0:
all_tags = list(set(all_tags))
all_tags = [t for t in all_tags if t not in p.prompt]
shared.log.debug(f"Load network: type=LoRA tags={all_tags} max={shared.opts.lora_apply_tags} apply")
all_tags = ', '.join(all_tags)
p.extra_generation_params["LoRA tags"] = all_tags
if '_tags_' in p.prompt:
p.prompt = p.prompt.replace('_tags_', all_tags)
else:
p.prompt = f"{p.prompt}, {all_tags}"
if p.all_prompts is not None:
for i in range(len(p.all_prompts)):
if '_tags_' in p.all_prompts[i]:
p.all_prompts[i] = p.all_prompts[i].replace('_tags_', all_tags)
else:
p.all_prompts[i] = f"{p.all_prompts[i]}, {all_tags}"
def infotext(p):
names = [i.name for i in networks.loaded_networks]
if len(names) > 0:
p.extra_generation_params["LoRA networks"] = ", ".join(names)
if shared.opts.lora_add_hashes_to_infotext:
network_hashes = []
for item in networks.loaded_networks:
if not item.network_on_disk.shorthash:
continue
network_hashes.append(item.network_on_disk.shorthash)
if len(network_hashes) > 0:
p.extra_generation_params["LoRA hashes"] = ", ".join(network_hashes)
def parse(p, params_list, step=0):
names = []
te_multipliers = []
unet_multipliers = []
dyn_dims = []
for params in params_list:
assert params.items
names.append(params.positional[0])
te_multiplier = params.named.get("te", params.positional[1] if len(params.positional) > 1 else shared.opts.extra_networks_default_multiplier)
if isinstance(te_multiplier, str) and "@" in te_multiplier:
te_multiplier = get_stepwise(te_multiplier, step, p.steps)
else:
te_multiplier = float(te_multiplier)
unet_multiplier = [params.positional[2] if len(params.positional) > 2 else te_multiplier] * 3
unet_multiplier = [params.named.get("unet", unet_multiplier[0])] * 3
unet_multiplier[0] = params.named.get("in", unet_multiplier[0])
unet_multiplier[1] = params.named.get("mid", unet_multiplier[1])
unet_multiplier[2] = params.named.get("out", unet_multiplier[2])
for i in range(len(unet_multiplier)):
if isinstance(unet_multiplier[i], str) and "@" in unet_multiplier[i]:
unet_multiplier[i] = get_stepwise(unet_multiplier[i], step, p.steps)
else:
unet_multiplier[i] = float(unet_multiplier[i])
dyn_dim = int(params.positional[3]) if len(params.positional) > 3 else None
dyn_dim = int(params.named["dyn"]) if "dyn" in params.named else dyn_dim
te_multipliers.append(te_multiplier)
unet_multipliers.append(unet_multiplier)
dyn_dims.append(dyn_dim)
return names, te_multipliers, unet_multipliers, dyn_dims
class ExtraNetworkLora(extra_networks.ExtraNetwork):
def __init__(self):
super().__init__('lora')
self.active = False
self.model = None
self.errors = {}
def activate(self, p, params_list, step=0, include=[], exclude=[]):
self.errors.clear()
if self.active:
if self.model != shared.opts.sd_model_checkpoint: # reset if model changed
self.active = False
if len(params_list) > 0 and not self.active: # activate patches once
# shared.log.debug(f'Activate network: type=LoRA model="{shared.opts.sd_model_checkpoint}"')
self.active = True
self.model = shared.opts.sd_model_checkpoint
if 'text_encoder' in include:
networks.timer.clear(complete=True)
names, te_multipliers, unet_multipliers, dyn_dims = parse(p, params_list, step)
networks.network_load(names, te_multipliers, unet_multipliers, dyn_dims) # load
networks.network_activate(include, exclude)
if len(networks.loaded_networks) > 0 and len(networks.applied_layers) > 0 and step == 0:
infotext(p)
prompt(p)
shared.log.info(f'Load network: type=LoRA apply={[n.name for n in networks.loaded_networks]} mode={"fuse" if shared.opts.lora_fuse_diffusers else "backup"} te={te_multipliers} unet={unet_multipliers} time={networks.timer.summary}')
def deactivate(self, p):
if shared.native and len(networks.diffuser_loaded) > 0:
if hasattr(shared.sd_model, "unload_lora_weights") and hasattr(shared.sd_model, "text_encoder"):
if not (shared.compiled_model_state is not None and shared.compiled_model_state.is_compiled is True):
try:
if shared.opts.lora_fuse_diffusers:
shared.sd_model.unfuse_lora()
shared.sd_model.unload_lora_weights() # fails for non-CLIP models
except Exception:
pass
networks.network_deactivate()
if self.active and networks.debug:
shared.log.debug(f"Network end: type=LoRA time={networks.timer.summary}")
if self.errors:
for k, v in self.errors.items():
shared.log.error(f'LoRA: name="{k}" errors={v}')
self.errors.clear()
modules/lora/lora_convert.py
+509
View File
@@ -0,0 +1,509 @@
import os
import re
import bisect
from typing import Dict
import torch
from modules import shared
debug = os.environ.get('SD_LORA_DEBUG', None) is not None
suffix_conversion = {
"attentions": {},
"resnets": {
"conv1": "in_layers_2",
"conv2": "out_layers_3",
"norm1": "in_layers_0",
"norm2": "out_layers_0",
"time_emb_proj": "emb_layers_1",
"conv_shortcut": "skip_connection",
}
}
re_digits = re.compile(r"\d+")
re_x_proj = re.compile(r"(.*)_([qkv]_proj)$")
re_compiled = {}
def make_unet_conversion_map() -> Dict[str, str]:
unet_conversion_map_layer = []
for i in range(3): # num_blocks is 3 in sdxl
# loop over downblocks/upblocks
for j in range(2):
# loop over resnets/attentions for downblocks
hf_down_res_prefix = f"down_blocks.{i}.resnets.{j}."
sd_down_res_prefix = f"input_blocks.{3 * i + j + 1}.0."
unet_conversion_map_layer.append((sd_down_res_prefix, hf_down_res_prefix))
if i < 3:
# no attention layers in down_blocks.3
hf_down_atn_prefix = f"down_blocks.{i}.attentions.{j}."
sd_down_atn_prefix = f"input_blocks.{3 * i + j + 1}.1."
unet_conversion_map_layer.append((sd_down_atn_prefix, hf_down_atn_prefix))
for j in range(3):
# loop over resnets/attentions for upblocks
hf_up_res_prefix = f"up_blocks.{i}.resnets.{j}."
sd_up_res_prefix = f"output_blocks.{3 * i + j}.0."
unet_conversion_map_layer.append((sd_up_res_prefix, hf_up_res_prefix))
# if i > 0: commentout for sdxl
# no attention layers in up_blocks.0
hf_up_atn_prefix = f"up_blocks.{i}.attentions.{j}."
sd_up_atn_prefix = f"output_blocks.{3 * i + j}.1."
unet_conversion_map_layer.append((sd_up_atn_prefix, hf_up_atn_prefix))
if i < 3:
# no downsample in down_blocks.3
hf_downsample_prefix = f"down_blocks.{i}.downsamplers.0.conv."
sd_downsample_prefix = f"input_blocks.{3 * (i + 1)}.0.op."
unet_conversion_map_layer.append((sd_downsample_prefix, hf_downsample_prefix))
# no upsample in up_blocks.3
hf_upsample_prefix = f"up_blocks.{i}.upsamplers.0."
sd_upsample_prefix = f"output_blocks.{3 * i + 2}.{2}." # change for sdxl
unet_conversion_map_layer.append((sd_upsample_prefix, hf_upsample_prefix))
hf_mid_atn_prefix = "mid_block.attentions.0."
sd_mid_atn_prefix = "middle_block.1."
unet_conversion_map_layer.append((sd_mid_atn_prefix, hf_mid_atn_prefix))
for j in range(2):
hf_mid_res_prefix = f"mid_block.resnets.{j}."
sd_mid_res_prefix = f"middle_block.{2 * j}."
unet_conversion_map_layer.append((sd_mid_res_prefix, hf_mid_res_prefix))
unet_conversion_map_resnet = [
# (stable-diffusion, HF Diffusers)
("in_layers.0.", "norm1."),
("in_layers.2.", "conv1."),
("out_layers.0.", "norm2."),
("out_layers.3.", "conv2."),
("emb_layers.1.", "time_emb_proj."),
("skip_connection.", "conv_shortcut."),
]
unet_conversion_map = []
for sd, hf in unet_conversion_map_layer:
if "resnets" in hf:
for sd_res, hf_res in unet_conversion_map_resnet:
unet_conversion_map.append((sd + sd_res, hf + hf_res))
else:
unet_conversion_map.append((sd, hf))
for j in range(2):
hf_time_embed_prefix = f"time_embedding.linear_{j + 1}."
sd_time_embed_prefix = f"time_embed.{j * 2}."
unet_conversion_map.append((sd_time_embed_prefix, hf_time_embed_prefix))
for j in range(2):
hf_label_embed_prefix = f"add_embedding.linear_{j + 1}."
sd_label_embed_prefix = f"label_emb.0.{j * 2}."
unet_conversion_map.append((sd_label_embed_prefix, hf_label_embed_prefix))
unet_conversion_map.append(("input_blocks.0.0.", "conv_in."))
unet_conversion_map.append(("out.0.", "conv_norm_out."))
unet_conversion_map.append(("out.2.", "conv_out."))
sd_hf_conversion_map = {sd.replace(".", "_")[:-1]: hf.replace(".", "_")[:-1] for sd, hf in unet_conversion_map}
return sd_hf_conversion_map
class KeyConvert:
def __init__(self):
self.is_sdxl = True if shared.sd_model_type == "sdxl" else False
self.UNET_CONVERSION_MAP = make_unet_conversion_map() if self.is_sdxl else None
self.LORA_PREFIX_UNET = "lora_unet_"
self.LORA_PREFIX_TEXT_ENCODER = "lora_te_"
self.OFT_PREFIX_UNET = "oft_unet_"
# SDXL: must starts with LORA_PREFIX_TEXT_ENCODER
self.LORA_PREFIX_TEXT_ENCODER1 = "lora_te1_"
self.LORA_PREFIX_TEXT_ENCODER2 = "lora_te2_"
def __call__(self, key):
if self.is_sdxl:
if "diffusion_model" in key: # Fix NTC Slider naming error
key = key.replace("diffusion_model", "lora_unet")
map_keys = list(self.UNET_CONVERSION_MAP.keys()) # prefix of U-Net modules
map_keys.sort()
search_key = key.replace(self.LORA_PREFIX_UNET, "").replace(self.OFT_PREFIX_UNET, "").replace(self.LORA_PREFIX_TEXT_ENCODER1, "").replace(self.LORA_PREFIX_TEXT_ENCODER2, "")
position = bisect.bisect_right(map_keys, search_key)
map_key = map_keys[position - 1]
if search_key.startswith(map_key):
key = key.replace(map_key, self.UNET_CONVERSION_MAP[map_key]).replace("oft", "lora") # pylint: disable=unsubscriptable-object
if "lycoris" in key and "transformer" in key:
key = key.replace("lycoris", "lora_transformer")
sd_module = shared.sd_model.network_layer_mapping.get(key, None)
if sd_module is None:
sd_module = shared.sd_model.network_layer_mapping.get(key.replace("guidance", "timestep"), None) # FLUX1 fix
if debug and sd_module is None:
raise RuntimeError(f"LoRA key not found in network_layer_mapping: key={key} mapping={shared.sd_model.network_layer_mapping.keys()}")
return key, sd_module
# Taken from https://github.com/huggingface/diffusers/blob/main/src/diffusers/loaders/lora_conversion_utils.py
# Modified from 'lora_A' and 'lora_B' to 'lora_down' and 'lora_up'
# Added early exit
# The utilities under `_convert_kohya_flux_lora_to_diffusers()`
# are taken from https://github.com/kohya-ss/sd-scripts/blob/a61cf73a5cb5209c3f4d1a3688dd276a4dfd1ecb/networks/convert_flux_lora.py
# All credits go to `kohya-ss`.
def _convert_to_ai_toolkit(sds_sd, ait_sd, sds_key, ait_key):
if sds_key + ".lora_down.weight" not in sds_sd:
return
down_weight = sds_sd.pop(sds_key + ".lora_down.weight")
# scale weight by alpha and dim
rank = down_weight.shape[0]
alpha = sds_sd.pop(sds_key + ".alpha").item() # alpha is scalar
scale = alpha / rank # LoRA is scaled by 'alpha / rank' in forward pass, so we need to scale it back here
# calculate scale_down and scale_up to keep the same value. if scale is 4, scale_down is 2 and scale_up is 2
scale_down = scale
scale_up = 1.0
while scale_down * 2 < scale_up:
scale_down *= 2
scale_up /= 2
ait_sd[ait_key + ".lora_down.weight"] = down_weight * scale_down
ait_sd[ait_key + ".lora_up.weight"] = sds_sd.pop(sds_key + ".lora_up.weight") * scale_up
def _convert_to_ai_toolkit_cat(sds_sd, ait_sd, sds_key, ait_keys, dims=None):
if sds_key + ".lora_down.weight" not in sds_sd:
return
down_weight = sds_sd.pop(sds_key + ".lora_down.weight")
up_weight = sds_sd.pop(sds_key + ".lora_up.weight")
sd_lora_rank = down_weight.shape[0]
# scale weight by alpha and dim
alpha = sds_sd.pop(sds_key + ".alpha")
scale = alpha / sd_lora_rank
# calculate scale_down and scale_up
scale_down = scale
scale_up = 1.0
while scale_down * 2 < scale_up:
scale_down *= 2
scale_up /= 2
down_weight = down_weight * scale_down
up_weight = up_weight * scale_up
# calculate dims if not provided
num_splits = len(ait_keys)
if dims is None:
dims = [up_weight.shape[0] // num_splits] * num_splits
else:
assert sum(dims) == up_weight.shape[0]
# check upweight is sparse or not
is_sparse = False
if sd_lora_rank % num_splits == 0:
ait_rank = sd_lora_rank // num_splits
is_sparse = True
i = 0
for j in range(len(dims)):
for k in range(len(dims)):
if j == k:
continue
is_sparse = is_sparse and torch.all(
up_weight[i : i + dims[j], k * ait_rank : (k + 1) * ait_rank] == 0
)
i += dims[j]
# if is_sparse:
# print(f"weight is sparse: {sds_key}")
# make ai-toolkit weight
ait_down_keys = [k + ".lora_down.weight" for k in ait_keys]
ait_up_keys = [k + ".lora_up.weight" for k in ait_keys]
if not is_sparse:
# down_weight is copied to each split
ait_sd.update({k: down_weight for k in ait_down_keys})
# up_weight is split to each split
ait_sd.update({k: v for k, v in zip(ait_up_keys, torch.split(up_weight, dims, dim=0))}) # noqa: C416 # pylint: disable=unnecessary-comprehension
else:
# down_weight is chunked to each split
ait_sd.update({k: v for k, v in zip(ait_down_keys, torch.chunk(down_weight, num_splits, dim=0))}) # noqa: C416 # pylint: disable=unnecessary-comprehension
# up_weight is sparse: only non-zero values are copied to each split
i = 0
for j in range(len(dims)):
ait_sd[ait_up_keys[j]] = up_weight[i : i + dims[j], j * ait_rank : (j + 1) * ait_rank].contiguous()
i += dims[j]
def _convert_text_encoder_lora_key(key, lora_name):
"""
Converts a text encoder LoRA key to a Diffusers compatible key.
"""
if lora_name.startswith(("lora_te_", "lora_te1_")):
key_to_replace = "lora_te_" if lora_name.startswith("lora_te_") else "lora_te1_"
else:
key_to_replace = "lora_te2_"
diffusers_name = key.replace(key_to_replace, "").replace("_", ".")
diffusers_name = diffusers_name.replace("text.model", "text_model")
diffusers_name = diffusers_name.replace("self.attn", "self_attn")
diffusers_name = diffusers_name.replace("q.proj.lora", "to_q_lora")
diffusers_name = diffusers_name.replace("k.proj.lora", "to_k_lora")
diffusers_name = diffusers_name.replace("v.proj.lora", "to_v_lora")
diffusers_name = diffusers_name.replace("out.proj.lora", "to_out_lora")
diffusers_name = diffusers_name.replace("text.projection", "text_projection")
if "self_attn" in diffusers_name or "text_projection" in diffusers_name:
pass
elif "mlp" in diffusers_name:
# Be aware that this is the new diffusers convention and the rest of the code might
# not utilize it yet.
diffusers_name = diffusers_name.replace(".lora.", ".lora_linear_layer.")
return diffusers_name
def _convert_kohya_flux_lora_to_diffusers(state_dict):
def _convert_sd_scripts_to_ai_toolkit(sds_sd):
ait_sd = {}
for i in range(19):
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_double_blocks_{i}_img_attn_proj",
f"transformer.transformer_blocks.{i}.attn.to_out.0",
)
_convert_to_ai_toolkit_cat(
sds_sd,
ait_sd,
f"lora_unet_double_blocks_{i}_img_attn_qkv",
[
f"transformer.transformer_blocks.{i}.attn.to_q",
f"transformer.transformer_blocks.{i}.attn.to_k",
f"transformer.transformer_blocks.{i}.attn.to_v",
],
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_double_blocks_{i}_img_mlp_0",
f"transformer.transformer_blocks.{i}.ff.net.0.proj",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_double_blocks_{i}_img_mlp_2",
f"transformer.transformer_blocks.{i}.ff.net.2",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_double_blocks_{i}_img_mod_lin",
f"transformer.transformer_blocks.{i}.norm1.linear",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_double_blocks_{i}_txt_attn_proj",
f"transformer.transformer_blocks.{i}.attn.to_add_out",
)
_convert_to_ai_toolkit_cat(
sds_sd,
ait_sd,
f"lora_unet_double_blocks_{i}_txt_attn_qkv",
[
f"transformer.transformer_blocks.{i}.attn.add_q_proj",
f"transformer.transformer_blocks.{i}.attn.add_k_proj",
f"transformer.transformer_blocks.{i}.attn.add_v_proj",
],
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_double_blocks_{i}_txt_mlp_0",
f"transformer.transformer_blocks.{i}.ff_context.net.0.proj",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_double_blocks_{i}_txt_mlp_2",
f"transformer.transformer_blocks.{i}.ff_context.net.2",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_double_blocks_{i}_txt_mod_lin",
f"transformer.transformer_blocks.{i}.norm1_context.linear",
)
for i in range(38):
_convert_to_ai_toolkit_cat(
sds_sd,
ait_sd,
f"lora_unet_single_blocks_{i}_linear1",
[
f"transformer.single_transformer_blocks.{i}.attn.to_q",
f"transformer.single_transformer_blocks.{i}.attn.to_k",
f"transformer.single_transformer_blocks.{i}.attn.to_v",
f"transformer.single_transformer_blocks.{i}.proj_mlp",
],
dims=[3072, 3072, 3072, 12288],
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_single_blocks_{i}_linear2",
f"transformer.single_transformer_blocks.{i}.proj_out",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_single_blocks_{i}_modulation_lin",
f"transformer.single_transformer_blocks.{i}.norm.linear",
)
if len(sds_sd) > 0:
return None
return ait_sd
return _convert_sd_scripts_to_ai_toolkit(state_dict)
def _convert_kohya_sd3_lora_to_diffusers(state_dict):
def _convert_sd_scripts_to_ai_toolkit(sds_sd):
ait_sd = {}
for i in range(38):
_convert_to_ai_toolkit_cat(
sds_sd,
ait_sd,
f"lora_unet_joint_blocks_{i}_context_block_attn_qkv",
[
f"transformer.transformer_blocks.{i}.attn.to_q",
f"transformer.transformer_blocks.{i}.attn.to_k",
f"transformer.transformer_blocks.{i}.attn.to_v",
],
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_joint_blocks_{i}_context_block_mlp_fc1",
f"transformer.transformer_blocks.{i}.ff_context.net.0.proj",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_joint_blocks_{i}_context_block_mlp_fc2",
f"transformer.transformer_blocks.{i}.ff_context.net.2",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_joint_blocks_{i}_x_block_mlp_fc1",
f"transformer.transformer_blocks.{i}.ff.net.0.proj",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_joint_blocks_{i}_x_block_mlp_fc2",
f"transformer.transformer_blocks.{i}.ff.net.2",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_joint_blocks_{i}_context_block_adaLN_modulation_1",
f"transformer.transformer_blocks.{i}.norm1_context.linear",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_joint_blocks_{i}_x_block_adaLN_modulation_1",
f"transformer.transformer_blocks.{i}.norm1.linear",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_joint_blocks_{i}_context_block_attn_proj",
f"transformer.transformer_blocks.{i}.attn.to_add_out",
)
_convert_to_ai_toolkit(
sds_sd,
ait_sd,
f"lora_unet_joint_blocks_{i}_x_block_attn_proj",
f"transformer.transformer_blocks.{i}.attn.to_out_0",
)
_convert_to_ai_toolkit_cat(
sds_sd,
ait_sd,
f"lora_unet_joint_blocks_{i}_x_block_attn_qkv",
[
f"transformer.transformer_blocks.{i}.attn.add_q_proj",
f"transformer.transformer_blocks.{i}.attn.add_k_proj",
f"transformer.transformer_blocks.{i}.attn.add_v_proj",
],
)
remaining_keys = list(sds_sd.keys())
te_state_dict = {}
if remaining_keys:
if not all(k.startswith("lora_te1") for k in remaining_keys):
raise ValueError(f"Incompatible keys detected: \n\n {', '.join(remaining_keys)}")
for key in remaining_keys:
if not key.endswith("lora_down.weight"):
continue
lora_name = key.split(".")[0]
lora_name_up = f"{lora_name}.lora_up.weight"
lora_name_alpha = f"{lora_name}.alpha"
diffusers_name = _convert_text_encoder_lora_key(key, lora_name)
sd_lora_rank = 1
if lora_name.startswith(("lora_te_", "lora_te1_")):
down_weight = sds_sd.pop(key)
sd_lora_rank = down_weight.shape[0]
te_state_dict[diffusers_name] = down_weight
te_state_dict[diffusers_name.replace(".down.", ".up.")] = sds_sd.pop(lora_name_up)
if lora_name_alpha in sds_sd:
alpha = sds_sd.pop(lora_name_alpha).item()
scale = alpha / sd_lora_rank
scale_down = scale
scale_up = 1.0
while scale_down * 2 < scale_up:
scale_down *= 2
scale_up /= 2
te_state_dict[diffusers_name] *= scale_down
te_state_dict[diffusers_name.replace(".down.", ".up.")] *= scale_up
if len(sds_sd) > 0:
print(f"Unsupported keys for ai-toolkit: {sds_sd.keys()}")
if te_state_dict:
te_state_dict = {f"text_encoder.{module_name}": params for module_name, params in te_state_dict.items()}
new_state_dict = {**ait_sd, **te_state_dict}
return new_state_dict
return _convert_sd_scripts_to_ai_toolkit(state_dict)
def assign_network_names_to_compvis_modules(sd_model):
if sd_model is None:
return
sd_model = getattr(shared.sd_model, "pipe", shared.sd_model) # wrapped model compatiblility
network_layer_mapping = {}
if hasattr(sd_model, 'text_encoder') and sd_model.text_encoder is not None:
for name, module in sd_model.text_encoder.named_modules():
prefix = "lora_te1_" if hasattr(sd_model, 'text_encoder_2') else "lora_te_"
network_name = prefix + name.replace(".", "_")
network_layer_mapping[network_name] = module
module.network_layer_name = network_name
if hasattr(sd_model, 'text_encoder_2'):
for name, module in sd_model.text_encoder_2.named_modules():
network_name = "lora_te2_" + name.replace(".", "_")
network_layer_mapping[network_name] = module
module.network_layer_name = network_name
if hasattr(sd_model, 'unet'):
for name, module in sd_model.unet.named_modules():
network_name = "lora_unet_" + name.replace(".", "_")
network_layer_mapping[network_name] = module
module.network_layer_name = network_name
if hasattr(sd_model, 'transformer'):
for name, module in sd_model.transformer.named_modules():
network_name = "lora_transformer_" + name.replace(".", "_")
network_layer_mapping[network_name] = module
if "norm" in network_name and "linear" not in network_name and shared.sd_model_type != "sd3":
continue
module.network_layer_name = network_name
shared.sd_model.network_layer_mapping = network_layer_mapping
modules/lora/lora_extract.py
+271
View File
@@ -0,0 +1,271 @@
import os
import time
import json
import datetime
import torch
from safetensors.torch import save_file
import gradio as gr
from rich import progress as p
from modules import shared, devices
from modules.ui_common import create_refresh_button
from modules.call_queue import wrap_gradio_gpu_call
class SVDHandler:
def __init__(self, maxrank=0, rank_ratio=1):
self.network_name: str = None
self.U: torch.Tensor = None
self.S: torch.Tensor = None
self.Vh: torch.Tensor = None
self.maxrank: int = maxrank
self.rank_ratio: float = rank_ratio
self.rank: int = 0
self.out_size: int = None
self.in_size: int = None
self.kernel_size: tuple[int, int] = None
self.conv2d: bool = False
def decompose(self, weight, backupweight):
self.conv2d = len(weight.size()) == 4
self.kernel_size = None if not self.conv2d else weight.size()[2:4]
self.out_size, self.in_size = weight.size()[0:2]
diffweight = weight.clone().to(devices.device)
diffweight -= backupweight.to(devices.device)
if self.conv2d:
if self.conv2d and self.kernel_size != (1, 1):
diffweight = diffweight.flatten(start_dim=1)
else:
diffweight = diffweight.squeeze()
self.U, self.S, self.Vh = torch.svd_lowrank(diffweight.to(device=devices.device, dtype=torch.float), self.maxrank, 2)
# del diffweight
self.U = self.U.to(device=devices.cpu, dtype=torch.bfloat16)
self.S = self.S.to(device=devices.cpu, dtype=torch.bfloat16)
self.Vh = self.Vh.t().to(device=devices.cpu, dtype=torch.bfloat16) # svd_lowrank outputs a transposed matrix
def findrank(self):
if self.rank_ratio < 1:
S_squared = self.S.pow(2)
S_fro_sq = float(torch.sum(S_squared))
sum_S_squared = torch.cumsum(S_squared, dim=0) / S_fro_sq
index = int(torch.searchsorted(sum_S_squared, self.rank_ratio ** 2)) + 1
index = max(1, min(index, len(self.S) - 1))
self.rank = index
if self.maxrank > 0:
self.rank = min(self.rank, self.maxrank)
else:
self.rank = min(self.in_size, self.out_size, self.maxrank)
def makeweights(self):
self.findrank()
up = self.U[:, :self.rank] @ torch.diag(self.S[:self.rank])
down = self.Vh[:self.rank, :]
if self.conv2d and self.kernel_size is not None:
up = up.reshape(self.out_size, self.rank, 1, 1)
down = down.reshape(self.rank, self.in_size, self.kernel_size[0], self.kernel_size[1]) # pylint: disable=unsubscriptable-object
return_dict = {f'{self.network_name}.lora_up.weight': up.contiguous(),
f'{self.network_name}.lora_down.weight': down.contiguous(),
f'{self.network_name}.alpha': torch.tensor(down.shape[0]),
}
return return_dict
def loaded_lora():
if not shared.sd_loaded:
return ""
loaded = set()
if hasattr(shared.sd_model, 'unet'):
for _name, module in shared.sd_model.unet.named_modules():
current = getattr(module, "network_current_names", None)
if current is not None:
current = [item[0] for item in current]
loaded.update(current)
return list(loaded)
def loaded_lora_str():
return ", ".join(loaded_lora())
def make_meta(fn, maxrank, rank_ratio):
meta = {
"model_spec.sai_model_spec": "1.0.0",
"model_spec.title": os.path.splitext(os.path.basename(fn))[0],
"model_spec.author": "SD.Next",
"model_spec.implementation": "https://github.com/vladmandic/automatic",
"model_spec.date": datetime.datetime.now().astimezone().replace(microsecond=0).isoformat(),
"model_spec.base_model": shared.opts.sd_model_checkpoint,
"model_spec.dtype": str(devices.dtype),
"model_spec.base_lora": json.dumps(loaded_lora()),
"model_spec.config": f"maxrank={maxrank} rank_ratio={rank_ratio}",
}
if shared.sd_model_type == "sdxl":
meta["model_spec.architecture"] = "stable-diffusion-xl-v1-base/lora" # sai standard
meta["ss_base_model_version"] = "sdxl_base_v1-0" # kohya standard
elif shared.sd_model_type == "sd":
meta["model_spec.architecture"] = "stable-diffusion-v1/lora"
meta["ss_base_model_version"] = "sd_v1"
elif shared.sd_model_type == "f1":
meta["model_spec.architecture"] = "flux-1-dev/lora"
meta["ss_base_model_version"] = "flux1"
elif shared.sd_model_type == "sc":
meta["model_spec.architecture"] = "stable-cascade-v1-prior/lora"
return meta
def make_lora(fn, maxrank, auto_rank, rank_ratio, modules, overwrite):
if not shared.sd_loaded or not shared.native:
msg = "LoRA extract: model not loaded"
shared.log.warning(msg)
yield msg
return
if loaded_lora() == "":
msg = "LoRA extract: no LoRA detected"
shared.log.warning(msg)
yield msg
return
if not fn:
msg = "LoRA extract: target filename required"
shared.log.warning(msg)
yield msg
return
t0 = time.time()
maxrank = int(maxrank)
rank_ratio = 1 if not auto_rank else rank_ratio
shared.log.debug(f'LoRA extract: modules={modules} maxrank={maxrank} auto={auto_rank} ratio={rank_ratio} fn="{fn}"')
shared.state.begin('LoRA extract')
with p.Progress(p.TextColumn('[cyan]LoRA extract'), p.BarColumn(), p.TaskProgressColumn(), p.TimeRemainingColumn(), p.TimeElapsedColumn(), p.TextColumn('[cyan]{task.description}'), console=shared.console) as progress:
if 'te' in modules and getattr(shared.sd_model, 'text_encoder', None) is not None:
modules = shared.sd_model.text_encoder.named_modules()
task = progress.add_task(description="te1 decompose", total=len(list(modules)))
for name, module in shared.sd_model.text_encoder.named_modules():
progress.update(task, advance=1)
weights_backup = getattr(module, "network_weights_backup", None)
if weights_backup is None or getattr(module, "network_current_names", None) is None:
continue
prefix = "lora_te1_" if hasattr(shared.sd_model, 'text_encoder_2') else "lora_te_"
module.svdhandler = SVDHandler(maxrank, rank_ratio)
module.svdhandler.network_name = prefix + name.replace(".", "_")
with devices.inference_context():
module.svdhandler.decompose(module.weight, weights_backup)
progress.remove_task(task)
t1 = time.time()
if 'te' in modules and getattr(shared.sd_model, 'text_encoder_2', None) is not None:
modules = shared.sd_model.text_encoder_2.named_modules()
task = progress.add_task(description="te2 decompose", total=len(list(modules)))
for name, module in shared.sd_model.text_encoder_2.named_modules():
progress.update(task, advance=1)
weights_backup = getattr(module, "network_weights_backup", None)
if weights_backup is None or getattr(module, "network_current_names", None) is None:
continue
module.svdhandler = SVDHandler(maxrank, rank_ratio)
module.svdhandler.network_name = "lora_te2_" + name.replace(".", "_")
with devices.inference_context():
module.svdhandler.decompose(module.weight, weights_backup)
progress.remove_task(task)
t2 = time.time()
if 'unet' in modules and getattr(shared.sd_model, 'unet', None) is not None:
modules = shared.sd_model.unet.named_modules()
task = progress.add_task(description="unet decompose", total=len(list(modules)))
for name, module in shared.sd_model.unet.named_modules():
progress.update(task, advance=1)
weights_backup = getattr(module, "network_weights_backup", None)
if weights_backup is None or getattr(module, "network_current_names", None) is None:
continue
module.svdhandler = SVDHandler(maxrank, rank_ratio)
module.svdhandler.network_name = "lora_unet_" + name.replace(".", "_")
with devices.inference_context():
module.svdhandler.decompose(module.weight, weights_backup)
progress.remove_task(task)
t3 = time.time()
# TODO: lora make support quantized flux
# if 'te' in modules and getattr(shared.sd_model, 'transformer', None) is not None:
# for name, module in shared.sd_model.transformer.named_modules():
# if "norm" in name and "linear" not in name:
# continue
# weights_backup = getattr(module, "network_weights_backup", None)
# if weights_backup is None:
# continue
# module.svdhandler = SVDHandler()
# module.svdhandler.network_name = "lora_transformer_" + name.replace(".", "_")
# module.svdhandler.decompose(module.weight, weights_backup)
# module.svdhandler.findrank(rank, rank_ratio)
lora_state_dict = {}
for sub in ['text_encoder', 'text_encoder_2', 'unet', 'transformer']:
submodel = getattr(shared.sd_model, sub, None)
if submodel is not None:
modules = submodel.named_modules()
task = progress.add_task(description=f"{sub} exctract", total=len(list(modules)))
for _name, module in submodel.named_modules():
progress.update(task, advance=1)
if not hasattr(module, "svdhandler"):
continue
lora_state_dict.update(module.svdhandler.makeweights())
del module.svdhandler
progress.remove_task(task)
t4 = time.time()
if not os.path.isabs(fn):
fn = os.path.join(shared.cmd_opts.lora_dir, fn)
if not fn.endswith('.safetensors'):
fn += '.safetensors'
if os.path.exists(fn):
if overwrite:
os.remove(fn)
else:
msg = f'LoRA extract: fn="{fn}" file exists'
shared.log.warning(msg)
yield msg
return
shared.state.end()
meta = make_meta(fn, maxrank, rank_ratio)
shared.log.debug(f'LoRA metadata: {meta}')
try:
save_file(tensors=lora_state_dict, metadata=meta, filename=fn)
except Exception as e:
msg = f'LoRA extract error: fn="{fn}" {e}'
shared.log.error(msg)
yield msg
return
t5 = time.time()
shared.log.debug(f'LoRA extract: time={t5-t0:.2f} te1={t1-t0:.2f} te2={t2-t1:.2f} unet={t3-t2:.2f} save={t5-t4:.2f}')
keys = list(lora_state_dict.keys())
msg = f'LoRA extract: fn="{fn}" keys={len(keys)}'
shared.log.info(msg)
yield msg
def create_ui():
def gr_show(visible=True):
return {"visible": visible, "__type__": "update"}
with gr.Tab(label="Extract LoRA"):
with gr.Row():
loaded = gr.Textbox(placeholder="Press refresh to query loaded LoRA", label="Loaded LoRA", interactive=False)
create_refresh_button(loaded, lambda: None, lambda: {'value': loaded_lora_str()}, "testid")
with gr.Group():
with gr.Row():
modules = gr.CheckboxGroup(label="Modules to extract", value=['unet'], choices=['te', 'unet'])
with gr.Row():
auto_rank = gr.Checkbox(value=False, label="Automatically determine rank")
rank_ratio = gr.Slider(label="Autorank ratio", value=1, minimum=0, maximum=1, step=0.05, visible=False)
rank = gr.Slider(label="Maximum rank", value=32, minimum=1, maximum=256)
with gr.Row():
filename = gr.Textbox(label="LoRA target filename")
overwrite = gr.Checkbox(value=False, label="Overwrite existing file")
with gr.Row():
extract = gr.Button(value="Extract LoRA", variant='primary')
status = gr.HTML(value="", show_label=False)
auto_rank.change(fn=lambda x: gr_show(x), inputs=[auto_rank], outputs=[rank_ratio])
extract.click(
fn=wrap_gradio_gpu_call(make_lora, extra_outputs=[]),
inputs=[filename, rank, auto_rank, rank_ratio, modules, overwrite],
outputs=[status]
)
modules/lora/lora_timers.py
+38
View File
@@ -0,0 +1,38 @@
class Timer():
list: float = 0
load: float = 0
backup: float = 0
calc: float = 0
apply: float = 0
move: float = 0
restore: float = 0
activate: float = 0
deactivate: float = 0
@property
def total(self):
return round(self.activate + self.deactivate, 2)
@property
def summary(self):
t = {}
for k, v in self.__dict__.items():
if v > 0.1:
t[k] = round(v, 2)
return t
def clear(self, complete: bool = False):
self.backup = 0
self.calc = 0
self.apply = 0
self.move = 0
self.restore = 0
if complete:
self.activate = 0
self.deactivate = 0
def add(self, name, t):
self.__dict__[name] += t
def __str__(self):
return f'{self.__class__.__name__}({self.summary})'
modules/lora/lyco_helpers.py
+66
View File
@@ -0,0 +1,66 @@
import torch
def make_weight_cp(t, wa, wb):
temp = torch.einsum('i j k l, j r -> i r k l', t, wb)
return torch.einsum('i j k l, i r -> r j k l', temp, wa)
def rebuild_conventional(up, down, shape, dyn_dim=None):
up = up.reshape(up.size(0), -1)
down = down.reshape(down.size(0), -1)
if dyn_dim is not None:
up = up[:, :dyn_dim]
down = down[:dyn_dim, :]
return (up @ down).reshape(shape).to(up.dtype)
def rebuild_cp_decomposition(up, down, mid):
up = up.reshape(up.size(0), -1)
down = down.reshape(down.size(0), -1)
return torch.einsum('n m k l, i n, m j -> i j k l', mid, up, down).to(up.dtype)
# copied from https://github.com/KohakuBlueleaf/LyCORIS/blob/dev/lycoris/modules/lokr.py
def factorization(dimension: int, factor:int=-1) -> tuple[int, int]:
"""
return a tuple of two value of input dimension decomposed by the number closest to factor
second value is higher or equal than first value.
In LoRA with Kroneckor Product, first value is a value for weight scale.
secon value is a value for weight.
Becuase of non-commutative property, A⊗B ≠ B⊗A. Meaning of two matrices is slightly different.
examples
factor
-1 2 4 8 16 ...
127 -> 1, 127 127 -> 1, 127 127 -> 1, 127 127 -> 1, 127 127 -> 1, 127
128 -> 8, 16 128 -> 2, 64 128 -> 4, 32 128 -> 8, 16 128 -> 8, 16
250 -> 10, 25 250 -> 2, 125 250 -> 2, 125 250 -> 5, 50 250 -> 10, 25
360 -> 8, 45 360 -> 2, 180 360 -> 4, 90 360 -> 8, 45 360 -> 12, 30
512 -> 16, 32 512 -> 2, 256 512 -> 4, 128 512 -> 8, 64 512 -> 16, 32
1024 -> 32, 32 1024 -> 2, 512 1024 -> 4, 256 1024 -> 8, 128 1024 -> 16, 64
"""
if factor > 0 and (dimension % factor) == 0:
m = factor
n = dimension // factor
if m > n:
n, m = m, n
return m, n
if factor < 0:
factor = dimension
m, n = 1, dimension
length = m + n
while m<n:
new_m = m + 1
while dimension%new_m != 0:
new_m += 1
new_n = dimension // new_m
if new_m + new_n > length or new_m>factor:
break
m, n = new_m, new_n
if m > n:
n, m = m, n
return m, n
modules/lora/network.py
+188
View File
@@ -0,0 +1,188 @@
import os
import enum
from typing import Union
from collections import namedtuple
from modules import sd_models, hashes, shared
NetworkWeights = namedtuple('NetworkWeights', ['network_key', 'sd_key', 'w', 'sd_module'])
metadata_tags_order = {"ss_sd_model_name": 1, "ss_resolution": 2, "ss_clip_skip": 3, "ss_num_train_images": 10, "ss_tag_frequency": 20}
class SdVersion(enum.Enum):
Unknown = 1
SD1 = 2
SD2 = 3
SD3 = 3
SDXL = 4
SC = 5
F1 = 6
class NetworkOnDisk:
def __init__(self, name, filename):
self.shorthash = None
self.hash = None
self.name = name
self.filename = filename
if filename.startswith(shared.cmd_opts.lora_dir):
self.fullname = os.path.splitext(filename[len(shared.cmd_opts.lora_dir):].strip("/"))[0]
else:
self.fullname = name
self.metadata = {}
self.is_safetensors = os.path.splitext(filename)[1].lower() == ".safetensors"
if self.is_safetensors:
self.metadata = sd_models.read_metadata_from_safetensors(filename)
if self.metadata:
m = {}
for k, v in sorted(self.metadata.items(), key=lambda x: metadata_tags_order.get(x[0], 999)):
m[k] = v
self.metadata = m
self.alias = self.metadata.get('ss_output_name', self.name)
sha256 = hashes.sha256_from_cache(self.filename, "lora/" + self.name) or hashes.sha256_from_cache(self.filename, "lora/" + self.name, use_addnet_hash=True) or self.metadata.get('sshs_model_hash')
self.set_hash(sha256)
self.sd_version = self.detect_version()
def detect_version(self):
base = str(self.metadata.get('ss_base_model_version', "")).lower()
arch = str(self.metadata.get('modelspec.architecture', "")).lower()
if base.startswith("sd_v1"):
return 'sd1'
if base.startswith("sdxl"):
return 'xl'
if base.startswith("stable_cascade"):
return 'sc'
if base.startswith("sd3"):
return 'sd3'
if base.startswith("flux"):
return 'f1'
if arch.startswith("stable-diffusion-v1"):
return 'sd1'
if arch.startswith("stable-diffusion-xl"):
return 'xl'
if arch.startswith("stable-cascade"):
return 'sc'
if arch.startswith("flux"):
return 'f1'
if "v1-5" in str(self.metadata.get('ss_sd_model_name', "")):
return 'sd1'
if str(self.metadata.get('ss_v2', "")) == "True":
return 'sd2'
if 'flux' in self.name.lower():
return 'f1'
if 'xl' in self.name.lower():
return 'xl'
return ''
def set_hash(self, v):
self.hash = v or ''
self.shorthash = self.hash[0:8]
def read_hash(self):
if not self.hash:
self.set_hash(hashes.sha256(self.filename, "lora/" + self.name, use_addnet_hash=self.is_safetensors) or '')
def get_alias(self):
import modules.lora.networks as networks
return self.name if shared.opts.lora_preferred_name == "filename" or self.alias.lower() in networks.forbidden_network_aliases else self.alias
class Network: # LoraModule
def __init__(self, name, network_on_disk: NetworkOnDisk):
self.name = name
self.network_on_disk = network_on_disk
self.te_multiplier = 1.0
self.unet_multiplier = [1.0] * 3
self.dyn_dim = None
self.modules = {}
self.bundle_embeddings = {}
self.mtime = None
self.mentioned_name = None
self.tags = None
"""the text that was used to add the network to prompt - can be either name or an alias"""
class ModuleType:
def create_module(self, net: Network, weights: NetworkWeights) -> Union[Network, None]: # pylint: disable=W0613
return None
class NetworkModule:
def __init__(self, net: Network, weights: NetworkWeights):
self.network = net
self.network_key = weights.network_key
self.sd_key = weights.sd_key
self.sd_module = weights.sd_module
if hasattr(self.sd_module, 'weight'):
self.shape = self.sd_module.weight.shape
self.dim = None
self.bias = weights.w.get("bias")
self.alpha = weights.w["alpha"].item() if "alpha" in weights.w else None
self.scale = weights.w["scale"].item() if "scale" in weights.w else None
self.dora_scale = weights.w.get("dora_scale", None)
self.dora_norm_dims = len(self.shape) - 1
def multiplier(self):
unet_multiplier = 3 * [self.network.unet_multiplier] if not isinstance(self.network.unet_multiplier, list) else self.network.unet_multiplier
if 'transformer' in self.sd_key[:20]:
return self.network.te_multiplier
if "down_blocks" in self.sd_key:
return unet_multiplier[0]
if "mid_block" in self.sd_key:
return unet_multiplier[1]
if "up_blocks" in self.sd_key:
return unet_multiplier[2]
else:
return unet_multiplier[0]
def calc_scale(self):
if self.scale is not None:
return self.scale
if self.dim is not None and self.alpha is not None:
return self.alpha / self.dim
return 1.0
def apply_weight_decompose(self, updown, orig_weight):
# Match the device/dtype
orig_weight = orig_weight.to(updown.dtype)
dora_scale = self.dora_scale.to(device=orig_weight.device, dtype=updown.dtype)
updown = updown.to(orig_weight.device)
merged_scale1 = updown + orig_weight
merged_scale1_norm = (
merged_scale1.transpose(0, 1)
.reshape(merged_scale1.shape[1], -1)
.norm(dim=1, keepdim=True)
.reshape(merged_scale1.shape[1], *[1] * self.dora_norm_dims)
.transpose(0, 1)
)
dora_merged = (
merged_scale1 * (dora_scale / merged_scale1_norm)
)
final_updown = dora_merged - orig_weight
return final_updown
def finalize_updown(self, updown, orig_weight, output_shape, ex_bias=None):
if self.bias is not None:
updown = updown.reshape(self.bias.shape)
updown += self.bias.to(orig_weight.device, dtype=orig_weight.dtype)
updown = updown.reshape(output_shape)
if len(output_shape) == 4:
updown = updown.reshape(output_shape)
if orig_weight.size().numel() == updown.size().numel():
updown = updown.reshape(orig_weight.shape)
if ex_bias is not None:
ex_bias = ex_bias * self.multiplier()
if self.dora_scale is not None:
updown = self.apply_weight_decompose(updown, orig_weight)
return updown * self.calc_scale() * self.multiplier(), ex_bias
def calc_updown(self, target):
raise NotImplementedError
def forward(self, x, y):
raise NotImplementedError
modules/lora/network_full.py
+26
View File
@@ -0,0 +1,26 @@
import modules.lora.network as network
class ModuleTypeFull(network.ModuleType):
def create_module(self, net: network.Network, weights: network.NetworkWeights):
if all(x in weights.w for x in ["diff"]):
return NetworkModuleFull(net, weights)
return None
class NetworkModuleFull(network.NetworkModule): # pylint: disable=abstract-method
def __init__(self, net: network.Network, weights: network.NetworkWeights):
super().__init__(net, weights)
self.weight = weights.w.get("diff")
self.ex_bias = weights.w.get("diff_b")
def calc_updown(self, target):
output_shape = self.weight.shape
updown = self.weight.to(target.device, dtype=target.dtype)
if self.ex_bias is not None:
ex_bias = self.ex_bias.to(target.device, dtype=target.dtype)
else:
ex_bias = None
return self.finalize_updown(updown, target, output_shape, ex_bias)
modules/lora/network_glora.py
+30
View File
@@ -0,0 +1,30 @@
import modules.lora.network as network
class ModuleTypeGLora(network.ModuleType):
def create_module(self, net: network.Network, weights: network.NetworkWeights):
if all(x in weights.w for x in ["a1.weight", "a2.weight", "alpha", "b1.weight", "b2.weight"]):
return NetworkModuleGLora(net, weights)
return None
# adapted from https://github.com/KohakuBlueleaf/LyCORIS
class NetworkModuleGLora(network.NetworkModule): # pylint: disable=abstract-method
def __init__(self, net: network.Network, weights: network.NetworkWeights):
super().__init__(net, weights)
if hasattr(self.sd_module, 'weight'):
self.shape = self.sd_module.weight.shape
self.w1a = weights.w["a1.weight"]
self.w1b = weights.w["b1.weight"]
self.w2a = weights.w["a2.weight"]
self.w2b = weights.w["b2.weight"]
def calc_updown(self, target): # pylint: disable=arguments-differ
w1a = self.w1a.to(target.device, dtype=target.dtype)
w1b = self.w1b.to(target.device, dtype=target.dtype)
w2a = self.w2a.to(target.device, dtype=target.dtype)
w2b = self.w2b.to(target.device, dtype=target.dtype)
output_shape = [w1a.size(0), w1b.size(1)]
updown = (w2b @ w1b) + ((target @ w2a) @ w1a)
return self.finalize_updown(updown, target, output_shape)
modules/lora/network_hada.py
+46
View File
@@ -0,0 +1,46 @@
import modules.lora.lyco_helpers as lyco_helpers
import modules.lora.network as network
class ModuleTypeHada(network.ModuleType):
def create_module(self, net: network.Network, weights: network.NetworkWeights):
if all(x in weights.w for x in ["hada_w1_a", "hada_w1_b", "hada_w2_a", "hada_w2_b"]):
return NetworkModuleHada(net, weights)
return None
class NetworkModuleHada(network.NetworkModule): # pylint: disable=abstract-method
def __init__(self, net: network.Network, weights: network.NetworkWeights):
super().__init__(net, weights)
if hasattr(self.sd_module, 'weight'):
self.shape = self.sd_module.weight.shape
self.w1a = weights.w["hada_w1_a"]
self.w1b = weights.w["hada_w1_b"]
self.dim = self.w1b.shape[0]
self.w2a = weights.w["hada_w2_a"]
self.w2b = weights.w["hada_w2_b"]
self.t1 = weights.w.get("hada_t1")
self.t2 = weights.w.get("hada_t2")
def calc_updown(self, target):
w1a = self.w1a.to(target.device, dtype=target.dtype)
w1b = self.w1b.to(target.device, dtype=target.dtype)
w2a = self.w2a.to(target.device, dtype=target.dtype)
w2b = self.w2b.to(target.device, dtype=target.dtype)
output_shape = [w1a.size(0), w1b.size(1)]
if self.t1 is not None:
output_shape = [w1a.size(1), w1b.size(1)]
t1 = self.t1.to(target.device, dtype=target.dtype)
updown1 = lyco_helpers.make_weight_cp(t1, w1a, w1b)
output_shape += t1.shape[2:]
else:
if len(w1b.shape) == 4:
output_shape += w1b.shape[2:]
updown1 = lyco_helpers.rebuild_conventional(w1a, w1b, output_shape)
if self.t2 is not None:
t2 = self.t2.to(target.device, dtype=target.dtype)
updown2 = lyco_helpers.make_weight_cp(t2, w2a, w2b)
else:
updown2 = lyco_helpers.rebuild_conventional(w2a, w2b, output_shape)
updown = updown1 * updown2
return self.finalize_updown(updown, target, output_shape)
modules/lora/network_ia3.py
+24
View File
@@ -0,0 +1,24 @@
import modules.lora.network as network
class ModuleTypeIa3(network.ModuleType):
def create_module(self, net: network.Network, weights: network.NetworkWeights):
if all(x in weights.w for x in ["weight"]):
return NetworkModuleIa3(net, weights)
return None
class NetworkModuleIa3(network.NetworkModule): # pylint: disable=abstract-method
def __init__(self, net: network.Network, weights: network.NetworkWeights):
super().__init__(net, weights)
self.w = weights.w["weight"]
self.on_input = weights.w["on_input"].item()
def calc_updown(self, target):
w = self.w.to(target.device, dtype=target.dtype)
output_shape = [w.size(0), target.size(1)]
if self.on_input:
output_shape.reverse()
else:
w = w.reshape(-1, 1)
updown = target * w
return self.finalize_updown(updown, target, output_shape)
modules/lora/network_lokr.py
+57
View File
@@ -0,0 +1,57 @@
import torch
import modules.lora.lyco_helpers as lyco_helpers
import modules.lora.network as network
class ModuleTypeLokr(network.ModuleType):
def create_module(self, net: network.Network, weights: network.NetworkWeights):
has_1 = "lokr_w1" in weights.w or ("lokr_w1_a" in weights.w and "lokr_w1_b" in weights.w)
has_2 = "lokr_w2" in weights.w or ("lokr_w2_a" in weights.w and "lokr_w2_b" in weights.w)
if has_1 and has_2:
return NetworkModuleLokr(net, weights)
return None
def make_kron(orig_shape, w1, w2):
if len(w2.shape) == 4:
w1 = w1.unsqueeze(2).unsqueeze(2)
w2 = w2.contiguous()
return torch.kron(w1, w2).reshape(orig_shape)
class NetworkModuleLokr(network.NetworkModule): # pylint: disable=abstract-method
def __init__(self, net: network.Network, weights: network.NetworkWeights):
super().__init__(net, weights)
self.w1 = weights.w.get("lokr_w1")
self.w1a = weights.w.get("lokr_w1_a")
self.w1b = weights.w.get("lokr_w1_b")
self.dim = self.w1b.shape[0] if self.w1b is not None else self.dim
self.w2 = weights.w.get("lokr_w2")
self.w2a = weights.w.get("lokr_w2_a")
self.w2b = weights.w.get("lokr_w2_b")
self.dim = self.w2b.shape[0] if self.w2b is not None else self.dim
self.t2 = weights.w.get("lokr_t2")
def calc_updown(self, target):
if self.w1 is not None:
w1 = self.w1.to(target.device, dtype=target.dtype)
else:
w1a = self.w1a.to(target.device, dtype=target.dtype)
w1b = self.w1b.to(target.device, dtype=target.dtype)
w1 = w1a @ w1b
if self.w2 is not None:
w2 = self.w2.to(target.device, dtype=target.dtype)
elif self.t2 is None:
w2a = self.w2a.to(target.device, dtype=target.dtype)
w2b = self.w2b.to(target.device, dtype=target.dtype)
w2 = w2a @ w2b
else:
t2 = self.t2.to(target.device, dtype=target.dtype)
w2a = self.w2a.to(target.device, dtype=target.dtype)
w2b = self.w2b.to(target.device, dtype=target.dtype)
w2 = lyco_helpers.make_weight_cp(t2, w2a, w2b)
output_shape = [w1.size(0) * w2.size(0), w1.size(1) * w2.size(1)]
if len(target.shape) == 4:
output_shape = target.shape
updown = make_kron(output_shape, w1, w2)
return self.finalize_updown(updown, target, output_shape)
modules/lora/network_lora.py
+75
View File
@@ -0,0 +1,75 @@
import torch
import diffusers.models.lora as diffusers_lora
import modules.lora.lyco_helpers as lyco_helpers
import modules.lora.network as network
from modules import devices
class ModuleTypeLora(network.ModuleType):
def create_module(self, net: network.Network, weights: network.NetworkWeights):
if all(x in weights.w for x in ["lora_up.weight", "lora_down.weight"]):
return NetworkModuleLora(net, weights)
return None
class NetworkModuleLora(network.NetworkModule):
def __init__(self, net: network.Network, weights: network.NetworkWeights):
super().__init__(net, weights)
self.up_model = self.create_module(weights.w, "lora_up.weight")
self.down_model = self.create_module(weights.w, "lora_down.weight")
self.mid_model = self.create_module(weights.w, "lora_mid.weight", none_ok=True)
self.dim = weights.w["lora_down.weight"].shape[0]
def create_module(self, weights, key, none_ok=False):
weight = weights.get(key)
if weight is None and none_ok:
return None
linear_modules = [torch.nn.Linear, torch.nn.modules.linear.NonDynamicallyQuantizableLinear, torch.nn.MultiheadAttention, diffusers_lora.LoRACompatibleLinear]
is_linear = type(self.sd_module) in linear_modules or self.sd_module.__class__.__name__ in {"NNCFLinear", "QLinear", "Linear4bit"}
is_conv = type(self.sd_module) in [torch.nn.Conv2d, diffusers_lora.LoRACompatibleConv] or self.sd_module.__class__.__name__ in {"NNCFConv2d", "QConv2d"}
if is_linear:
weight = weight.reshape(weight.shape[0], -1)
module = torch.nn.Linear(weight.shape[1], weight.shape[0], bias=False)
elif is_conv and (key == "lora_down.weight" or key == "dyn_up"):
if len(weight.shape) == 2:
weight = weight.reshape(weight.shape[0], -1, 1, 1)
if weight.shape[2] != 1 or weight.shape[3] != 1:
module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], self.sd_module.kernel_size, self.sd_module.stride, self.sd_module.padding, bias=False)
else:
module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], (1, 1), bias=False)
elif is_conv and key == "lora_mid.weight":
module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], self.sd_module.kernel_size, self.sd_module.stride, self.sd_module.padding, bias=False)
elif is_conv and (key == "lora_up.weight" or key == "dyn_down"):
module = torch.nn.Conv2d(weight.shape[1], weight.shape[0], (1, 1), bias=False)
else:
raise AssertionError(f'Lora unsupported: layer={self.network_key} type={type(self.sd_module).__name__}')
with torch.no_grad():
if weight.shape != module.weight.shape:
weight = weight.reshape(module.weight.shape)
module.weight.copy_(weight)
module.weight.requires_grad_(False)
return module
def calc_updown(self, target): # pylint: disable=W0237
target_dtype = target.dtype if target.dtype != torch.uint8 else self.up_model.weight.dtype
up = self.up_model.weight.to(target.device, dtype=target_dtype)
down = self.down_model.weight.to(target.device, dtype=target_dtype)
output_shape = [up.size(0), down.size(1)]
if self.mid_model is not None:
# cp-decomposition
mid = self.mid_model.weight.to(target.device, dtype=target_dtype)
updown = lyco_helpers.rebuild_cp_decomposition(up, down, mid)
output_shape += mid.shape[2:]
else:
if len(down.shape) == 4:
output_shape += down.shape[2:]
updown = lyco_helpers.rebuild_conventional(up, down, output_shape, self.network.dyn_dim)
return self.finalize_updown(updown, target, output_shape)
def forward(self, x, y):
self.up_model.to(device=devices.device)
self.down_model.to(device=devices.device)
if hasattr(y, "scale"):
return y(scale=1) + self.up_model(self.down_model(x)) * self.multiplier() * self.calc_scale()
return y + self.up_model(self.down_model(x)) * self.multiplier() * self.calc_scale()
modules/lora/network_norm.py
+24
View File
@@ -0,0 +1,24 @@
import modules.lora.network as network
class ModuleTypeNorm(network.ModuleType):
def create_module(self, net: network.Network, weights: network.NetworkWeights):
if all(x in weights.w for x in ["w_norm", "b_norm"]):
return NetworkModuleNorm(net, weights)
return None
class NetworkModuleNorm(network.NetworkModule): # pylint: disable=abstract-method
def __init__(self, net: network.Network, weights: network.NetworkWeights):
super().__init__(net, weights)
self.w_norm = weights.w.get("w_norm")
self.b_norm = weights.w.get("b_norm")
def calc_updown(self, target):
output_shape = self.w_norm.shape
updown = self.w_norm.to(target.device, dtype=target.dtype)
if self.b_norm is not None:
ex_bias = self.b_norm.to(target.device, dtype=target.dtype)
else:
ex_bias = None
return self.finalize_updown(updown, target, output_shape, ex_bias)

Some files were not shown because too many files have changed in this diff Show More