Public/automatic
1
0
Fork 0
mirror of https://github.com/vladmandic/automatic synced 2026-09-19 01:04:32 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #2653 from vladmandic/dev

Browse Source 
merge dev to master
This commit is contained in:
Vladimir Mandic
2023-12-29 09:32:52 -05:00
committed by GitHub
parent 00281e15d2 f4f4f42770
commit f4d4f8dad5
321 changed files with 39273 additions and 1680 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -32,6 +32,7 @@ cache
*.lock
*.zip
*.rar
*.7z
*.pyc
/*.bat
/*.sh
.vscode/settings.json
Vendored
+3 -2
View File
@@ -10,5 +10,6 @@
"./repositories/stable-diffusion-stability-ai",
"./repositories/stable-diffusion-stability-ai/ldm"
],
"python.analysis.typeCheckingMode": "off"
}
"python.analysis.typeCheckingMode": "off",
"editor.formatOnSave": false
}
CHANGELOG.md
+141
View File
@@ -1,5 +1,146 @@
# Change Log for SD.Next
## Update for 2023-12-29
- **Control**
- native implementation of all image control methods:
**ControlNet**, **ControlNet XS**, **Control LLLite**, **T2I Adapters** and **IP Adapters**
- top-level **Control** next to **Text** and **Image** generate
- supports all variations of **SD15** and **SD-XL** models
- supports *Text*, *Image*, *Batch* and *Video* processing
- for details and list of supported models and workflows, see Wiki documentation:
<https://github.com/vladmandic/automatic/wiki/Control>
- **Diffusers**
- [Segmind Vega](https://huggingface.co/segmind/Segmind-Vega) model support
- small and fast version of **SDXL**, only 3.1GB in size!
- select from *networks -> reference*
- [aMUSEd 256](https://huggingface.co/amused/amused-256) and [aMUSEd 512](https://huggingface.co/amused/amused-512) model support
- lightweigt models that excel at fast image generation
- *note*: must select: settings -> diffusers -> generator device: unset
- select from *networks -> reference*
- [Playground v1](https://huggingface.co/playgroundai/playground-v1), [Playground v2 256](https://huggingface.co/playgroundai/playground-v2-256px-base), [Playground v2 512](https://huggingface.co/playgroundai/playground-v2-512px-base), [Playground v2 1024](https://huggingface.co/playgroundai/playground-v2-1024px-aesthetic) model support
- comparable to SD15 and SD-XL, trained from scratch for highly aesthetic images
- simply select from *networks -> reference* and use as usual
- [BLIP-Diffusion](https://dxli94.github.io/BLIP-Diffusion-website/)
- img2img model that can replace subjects in images using prompt keywords
- download and load by selecting from *networks -> reference -> blip diffusion*
- in image tab, select `blip diffusion` script
- [DemoFusion](https://github.com/PRIS-CV/DemoFusion) run your SDXL generations at any resolution!
- in **Text** tab select *script* -> *demofusion*
- *note*: GPU VRAM limits do not automatically go away so be careful when using it with large resolutions
in the future, expect more optimizations, especially related to offloading/slicing/tiling,
but at the moment this is pretty much experimental-only
- [AnimateDiff](https://github.com/guoyww/animatediff/)
- overall improved quality
- can now be used with *second pass* - enhance, upscale and hires your videos!
- [IP Adapter](https://github.com/tencent-ailab/IP-Adapter)
- add support for **ip-adapter-plus_sd15, ip-adapter-plus-face_sd15 and ip-adapter-full-face_sd15**
- can now be used in *xyz-grid*
- **Text-to-Video**
- in text tab, select `text-to-video` script
- supported models: **ModelScope v1.7b, ZeroScope v1, ZeroScope v1.1, ZeroScope v2, ZeroScope v2 Dark, Potat v1**
*if you know of any other t2v models youd like to see supported, let me know!*
- models are auto-downloaded on first use
- *note*: current base model will be unloaded to free up resources
- **Prompt scheduling** now implemented for Diffusers backend, thanks @AI-Casanova
- **Custom pipelines** contribute by adding your own custom pipelines!
- for details, see fully documented example:
<https://github.com/vladmandic/automatic/blob/dev/scripts/example.py>
- **Schedulers**
- add timesteps range, changing it will make scheduler to be over-complete or under-complete
- add rescale betas with zero SNR option (applicable to Euler, Euler a and DDIM, allows for higher dynamic range)
- **Inpaint**
- improved quality when using mask blur and padding
- **UI**
- 3 new native UI themes: **orchid-dreams**, **emerald-paradise** and **timeless-beige**, thanks @illu_Zn
- more dynamic controls depending on the backend (original or diffusers)
controls that are not applicable in current mode are now hidden
- allow setting of resize method directly in image tab
(previously via settings -> upscaler_for_img2img)
- **Optional**
- **FaceID** face guidance during generation
- also based on IP adapters, but with additional face detection and external embeddings calculation
- calculates face embeds based on input image and uses it to guide generation
- simply select from *scripts -> faceid*
- *experimental module*: requirements must be installed manually:
> pip install insightface ip_adapter
- **Depth 3D** image to 3D scene
- delivered as an extension, install from extensions tab
<https://github.com/vladmandic/sd-extension-depth3d>
- creates fully compatible 3D scene from any image by using depth estimation
and creating a fully populated mesh
- scene can be freely viewed in 3D in the UI itself or downloaded for use in other applications
- [ONNX/Olive](https://github.com/vladmandic/automatic/wiki/ONNX-Olive)
- major work continues in olive branch, see wiki for details, thanks @lshqqytiger
as a highlight, 4-5 it/s using DirectML on AMD GPU translates to 23-25 it/s using ONNX/Olive!
- **General**
- new **onboarding**
- if no models are found during startup, app will no longer ask to download default checkpoint
instead, it will show message in UI with options to change model path or download any of the reference checkpoints
- *extra networks -> models -> reference* section is now enabled for both original and diffusers backend
- support for **Torch 2.1.2** (release) and **Torch 2.3** (dev)
- **Process** create videos from batch or folder processing
supports *GIF*, *PNG* and *MP4* with full interpolation, scene change detection, etc.
- **LoRA**
- add support for block weights, thanks @AI-Casanova
example `<lora:SDXL_LCM_LoRA:1.0:in=0:mid=1:out=0>`
- add support for LyCORIS GLora networks
- add support for LoRA PEFT (*Diffusers*) networks
- add support for Lora-OFT (*Kohya*) and Lyco-OFT (*Kohaku*) networks
- reintroduce alternative loading method in settings: `lora_force_diffusers`
- add support for `lora_fuse_diffusers` if using alternative method
use if you have multiple complex loras that may be causing performance degradation
as it fuses lora with model during load instead of interpreting lora on-the-fly
- **CivitAI downloader** allow usage of access tokens for download of gated or private models
- **Extra networks** new *settting -> extra networks -> build info on first access*
indexes all networks on first access instead of server startup
- **IPEX**, thanks @disty0
- update to **Torch 2.1**
if you get file not found errors, set `DISABLE_IPEXRUN=1` and run the webui with `--reinstall`
- built-in *MKL* and *DPCPP* for IPEX, no need to install OneAPI anymore
- **StableVideoDiffusion** is now supported with IPEX
- **8 bit support with NNCF** on Diffusers backend
- fix IPEX Optimize not applying with Diffusers backend
- disable 32bit workarounds if the GPU supports 64bit
- add `DISABLE_IPEXRUN` and `DISABLE_IPEX_1024_WA` environment variables
- performance and compatibility improvements
- **OpenVINO**, thanks @disty0
- **8 bit support for CPUs**
- reduce System RAM usage
- update to Torch 2.1.2
- add *Directory for OpenVINO cache* option to *System Paths*
- remove Intel ARC specific 1024x1024 workaround
- **HDR controls**
- batch-aware for enhancement of multiple images or video frames
- available in image tab
- **Logging**
- additional *TRACE* logging enabled via specific env variables
see <https://github.com/vladmandic/automatic/wiki/Debug> for details
- improved profiling
use with `--debug --profile`
- log output file sizes
- **Other**
- **API** several minor but breaking changes to API behavior to better align response fields, thanks @Trojaner
- **Inpaint** add option `apply_overlay` to control if inpaint result should be applied as overlay or as-is
can remove artifacts and hard edges of inpaint area but also remove some details from original
- **chaiNNer** fix `NaN` issues due to autocast
- **Upscale** increase limit from 4x to 8x given the quality of some upscalers
- **Extra Networks** fix sort
- reduced default **CFG scale** from 6 to 4 to be more out-of-the-box compatibile with LCM/Turbo models
- disable google fonts check on server startup
- fix torchvision/basicsr compatibility
- fix styles quick save
- add hdr settings to metadata
- improve handling of long filenames and filenames during batch processing
- do not set preview samples when using via api
- avoid unnecessary resizes in img2img and inpaint
- safe handling of config updates avoid file corruption on I/O errors
- updated `cli/simple-txt2img.py` and `cli/simple-img2img.py` scripts
- save `params.txt` regardless of image save status
- update built-in log monitor in ui, thanks @midcoastal
- major CHANGELOG doc cleanup, thanks @JetVarimax
- major INSTALL doc cleanup, thanks JetVarimax
## Update for 2023-12-04
Whats new? Native video in SD.Next via both **AnimateDiff** and **Stable-Video-Diffusion** - and including native MP4 encoding and smooth video outputs out-of-the-box, not just animated-GIFs.
CITATION.cff
+28
View File
@@ -0,0 +1,28 @@
cff-version: 1.2.0
title: SD.Next
url: 'https://github.com/vladmandic/automatic'
message: >-
If you use this software, please cite it using the
metadata from this file
type: software
authors:
- given-names: Vladimir
name-particle: Vlado
family-names: Mandic
orcid: 'https://orcid.org/0009-0003-4592-5074'
identifiers:
- type: url
value: 'https://github.com/vladmandic'
description: GitHub
- type: url
value: 'https://www.linkedin.com/in/cyan051/'
description: LinkedIn
repository-code: 'https://github.com/vladmandic/automatic'
abstract: >-
SD.Next: Advanced Implementation of Stable Diffusion and
other diffusion models for text, image and video
generation
keywords:
- stablediffusion diffusers sdnext
license: AGPL-3.0
date-released: 2022-12-24
README.md
+21 -10
View File
@@ -20,7 +20,9 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
- Multiple backends!
▹ **Original | Diffusers**
- Multiple diffusion models!
▹ **Stable Diffusion | SD-XL | LCM | Segmind | Kandinsky | Pixart-α | Würstchen | DeepFloyd IF | UniDiffusion | SD-Distilled | etc.**
▹ **Stable Diffusion 1.5/2.1 | SD-XL | LCM | Segmind | Kandinsky | Pixart-α | Würstchen | aMUSEd | DeepFloyd IF | UniDiffusion | SD-Distilled | BLiP Diffusion | etc.**
- Built-in Control for Text, Image, Batch and video processing!
▹ **ControlNet | ControlNet XS | Control LLLite | T2I Adapters | IP Adapters**
- Multiplatform!
▹ **Windows | Linux | MacOS with CPU | nVidia | AMD | IntelArc | DirectML | OpenVINO | ONNX+Olive**
- Platform specific autodetection and tuning performed on install
@@ -28,7 +30,6 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
- Improved prompt parser
- Enhanced *Lora*/*LoCon*/*Lyco* code supporting latest trends in training
- Built-in queue management
- Advanced metadata caching and handling to speed up operations
- Enterprise level logging and hardened API
- Modern localization and hints engine
- Broad compatibility with existing extensions ecosystem and new extensions manager
@@ -37,7 +38,8 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
<br>
![Screenshot-Dark](html/black-teal.jpg)
![Screenshot-Dark](html/xmas-default.jpg)
![Screenshot-Control](html/xmas-control.jpg)
![Screenshot-Light](html/light-teal.jpg)
<br>
@@ -58,17 +60,23 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
Additional models will be added as they become available and there is public interest in them
- [RunwayML Stable Diffusion](https://github.com/Stability-AI/stablediffusion/) 1.x and 2.x *(all variants)*
- [StabilityAI Stable Diffusion XL](https://github.com/Stability-AI/generative-models)
- [StabilityAI Stable Video Diffusion Base and XT](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid)
- [Segmind SSD-1B](https://huggingface.co/segmind/SSD-1B)
- [LCM: Latent Consistency Models](https://github.com/openai/consistency_models)
- [Kandinsky](https://github.com/ai-forever/Kandinsky-2) *2.1 and 2.2 and latest 3.0*
- [PixArt-α XL 2](https://github.com/PixArt-alpha/PixArt-alpha) *Medium and Large*
- [RunwayML Stable Diffusion](https://github.com/Stability-AI/stablediffusion/) 1.x and 2.x *(all variants)*
- [StabilityAI Stable Diffusion XL](https://github.com/Stability-AI/generative-models)
- [StabilityAI Stable Video Diffusion](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid) Base and XT
- [LCM: Latent Consistency Models](https://github.com/openai/consistency_models)
- [aMUSEd 256](https://huggingface.co/amused/amused-256) 256 and 512
- [Segmind Vega](https://huggingface.co/segmind/Segmind-Vega)
- [Segmind SSD-1B](https://huggingface.co/segmind/SSD-1B)
- [Kandinsky](https://github.com/ai-forever/Kandinsky-2) *2.1 and 2.2 and latest 3.0*
- [PixArt-α XL 2](https://github.com/PixArt-alpha/PixArt-alpha) *Medium and Large*
- [Warp Wuerstchen](https://huggingface.co/blog/wuertschen)
- [Playground](https://huggingface.co/playgroundai/playground-v2-256px-base) *v1, v2 256, v2 512, v2 1024*
- [Tsinghua UniDiffusion](https://github.com/thu-ml/unidiffuser)
- [DeepFloyd IF](https://github.com/deep-floyd/IF) *Medium and Large*
- [ModelScope T2V](https://huggingface.co/damo-vilab/text-to-video-ms-1.7b)
- [Segmind SD Distilled](https://huggingface.co/blog/sd_distillation) *(all variants)*
- [BLIP-Diffusion](https://dxli94.github.io/BLIP-Diffusion-website/)
Also supported are modifiers such as:
- **LCM** and **Turbo** (Adversarial Diffusion Distillation) networks
@@ -209,6 +217,9 @@ General goals:
### **Docs**
If you're unsure how to use a feature, best place to start is [Wiki](https://github.com/vladmandic/automatic/wiki) and if its not there,
check [ChangeLog](CHANGELOG.md) for when feature was first introduced as it will always have a short note on how to use it
- [Wiki](https://github.com/vladmandic/automatic/wiki)
- [ReadMe](README.md)
- [ToDo](TODO.md)
SECURITY.md
+1 -1
View File
@@ -32,5 +32,5 @@ Any code commit is validated before merge
- Download extensions and themes indexes from automatically updated indexes
- Download required packages and repositories from GitHub during installation/upgrade
- Download installed/enabled extensions
- Download default model from official repository
- Download models from CivitAI and/or Huggingface when instructed by user
- Submit benchmark info upon user interaction
cli/image-grid.py
+1 -1
View File
@@ -56,7 +56,7 @@ def grid(images, labels = None, width = 0, height = 0, border = 0, square = Fals
for i, img in enumerate(images): # pylint: disable=redefined-outer-name
x = (i % cols * w) + (i % cols * border)
y = (i // cols * h) + (i // cols * border)
img.thumbnail((w, h), Image.HAMMING)
img.thumbnail((w, h), Image.Resampling.HAMMING)
image.paste(img, box=(x, y))
if labels is not None and len(images) == len(labels):
ctx = ImageDraw.Draw(image)
cli/simple-img2img.py
+40 -23
View File
@@ -1,9 +1,10 @@
#!/usr/bin/env python
import os
import io
import sys
import time
import base64
import logging
import argparse
import requests
import urllib3
from PIL import Image
@@ -14,22 +15,13 @@ sd_password = os.environ.get('SDAPI_PWD', None)
logging.basicConfig(level = logging.INFO, format = '%(asctime)s %(levelname)s: %(message)s')
log = logging.getLogger(__name__)
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
filename='/tmp/simple-img2img.jpg'
options = {
"init_images": [],
"prompt": "city at night",
"negative_prompt": "foggy, blurry",
"steps": 20,
"batch_size": 1,
"n_iter": 1,
"seed": -1,
"sampler_name": "Euler a",
"cfg_scale": 6,
"width": 512,
"height": 512,
"save_images": False,
"send_images": True,
}
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
def auth():
@@ -51,26 +43,51 @@ def encode(f):
image = image.convert('RGB')
with io.BytesIO() as stream:
image.save(stream, 'JPEG')
image.close()
values = stream.getvalue()
encoded = base64.b64encode(values).decode()
return encoded
def generate(num: int = 0):
log.info(f'sending generate request: {num+1} {options}')
options['init_images'] = [encode('html/logo-dark.png')]
options['batch_size'] = len(options['init_images'])
def generate(args): # pylint: disable=redefined-outer-name
t0 = time.time()
if args.model is not None:
post('/sdapi/v1/options', { 'sd_model_checkpoint': args.model })
post('/sdapi/v1/reload-checkpoint') # needed if running in api-only to trigger new model load
options['prompt'] = args.prompt
options['negative_prompt'] = args.negative
options['steps'] = int(args.steps)
options['seed'] = int(args.seed)
options['sampler_name'] = args.sampler
options['init_images'] = [encode(args.init)]
image = Image.open(args.init)
options['width'] = image.width
options['height'] = image.height
image.close()
if args.mask is not None:
options['mask'] = encode(args.mask)
data = post('/sdapi/v1/img2img', options)
t1 = time.time()
if 'images' in data:
for i in range(len(data['images'])):
b64 = data['images'][i].split(',',1)[0]
info = data['info']
image = Image.open(io.BytesIO(base64.b64decode(b64)))
log.info(f'received image: {image.size}')
image.save(filename)
log.info(f'received image: size={image.size} file={filename} time={t1-t0:.2f} info="{info}"')
else:
log.warning(f'no images received: {data}')
if __name__ == "__main__":
sys.argv.pop(0)
repeats = int(''.join(sys.argv) or '1')
log.info(f'repeats: {repeats}')
for n in range(repeats):
generate(n)
parser = argparse.ArgumentParser(description = 'simple-img2img')
parser.add_argument('--init', required=True, help='init image')
parser.add_argument('--mask', required=False, help='mask image')
parser.add_argument('--prompt', required=False, default='', help='prompt text')
parser.add_argument('--negative', required=False, default='', help='negative prompt text')
parser.add_argument('--steps', required=False, default=20, help='number of steps')
parser.add_argument('--seed', required=False, default=-1, help='initial seed')
parser.add_argument('--sampler', required=False, default='Euler a', help='sampler name')
parser.add_argument('--model', required=False, help='model name')
args = parser.parse_args()
log.info(f'img2img: {args}')
generate(args)
cli/simple-txt2img.py
+28 -22
View File
@@ -1,9 +1,10 @@
#!/usr/bin/env python
import io
import os
import sys
import time
import base64
import logging
import argparse
import requests
import urllib3
from PIL import Image
@@ -17,18 +18,7 @@ log = logging.getLogger(__name__)
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
filename='/tmp/simple-txt2img.jpg'
model = None # desired model name, will be set if not none
options = {
"prompt": "city at night",
"negative_prompt": "foggy, blurry",
"steps": 20,
"batch_size": 1,
"n_iter": 1,
"seed": -1,
"sampler_name": "UniPC",
"cfg_scale": 6,
"width": 512,
"height": 512,
"save_images": False,
"send_images": True,
}
@@ -48,25 +38,41 @@ def post(endpoint: str, dct: dict = None):
return req.json()
def generate(num: int = 0):
log.info(f'sending generate request: {num+1} {options}')
if model is not None:
post('/sdapi/v1/options', { 'sd_model_checkpoint': model })
def generate(args): # pylint: disable=redefined-outer-name
t0 = time.time()
if args.model is not None:
post('/sdapi/v1/options', { 'sd_model_checkpoint': args.model })
post('/sdapi/v1/reload-checkpoint') # needed if running in api-only to trigger new model load
options['prompt'] = args.prompt
options['negative_prompt'] = args.negative
options['steps'] = int(args.steps)
options['seed'] = int(args.seed)
options['sampler_name'] = args.sampler
options['width'] = int(args.width)
options['height'] = int(args.height)
data = post('/sdapi/v1/txt2img', options)
t1 = time.time()
if 'images' in data:
for i in range(len(data['images'])):
b64 = data['images'][i].split(',',1)[0]
image = Image.open(io.BytesIO(base64.b64decode(b64)))
info = data['info']
image.save(filename)
log.info(f'received image: size={image.size} file={filename}')
log.info(f'received image: size={image.size} file={filename} time={t1-t0:.2f} info="{info}"')
else:
log.warning(f'no images received: {data}')
if __name__ == "__main__":
sys.argv.pop(0)
repeats = int(''.join(sys.argv) or '1')
log.info(f'repeats: {repeats}')
for n in range(repeats):
generate(n)
parser = argparse.ArgumentParser(description = 'simple-txt2img')
parser.add_argument('--prompt', required=False, default='', help='prompt text')
parser.add_argument('--negative', required=False, default='', help='negative prompt text')
parser.add_argument('--width', required=False, default=512, help='image width')
parser.add_argument('--height', required=False, default=512, help='image height')
parser.add_argument('--steps', required=False, default=20, help='number of steps')
parser.add_argument('--seed', required=False, default=-1, help='initial seed')
parser.add_argument('--sampler', required=False, default='Euler a', help='sampler name')
parser.add_argument('--model', required=False, help='model name')
args = parser.parse_args()
log.info(f'txt2img: {args}')
generate(args)
extensions-builtin/Lora/extra_networks_lora.py
+8 -3
View File
@@ -35,8 +35,11 @@ class ExtraNetworkLora(extra_networks.ExtraNetwork):
names.append(params.positional[0])
te_multiplier = float(params.positional[1]) if len(params.positional) > 1 else 1.0
te_multiplier = float(params.named.get("te", te_multiplier))
unet_multiplier = float(params.positional[2]) if len(params.positional) > 2 else te_multiplier
unet_multiplier = float(params.named.get("unet", unet_multiplier))
unet_multiplier = [float(params.positional[2]) if len(params.positional) > 2 else te_multiplier] * 3
unet_multiplier = [float(params.named.get("unet", unet_multiplier[0]))] * 3
unet_multiplier[0] = float(params.named.get("in", unet_multiplier[0]))
unet_multiplier[1] = float(params.named.get("mid", unet_multiplier[1]))
unet_multiplier[2] = float(params.named.get("out", unet_multiplier[2]))
dyn_dim = int(params.positional[3]) if len(params.positional) > 3 else None
dyn_dim = int(params.named["dyn"]) if "dyn" in params.named else dyn_dim
te_multipliers.append(te_multiplier)
@@ -59,13 +62,15 @@ class ExtraNetworkLora(extra_networks.ExtraNetwork):
if network_hashes:
p.extra_generation_params["Lora hashes"] = ", ".join(network_hashes)
if len(names) > 0:
shared.log.info(f'Applying LoRA: {names} patch={t1-t0:.2f} load={t2-t1:.2f}')
shared.log.info(f'LoRA apply: {names} patch={t1-t0:.2f} load={t2-t1:.2f}')
elif self.active:
self.active = False
def deactivate(self, p):
if shared.backend == shared.Backend.DIFFUSERS and hasattr(shared.sd_model, "unload_lora_weights") and hasattr(shared.sd_model, "text_encoder"):
if 'CLIP' in shared.sd_model.text_encoder.__class__.__name__ and not (shared.opts.cuda_compile and shared.opts.cuda_compile_backend == "openvino_fx"):
if shared.opts.lora_fuse_diffusers:
shared.sd_model.unfuse_lora()
shared.sd_model.unload_lora_weights()
if not self.active and getattr(networks, "originals", None ) is not None:
networks.originals.undo() # remove patches
extensions-builtin/Lora/lora_convert.py
+6 -3
View File
@@ -1,9 +1,11 @@
from typing import Dict
import os
import re
import bisect
from typing import Dict
from modules import shared
debug = os.environ.get('SD_LORA_DEBUG', None) is not None
suffix_conversion = {
"attentions": {},
"resnets": {
@@ -144,12 +146,13 @@ class KeyConvert:
map_keys = list(self.UNET_CONVERSION_MAP.keys()) # prefix of U-Net modules
map_keys.sort()
search_key = key.replace(self.LORA_PREFIX_UNET, "").replace(self.OFT_PREFIX_UNET, "").replace(self.LORA_PREFIX_TEXT_ENCODER1, "").replace(self.LORA_PREFIX_TEXT_ENCODER2, "")
position = bisect.bisect_right(map_keys, search_key)
map_key = map_keys[position - 1]
if search_key.startswith(map_key):
key = key.replace(map_key, self.UNET_CONVERSION_MAP[map_key]).replace("oft","lora") # pylint: disable=unsubscriptable-object
key = key.replace(map_key, self.UNET_CONVERSION_MAP[map_key]).replace("oft", "lora") # pylint: disable=unsubscriptable-object
sd_module = shared.sd_model.network_layer_mapping.get(key, None)
if debug and sd_module is None:
raise RuntimeError(f"LoRA key not found in network_layer_mapping: key={key} mapping={shared.sd_model.network_layer_mapping.keys()}")
return key, sd_module
def __call__(self, key):
extensions-builtin/Lora/lora_patches.py
+2 -5
View File
@@ -1,10 +1,7 @@
import os
import torch
import networks
from modules import patches, shared
# OpenVINO only works with Diffusers LoRa loading
force_lora_diffusers = os.environ.get('SD_LORA_DIFFUSERS', None) is not None
class LoraPatches:
def __init__(self):
@@ -21,7 +18,7 @@ class LoraPatches:
self.MultiheadAttention_load_state_dict = None
def apply(self):
if self.active or force_lora_diffusers:
if self.active or shared.opts.lora_force_diffusers:
return
self.Linear_forward = patches.patch(__name__, torch.nn.Linear, 'forward', networks.network_Linear_forward)
self.Linear_load_state_dict = patches.patch(__name__, torch.nn.Linear, '_load_from_state_dict', networks.network_Linear_load_state_dict)
@@ -39,7 +36,7 @@ class LoraPatches:
self.active = True
def undo(self):
if not self.active or force_lora_diffusers:
if not self.active or shared.opts.lora_force_diffusers:
return
self.Linear_forward = patches.undo(__name__, torch.nn.Linear, 'forward') # pylint: disable=E1128
self.Linear_load_state_dict = patches.undo(__name__, torch.nn.Linear, '_load_from_state_dict') # pylint: disable=E1128
extensions-builtin/Lora/lyco_helpers.py
+47
View File
@@ -19,3 +19,50 @@ def rebuild_cp_decomposition(up, down, mid):
up = up.reshape(up.size(0), -1)
down = down.reshape(down.size(0), -1)
return torch.einsum('n m k l, i n, m j -> i j k l', mid, up, down)
# copied from https://github.com/KohakuBlueleaf/LyCORIS/blob/dev/lycoris/modules/lokr.py
def factorization(dimension: int, factor:int=-1) -> tuple[int, int]:
'''
return a tuple of two value of input dimension decomposed by the number closest to factor
second value is higher or equal than first value.
In LoRA with Kroneckor Product, first value is a value for weight scale.
secon value is a value for weight.
Becuase of non-commutative property, A⊗B ≠ B⊗A. Meaning of two matrices is slightly different.
examples)
factor
-1 2 4 8 16 ...
127 -> 1, 127 127 -> 1, 127 127 -> 1, 127 127 -> 1, 127 127 -> 1, 127
128 -> 8, 16 128 -> 2, 64 128 -> 4, 32 128 -> 8, 16 128 -> 8, 16
250 -> 10, 25 250 -> 2, 125 250 -> 2, 125 250 -> 5, 50 250 -> 10, 25
360 -> 8, 45 360 -> 2, 180 360 -> 4, 90 360 -> 8, 45 360 -> 12, 30
512 -> 16, 32 512 -> 2, 256 512 -> 4, 128 512 -> 8, 64 512 -> 16, 32
1024 -> 32, 32 1024 -> 2, 512 1024 -> 4, 256 1024 -> 8, 128 1024 -> 16, 64
'''
if factor > 0 and (dimension % factor) == 0:
m = factor
n = dimension // factor
if m > n:
n, m = m, n
return m, n
if factor < 0:
factor = dimension
m, n = 1, dimension
length = m + n
while m<n:
new_m = m + 1
while dimension%new_m != 0:
new_m += 1
new_n = dimension // new_m
if new_m + new_n > length or new_m>factor:
break
else:
m, n = new_m, new_n
if m > n:
n, m = m, n
return m, n
extensions-builtin/Lora/network.py
+8 -2
View File
@@ -83,7 +83,7 @@ class Network: # LoraModule
self.name = name
self.network_on_disk = network_on_disk
self.te_multiplier = 1.0
self.unet_multiplier = 1.0
self.unet_multiplier = [1.0] * 3
self.dyn_dim = None
self.modules = {}
self.mtime = None
@@ -112,8 +112,14 @@ class NetworkModule:
def multiplier(self):
if 'transformer' in self.sd_key[:20]:
return self.network.te_multiplier
if "down_blocks" in self.sd_key:
return self.network.unet_multiplier[0]
if "mid_block" in self.sd_key:
return self.network.unet_multiplier[1]
if "up_blocks" in self.sd_key:
return self.network.unet_multiplier[2]
else:
return self.network.unet_multiplier
return self.network.unet_multiplier[0]
def calc_scale(self):
if self.scale is not None:
extensions-builtin/Lora/network_full.py
+4 -4
View File
@@ -16,12 +16,12 @@ class NetworkModuleFull(network.NetworkModule):
self.weight = weights.w.get("diff")
self.ex_bias = weights.w.get("diff_b")
def calc_updown(self, orig_weight):
def calc_updown(self, target):
output_shape = self.weight.shape
updown = self.weight.to(orig_weight.device, dtype=orig_weight.dtype)
updown = self.weight.to(target.device, dtype=target.dtype)
if self.ex_bias is not None:
ex_bias = self.ex_bias.to(orig_weight.device, dtype=orig_weight.dtype)
ex_bias = self.ex_bias.to(target.device, dtype=target.dtype)
else:
ex_bias = None
return self.finalize_updown(updown, orig_weight, output_shape, ex_bias)
return self.finalize_updown(updown, target, output_shape, ex_bias)
extensions-builtin/Lora/network_glora.py
+30
View File
@@ -0,0 +1,30 @@
import network
class ModuleTypeGLora(network.ModuleType):
def create_module(self, net: network.Network, weights: network.NetworkWeights):
if all(x in weights.w for x in ["a1.weight", "a2.weight", "alpha", "b1.weight", "b2.weight"]):
return NetworkModuleGLora(net, weights)
return None
# adapted from https://github.com/KohakuBlueleaf/LyCORIS
class NetworkModuleGLora(network.NetworkModule): # pylint: disable=abstract-method
def __init__(self, net: network.Network, weights: network.NetworkWeights):
super().__init__(net, weights)
if hasattr(self.sd_module, 'weight'):
self.shape = self.sd_module.weight.shape
self.w1a = weights.w["a1.weight"]
self.w1b = weights.w["b1.weight"]
self.w2a = weights.w["a2.weight"]
self.w2b = weights.w["b2.weight"]
def calc_updown(self, target): # pylint: disable=arguments-differ
w1a = self.w1a.to(target.device, dtype=target.dtype)
w1b = self.w1b.to(target.device, dtype=target.dtype)
w2a = self.w2a.to(target.device, dtype=target.dtype)
w2b = self.w2b.to(target.device, dtype=target.dtype)
output_shape = [w1a.size(0), w1b.size(1)]
updown = (w2b @ w1b) + ((target @ w2a) @ w1a)
return self.finalize_updown(updown, target, output_shape)
extensions-builtin/Lora/network_hada.py
+8 -8
View File
@@ -22,15 +22,15 @@ class NetworkModuleHada(network.NetworkModule):
self.t1 = weights.w.get("hada_t1")
self.t2 = weights.w.get("hada_t2")
def calc_updown(self, orig_weight):
w1a = self.w1a.to(orig_weight.device, dtype=orig_weight.dtype)
w1b = self.w1b.to(orig_weight.device, dtype=orig_weight.dtype)
w2a = self.w2a.to(orig_weight.device, dtype=orig_weight.dtype)
w2b = self.w2b.to(orig_weight.device, dtype=orig_weight.dtype)
def calc_updown(self, target):
w1a = self.w1a.to(target.device, dtype=target.dtype)
w1b = self.w1b.to(target.device, dtype=target.dtype)
w2a = self.w2a.to(target.device, dtype=target.dtype)
w2b = self.w2b.to(target.device, dtype=target.dtype)
output_shape = [w1a.size(0), w1b.size(1)]
if self.t1 is not None:
output_shape = [w1a.size(1), w1b.size(1)]
t1 = self.t1.to(orig_weight.device, dtype=orig_weight.dtype)
t1 = self.t1.to(target.device, dtype=target.dtype)
updown1 = lyco_helpers.make_weight_cp(t1, w1a, w1b)
output_shape += t1.shape[2:]
else:
@@ -38,9 +38,9 @@ class NetworkModuleHada(network.NetworkModule):
output_shape += w1b.shape[2:]
updown1 = lyco_helpers.rebuild_conventional(w1a, w1b, output_shape)
if self.t2 is not None:
t2 = self.t2.to(orig_weight.device, dtype=orig_weight.dtype)
t2 = self.t2.to(target.device, dtype=target.dtype)
updown2 = lyco_helpers.make_weight_cp(t2, w2a, w2b)
else:
updown2 = lyco_helpers.rebuild_conventional(w2a, w2b, output_shape)
updown = updown1 * updown2
return self.finalize_updown(updown, orig_weight, output_shape)
return self.finalize_updown(updown, target, output_shape)
extensions-builtin/Lora/network_ia3.py
+5 -5
View File
@@ -15,12 +15,12 @@ class NetworkModuleIa3(network.NetworkModule):
self.w = weights.w["weight"]
self.on_input = weights.w["on_input"].item()
def calc_updown(self, orig_weight):
w = self.w.to(orig_weight.device, dtype=orig_weight.dtype)
output_shape = [w.size(0), orig_weight.size(1)]
def calc_updown(self, target):
w = self.w.to(target.device, dtype=target.dtype)
output_shape = [w.size(0), target.size(1)]
if self.on_input:
output_shape.reverse()
else:
w = w.reshape(-1, 1)
updown = orig_weight * w
return self.finalize_updown(updown, orig_weight, output_shape)
updown = target * w
return self.finalize_updown(updown, target, output_shape)
extensions-builtin/Lora/network_lokr.py
+13 -13
View File
@@ -32,26 +32,26 @@ class NetworkModuleLokr(network.NetworkModule):
self.dim = self.w2b.shape[0] if self.w2b is not None else self.dim
self.t2 = weights.w.get("lokr_t2")
def calc_updown(self, orig_weight):
def calc_updown(self, target):
if self.w1 is not None:
w1 = self.w1.to(orig_weight.device, dtype=orig_weight.dtype)
w1 = self.w1.to(target.device, dtype=target.dtype)
else:
w1a = self.w1a.to(orig_weight.device, dtype=orig_weight.dtype)
w1b = self.w1b.to(orig_weight.device, dtype=orig_weight.dtype)
w1a = self.w1a.to(target.device, dtype=target.dtype)
w1b = self.w1b.to(target.device, dtype=target.dtype)
w1 = w1a @ w1b
if self.w2 is not None:
w2 = self.w2.to(orig_weight.device, dtype=orig_weight.dtype)
w2 = self.w2.to(target.device, dtype=target.dtype)
elif self.t2 is None:
w2a = self.w2a.to(orig_weight.device, dtype=orig_weight.dtype)
w2b = self.w2b.to(orig_weight.device, dtype=orig_weight.dtype)
w2a = self.w2a.to(target.device, dtype=target.dtype)
w2b = self.w2b.to(target.device, dtype=target.dtype)
w2 = w2a @ w2b
else:
t2 = self.t2.to(orig_weight.device, dtype=orig_weight.dtype)
w2a = self.w2a.to(orig_weight.device, dtype=orig_weight.dtype)
w2b = self.w2b.to(orig_weight.device, dtype=orig_weight.dtype)
t2 = self.t2.to(target.device, dtype=target.dtype)
w2a = self.w2a.to(target.device, dtype=target.dtype)
w2b = self.w2b.to(target.device, dtype=target.dtype)
w2 = lyco_helpers.make_weight_cp(t2, w2a, w2b)
output_shape = [w1.size(0) * w2.size(0), w1.size(1) * w2.size(1)]
if len(orig_weight.shape) == 4:
output_shape = orig_weight.shape
if len(target.shape) == 4:
output_shape = target.shape
updown = make_kron(output_shape, w1, w2)
return self.finalize_updown(updown, orig_weight, output_shape)
return self.finalize_updown(updown, target, output_shape)
extensions-builtin/Lora/network_lora.py
+5 -5
View File
@@ -51,20 +51,20 @@ class NetworkModuleLora(network.NetworkModule):
module.weight.requires_grad_(False)
return module
def calc_updown(self, orig_weight): # pylint: disable=W0237
up = self.up_model.weight.to(orig_weight.device, dtype=orig_weight.dtype)
down = self.down_model.weight.to(orig_weight.device, dtype=orig_weight.dtype)
def calc_updown(self, target): # pylint: disable=W0237
up = self.up_model.weight.to(target.device, dtype=target.dtype)
down = self.down_model.weight.to(target.device, dtype=target.dtype)
output_shape = [up.size(0), down.size(1)]
if self.mid_model is not None:
# cp-decomposition
mid = self.mid_model.weight.to(orig_weight.device, dtype=orig_weight.dtype)
mid = self.mid_model.weight.to(target.device, dtype=target.dtype)
updown = lyco_helpers.rebuild_cp_decomposition(up, down, mid)
output_shape += mid.shape[2:]
else:
if len(down.shape) == 4:
output_shape += down.shape[2:]
updown = lyco_helpers.rebuild_conventional(up, down, output_shape, self.network.dyn_dim)
return self.finalize_updown(updown, orig_weight, output_shape)
return self.finalize_updown(updown, target, output_shape)
def forward(self, x, y):
self.up_model.to(device=devices.device)
extensions-builtin/Lora/network_norm.py
+4 -4
View File
@@ -14,11 +14,11 @@ class NetworkModuleNorm(network.NetworkModule):
self.w_norm = weights.w.get("w_norm")
self.b_norm = weights.w.get("b_norm")
def calc_updown(self, orig_weight):
def calc_updown(self, target):
output_shape = self.w_norm.shape
updown = self.w_norm.to(orig_weight.device, dtype=orig_weight.dtype)
updown = self.w_norm.to(target.device, dtype=target.dtype)
if self.b_norm is not None:
ex_bias = self.b_norm.to(orig_weight.device, dtype=orig_weight.dtype)
ex_bias = self.b_norm.to(target.device, dtype=target.dtype)
else:
ex_bias = None
return self.finalize_updown(updown, orig_weight, output_shape, ex_bias)
return self.finalize_updown(updown, target, output_shape, ex_bias)
extensions-builtin/Lora/network_oft.py
+71 -35
View File
@@ -1,49 +1,85 @@
import torch
import diffusers.models.lora as diffusers_lora
import network
from modules import devices
from lyco_helpers import factorization
from einops import rearrange
class ModuleTypeOFT(network.ModuleType):
def create_module(self, net: network.Network, weights: network.NetworkWeights):
"""
weights.w.items()
alpha : tensor(0.0010, dtype=torch.bfloat16)
oft_blocks : tensor([[[ 0.0000e+00, 1.4400e-04, 1.7319e-03, ..., -8.8882e-04,
5.7373e-03, -4.4250e-03],
[-1.4400e-04, 0.0000e+00, 8.6594e-04, ..., 1.5945e-03,
-8.5449e-04, 1.9684e-03], ...etc...
, dtype=torch.bfloat16)"""
if "oft_blocks" in weights.w.keys():
module = NetworkModuleOFT(net, weights)
return module
else:
return None
if all(x in weights.w for x in ["oft_blocks"]) or all(x in weights.w for x in ["oft_diag"]):
return NetworkModuleOFT(net, weights)
return None
# Supports both kohya-ss' implementation of COFT https://github.com/kohya-ss/sd-scripts/blob/main/networks/oft.py
# and KohakuBlueleaf's implementation of OFT/COFT https://github.com/KohakuBlueleaf/LyCORIS/blob/dev/lycoris/modules/diag_oft.py
class NetworkModuleOFT(network.NetworkModule):
def __init__(self, net: network.Network, weights: network.NetworkWeights):
def __init__(self, net: network.Network, weights: network.NetworkWeights):
super().__init__(net, weights)
self.weights = weights.w.get("oft_blocks").to(device=devices.device)
self.dim = self.weights.shape[0] # num blocks
self.alpha = self.multiplier()
self.block_size = self.weights.shape[-1]
self.lin_module = None
self.org_module: list[torch.Module] = [self.sd_module]
def get_weight(self):
block_Q = self.weights - self.weights.transpose(1, 2)
I = torch.eye(self.block_size, device=devices.device).unsqueeze(0).repeat(self.dim, 1, 1)
block_R = torch.matmul(I + block_Q, (I - block_Q).inverse())
block_R_weighted = self.alpha * block_R + (1 - self.alpha) * I
R = torch.block_diag(*block_R_weighted)
return R
self.scale = 1.0
def calc_updown(self, orig_weight):
R = self.get_weight().to(device=devices.device, dtype=orig_weight.dtype)
if orig_weight.dim() == 4:
updown = torch.einsum("oihw, op -> pihw", orig_weight, R) * self.calc_scale()
# kohya-ss
if "oft_blocks" in weights.w.keys():
self.is_kohya = True
self.oft_blocks = weights.w["oft_blocks"] # (num_blocks, block_size, block_size)
self.alpha = weights.w["alpha"] # alpha is constraint
self.dim = self.oft_blocks.shape[0] # lora dim
# LyCORIS
elif "oft_diag" in weights.w.keys():
self.is_kohya = False
self.oft_blocks = weights.w["oft_diag"]
# self.alpha is unused
self.dim = self.oft_blocks.shape[1] # (num_blocks, block_size, block_size)
is_linear = type(self.sd_module) in [torch.nn.Linear, torch.nn.modules.linear.NonDynamicallyQuantizableLinear]
is_conv = type(self.sd_module) in [torch.nn.Conv2d]
is_other_linear = type(self.sd_module) in [torch.nn.MultiheadAttention] # unsupported
if is_linear:
self.out_dim = self.sd_module.out_features
elif is_conv:
self.out_dim = self.sd_module.out_channels
elif is_other_linear:
self.out_dim = self.sd_module.embed_dim
if self.is_kohya:
self.constraint = self.alpha * self.out_dim
self.num_blocks = self.dim
self.block_size = self.out_dim // self.dim
else:
updown = torch.einsum("oi, op -> pi", orig_weight, R) * self.calc_scale()
self.constraint = None
self.block_size, self.num_blocks = factorization(self.out_dim, self.dim)
return self.finalize_updown(updown, orig_weight, orig_weight.shape)
def calc_updown(self, target):
oft_blocks = self.oft_blocks.to(target.device, dtype=target.dtype)
eye = torch.eye(self.block_size, device=target.device)
constraint = self.constraint.to(target.device)
if self.is_kohya:
block_Q = oft_blocks - oft_blocks.transpose(1, 2) # ensure skew-symmetric orthogonal matrix
norm_Q = torch.norm(block_Q.flatten()).to(target.device)
new_norm_Q = torch.clamp(norm_Q, max=constraint)
block_Q = block_Q * ((new_norm_Q + 1e-8) / (norm_Q + 1e-8))
mat1 = eye + block_Q
mat2 = (eye - block_Q).float().inverse()
oft_blocks = torch.matmul(mat1, mat2)
R = oft_blocks.to(target.device, dtype=target.dtype)
# This errors out for MultiheadAttention, might need to be handled up-stream
merged_weight = rearrange(target, '(k n) ... -> k n ...', k=self.num_blocks, n=self.block_size)
merged_weight = torch.einsum(
'k n m, k n ... -> k m ...',
R,
merged_weight
)
merged_weight = rearrange(merged_weight, 'k m ... -> (k m) ...')
updown = merged_weight.to(target.device, dtype=target.dtype) - target
output_shape = target.shape
return self.finalize_updown(updown, target, output_shape)
extensions-builtin/Lora/networks.py
+41 -20
View File
@@ -1,4 +1,4 @@
from typing import Union
from typing import Union, List
import os
import re
import time
@@ -11,18 +11,19 @@ import network_oft
import network_lokr
import network_full
import network_norm
import network_glora
import lora_convert
import torch
import diffusers.models.lora
from modules import shared, devices, sd_models, sd_models_compile, errors, scripts, sd_hijack
debug = os.environ.get('SD_LORA_DEBUG', None)
debug = os.environ.get('SD_LORA_DEBUG', None) is not None
originals: lora_patches.LoraPatches = None
extra_network_lora = None
available_networks = {}
available_network_aliases = {}
loaded_networks = []
loaded_networks: List[network.Network] = []
timer = { 'load': 0, 'apply': 0, 'restore': 0 }
# networks_in_memory = {}
lora_cache = {}
@@ -37,6 +38,7 @@ module_types = [
network_lokr.ModuleTypeLokr(),
network_full.ModuleTypeFull(),
network_norm.ModuleTypeNorm(),
network_glora.ModuleTypeGLora(),
]
convert_diffusers_name_to_compvis = lora_convert.convert_diffusers_name_to_compvis # supermerger compatibility item
@@ -74,17 +76,17 @@ def assign_network_names_to_compvis_modules(sd_model):
sd_model.network_layer_mapping = network_layer_mapping
def load_diffusers(name, network_on_disk, lora_scale=1.0):
def load_diffusers(name, network_on_disk, lora_scale=1.0) -> network.Network:
t0 = time.time()
cached = lora_cache.get(name, None)
# if debug:
shared.log.debug(f'LoRA load: name={name} file={network_on_disk.filename} type=diffusers {"cached" if cached else ""}')
shared.log.debug(f'LoRA load: name="{name}" file="{network_on_disk.filename}" type=diffusers {"cached" if cached else ""} fuse={shared.opts.lora_fuse_diffusers}')
if cached is not None:
return cached
if shared.backend != shared.Backend.DIFFUSERS:
return None
shared.sd_model.load_lora_weights(network_on_disk.filename)
if shared.opts.cuda_compile and shared.opts.cuda_compile_backend == "openvino_fx":
if shared.opts.lora_fuse_diffusers:
shared.sd_model.fuse_lora(lora_scale=lora_scale)
net = network.Network(name, network_on_disk)
net.mtime = os.path.getmtime(network_on_disk.filename)
@@ -94,11 +96,11 @@ def load_diffusers(name, network_on_disk, lora_scale=1.0):
return net
def load_network(name, network_on_disk):
def load_network(name, network_on_disk) -> network.Network:
t0 = time.time()
cached = lora_cache.get(name, None)
if debug:
shared.log.debug(f'LoRA load: name={name} file={network_on_disk.filename} {"cached" if cached else ""}')
shared.log.debug(f'LoRA load: name="{name}" file="{network_on_disk.filename}" type=lora {"cached" if cached else ""}')
if cached is not None:
return cached
net = network.Network(name, network_on_disk)
@@ -109,7 +111,16 @@ def load_network(name, network_on_disk):
matched_networks = {}
convert = lora_convert.KeyConvert()
for key_network, weight in sd.items():
key_network_without_network_parts, network_part = key_network.split(".", 1)
parts = key_network.split('.')
if len(parts) > 5: # messy handler for diffusers peft lora
key_network_without_network_parts = '_'.join(parts[:-2])
if not key_network_without_network_parts.startswith('lora_'):
key_network_without_network_parts = 'lora_' + key_network_without_network_parts
network_part = '.'.join(parts[-2:]).replace('lora_A', 'lora_down').replace('lora_B', 'lora_up')
else:
key_network_without_network_parts, network_part = key_network.split(".", 1)
# if debug:
# shared.log.debug(f'LoRA load: name="{name}" full={key_network} network={network_part} key={key_network_without_network_parts}')
key, sd_module = convert(key_network_without_network_parts)
if sd_module is None:
keys_failed_to_match[key_network] = key
@@ -124,12 +135,15 @@ def load_network(name, network_on_disk):
if net_module is not None:
break
if net_module is None:
raise AssertionError(f"Could not find a module type (out of {', '.join([x.__class__.__name__ for x in module_types])}) that would accept those keys: {', '.join(weights.w)}")
net.modules[key] = net_module
if keys_failed_to_match:
shared.log.warning(f"LoRA unmatched keys: file={network_on_disk.filename} keys={len(keys_failed_to_match)}")
shared.log.error(f'LoRA unhandled: name={name} key={key} weights={weights.w.keys()}')
else:
net.modules[key] = net_module
if len(keys_failed_to_match) > 0:
shared.log.warning(f"LoRA file={network_on_disk.filename} unmatched={len(keys_failed_to_match)} matched={len(matched_networks)}")
if debug:
shared.log.debug(f"LoRA unmatched keys: file={network_on_disk.filename} keys={keys_failed_to_match}")
shared.log.debug(f"LoRA file={network_on_disk.filename} unmatched={keys_failed_to_match}")
elif debug:
shared.log.debug(f"LoRA file={network_on_disk.filename} unmatched={len(keys_failed_to_match)} matched={len(matched_networks)}")
lora_cache[name] = net
t1 = time.time()
timer['load'] += t1 - t0
@@ -167,10 +181,12 @@ def load_networks(names, te_multipliers=None, unet_multipliers=None, dyn_dims=No
for i, (network_on_disk, name) in enumerate(zip(networks_on_disk, names)):
net = None
if network_on_disk is not None:
if debug:
shared.log.debug(f'LoRA load start: name="{name}" file="{network_on_disk.filename}"')
try:
if recompile_model:
shared.compiled_model_state.lora_model.append(f"{name}:{te_multipliers[i] if te_multipliers else 1.0}")
if shared.backend == shared.Backend.DIFFUSERS and (os.environ.get('SD_LORA_DIFFUSERS', None) is not None): # OpenVINO only works with Diffusers LoRa loading.
if shared.backend == shared.Backend.DIFFUSERS and shared.opts.lora_force_diffusers: # OpenVINO only works with Diffusers LoRa loading.
# or getattr(network_on_disk, 'shorthash', '').lower() == 'aaebf6360f7d' # sd15-lcm
# or getattr(network_on_disk, 'shorthash', '').lower() == '3d18b05e4f56' # sdxl-lcm
# or getattr(network_on_disk, 'shorthash', '').lower() == '813ea5fb1c67' # turbo sdxl-turbo
@@ -186,7 +202,7 @@ def load_networks(names, te_multipliers=None, unet_multipliers=None, dyn_dims=No
network_on_disk.read_hash()
if net is None:
failed_to_load_networks.append(name)
shared.log.error(f"LoRA unknown: network={name}")
shared.log.error(f"LoRA unknown type: network={name}")
continue
net.te_multiplier = te_multipliers[i] if te_multipliers else 1.0
net.unet_multiplier = unet_multipliers[i] if unet_multipliers else 1.0
@@ -269,10 +285,11 @@ def network_apply_weights(self: Union[torch.nn.Conv2d, torch.nn.Linear, torch.nn
if current_names != wanted_names:
network_restore_weights_from_backup(self)
for net in loaded_networks:
# default workflow where module is known and has weights
module = net.modules.get(network_layer_name, None)
if module is not None and hasattr(self, 'weight'):
try:
with torch.no_grad():
with devices.inference_context():
updown, ex_bias = module.calc_updown(self.weight)
if len(self.weight.shape) == 4 and self.weight.shape[1] == 9:
# inpainting model. zero pad updown to make channel[1] 4 to 9
@@ -284,17 +301,21 @@ def network_apply_weights(self: Union[torch.nn.Conv2d, torch.nn.Linear, torch.nn
else:
self.bias += ex_bias
except RuntimeError as e:
if debug:
shared.log.debug(f"LoRA apply weight network={net.name} layer={network_layer_name} {e}")
extra_network_lora.errors[net.name] = extra_network_lora.errors.get(net.name, 0) + 1
if debug:
module_name = net.modules.get(network_layer_name, None)
shared.log.error(f"LoRA apply weight name={net.name} module={module_name} layer={network_layer_name} {e}")
errors.display(e, 'LoRA apply weight')
raise RuntimeError('LoRA apply weight') from e
continue
# alternative workflow looking at _*_proj layers
module_q = net.modules.get(network_layer_name + "_q_proj", None)
module_k = net.modules.get(network_layer_name + "_k_proj", None)
module_v = net.modules.get(network_layer_name + "_v_proj", None)
module_out = net.modules.get(network_layer_name + "_out_proj", None)
if isinstance(self, torch.nn.MultiheadAttention) and module_q and module_k and module_v and module_out:
try:
with torch.no_grad():
with devices.inference_context():
updown_q, _ = module_q.calc_updown(self.in_proj_weight)
updown_k, _ = module_k.calc_updown(self.in_proj_weight)
updown_v, _ = module_v.calc_updown(self.in_proj_weight)
extensions-builtin/Lora/ui_extra_networks_lora.py
-1
View File
@@ -19,7 +19,6 @@ class ExtraNetworksPageLora(ui_extra_networks.ExtraNetworksPage):
try:
path, _ext = os.path.splitext(l.filename)
name = os.path.splitext(os.path.relpath(l.filename, shared.cmd_opts.lora_dir))[0]
if shared.backend == shared.Backend.ORIGINAL:
if l.sd_version == network.SdVersion.SDXL:
return None
extensions-builtin/sd-extension-chainner
+1 -1
Submodule extensions-builtin/sd-extension-chainner updated: 1cdc8578a3...e48020d035
extensions-builtin/sd-extension-system-info
+1 -1
Submodule extensions-builtin/sd-extension-system-info updated: 1841cf7627...f934808698
extensions-builtin/sd-webui-agent-scheduler
+1 -1
Submodule extensions-builtin/sd-webui-agent-scheduler updated: a8d527b269...435dd9bdec
extensions-builtin/sd-webui-controlnet
+1 -1
Submodule extensions-builtin/sd-webui-controlnet updated: 10bd9b25f6...dc0d590faf
extensions-builtin/stable-diffusion-webui-images-browser
+1 -1
Submodule extensions-builtin/stable-diffusion-webui-images-browser updated: 08fc2647f1...27fe4a713d
html/emerald-paradise.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 181 KiB

html/locale_en.json
+13 -12
View File
@@ -43,17 +43,18 @@
"tabs": [
{"id":"","label":"Text","localized":"","hint":"Create image from text"},
{"id":"","label":"Image","localized":"","hint":"Create image from image"},
{"id":"","label":"Control","localized":"","hint":"Create image with additional control"},
{"id":"","label":"Process","localized":"","hint":"Process existing image"},
{"id":"","label":"Train","localized":"","hint":"Run training or model merging"},
{"id":"","label":"Interrogate","localized":"","hint":"Run interrogate to get description of your image"},
{"id":"","label":"Train","localized":"","hint":"Run training"},
{"id":"","label":"Models","localized":"","hint":"Convert or merge your models"},
{"id":"","label":"Interrogator","localized":"","hint":"Run interrogate to get description of your image"},
{"id":"","label":"System Info","localized":"","hint":"System information"},
{"id":"","label":"Agent Scheduler","localized":"","hint":"Enqueue your generate requests and run them in the background"},
{"id":"","label":"Image Browser","localized":"","hint":"Browse through your generated image database"},
{"id":"","label":"System","localized":"","hint":"System settings and information"},
{"id":"","label":"System Info","localized":"","hint":"System information"},
{"id":"","label":"Settings","localized":"","hint":"Application settings"},
{"id":"","label":"Extensions","localized":"","hint":"Application extensions"},
{"id":"","label":"Script","localized":"","hint":"Addtional scripts to be used"}
{"id":"","label":"Script","localized":"","hint":"Addtional scripts to be used"},
{"id":"","label":"Extensions","localized":"","hint":"Application extensions"}
],
"action panel": [
{"id":"","label":"Generate","localized":"","hint":"Start processing"},
@@ -226,10 +227,10 @@
{"id":"","label":"Inpaint batch input directory","localized":"","hint":""},
{"id":"","label":"Inpaint batch output directory","localized":"","hint":""},
{"id":"","label":"Inpaint batch mask directory","localized":"","hint":""},
{"id":"","label":"Resize fixed","localized":"","hint":"Resize image to target resolution. Unless height and width match, you will get incorrect aspect ratio"},
{"id":"","label":"Crop and resize","localized":"","hint":"Resize the image so that entirety of target resolution is filled with the image. Crop parts that stick out"},
{"id":"","label":"Resize and fill","localized":"","hint":"Resize the image so that entirety of image is inside target resolution. Fill empty space with image's colors"},
{"id":"","label":"Latent upscale","localized":"","hint":""},
{"id":"","label":"Fixed","localized":"","hint":"Resize image to target resolution. Unless height and width match, you will get incorrect aspect ratio"},
{"id":"","label":"Crop","localized":"","hint":"Resize the image so that entirety of target resolution is filled with the image. Crop parts that stick out"},
{"id":"","label":"Fill","localized":"","hint":"Resize the image so that entirety of image is inside target resolution. Fill empty space with image's colors"},
{"id":"","label":"Latent","localized":"","hint":""},
{"id":"","label":"Mask blur","localized":"","hint":"How much to blur the mask before processing, in pixels"},
{"id":"","label":"Mask transparency","localized":"","hint":""},
{"id":"","label":"Inpaint masked","localized":"","hint":""},
@@ -484,8 +485,8 @@
{"id":"","label":"Dark","localized":"","hint":""},
{"id":"","label":"Light","localized":"","hint":""},
{"id":"","label":"Show grid in results","localized":"","hint":""},
{"id":"","label":"For inpainting, include the greyscale mask in results","localized":"","hint":""},
{"id":"","label":"For inpainting, include masked composite in results","localized":"","hint":""},
{"id":"","label":"Inpainting include greyscale mask in results","localized":"","hint":""},
{"id":"","label":"Inpainting include masked composite in results","localized":"","hint":""},
{"id":"","label":"Do not change selected model when reading generation parameters","localized":"","hint":""},
{"id":"","label":"Send seed when sending prompt or image to other interface","localized":"","hint":""},
{"id":"","label":"Send size when sending prompt or image to another interface","localized":"","hint":""},
@@ -500,7 +501,7 @@
{"id":"","label":"Show previews of all images generated in a batch as a grid","localized":"","hint":""},
{"id":"","label":"Play a sound when images are finished generating","localized":"","hint":""},
{"id":"","label":"Path to notification sound","localized":"","hint":""},
{"id":"","label":"Live preview display period","localized":"","hint":""},
{"id":"","label":"Live preview display period","localized":"","hint":"Request preview image every n steps, set to 0 to disable"},
{"id":"","label":"Full VAE","localized":"","hint":""},
{"id":"","label":"Approximate","localized":"","hint":"Cheap neural network approximation. Very fast compared to VAE, but produces pictures with 4 times smaller horizontal/vertical resolution and lower quality"},
{"id":"","label":"Simple","localized":"","hint":"Very cheap approximation. Very fast compared to VAE, but produces pictures with 8 times smaller horizontal/vertical resolution and extremely low quality"},
html/orchid-dreams.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 157 KiB

html/reference.json
+80 -5
View File
@@ -1,13 +1,46 @@
{
"DreamShaper SD 1.5 v8": {
"path": "dreamshaper_8.safetensors@https://civitai.com/api/download/models/128713",
"desc": "Showcase finetuned model based on Stable diffusion 1.5",
"preview": "dreamshaper_8.jpg",
"original": true
},
"DreamShaper SD XL Turbo": {
"path": "dreamshaperXL_turboDpmppSDE.safetensors@https://civitai.com/api/download/models/251662",
"desc": "Showcase finetuned model based on Stable diffusion XL",
"preview": "dreamshaperXL_turboDpmppSDE.jpg"
},
"Juggernaut Reborn": {
"path": "juggernaut_reborn.safetensors@https://civitai.com/api/download/models/274039",
"desc": "Showcase finetuned model based on Stable diffusion 1.5",
"preview": "juggernaut_reborn.jpg",
"original": true
},
"Juggernaut XL v7 RunDiffusion": {
"path": "juggernautXL_v7Rundiffusion.safetensors@https://civitai.com/api/download/models/240840",
"desc": "Showcase finetuned model based on Stable diffusion XL",
"preview": "juggernautXL_v7Rundiffusion.jpg"
},
"RunwayML SD 1.5": {
"path": "runwayml/stable-diffusion-v1-5",
"alt": "v1-5-pruned-emaonly.safetensors@https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.safetensors?download=true",
"desc": "Stable Diffusion 1.5 is the base model all other 1.5 checkpoint were trained from. It's a latent text-to-image diffusion model capable of generating photo-realistic images given any text input. The Stable-Diffusion-v1-5 checkpoint was initialized with the weights of the Stable-Diffusion-v1-2 checkpoint and subsequently fine-tuned on 595k steps at resolution 512x512.",
"preview": "runwayml--stable-diffusion-v1-5.jpg"
"preview": "runwayml--stable-diffusion-v1-5.jpg",
"original": true
},
"StabilityAI SD 2.1": {
"StabilityAI SD 2.1 EMA": {
"path": "stabilityai/stable-diffusion-2-1-base",
"desc": "This stable-diffusion-2-1 model is fine-tuned from stable-diffusion-2 (768-v-ema.ckpt) with an additional 55k steps on the same dataset. Improvement over base 1.5 model, but never really took off.",
"preview": "stabilityai--stable-diffusion-2.1-base.jpg"
"alt": "v2-1_512-ema-pruned.safetensors@https://huggingface.co/stabilityai/stable-diffusion-2-1-base/resolve/main/v2-1_512-ema-pruned.safetensors?download=true",
"desc": "This stable-diffusion-2-1-base model fine-tunes stable-diffusion-2-base (512-base-ema.ckpt) with 220k extra steps taken",
"preview": "stabilityai--stable-diffusion-2.1-base.jpg",
"original": true
},
"StabilityAI SD 2.1 V": {
"path": "stabilityai/stable-diffusion-2-1-base",
"alt": "v2-1_768-ema-pruned.safetensors@https://huggingface.co/stabilityai/stable-diffusion-2-1/resolve/main/v2-1_768-ema-pruned.safetensors?download=true",
"desc": "This stable-diffusion-2 model is resumed from stable-diffusion-2-base (512-base-ema.ckpt) and trained for 150k steps using a v-objective on the same dataset. Resumed for another 140k steps on 768x768 images",
"preview": "stabilityai--stable-diffusion-2.1-base.jpg",
"original": true
},
"StabilityAI SD-XL 1.0 Base": {
"path": "stabilityai/stable-diffusion-xl-base-1.0",
@@ -16,8 +49,10 @@
},
"StabilityAI SD 2.1 Turbo": {
"path": "stabilityai/sd-turbo",
"alt": "sd_turbo.safetensors@https://huggingface.co/stabilityai/sd-turbo/resolve/main/sd_turbo.safetensors?download=true",
"desc": "SD-Turbo is a distilled version of Stable Diffusion 2.1, trained for real-time synthesis. SD-Turbo is based on a novel training method called Adversarial Diffusion Distillation (ADD) (see the technical report), which allows sampling large-scale foundational image diffusion models in 1 to 4 steps at high image quality. This approach uses score distillation to leverage large-scale off-the-shelf image diffusion models as a teacher signal and combines this with an adversarial loss to ensure high image fidelity even in the low-step regime of one or two sampling steps.",
"preview": "stabilityai--sd-turbo.jpg"
"preview": "stabilityai--sd-turbo.jpg",
"original": true
},
"StabilityAI SD-XL Turbo": {
"path": "stabilityai/sdxl-turbo",
@@ -34,6 +69,11 @@
"desc": "(SVD) Image-to-Video is a latent diffusion model trained to generate short video clips from an image conditioning. This model was trained to generate 25 frames at resolution 576x1024 given a context frame of the same size, finetuned from SVD Image-to-Video [14 frames]. We also finetune the widely used f8-decoder for temporal consistency.",
"preview": "stabilityai--stable-video-diffusion-img2vid-xt.jpg"
},
"Segmind Vega": {
"path": "segmind/Segmind-Vega",
"desc": "The Segmind-Vega Model is a distilled version of the Stable Diffusion XL (SDXL), offering a remarkable 70% reduction in size and an impressive 100% speedup while retaining high-quality text-to-image generation capabilities. Trained on diverse datasets, including Grit and Midjourney scrape data, it excels at creating a wide range of visual content based on textual prompts. Employing a knowledge distillation strategy, Segmind-Vega leverages the teachings of several expert models, including SDXL, ZavyChromaXL, and JuggernautXL, to combine their strengths and produce compelling visual outputs.",
"preview": "segmind--Segmind-Vega.jpg"
},
"Segmind SSD-1B": {
"path": "segmind/SSD-1B",
"desc": "The Segmind Stable Diffusion Model (SSD-1B) offers a compact, efficient, and distilled version of the SDXL model. At 50% smaller and 60% faster than Stable Diffusion XL (SDXL), it provides quick and seamless performance without sacrificing image quality.",
@@ -79,14 +119,49 @@
"desc": "Kandinsky 3.0 is an open-source text-to-image diffusion model built upon the Kandinsky2-x model family. In comparison to its predecessors, Kandinsky 3.0 incorporates more data and specifically related to Russian culture, which allows to generate pictures related to Russin culture. Furthermore, enhancements have been made to the text understanding and visual quality of the model, achieved by increasing the size of the text encoder and Diffusion U-Net models, respectively.",
"preview": "kandinsky-community--kandinsky-3.jpg"
},
"Playground v1": {
"path": "playgroundai/playground-v1",
"desc": "Playground v1 is a latent diffusion model that improves the overall HDR quality to get more stunning images.",
"preview": "playgroundai--playground-v1.jpg"
},
"Playground v2 256": {
"path": "playgroundai/playground-v2-256px-base",
"desc": "Playground v2 is a diffusion-based text-to-image generative model. The model was trained from scratch by the research team at Playground. Images generated by Playground v2 are favored 2.5 times more than those produced by Stable Diffusion XL, according to Playground’s user study.",
"preview": "playgroundai--playground-v2-256px-base.jpg"
},
"Playground v2 512": {
"path": "playgroundai/playground-v2-512px-base",
"desc": "Playground v2 is a diffusion-based text-to-image generative model. The model was trained from scratch by the research team at Playground. Images generated by Playground v2 are favored 2.5 times more than those produced by Stable Diffusion XL, according to Playground’s user study.",
"preview": "playgroundai--playground-v2-512px-base.jpg"
},
"Playground v2 1024": {
"path": "playgroundai/playground-v2-1024px-aesthetic",
"desc": "Playground v2 is a diffusion-based text-to-image generative model. The model was trained from scratch by the research team at Playground. Images generated by Playground v2 are favored 2.5 times more than those produced by Stable Diffusion XL, according to Playground’s user study.",
"preview": "playgroundai--playground-v2-1024px-aesthetic.jpg"
},
"DeepFloyd IF Medium": {
"path": "DeepFloyd/IF-I-M-v1.0",
"desc": "DeepFloyd-IF is a pixel-based text-to-image triple-cascaded diffusion model, that can generate pictures with new state-of-the-art for photorealism and language understanding. The result is a highly efficient model that outperforms current state-of-the-art models, achieving a zero-shot FID-30K score of 6.66 on the COCO dataset. It is modular and composed of frozen text mode and three pixel cascaded diffusion modules, each designed to generate images of increasing resolution: 64x64, 256x256, and 1024x1024.",
"preview": "DeepFloyd--IF-I-M-v1.0.jpg"
},
"aMUSEd 256": {
"path": "amused/amused-256",
"desc": "Amused is a lightweight text to image model based off of the muse architecture. Amused is particularly useful in applications that require a lightweight and fast model such as generating many images quickly at once.",
"preview": "amused--amused-256.jpg"
},
"aMUSEd 512": {
"path": "amused/amused-512",
"desc": "Amused is a lightweight text to image model based off of the muse architecture. Amused is particularly useful in applications that require a lightweight and fast model such as generating many images quickly at once.",
"preview": "amused--amused-512.jpg"
},
"Tsinghua UniDiffuser": {
"path": "thu-ml/unidiffuser-v1",
"desc": "UniDiffuser is a unified diffusion framework to fit all distributions relevant to a set of multi-modal data in one transformer. UniDiffuser is able to perform image, text, text-to-image, image-to-text, and image-text pair generation by setting proper timesteps without additional overhead.\nSpecifically, UniDiffuser employs a variation of transformer, called U-ViT, which parameterizes the joint noise prediction network. Other components perform as encoders and decoders of different modalities, including a pretrained image autoencoder from Stable Diffusion, a pretrained image ViT-B/32 CLIP encoder, a pretrained text ViT-L CLIP encoder, and a GPT-2 text decoder finetuned by ourselves.",
"preview": "thu-ml--unidiffuser-v1.jpg"
},
"SalesForce BLIP-Diffusion": {
"path": "salesforce/blipdiffusion",
"desc": "BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation.",
"preview": "salesforce--blipdiffusion.jpg"
}
}
html/timeless-beige.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 160 KiB

html/xmas-control.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 143 KiB

html/xmas-default.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 196 KiB

installer.py
+34 -43
View File
@@ -67,6 +67,7 @@ def setup_logging():
def get(self):
return self.buffer
from functools import partial, partialmethod
from logging.handlers import RotatingFileHandler
from rich.theme import Theme
from rich.logging import RichHandler
@@ -78,6 +79,11 @@ def setup_logging():
global log_file # pylint: disable=global-statement
log_file = args.log
logging.TRACE = 25
logging.addLevelName(logging.TRACE, 'TRACE')
logging.Logger.trace = partialmethod(logging.Logger.log, logging.TRACE)
logging.trace = partial(logging.log, logging.TRACE)
level = logging.DEBUG if args.debug else logging.INFO
log.setLevel(logging.DEBUG) # log to file is always at level debug for facility `sd`
console = Console(log_time=True, log_time_format='%H:%M:%S-%f', theme=Theme({
@@ -357,7 +363,8 @@ def check_torch():
log.debug(f'Torch allowed: cuda={allow_cuda} rocm={allow_rocm} ipex={allow_ipex} diml={allow_directml} openvino={allow_openvino}')
torch_command = os.environ.get('TORCH_COMMAND', '')
xformers_package = os.environ.get('XFORMERS_PACKAGE', 'none')
install('onnxruntime', 'onnxruntime', ignore=True)
if not installed('onnxruntime', quiet=True) and not installed('onnxruntime-gpu', quiet=True): # allow either
install('onnxruntime', 'onnxruntime', ignore=True)
if torch_command != '':
pass
elif allow_cuda and (shutil.which('nvidia-smi') is not None or args.use_xformers or os.path.exists(os.path.join(os.environ.get('SystemRoot') or r'C:\Windows', 'System32', 'nvidia-smi.exe'))):
@@ -424,18 +431,32 @@ def check_torch():
os.environ.setdefault('NEOReadDebugKeys', '1')
os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
if "linux" in sys.platform:
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.0.1a0 torchvision==0.15.2a0 intel_extension_for_pytorch==2.0.110+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/')
os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow==2.13.0 intel-extension-for-tensorflow[gpu]')
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.1.0a0 torchvision==0.16.0a0 intel-extension-for-pytorch==2.1.10+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/')
os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow==2.14.0 intel-extension-for-tensorflow[xpu]==2.14.0.1')
install(os.environ.get('MKL_PACKAGE', 'mkl==2024.0.0'), 'mkl')
install(os.environ.get('DPCPP_PACKAGE', 'mkl-dpcpp==2024.0.0'), 'mkl-dpcpp')
else:
pytorch_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.0.110%2Bxpu-master%2Bdll-bundle/torch-2.0.0a0+gite9ebda2-cp310-cp310-win_amd64.whl'
torchvision_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.0.110%2Bxpu-master%2Bdll-bundle/torchvision-0.15.2a0+fa99a53-cp310-cp310-win_amd64.whl'
ipex_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.0.110%2Bxpu-master%2Bdll-bundle/intel_extension_for_pytorch-2.0.110+gitc6ea20b-cp310-cp310-win_amd64.whl'
if sys.version_info[1] == 11:
pytorch_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torch-2.1.0a0+cxx11.abi-cp311-cp311-win_amd64.whl'
torchvision_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torchvision-0.16.0a0+cxx11.abi-cp311-cp311-win_amd64.whl'
ipex_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/intel_extension_for_pytorch-2.1.10+xpu-cp311-cp311-win_amd64.whl'
elif sys.version_info[1] == 10:
pytorch_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torch-2.1.0a0+cxx11.abi-cp310-cp310-win_amd64.whl'
torchvision_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torchvision-0.16.0a0+cxx11.abi-cp310-cp310-win_amd64.whl'
ipex_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/intel_extension_for_pytorch-2.1.10+xpu-cp310-cp310-win_amd64.whl'
else:
pytorch_pip = 'torch==2.1.0a0'
torchvision_pip = 'torchvision==0.16.0a0'
ipex_pip = 'intel-extension-for-pytorch==2.1.10+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/'
install(os.environ.get('MKL_PACKAGE', 'mkl==2024.0.0'), 'mkl')
install(os.environ.get('DPCPP_PACKAGE', 'mkl-dpcpp==2024.0.0'), 'mkl-dpcpp')
torch_command = os.environ.get('TORCH_COMMAND', f'{pytorch_pip} {torchvision_pip} {ipex_pip}')
install('openvino', 'openvino', ignore=True)
install(os.environ.get('OPENVINO_PACKAGE', 'openvino==2023.2.0'), 'openvino', ignore=True)
install('nncf==2.7.0', 'nncf', ignore=True)
install('onnxruntime-openvino', 'onnxruntime-openvino', ignore=True)
elif allow_openvino and args.use_openvino:
log.info('Using OpenVINO')
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.1.1 torchvision==0.16.1 --index-url https://download.pytorch.org/whl/cpu')
torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cpu')
else:
machine = platform.machine()
if sys.platform == 'darwin':
@@ -503,11 +524,10 @@ def check_torch():
if opts.get('cuda_compile_backend', '') == 'hidet':
install('hidet', 'hidet')
if args.use_openvino or opts.get('cuda_compile_backend', '') == 'openvino_fx':
uninstall('openvino-nightly') # TODO openvino: remove after people had enough time upgrading
install('openvino==2023.2.0', 'openvino')
install(os.environ.get('OPENVINO_PACKAGE', 'openvino==2023.2.0'), 'openvino')
install('nncf==2.7.0', 'nncf')
install('onnxruntime-openvino', 'onnxruntime-openvino', ignore=True) # TODO openvino: numpy version conflicts with tensorflow and doesn't support Python 3.11
os.environ.setdefault('PYTORCH_TRACING_MODE', 'TORCHFX')
os.environ.setdefault('SD_LORA_DIFFUSERS', '1')
os.environ.setdefault('NEOReadDebugKeys', '1')
os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
if args.profile:
@@ -556,36 +576,6 @@ def install_packages():
print_profile(pr, 'Packages')
# clone required repositories
def install_repositories():
"""
if args.profile:
pr = cProfile.Profile()
pr.enable()
def d(name):
return os.path.join(os.path.dirname(__file__), 'repositories', name)
log.info('Verifying repositories')
os.makedirs(os.path.join(os.path.dirname(__file__), 'repositories'), exist_ok=True)
stable_diffusion_repo = os.environ.get('STABLE_DIFFUSION_REPO', "https://github.com/Stability-AI/stablediffusion.git")
stable_diffusion_commit = os.environ.get('STABLE_DIFFUSION_COMMIT_HASH', None)
clone(stable_diffusion_repo, d('stable-diffusion-stability-ai'), stable_diffusion_commit)
taming_transformers_repo = os.environ.get('TAMING_TRANSFORMERS_REPO', "https://github.com/CompVis/taming-transformers.git")
taming_transformers_commit = os.environ.get('TAMING_TRANSFORMERS_COMMIT_HASH', None)
clone(taming_transformers_repo, d('taming-transformers'), taming_transformers_commit)
k_diffusion_repo = os.environ.get('K_DIFFUSION_REPO', 'https://github.com/crowsonkb/k-diffusion.git')
k_diffusion_commit = os.environ.get('K_DIFFUSION_COMMIT_HASH', '0455157')
clone(k_diffusion_repo, d('k-diffusion'), k_diffusion_commit)
codeformer_repo = os.environ.get('CODEFORMER_REPO', 'https://github.com/sczhou/CodeFormer.git')
codeformer_commit = os.environ.get('CODEFORMER_COMMIT_HASH', "7a584fd")
clone(codeformer_repo, d('CodeFormer'), codeformer_commit)
blip_repo = os.environ.get('BLIP_REPO', 'https://github.com/salesforce/BLIP.git')
blip_commit = os.environ.get('BLIP_COMMIT_HASH', None)
clone(blip_repo, d('BLIP'), blip_commit)
if args.profile:
print_profile(pr, 'Repositories')
"""
# run extension installer
def run_extension_installer(folder):
path_installer = os.path.realpath(os.path.join(folder, "install.py"))
@@ -614,7 +604,7 @@ def list_extensions_folder(folder, quiet=False):
if disabled_extensions_all != 'none':
return []
disabled_extensions = opts.get('disabled_extensions', [])
enabled_extensions = [x for x in os.listdir(folder) if x not in disabled_extensions and not x.startswith('.')]
enabled_extensions = [x for x in os.listdir(folder) if os.path.isdir(os.path.join(folder, x)) and x not in disabled_extensions and not x.startswith('.')]
if not quiet:
log.info(f'Extensions: enabled={enabled_extensions} {name}')
return enabled_extensions
@@ -758,6 +748,7 @@ def set_environment():
os.environ.setdefault('USE_TORCH', '1')
os.environ.setdefault('UVICORN_TIMEOUT_KEEP_ALIVE', '60')
os.environ.setdefault('KINETO_LOG_LEVEL', '3')
os.environ.setdefault('DO_NOT_TRACK', '1')
os.environ.setdefault('HF_HUB_CACHE', opts.get('hfcache_dir', os.path.join(os.path.expanduser('~'), '.cache', 'huggingface', 'hub')))
log.debug(f'Cache folder: {os.environ.get("HF_HUB_CACHE")}')
if sys.platform == 'darwin':
@@ -919,7 +910,7 @@ def add_args(parser):
group.add_argument('--reset', default = os.environ.get("SD_RESET",False), action='store_true', help = "Reset main repository to latest version, default: %(default)s")
group.add_argument('--upgrade', default = os.environ.get("SD_UPGRADE",False), action='store_true', help = "Upgrade main repository to latest version, default: %(default)s")
group.add_argument('--requirements', default = os.environ.get("SD_REQUIREMENTS",False), action='store_true', help = "Force re-check of requirements, default: %(default)s")
group.add_argument('--quick', default = os.environ.get("SD_QUICK",False), action='store_true', help = "Run with startup sequence only, default: %(default)s")
group.add_argument('--quick', default = os.environ.get("SD_QUICK",False), action='store_true', help = "Bypass version checks, default: %(default)s")
group.add_argument('--use-directml', default = os.environ.get("SD_USEDIRECTML",False), action='store_true', help = "Use DirectML if no compatible GPU is detected, default: %(default)s")
group.add_argument("--use-openvino", default = os.environ.get("SD_USEOPENVINO",False), action='store_true', help="Use Intel OpenVINO backend, default: %(default)s")
group.add_argument("--use-ipex", default = os.environ.get("SD_USEIPEX",False), action='store_true', help="Force use Intel OneAPI XPU backend, default: %(default)s")
javascript/amethyst-nightfall.css
-3
View File
@@ -101,9 +101,6 @@ svg.feather.feather-image, .feather .feather-image { display: none }
#txt2img_checkboxes, #img2img_checkboxes { background-color: transparent; }
#txt2img_checkboxes, #img2img_checkboxes { margin-bottom: 0.2em; }
#txt2img_gallery, #img2img_gallery, #extras_gallery { padding: 0; margin: 0; object-fit: contain; box-shadow: none; min-height: 0; }
#txt2img_actions_column, #img2img_actions_column { flex-flow: wrap; justify-content: space-between; }
#txt2img_enqueue_wrapper, #img2img_enqueue_wrapper { min-width: unset; width: 48%; }
#txt2img_generate_box, #img2img_generate_box { min-width: unset; width: 48%; }
#extras_upscale { margin-top: 10px }
#txt2img_progress_row > div { min-width: var(--left-column); max-width: var(--left-column); }
javascript/black-orange.css
-3
View File
@@ -117,9 +117,6 @@ svg.feather.feather-image, .feather .feather-image { display: none }
#txt2img_cfg_scale { min-width: 200px; }
#txt2img_checkboxes, #img2img_checkboxes { background-color: transparent; }
#txt2img_checkboxes, #img2img_checkboxes { margin-bottom: 0.2em; }
#txt2img_actions_column, #img2img_actions_column { flex-flow: wrap; justify-content: space-between; }
#txt2img_enqueue_wrapper, #img2img_enqueue_wrapper { min-width: unset; width: 48%; }
#txt2img_generate_box, #img2img_generate_box { min-width: unset; width: 48%; }
#extras_upscale { margin-top: 10px }
#txt2img_progress_row > div { min-width: var(--left-column); max-width: var(--left-column); }
javascript/black-teal.css
+2 -6
View File
@@ -51,7 +51,7 @@ input[type=range]::-moz-range-thumb { box-shadow: 2px 2px 3px #111111 !important
::-webkit-scrollbar { width: 12px; }
::-webkit-scrollbar-track { background: #333333; }
::-webkit-scrollbar-thumb { background-color: var(--highlight-color); border-radius: var(--radius-lg); border-width: 0; box-shadow: 2px 2px 3px #111111; }
div.form { border-width: 0; box-shadow: none; background: transparent; overflow: visible; margin-bottom: 6px; }
div.form { border-width: 0; box-shadow: none; background: transparent; overflow: visible; }
/* gradio style classes */
fieldset .gr-block.gr-box, label.block span { padding: 0; margin-top: -4px; }
@@ -78,7 +78,7 @@ svg.feather.feather-image, .feather .feather-image { display: none }
.px-4 { padding-lefT: 1rem; padding-right: 1rem; }
.py-6 { padding-bottom: 0; }
.tabs { background-color: var(--background-color); }
.block.token-counter span { background-color: var(--input-background-fill) !important; box-shadow: 2px 2px 2px #111; border: none !important; font-size: 0.8rem; }
.block.token-counter span { background-color: var(--input-background-fill) !important; box-shadow: 2px 2px 2px #111; border: none !important; font-size: 0.7rem; }
.tab-nav { zoom: 120%; margin-top: 10px; margin-bottom: 10px; border-bottom: 2px solid var(--highlight-color) !important; padding-bottom: 2px; }
.label-wrap { margin: 8px 0px 4px 0px; }
.gradio-button.tool { border: none; background: none; box-shadow: none; filter: hue-rotate(340deg) saturate(0.5); }
@@ -118,11 +118,7 @@ svg.feather.feather-image, .feather .feather-image { display: none }
#txt2img_cfg_scale { min-width: 200px; }
#txt2img_checkboxes, #img2img_checkboxes { background-color: transparent; }
#txt2img_checkboxes, #img2img_checkboxes { margin-bottom: 0.2em; }
#txt2img_actions_column, #img2img_actions_column { flex-flow: wrap; justify-content: space-between; }
#txt2img_enqueue_wrapper, #img2img_enqueue_wrapper { min-width: unset; width: 48%; }
#txt2img_generate_box, #img2img_generate_box { min-width: unset; width: 48%; }
textarea[rows="1"] { height: 33px !important; width: 99% !important; padding: 8px !important; }
#extras_upscale { margin-top: 10px }
#txt2img_progress_row > div { min-width: var(--left-column); max-width: var(--left-column); }
#txt2img_settings { min-width: var(--left-column); max-width: var(--left-column); background-color: #111111; padding-top: 16px; }
javascript/control.js
+17
View File
@@ -0,0 +1,17 @@
function setupControlUI() {
const tabs = ['input', 'output', 'preview'];
for (const tab of tabs) {
const btn = gradioApp().getElementById(`control-${tab}-button`);
if (!btn) continue; // eslint-disable-line no-continue
btn.style.cursor = 'pointer';
btn.onclick = () => {
const t = gradioApp().getElementById(`control-tab-${tab}`);
t.style.display = t.style.display === 'none' ? 'block' : 'none';
const c = gradioApp().getElementById(`control-${tab}-column`);
c.style.flexGrow = c.style.flexGrow === '0' ? '9' : '0';
};
}
log('initControlUI');
}
onUiLoaded(setupControlUI);
javascript/dragDrop.js
+10 -7
View File
@@ -48,9 +48,11 @@ window.document.addEventListener('dragover', (e) => {
const target = e.composedPath()[0];
const imgWrap = target.closest('[data-testid="image"]');
if (!imgWrap && target.placeholder && target.placeholder.indexOf('Prompt') === -1) return;
e.stopPropagation();
e.preventDefault();
e.dataTransfer.dropEffect = 'copy';
if ((e.dataTransfer?.files?.length || 0) > 0) {
e.stopPropagation();
e.preventDefault();
e.dataTransfer.dropEffect = 'copy';
}
});
window.document.addEventListener('drop', (e) => {
@@ -59,10 +61,11 @@ window.document.addEventListener('drop', (e) => {
if (target.placeholder.indexOf('Prompt') === -1) return;
const imgWrap = target.closest('[data-testid="image"]');
if (!imgWrap) return;
e.stopPropagation();
e.preventDefault();
const { files } = e.dataTransfer;
dropReplaceImage(imgWrap, files);
if ((e.dataTransfer?.files?.length || 0) > 0) {
e.stopPropagation();
e.preventDefault();
dropReplaceImage(imgWrap, e.dataTransfer.files);
}
});
window.addEventListener('paste', (e) => {
javascript/emerald-paradise.css
+297
View File
@@ -0,0 +1,297 @@
/* generic html tags */
:root, .light, .dark {
--font: 'system-ui', 'ui-sans-serif', 'system-ui', "Roboto", sans-serif;
--font-mono: 'ui-monospace', 'Consolas', monospace;
--font-size: 16px;
--primary-100: #1e2223; /* bg color*/
--primary-200: #242a2c; /* drop down menu/ prompt window fill*/
--primary-300: #0a0c0e; /* black */
--primary-400: #2a302c; /* small buttons*/
--primary-500: #4b695d; /* main accent color green*/
--primary-700: #273538; /* extension box fill*/
--primary-800: #d15e84; /* pink(hover accent)*/
--highlight-color: var(--primary-500);
--inactive-color: var(--primary--800);
--body-text-color: var(--neutral-100);
--body-text-color-subdued: var(--neutral-300);
--background-color: var(--primary-100);
--background-fill-primary: var(--input-background-fill);
--input-padding: 8px;
--input-background-fill: var(--primary-200);
--input-shadow: none;
--button-secondary-text-color: white;
--button-secondary-background-fill: var(--primary-400);
--button-secondary-background-fill-hover: var(--primary-700);
--block-title-text-color: var(--neutral-300);
--radius-sm: 1px;
--radius-lg: 6px;
--spacing-md: 4px;
--spacing-xxl: 8px;
--line-sm: 1.2em;
--line-md: 1.4em;
}
html { font-size: var(--font-size); }
body, button, input, select, textarea { font-family: var(--font);}
button { font-size: 1.2rem; max-width: 400px; }
img { background-color: var(--background-color); }
input[type=range] { height: var(--line-sm); appearance: none; margin-top: 0; min-width: 160px; background-color: var(--background-color); width: 100%; background: transparent; }
input[type=range]::-webkit-slider-runnable-track, input[type=range]::-moz-range-track { width: 100%; height: 6px; cursor: pointer; background: var(--primary-400); border-radius: var(--radius-lg); border: 0px solid #222222; }
input[type=range]::-webkit-slider-thumb, input[type=range]::-moz-range-thumb { border: 0px solid #000000; height: var(--line-sm); width: 8px; border-radius: var(--radius-lg); background: white; cursor: pointer; appearance: none; margin-top: 0px; }
input[type=range]::-moz-range-progress { background-color: var(--primary-500); height: 6px; border-radius: var(--radius-lg); }
::-webkit-scrollbar-track { background: #333333; }
::-webkit-scrollbar-thumb { background-color: var(--highlight-color); border-radius: var(--radius-lg); border-width: 0; box-shadow: 2px 2px 3px #111111; }
div.form { border-width: 0; box-shadow: none; background: transparent; overflow: visible; margin-bottom: 6px; }
div.compact { gap: 1em; }
/* gradio style classes */
fieldset .gr-block.gr-box, label.block span { padding: 0; margin-top: -4px; }
.border-2 { border-width: 0; }
.border-b-2 { border-bottom-width: 2px; border-color: var(--highlight-color) !important; padding-bottom: 2px; margin-bottom: 8px; }
.bg-white { color: lightyellow; background-color: var(--inactive-color); }
.gr-box { border-radius: var(--radius-sm) !important; background-color: #111111 !important; box-shadow: 2px 2px 3px #111111; border-width: 0; padding: 4px; margin: 12px 0px 12px 0px }
.gr-button { font-weight: normal; box-shadow: 2px 2px 3px #111111; font-size: 0.8rem; min-width: 32px; min-height: 32px; padding: 3px; margin: 3px; }
.gr-check-radio { background-color: var(--inactive-color); border-width: 0; border-radius: var(--radius-lg); box-shadow: 2px 2px 3px #111111; }
.gr-check-radio:checked { background-color: var(--highlight-color); }
.gr-compact { background-color: var(--background-color); }
.gr-form { border-width: 0; }
.gr-input { background-color: #333333 !important; padding: 4px; margin: 4px; }
.gr-input-label { color: lightyellow; border-width: 0; background: transparent; padding: 2px !important; }
.gr-panel { background-color: var(--background-color); }
.eta-bar { display: none !important }
svg.feather.feather-image, .feather .feather-image { display: none }
.gap-2 { padding-top: 8px; }
.gr-box > div > div > input.gr-text-input { right: 0; width: 4em; padding: 0; top: -12px; border: none; max-height: 20px; }
.output-html { line-height: 1.2rem; overflow-x: hidden; }
.output-html > div { margin-bottom: 8px; }
.overflow-hidden .flex .flex-col .relative col .gap-4 { min-width: var(--left-column); max-width: var(--left-column); } /* this is a problematic one */
.p-2 { padding: 0; }
.px-4 { padding-lefT: 1rem; padding-right: 1rem; }
.py-6 { padding-bottom: 0; }
.tabs { background-color: var(--background-color); }
.block.token-counter span { background-color: var(--input-background-fill) !important; box-shadow: 2px 2px 2px #111; border: none !important; font-size: 0.8rem; }
.tab-nav { zoom: 120%; margin-top: 10px; margin-bottom: 10px; border-bottom: 2px solid var(--highlight-color) !important; padding-bottom: 2px; }
div.tab-nav button.selected {background-color: var(--button-primary-background-fill);}
#settings div.tab-nav button.selected {background-color: var(--background-color); color: var(--primary-800); font-weight: bold;}
.label-wrap { background-color: #191919; /* extension tab color*/ padding: 16px 8px 8px 8px; border-radius: var(--radius-lg); padding-left: 8px !important; }
.small-accordion .label-wrap { padding: 8px 0px 8px 0px; }
.small-accordion .label-wrap .icon { margin-right: 1em; }
.gradio-button.tool { border: none; box-shadow: none; border-radius: var(--radius-lg);}
button.selected {background: var(--button-primary-background-fill);}
.center.boundedheight.flex {background-color: var(--input-background-fill);}
.compact {border-radius: var(--border-radius-lg);}
#logMonitorData {background-color: var(--input-background-fill);}
#tab_extensions table td, #tab_extensions table th, #tab_config table td, #tab_config table th { border: none; padding: 0.5em; background-color: var(--primary-200); }
#tab_extensions table, #tab_config table { width: 96vw; }
#tab_extensions table input[type=checkbox] {appearance: none; border-radius: 0px;}
#tab_extensions button:hover { background-color: var(--button-secondary-background-fill-hover);}
/* automatic style classes */
.progressDiv { border-radius: var(--radius-sm) !important; position: fixed; top: 44px; right: 26px; max-width: 262px; height: 48px; z-index: 99; box-shadow: var(--button-shadow); }
.progressDiv .progress { border-radius: var(--radius-lg) !important; background: var(--highlight-color); line-height: 3rem; height: 48px; }
.gallery-item { box-shadow: none !important; }
.performance { color: #888; }
.extra-networks { border-left: 2px solid var(--highlight-color) !important; padding-left: 4px; }
.image-buttons { gap: 10px !important; justify-content: center; }
.image-buttons > button { max-width: 160px; }
.tooltip { background: var(--primary-800); color: white; border: none; border-radius: var(--radius-lg) }
#system_row > button, #settings_row > button, #config_row > button { max-width: 190px; }
/* gradio elements overrides */
#div.gradio-container { overflow-x: hidden; }
#img2img_label_copy_to_img2img { font-weight: normal; }
#txt2img_prompt, #txt2img_neg_prompt, #img2img_prompt, #img2img_neg_prompt { background-color: var(--background-color); box-shadow: 4px 4px 4px 0px #333333 !important; }
#txt2img_prompt > label > textarea, #txt2img_neg_prompt > label > textarea, #img2img_prompt > label > textarea, #img2img_neg_prompt > label > textarea { font-size: 1.1rem; }
#img2img_settings { min-width: calc(2 * var(--left-column)); max-width: calc(2 * var(--left-column)); background-color: #111111; padding-top: 16px; }
#interrogate, #deepbooru { margin: 0 0px 10px 0px; max-width: 80px; max-height: 80px; font-weight: normal; font-size: 0.95em; }
#quicksettings .gr-button-tool { font-size: 1.6rem; box-shadow: none; margin-top: -2px; height: 2.4em; }
#quicksettings button {padding: 0 0.5em 0.1em 0.5em;}
#open_folder_extras, #footer, #style_pos_col, #style_neg_col, #roll_col, #extras_upscaler_2, #extras_upscaler_2_visibility, #txt2img_seed_resize_from_w, #txt2img_seed_resize_from_h { display: none; }
#save-animation { border-radius: var(--radius-sm) !important; margin-bottom: 16px; background-color: #111111; }
#script_list { padding: 4px; margin-top: 16px; margin-bottom: 8px; }
#settings > div.flex-wrap { width: 15em; }
#txt2img_cfg_scale { min-width: 200px; }
#txt2img_checkboxes, #img2img_checkboxes { background-color: transparent; }
#txt2img_checkboxes, #img2img_checkboxes { margin-bottom: 0.2em; }
#txt2img_actions_column, #img2img_actions_column { flex-flow: wrap; justify-content: space-between; }
#txt2img_enqueue_wrapper, #img2img_enqueue_wrapper { min-width: unset; width: 48%; }
#txt2img_generate_box, #img2img_generate_box { min-width: unset; width: 48%; }
#extras_upscale { margin-top: 10px }
#txt2img_progress_row > div { min-width: var(--left-column); max-width: var(--left-column); }
#txt2img_settings { min-width: var(--left-column); max-width: var(--left-column); background-color: #111111; padding-top: 16px; }
#pnginfo_html2_info { margin-top: -18px; background-color: var(--input-background-fill); padding: var(--input-padding) }
#txt2img_tools, #img2img_tools { margin-top: -4px; margin-bottom: -4px; }
#txt2img_styles_row, #img2img_styles_row { margin-top: -6px; z-index: 200; }
/* based on gradio built-in dark theme */
:root, .light, .dark {
--body-background-fill: var(--background-color);
--color-accent-soft: var(--neutral-700);
--background-fill-secondary: none;
--border-color-accent: var(--background-color);
--border-color-primary: var(--background-color);
--link-text-color-active: var(--primary-500);
--link-text-color: var(--secondary-500);
--link-text-color-hover: var(--secondary-400);
--link-text-color-visited: var(--secondary-600);
--shadow-spread: 1px;
--block-background-fill: None;
--block-border-color: var(--border-color-primary);
--block_border_width: None;
--block-info-text-color: var(--body-text-color-subdued);
--block-label-background-fill: var(--background-fill-secondary);
--block-label-border-color: var(--border-color-primary);
--block_label_border_width: None;
--block-label-text-color: var(--neutral-200);
--block_shadow: None;
--block_title_background_fill: None;
--block_title_border_color: None;
--block_title_border_width: None;
--panel-background-fill: var(--background-fill-secondary);
--panel-border-color: var(--border-color-primary);
--panel_border_width: None;
--checkbox-background-color: var(--primary-200);
--checkbox-background-color-focus: var(--primary-700);
--checkbox-background-color-hover: var(--primary-700);
--checkbox-background-color-selected: var(--primary-500);
--checkbox-border-color: transparent;
--checkbox-border-color-focus: var(--primary-800);
--checkbox-border-color-hover: var(--primary-800);
--checkbox-border-color-selected: var(--primary-800);
--checkbox-border-width: var(--input-border-width);
--checkbox-label-background-fill: None;
--checkbox-label-background-fill-hover: None;
--checkbox-label-background-fill-selected: var(--checkbox-label-background-fill);
--checkbox-label-border-color: var(--border-color-primary);
--checkbox-label-border-color-hover: var(--checkbox-label-border-color);
--checkbox-label-border-width: var(--input-border-width);
--checkbox-label-text-color: var(--body-text-color);
--checkbox-label-text-color-selected: var(--checkbox-label-text-color);
--error-background-fill: var(--background-fill-primary);
--error-border-color: var(--border-color-primary);
--error-text-color: #f768b7; /*was ef4444*/
--input-background-fill-focus: var(--secondary-600);
--input-background-fill-hover: var(--input-background-fill);
--input-border-color: var(--background-color);
--input-border-color-focus: var(--primary-800);
--input-placeholder-color: var(--neutral-500);
--input-shadow-focus: None;
--loader_color: None;
--slider_color: None;
--stat-background-fill: linear-gradient(to right, var(--primary-400), var(--primary-800));
--table-border-color: var(--neutral-700);
--table-even-background-fill: var(--primary-300);
--table-odd-background-fill: var(--primary-200);
--table-row-focus: var(--color-accent-soft);
--button-border-width: var(--input-border-width);
--button-cancel-background-fill: linear-gradient(to bottom right, #dc2626, #b91c1c);
--button-cancel-background-fill-hover: linear-gradient(to bottom right, #dc2626, #dc2626);
--button-cancel-border-color: #dc2626;
--button-cancel-border-color-hover: var(--button-cancel-border-color);
--button-cancel-text-color: white;
--button-cancel-text-color-hover: var(--button-cancel-text-color);
--button-primary-background-fill: var(--primary-500);
--button-primary-background-fill-hover: var(--primary-800);
--button-primary-border-color: var(--primary-500);
--button-primary-border-color-hover: var(--button-primary-border-color);
--button-primary-text-color: white;
--button-primary-text-color-hover: var(--button-primary-text-color);
--button-secondary-border-color: var(--neutral-600);
--button-secondary-border-color-hover: var(--button-secondary-border-color);
--button-secondary-text-color-hover: var(--button-secondary-text-color);
--secondary-50: #eff6ff;
--secondary-100: #dbeafe;
--secondary-200: #bfdbfe;
--secondary-300: #93c5fd;
--secondary-400: #60a5fa;
--secondary-500: #3b82f6;
--secondary-600: #2563eb;
--secondary-700: #1d4ed8;
--secondary-800: #1e40af;
--secondary-900: #1e3a8a;
--secondary-950: #1d3660;
--neutral-50: #f0f0f0; /* */
--neutral-100: #e8e8e3;/* majority of text (neutral gray yellow) */
--neutral-200: #d0d0d0;
--neutral-300: #b3b5ac; /* top tab /sub text (light accent) */
--neutral-400: #ffba85;/* tab title (bright orange) */
--neutral-500: #48665b; /* prompt text (desat accent)*/
--neutral-600: #373f39; /* tab outline color (accent color)*/
--neutral-700: #2b373b; /* small settings tab accent */
--neutral-800: #f379c2; /* bright pink accent */
--neutral-900: #111827;
--neutral-950: #0b0f19;
--radius-xxs: 0;
--radius-xs: 0;
--radius-md: 0;
--radius-xl: 0;
--radius-xxl: 0;
--body-text-size: var(--text-md);
--body-text-weight: 400;
--embed-radius: var(--radius-lg);
--color-accent: var(--primary-500);
--shadow-drop: 0;
--shadow-drop-lg: 0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1);
--shadow-inset: rgba(0,0,0,0.05) 0px 2px 4px 0px inset;
--block-border-width: 1px;
--block-info-text-size: var(--text-sm);
--block-info-text-weight: 400;
--block-label-border-width: 1px;
--block-label-margin: 0;
--block-label-padding: var(--spacing-sm) var(--spacing-lg);
--block-label-radius: calc(var(--radius-lg) - 1px) 0 calc(var(--radius-lg) - 1px) 0;
--block-label-right-radius: 0 calc(var(--radius-lg) - 1px) 0 calc(var(--radius-lg) - 1px);
--block-label-text-size: var(--text-sm);
--block-label-text-weight: 400;
--block-padding: var(--spacing-xl) calc(var(--spacing-xl) + 2px);
--block-radius: var(--radius-lg);
--block-shadow: var(--shadow-drop);
--block-title-background-fill: none;
--block-title-border-color: none;
--block-title-border-width: 0;
--block-title-padding: 0;
--block-title-radius: none;
--block-title-text-size: var(--text-md);
--block-title-text-weight: 400;
--container-radius: var(--radius-lg);
--form-gap-width: 1px;
--layout-gap: var(--spacing-xxl);
--panel-border-width: 0;
--section-header-text-size: var(--text-md);
--section-header-text-weight: 400;
--checkbox-border-radius: var(--radius-sm);
--checkbox-label-gap: 2px;
--checkbox-label-padding: var(--spacing-md);
--checkbox-label-shadow: var(--shadow-drop);
--checkbox-label-text-size: var(--text-md);
--checkbox-label-text-weight: 400;
--checkbox-check: url("data:image/svg+xml,%3csvg viewBox='0 0 16 16' fill='white' xmlns='http://www.w3.org/2000/svg'%3e%3cpath d='M12.207 4.793a1 1 0 010 1.414l-5 5a1 1 0 01-1.414 0l-2-2a1 1 0 011.414-1.414L6.5 9.086l4.293-4.293a1 1 0 011.414 0z'/%3e%3c/svg%3e");
--radio-circle: url("data:image/svg+xml,%3csvg viewBox='0 0 16 16' fill='white' xmlns='http://www.w3.org/2000/svg'%3e%3ccircle cx='8' cy='8' r='3'/%3e%3c/svg%3e");
--checkbox-shadow: var(--input-shadow);
--error-border-width: 1px;
--input-border-width: 1px;
--input-radius: var(--radius-lg);
--input-text-size: var(--text-md);
--input-text-weight: 400;
--loader-color: var(--color-accent);
--prose-text-size: var(--text-md);
--prose-text-weight: 400;
--prose-header-text-weight: 600;
--slider-color: ;
--table-radius: var(--radius-lg);
--button-large-padding: 2px 6px;
--button-large-radius: var(--radius-lg);
--button-large-text-size: var(--text-lg);
--button-large-text-weight: 400;
--button-shadow: none;
--button-shadow-active: none;
--button-shadow-hover: none;
--button-small-padding: var(--spacing-sm) calc(2 * var(--spacing-sm));
--button-small-radius: var(--radius-lg);
--button-small-text-size: var(--text-md);
--button-small-text-weight: 400;
--button-transition: none;
--size-9: 64px;
--size-14: 64px;
}
javascript/extraNetworks.js
+39 -9
View File
@@ -16,7 +16,12 @@ const requestGet = (url, data, handler) => {
xhr.send(JSON.stringify(data));
};
const getENActiveTab = () => gradioApp().getElementById('tab_txt2img').style.display === 'block' ? 'txt2img' : 'img2img';
const getENActiveTab = () => {
if (gradioApp().getElementById('tab_txt2img').style.display === 'block') return 'txt2img';
if (gradioApp().getElementById('tab_img2img').style.display === 'block') return 'img2img';
if (gradioApp().getElementById('tab_control').style.display === 'block') return 'control';
return '';
};
const getENActivePage = () => {
const tabname = getENActiveTab();
@@ -90,14 +95,23 @@ function readCardDescription(page, item) {
});
}
async function filterExtraNetworksForTab(tabname, searchTerm) {
function getCardsForActivePage() {
const pagename = getENActivePage();
if (!pagename) return [];
const allCards = Array.from(gradioApp().querySelectorAll('.extra-network-cards > .card'));
const cards = allCards.filter((el) => el.dataset.page.toLowerCase().includes(pagename.toLowerCase()));
log('getCardsForActivePage', pagename, cards.length);
return allCards;
}
async function filterExtraNetworksForTab(searchTerm) {
let found = 0;
let items = 0;
const t0 = performance.now();
const pagename = getENActivePage();
if (!pagename) return;
const allPages = Array.from(gradioApp().querySelectorAll('.extra-network-cards'));
const pages = allPages.filter((el) => el.id.includes(pagename.toLowerCase()));
const pages = allPages.filter((el) => el.id.toLowerCase().includes(pagename.toLowerCase()));
for (const pg of pages) {
const cards = Array.from(pg.querySelectorAll('.card') || []);
cards.forEach((elem) => {
@@ -158,7 +172,7 @@ function sortExtraNetworks() {
const pagename = getENActivePage();
if (!pagename) return 'sort error: unknown page';
const allPages = Array.from(gradioApp().querySelectorAll('.extra-network-cards'));
const pages = allPages.filter((el) => el.id.includes(pagename.toLowerCase()));
const pages = allPages.filter((el) => el.id.toLowerCase().includes(pagename.toLowerCase()));
let num = 0;
for (const pg of pages) {
const cards = Array.from(pg.querySelectorAll('.card') || []);
@@ -255,12 +269,23 @@ function refeshDetailsEN(args) {
return args;
}
// init
// refresh on en show
function refreshENpage() {
if (getCardsForActivePage().length === 0) {
log('refreshENpage');
const tabname = getENActiveTab();
const btnRefresh = gradioApp().getElementById(`${tabname}_extra_refresh`);
if (btnRefresh) btnRefresh.click();
}
}
// init
function setupExtraNetworksForTab(tabname) {
gradioApp().querySelector(`#${tabname}_extra_tabs`).classList.add('extra-networks');
let tabs = gradioApp().querySelector(`#${tabname}_extra_tabs`);
if (tabs) tabs.classList.add('extra-networks');
const en = gradioApp().getElementById(`${tabname}_extra_networks`);
const tabs = gradioApp().querySelector(`#${tabname}_extra_tabs > div`);
tabs = gradioApp().querySelector(`#${tabname}_extra_tabs > div`);
if (!tabs) return;
// buttons
const btnRefresh = gradioApp().getElementById(`${tabname}_extra_refresh`);
@@ -307,7 +332,7 @@ function setupExtraNetworksForTab(tabname) {
txtSearchValue.addEventListener('input', (evt) => {
if (searchTimer) clearTimeout(searchTimer);
searchTimer = setTimeout(() => {
filterExtraNetworksForTab(tabname, txtSearchValue.value.toLowerCase());
filterExtraNetworksForTab(txtSearchValue.value.toLowerCase());
searchTimer = null;
}, 150);
});
@@ -332,13 +357,14 @@ function setupExtraNetworksForTab(tabname) {
};
// en style
if (!en) return;
const intersectionObserver = new IntersectionObserver((entries) => {
if (!en) return;
for (const el of Array.from(gradioApp().querySelectorAll('.extra-networks-page'))) {
el.style.height = `${window.opts.extra_networks_height}vh`;
el.parentElement.style.width = '-webkit-fill-available';
}
if (entries[0].intersectionRatio > 0) {
refreshENpage();
if (window.opts.extra_networks_card_cover === 'cover') {
en.style.transition = '';
en.style.zIndex = 100;
@@ -375,9 +401,11 @@ function setupExtraNetworksForTab(tabname) {
function setupExtraNetworks() {
setupExtraNetworksForTab('txt2img');
setupExtraNetworksForTab('img2img');
setupExtraNetworksForTab('control');
function registerPrompt(tabname, id) {
const textarea = gradioApp().querySelector(`#${id} > label > textarea`);
if (!textarea) return;
if (!activePromptTextarea[tabname]) activePromptTextarea[tabname] = textarea;
textarea.addEventListener('focus', () => { activePromptTextarea[tabname] = textarea; });
}
@@ -386,6 +414,8 @@ function setupExtraNetworks() {
registerPrompt('txt2img', 'txt2img_neg_prompt');
registerPrompt('img2img', 'img2img_prompt');
registerPrompt('img2img', 'img2img_neg_prompt');
registerPrompt('control', 'control_prompt');
registerPrompt('control', 'control_neg_prompt');
log('initExtraNetworks');
}
javascript/imageParams.js
+6 -7
View File
@@ -9,16 +9,15 @@ async function initDragDrop() {
if (!target.placeholder) return;
if (target.placeholder.indexOf('Prompt') === -1) return;
const promptTarget = get_tab_index('tabs') === 1 ? 'img2img_prompt_image' : 'txt2img_prompt_image';
e.stopPropagation();
e.preventDefault();
const imgParent = gradioApp().getElementById(promptTarget);
if (!imgParent) return;
const { files } = e.dataTransfer;
const fileInput = imgParent.querySelector('input[type="file"]');
if (fileInput) {
fileInput.files = files;
if (!imgParent || !fileInput) return;
if ((e.dataTransfer?.files?.length || 0) > 0) {
e.stopPropagation();
e.preventDefault();
fileInput.files = e.dataTransfer.files;
fileInput.dispatchEvent(new Event('change'));
log('dropEvent');
log('dropEvent files', fileInput.files);
}
});
}
javascript/invoked.css
-3
View File
@@ -113,9 +113,6 @@ button.selected {background: var(--button-primary-background-fill);}
#txt2img_cfg_scale { min-width: 200px; }
#txt2img_checkboxes, #img2img_checkboxes { background-color: transparent; }
#txt2img_checkboxes, #img2img_checkboxes { margin-bottom: 0.2em; }
#txt2img_actions_column, #img2img_actions_column { flex-flow: wrap; justify-content: space-between; }
#txt2img_enqueue_wrapper, #img2img_enqueue_wrapper { min-width: unset; width: 48%; }
#txt2img_generate_box, #img2img_generate_box { min-width: unset; width: 48%; }
#extras_upscale { margin-top: 10px }
#txt2img_progress_row > div { min-width: var(--left-column); max-width: var(--left-column); }
javascript/light-teal.css
+1 -4
View File
@@ -79,7 +79,7 @@ svg.feather.feather-image, .feather .feather-image { display: none }
.px-4 { padding-lefT: 1rem; padding-right: 1rem; }
.py-6 { padding-bottom: 0; }
.tabs { background-color: var(--background-color); }
.block.token-counter span { background-color: var(--input-background-fill) !important; box-shadow: 2px 2px 2px #111; border: none !important; font-size: 0.8rem; }
.block.token-counter span { background-color: var(--input-background-fill) !important; box-shadow: 2px 2px 2px #111; border: none !important; font-size: 0.7rem; }
.tab-nav { zoom: 120%; margin-top: 10px; margin-bottom: 10px; border-bottom: 2px solid var(--highlight-color) !important; padding-bottom: 2px; }
.label-wrap { margin: 16px 0px 8px 0px; }
.gradio-button.tool { border: none; background: none; box-shadow: none; filter: hue-rotate(340deg) saturate(0.5); }
@@ -115,9 +115,6 @@ svg.feather.feather-image, .feather .feather-image { display: none }
#txt2img_cfg_scale { min-width: 200px; }
#txt2img_checkboxes, #img2img_checkboxes { background-color: transparent; }
#txt2img_checkboxes, #img2img_checkboxes { margin-bottom: 0.2em; }
#txt2img_actions_column, #img2img_actions_column { flex-flow: wrap; justify-content: space-between; }
#txt2img_enqueue_wrapper, #img2img_enqueue_wrapper { min-width: unset; width: 48%; }
#txt2img_generate_box, #img2img_generate_box { min-width: unset; width: 48%; }
#extras_upscale { margin-top: 10px }
#txt2img_progress_row > div { min-width: var(--left-column); max-width: var(--left-column); }
javascript/logMonitor.js
+10 -2
View File
@@ -9,8 +9,15 @@ async function logMonitor() {
try { res = await fetch('/sdapi/v1/log?clear=True'); } catch {}
if (res?.ok) {
logMonitorStatus = true;
if (!logMonitorEl) logMonitorEl = document.getElementById('logMonitorData');
if (!logMonitorEl) {
logMonitorEl = document.getElementById('logMonitorData');
logMonitorEl.onscrollend = () => {
const at_bottom = logMonitorEl.scrollHeight <= (logMonitorEl.scrollTop + logMonitorEl.clientHeight);
if (at_bottom) logMonitorEl.parentElement.style = '';
};
}
if (!logMonitorEl) return;
const at_bottom = logMonitorEl.scrollHeight <= (logMonitorEl.scrollTop + logMonitorEl.clientHeight);
const lines = await res.json();
if (logMonitorEl && lines?.length > 0) logMonitorEl.parentElement.parentElement.style.display = opts.logmonitor_show ? 'block' : 'none';
for (const line of lines) {
@@ -23,7 +30,8 @@ async function logMonitor() {
} catch {}
}
while (logMonitorEl.childElementCount > 100) logMonitorEl.removeChild(logMonitorEl.firstChild);
logMonitorEl.scrollTop = logMonitorEl.scrollHeight;
if (at_bottom) logMonitorEl.scrollTop = logMonitorEl.scrollHeight;
else if (lines?.length > 0) logMonitorEl.parentElement.style = 'border-bottom: 2px solid var(--highlight-color);';
}
}
javascript/midnight-barbie.css
-4
View File
@@ -106,10 +106,6 @@ svg.feather.feather-image, .feather .feather-image { display: none }
#txt2img_cfg_scale { min-width: 200px; }
#txt2img_checkboxes, #img2img_checkboxes { background-color: transparent; }
#txt2img_checkboxes, #img2img_checkboxes { margin-bottom: 0.2em; }
#txt2img_actions_column, #img2img_actions_column { flex-flow: wrap; justify-content: space-between; }
#txt2img_enqueue_wrapper, #img2img_enqueue_wrapper { min-width: unset; width: 48%; }
#txt2img_generate_box, #img2img_generate_box { min-width: unset; width: 48%; }
#extras_upscale { margin-top: 10px }
#txt2img_progress_row > div { min-width: var(--left-column); max-width: var(--left-column); }
#txt2img_settings { min-width: var(--left-column); max-width: var(--left-column); background-color: #111111; padding-top: 16px; }
javascript/orchid-dreams.css
+297
View File
@@ -0,0 +1,297 @@
/* generic html tags */
:root, .light, .dark {
--font: 'system-ui', 'ui-sans-serif', 'system-ui', "Roboto", sans-serif;
--font-mono: 'ui-monospace', 'Consolas', monospace;
--font-size: 16px;
--primary-100: #2a2a34; /* bg color*/
--primary-200: #1f2028; /* drop down menu/ prompt*/
--primary-300: #0a0c0e; /* black */
--primary-400: #40435c; /* small buttons*/
--primary-500: #4c48b5; /* main accent color purple*/
--primary-700: #1f2028; /* darker hover accent*/
--primary-800: #e95ee3; /* pink accent*/
--highlight-color: var(--primary-500);
--inactive-color: var(--primary--800);
--body-text-color: var(--neutral-100);
--body-text-color-subdued: var(--neutral-300);
--background-color: var(--primary-100);
--background-fill-primary: var(--input-background-fill);
--input-padding: 8px;
--input-background-fill: var(--primary-200);
--input-shadow: none;
--button-secondary-text-color: white;
--button-secondary-background-fill: var(--primary-400);
--button-secondary-background-fill-hover: var(--primary-700);
--block-title-text-color: var(--neutral-300);
--radius-sm: 1px;
--radius-lg: 6px;
--spacing-md: 4px;
--spacing-xxl: 8px;
--line-sm: 1.2em;
--line-md: 1.4em;
}
html { font-size: var(--font-size); }
body, button, input, select, textarea { font-family: var(--font);}
button { font-size: 1.2rem; max-width: 400px; }
img { background-color: var(--background-color); }
input[type=range] { height: var(--line-sm); appearance: none; margin-top: 0; min-width: 160px; background-color: var(--background-color); width: 100%; background: transparent; }
input[type=range]::-webkit-slider-runnable-track, input[type=range]::-moz-range-track { width: 100%; height: 6px; cursor: pointer; background: var(--primary-400); border-radius: var(--radius-lg); border: 0px solid #222222; }
input[type=range]::-webkit-slider-thumb, input[type=range]::-moz-range-thumb { border: 0px solid #000000; height: var(--line-sm); width: 8px; border-radius: var(--radius-lg); background: white; cursor: pointer; appearance: none; margin-top: 0px; }
input[type=range]::-moz-range-progress { background-color: var(--primary-500); height: 6px; border-radius: var(--radius-lg); }
::-webkit-scrollbar-track { background: #333333; }
::-webkit-scrollbar-thumb { background-color: var(--highlight-color); border-radius: var(--radius-lg); border-width: 0; box-shadow: 2px 2px 3px #111111; }
div.form { border-width: 0; box-shadow: none; background: transparent; overflow: visible; margin-bottom: 6px; }
div.compact { gap: 1em; }
/* gradio style classes */
fieldset .gr-block.gr-box, label.block span { padding: 0; margin-top: -4px; }
.border-2 { border-width: 0; }
.border-b-2 { border-bottom-width: 2px; border-color: var(--highlight-color) !important; padding-bottom: 2px; margin-bottom: 8px; }
.bg-white { color: lightyellow; background-color: var(--inactive-color); }
.gr-box { border-radius: var(--radius-sm) !important; background-color: #111111 !important; box-shadow: 2px 2px 3px #111111; border-width: 0; padding: 4px; margin: 12px 0px 12px 0px }
.gr-button { font-weight: normal; box-shadow: 2px 2px 3px #111111; font-size: 0.8rem; min-width: 32px; min-height: 32px; padding: 3px; margin: 3px; }
.gr-check-radio { background-color: var(--inactive-color); border-width: 0; border-radius: var(--radius-lg); box-shadow: 2px 2px 3px #111111; }
.gr-check-radio:checked { background-color: var(--highlight-color); }
.gr-compact { background-color: var(--background-color); }
.gr-form { border-width: 0; }
.gr-input { background-color: #333333 !important; padding: 4px; margin: 4px; }
.gr-input-label { color: lightyellow; border-width: 0; background: transparent; padding: 2px !important; }
.gr-panel { background-color: var(--background-color); }
.eta-bar { display: none !important }
svg.feather.feather-image, .feather .feather-image { display: none }
.gap-2 { padding-top: 8px; }
.gr-box > div > div > input.gr-text-input { right: 0; width: 4em; padding: 0; top: -12px; border: none; max-height: 20px; }
.output-html { line-height: 1.2rem; overflow-x: hidden; }
.output-html > div { margin-bottom: 8px; }
.overflow-hidden .flex .flex-col .relative col .gap-4 { min-width: var(--left-column); max-width: var(--left-column); } /* this is a problematic one */
.p-2 { padding: 0; }
.px-4 { padding-lefT: 1rem; padding-right: 1rem; }
.py-6 { padding-bottom: 0; }
.tabs { background-color: var(--background-color); }
.block.token-counter span { background-color: var(--input-background-fill) !important; box-shadow: 2px 2px 2px #111; border: none !important; font-size: 0.8rem; }
.tab-nav { zoom: 120%; margin-top: 10px; margin-bottom: 10px; border-bottom: 2px solid var(--highlight-color) !important; padding-bottom: 2px; }
div.tab-nav button.selected {background-color: var(--button-primary-background-fill);}
#settings div.tab-nav button.selected {background-color: var(--background-color); color: var(--primary-800); font-weight: bold;}
.label-wrap { background-color: #18181e; /* extension tab color*/ padding: 16px 8px 8px 8px; border-radius: var(--radius-lg); padding-left: 8px !important; }
.small-accordion .label-wrap { padding: 8px 0px 8px 0px; }
.small-accordion .label-wrap .icon { margin-right: 1em; }
.gradio-button.tool { border: none; box-shadow: none; border-radius: var(--radius-lg);}
button.selected {background: var(--button-primary-background-fill);}
.center.boundedheight.flex {background-color: var(--input-background-fill);}
.compact {border-radius: var(--border-radius-lg);}
#logMonitorData {background-color: var(--input-background-fill);}
#tab_extensions table td, #tab_extensions table th, #tab_config table td, #tab_config table th { border: none; padding: 0.5em; background-color: var(--primary-200); }
#tab_extensions table, #tab_config table { width: 96vw; }
#tab_extensions table input[type=checkbox] {appearance: none; border-radius: 0px;}
#tab_extensions button:hover { background-color: var(--button-secondary-background-fill-hover);}
/* automatic style classes */
.progressDiv { border-radius: var(--radius-sm) !important; position: fixed; top: 44px; right: 26px; max-width: 262px; height: 48px; z-index: 99; box-shadow: var(--button-shadow); }
.progressDiv .progress { border-radius: var(--radius-lg) !important; background: var(--highlight-color); line-height: 3rem; height: 48px; }
.gallery-item { box-shadow: none !important; }
.performance { color: #888; }
.extra-networks { border-left: 2px solid var(--highlight-color) !important; padding-left: 4px; }
.image-buttons { gap: 10px !important; justify-content: center; }
.image-buttons > button { max-width: 160px; }
.tooltip { background: var(--primary-800); color: white; border: none; border-radius: var(--radius-lg) }
#system_row > button, #settings_row > button, #config_row > button { max-width: 190px; }
/* gradio elements overrides */
#div.gradio-container { overflow-x: hidden; }
#img2img_label_copy_to_img2img { font-weight: normal; }
#txt2img_prompt, #txt2img_neg_prompt, #img2img_prompt, #img2img_neg_prompt { background-color: var(--background-color); box-shadow: 4px 4px 4px 0px #333333 !important; }
#txt2img_prompt > label > textarea, #txt2img_neg_prompt > label > textarea, #img2img_prompt > label > textarea, #img2img_neg_prompt > label > textarea { font-size: 1.1rem; }
#img2img_settings { min-width: calc(2 * var(--left-column)); max-width: calc(2 * var(--left-column)); background-color: #111111; padding-top: 16px; }
#interrogate, #deepbooru { margin: 0 0px 10px 0px; max-width: 80px; max-height: 80px; font-weight: normal; font-size: 0.95em; }
#quicksettings .gr-button-tool { font-size: 1.6rem; box-shadow: none; margin-top: -2px; height: 2.4em; }
#quicksettings button {padding: 0 0.5em 0.1em 0.5em;}
#open_folder_extras, #footer, #style_pos_col, #style_neg_col, #roll_col, #extras_upscaler_2, #extras_upscaler_2_visibility, #txt2img_seed_resize_from_w, #txt2img_seed_resize_from_h { display: none; }
#save-animation { border-radius: var(--radius-sm) !important; margin-bottom: 16px; background-color: #111111; }
#script_list { padding: 4px; margin-top: 16px; margin-bottom: 8px; }
#settings > div.flex-wrap { width: 15em; }
#txt2img_cfg_scale { min-width: 200px; }
#txt2img_checkboxes, #img2img_checkboxes { background-color: transparent; }
#txt2img_checkboxes, #img2img_checkboxes { margin-bottom: 0.2em; }
#txt2img_actions_column, #img2img_actions_column { flex-flow: wrap; justify-content: space-between; }
#txt2img_enqueue_wrapper, #img2img_enqueue_wrapper { min-width: unset; width: 48%; }
#txt2img_generate_box, #img2img_generate_box { min-width: unset; width: 48%; }
#extras_upscale { margin-top: 10px }
#txt2img_progress_row > div { min-width: var(--left-column); max-width: var(--left-column); }
#txt2img_settings { min-width: var(--left-column); max-width: var(--left-column); background-color: #111111; padding-top: 16px; }
#pnginfo_html2_info { margin-top: -18px; background-color: var(--input-background-fill); padding: var(--input-padding) }
#txt2img_tools, #img2img_tools { margin-top: -4px; margin-bottom: -4px; }
#txt2img_styles_row, #img2img_styles_row { margin-top: -6px; z-index: 200; }
/* based on gradio built-in dark theme */
:root, .light, .dark {
--body-background-fill: var(--background-color);
--color-accent-soft: var(--neutral-700);
--background-fill-secondary: none;
--border-color-accent: var(--background-color);
--border-color-primary: var(--background-color);
--link-text-color-active: var(--primary-500);
--link-text-color: var(--secondary-500);
--link-text-color-hover: var(--secondary-400);
--link-text-color-visited: var(--secondary-600);
--shadow-spread: 1px;
--block-background-fill: None;
--block-border-color: var(--border-color-primary);
--block_border_width: None;
--block-info-text-color: var(--body-text-color-subdued);
--block-label-background-fill: var(--background-fill-secondary);
--block-label-border-color: var(--border-color-primary);
--block_label_border_width: None;
--block-label-text-color: var(--neutral-200);
--block_shadow: None;
--block_title_background_fill: None;
--block_title_border_color: None;
--block_title_border_width: None;
--panel-background-fill: var(--background-fill-secondary);
--panel-border-color: var(--border-color-primary);
--panel_border_width: None;
--checkbox-background-color: var(--primary-200);
--checkbox-background-color-focus: var(--primary-400);
--checkbox-background-color-hover: var(--primary-200);
--checkbox-background-color-selected: var(--primary-400);
--checkbox-border-color: transparent;
--checkbox-border-color-focus: var(--primary-800);
--checkbox-border-color-hover: var(--primary-800);
--checkbox-border-color-selected: var(--primary-800);
--checkbox-border-width: var(--input-border-width);
--checkbox-label-background-fill: None;
--checkbox-label-background-fill-hover: None;
--checkbox-label-background-fill-selected: var(--checkbox-label-background-fill);
--checkbox-label-border-color: var(--border-color-primary);
--checkbox-label-border-color-hover: var(--checkbox-label-border-color);
--checkbox-label-border-width: var(--input-border-width);
--checkbox-label-text-color: var(--body-text-color);
--checkbox-label-text-color-selected: var(--checkbox-label-text-color);
--error-background-fill: var(--background-fill-primary);
--error-border-color: var(--border-color-primary);
--error-text-color: #f768b7; /*was ef4444*/
--input-background-fill-focus: var(--secondary-600);
--input-background-fill-hover: var(--input-background-fill);
--input-border-color: var(--background-color);
--input-border-color-focus: var(--primary-800);
--input-placeholder-color: var(--neutral-500);
--input-shadow-focus: None;
--loader_color: None;
--slider_color: None;
--stat-background-fill: linear-gradient(to right, var(--primary-400), var(--primary-800));
--table-border-color: var(--neutral-700);
--table-even-background-fill: var(--primary-300);
--table-odd-background-fill: var(--primary-200);
--table-row-focus: var(--color-accent-soft);
--button-border-width: var(--input-border-width);
--button-cancel-background-fill: linear-gradient(to bottom right, #dc2626, #b91c1c);
--button-cancel-background-fill-hover: linear-gradient(to bottom right, #dc2626, #dc2626);
--button-cancel-border-color: #dc2626;
--button-cancel-border-color-hover: var(--button-cancel-border-color);
--button-cancel-text-color: white;
--button-cancel-text-color-hover: var(--button-cancel-text-color);
--button-primary-background-fill: var(--primary-500);
--button-primary-background-fill-hover: var(--primary-800);
--button-primary-border-color: var(--primary-500);
--button-primary-border-color-hover: var(--button-primary-border-color);
--button-primary-text-color: white;
--button-primary-text-color-hover: var(--button-primary-text-color);
--button-secondary-border-color: var(--neutral-600);
--button-secondary-border-color-hover: var(--button-secondary-border-color);
--button-secondary-text-color-hover: var(--button-secondary-text-color);
--secondary-50: #eff6ff;
--secondary-100: #dbeafe;
--secondary-200: #bfdbfe;
--secondary-300: #93c5fd;
--secondary-400: #60a5fa;
--secondary-500: #3b82f6;
--secondary-600: #2563eb;
--secondary-700: #1d4ed8;
--secondary-800: #1e40af;
--secondary-900: #1e3a8a;
--secondary-950: #1d3660;
--neutral-50: #f0f0f0; /* */
--neutral-100: #ddd5e8;/* majority of text (neutral gray purple) */
--neutral-200: #d0d0d0;
--neutral-300: #bfbad6; /* top tab text (light accent) */
--neutral-400: #ffba85;/* tab title (bright orange) */
--neutral-500: #545b94; /* prompt text (desat accent)*/
--neutral-600: #1f2028; /* tab outline color (accent color)*/
--neutral-700: #20212c; /* unchanged settings tab accent (dark)*/
--neutral-800: #e055dc; /* bright pink accent */
--neutral-900: #111827;
--neutral-950: #0b0f19;
--radius-xxs: 0;
--radius-xs: 0;
--radius-md: 0;
--radius-xl: 0;
--radius-xxl: 0;
--body-text-size: var(--text-md);
--body-text-weight: 400;
--embed-radius: var(--radius-lg);
--color-accent: var(--primary-500);
--shadow-drop: 0;
--shadow-drop-lg: 0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1);
--shadow-inset: rgba(0,0,0,0.05) 0px 2px 4px 0px inset;
--block-border-width: 1px;
--block-info-text-size: var(--text-sm);
--block-info-text-weight: 400;
--block-label-border-width: 1px;
--block-label-margin: 0;
--block-label-padding: var(--spacing-sm) var(--spacing-lg);
--block-label-radius: calc(var(--radius-lg) - 1px) 0 calc(var(--radius-lg) - 1px) 0;
--block-label-right-radius: 0 calc(var(--radius-lg) - 1px) 0 calc(var(--radius-lg) - 1px);
--block-label-text-size: var(--text-sm);
--block-label-text-weight: 400;
--block-padding: var(--spacing-xl) calc(var(--spacing-xl) + 2px);
--block-radius: var(--radius-lg);
--block-shadow: var(--shadow-drop);
--block-title-background-fill: none;
--block-title-border-color: none;
--block-title-border-width: 0;
--block-title-padding: 0;
--block-title-radius: none;
--block-title-text-size: var(--text-md);
--block-title-text-weight: 400;
--container-radius: var(--radius-lg);
--form-gap-width: 1px;
--layout-gap: var(--spacing-xxl);
--panel-border-width: 0;
--section-header-text-size: var(--text-md);
--section-header-text-weight: 400;
--checkbox-border-radius: var(--radius-sm);
--checkbox-label-gap: 2px;
--checkbox-label-padding: var(--spacing-md);
--checkbox-label-shadow: var(--shadow-drop);
--checkbox-label-text-size: var(--text-md);
--checkbox-label-text-weight: 400;
--checkbox-check: url("data:image/svg+xml,%3csvg viewBox='0 0 16 16' fill='white' xmlns='http://www.w3.org/2000/svg'%3e%3cpath d='M12.207 4.793a1 1 0 010 1.414l-5 5a1 1 0 01-1.414 0l-2-2a1 1 0 011.414-1.414L6.5 9.086l4.293-4.293a1 1 0 011.414 0z'/%3e%3c/svg%3e");
--radio-circle: url("data:image/svg+xml,%3csvg viewBox='0 0 16 16' fill='white' xmlns='http://www.w3.org/2000/svg'%3e%3ccircle cx='8' cy='8' r='3'/%3e%3c/svg%3e");
--checkbox-shadow: var(--input-shadow);
--error-border-width: 1px;
--input-border-width: 1px;
--input-radius: var(--radius-lg);
--input-text-size: var(--text-md);
--input-text-weight: 400;
--loader-color: var(--color-accent);
--prose-text-size: var(--text-md);
--prose-text-weight: 400;
--prose-header-text-weight: 600;
--slider-color: ;
--table-radius: var(--radius-lg);
--button-large-padding: 2px 6px;
--button-large-radius: var(--radius-lg);
--button-large-text-size: var(--text-lg);
--button-large-text-weight: 400;
--button-shadow: none;
--button-shadow-active: none;
--button-shadow-hover: none;
--button-small-padding: var(--spacing-sm) calc(2 * var(--spacing-sm));
--button-small-radius: var(--radius-lg);
--button-small-text-size: var(--text-md);
--button-small-text-weight: 400;
--button-transition: none;
--size-9: 64px;
--size-14: 64px;
}
javascript/progressBar.js
+7 -5
View File
@@ -40,7 +40,7 @@ function checkPaused(state) {
}
function setProgress(res) {
const elements = ['txt2img_generate', 'img2img_generate', 'extras_generate'];
const elements = ['txt2img_generate', 'img2img_generate', 'extras_generate', 'control_generate'];
const progress = (res?.progress || 0);
const job = res?.job || '';
const perc = res && (progress > 0) ? `${Math.round(100.0 * progress)}%` : '';
@@ -57,10 +57,12 @@ function setProgress(res) {
document.title = `SD.Next ${perc}`;
for (const elId of elements) {
const el = document.getElementById(elId);
el.innerText = (res ? `${job} ${perc} ${eta}` : 'Generate');
el.style.background = res && (progress > 0)
? `linear-gradient(to right, var(--primary-500) 0%, var(--primary-800) ${perc}, var(--neutral-700) ${perc})`
: 'var(--button-primary-background-fill)';
if (el) {
el.innerText = (res ? `${job} ${perc} ${eta}` : 'Generate');
el.style.background = res && (progress > 0)
? `linear-gradient(to right, var(--primary-500) 0%, var(--primary-800) ${perc}, var(--neutral-700) ${perc})`
: 'var(--button-primary-background-fill)';
}
}
}
javascript/promptChecker.js
+2
View File
@@ -36,4 +36,6 @@ onAfterUiUpdate(() => {
setupBracketChecking('txt2img_neg_prompt', 'txt2img_negative_token_counter');
setupBracketChecking('img2img_prompt', 'img2img_token_counter');
setupBracketChecking('img2img_neg_prompt', 'img2img_negative_token_counter');
setupBracketChecking('control_prompt', 'control_token_counter');
setupBracketChecking('control_neg_prompt', 'control_negative_token_counter');
});
javascript/sdnext.css
+76 -96
View File
@@ -48,11 +48,11 @@ textarea { overflow-y: auto !important; }
/* custom gradio elements */
.accordion-compact { padding: 8px 0px 4px 0px !important; }
.settings-accordion > div { flex-flow: wrap; }
.settings-accordion>div { flex-flow: wrap; }
.small-accordion .form { min-width: var(--left-column) !important; max-width: max-content; }
.small-accordion .label-wrap .icon { margin-right: 1.6em; margin-left: 0.6em; color: var(--button-primary-border-color); }
.small-accordion .label-wrap { padding: 16px 0px 8px 0px; margin: 0; border-top: 2px solid var(--button-secondary-border-color); }
.small-accordion { width: fit-content !important; padding-left: 0 !important; }
.small-accordion { width: fit-content !important; min-width: fit-content !important; padding-left: 0 !important; }
.extension-script { max-width: 48vw; }
button.custom-button{ border-radius: var(--button-large-radius); padding: var(--button-large-padding); font-weight: var(--button-large-text-weight); border: var(--button-border-width) solid var(--button-secondary-border-color);
background: var(--button-secondary-background-fill); color: var(--button-secondary-text-color); font-size: var(--button-large-text-size);
@@ -62,7 +62,7 @@ button.custom-button{ border-radius: var(--button-large-radius); padding: var(--
.theme-preview { display: none; position: fixed; border: var(--spacing-sm) solid var(--neutral-600); box-shadow: 2px 2px 2px 2px var(--neutral-700); top: 0; bottom: 0; left: 0; right: 0; margin: auto; max-width: 75vw; z-index: 999; }
/* txt2img/img2img specific */
.block.token-counter{ position: absolute; display: inline-block; right: 0; min-width: 0 !important; width: auto; z-index: 100; top: -0.75em; }
.block.token-counter{ position: absolute; display: inline-block; right: 1em; min-width: 0 !important; width: auto; z-index: 100; top: -0.5em; }
.block.token-counter span{ background: var(--input-background-fill) !important; box-shadow: 0 0 0.0 0.3em rgba(192,192,192,0.15), inset 0 0 0.6em rgba(192,192,192,0.075); border: 2px solid rgba(192,192,192,0.4) !important; }
.block.token-counter.error span{ box-shadow: 0 0 0.0 0.3em rgba(255,0,0,0.15), inset 0 0 0.6em rgba(255,0,0,0.075); border: 2px solid rgba(255,0,0,0.4) !important; }
.block.token-counter div{ display: inline; }
@@ -70,18 +70,24 @@ button.custom-button{ border-radius: var(--button-large-radius); padding: var(--
.performance { font-size: 0.85em; color: #444; }
.performance p { display: inline-block; color: var(--body-text-color-subdued) !important }
.performance .time { margin-right: 0; }
#txt2img_prompt_container, #img2img_prompt_container { margin-right: var(--layout-gap) }
#txt2img_footer, #img2img_footer, #extras_footer { height: fit-content; }
#txt2img_footer, #img2img_footer { height: fit-content; display: none; }
#txt2img_generate_box, #img2img_generate_box { gap: 0.5em; flex-wrap: wrap-reverse; height: fit-content; }
#txt2img_actions_column, #img2img_actions_column { gap: 0.5em; height: fit-content; }
#txt2img_generate_box > button, #img2img_generate_box > button, #txt2img_enqueue, #img2img_enqueue { min-height: 42px; max-height: 42px; line-height: 1em; }
#txt2img_generate_line2, #img2img_generate_line2, #txt2img_tools, #img2img_tools { display: flex; }
#txt2img_generate_line2 > button, #img2img_generate_line2 > button, #extras_generate_box > button, #txt2img_tools > button, #img2img_tools > button { height: 2em; line-height: 0; font-size: var(--input-text-size);
.thumbnails { background: var(--body-background-fill); }
#control_gallery { height: 564px; }
#control-result { padding: 0.5em; }
#control-inputs { margin-top: 1em; }
#txt2img_prompt_container, #img2img_prompt_container, #control_prompt_container { margin-right: var(--layout-gap) }
#txt2img_footer, #img2img_footer, #extras_footer, #control_footer { height: fit-content; display: none; }
#txt2img_generate_box, #img2img_generate_box, #control_general_box { gap: 0.5em; flex-wrap: wrap-reverse; height: fit-content; }
#txt2img_actions_column, #img2img_actions_column, #control_actions_column { gap: 0.3em; height: fit-content; }
#txt2img_generate_box>button, #img2img_generate_box>button, #control_generate_box>button, #txt2img_enqueue, #img2img_enqueue { min-height: 42px; max-height: 42px; line-height: 1em; }
#txt2img_generate_line2, #img2img_generate_line2, #txt2img_tools, #img2img_tools, #control_generate_line2, #control_tools { display: flex; }
#txt2img_generate_line2>button, #img2img_generate_line2>button, #extras_generate_box>button, #control_generate_line2>button, #txt2img_tools>button, #img2img_tools>button, #control_tools>button { height: 2em; line-height: 0; font-size: var(--input-text-size);
min-width: unset; display: block !important; }
#txt2img_prompt, #txt2img_neg_prompt, #img2img_prompt, #img2img_neg_prompt { display: contents; }
#txt2img_prompt, #txt2img_neg_prompt, #img2img_prompt, #img2img_neg_prompt, #control_prompt, #control_neg_prompt { display: contents; }
#txt2img_generate_box, #img2img_generate_box, #control_generate_box { min-width: unset; width: 48%; }
#txt2img_actions_column, #img2img_actions_column, #control_actions { flex-flow: wrap; justify-content: space-between; }
#txt2img_enqueue_wrapper, #img2img_enqueue_wrapper, #control_enqueue_wrapper { min-width: unset !important; width: 48%; }
.interrogate-col{ min-width: 0 !important; max-width: fit-content; margin-right: var(--spacing-xxl); }
.interrogate-col > button{ flex: 1; }
.interrogate-col>button{ flex: 1; }
#sampler_selection_img2img { margin-top: 1em; }
#txtimg_hr_finalres{ min-height: 0 !important; }
#img2img_scale_resolution_preview.block{ display: flex; align-items: end; }
@@ -89,13 +95,12 @@ button.custom-button{ border-radius: var(--button-large-radius); padding: var(--
div#extras_scale_to_tab div.form{ flex-direction: row; }
#img2img_unused_scale_by_slider { visibility: hidden; width: 0.5em; max-width: 0.5em; min-width: 0.5em; }
.inactive{ opacity: 0.5; }
div.dimensions-tools { min-width: 0 !important; max-width: fit-content; flex-direction: row; align-content: center; }
div#extras_scale_to_tab div.form{ flex-direction: row; }
#mode_img2img .gradio-image > div.fixed-height, #mode_img2img .gradio-image > div.fixed-height img{ height: 480px !important; max-height: 480px !important; min-height: 480px !important; }
#mode_img2img .gradio-image>div.fixed-height, #mode_img2img .gradio-image>div.fixed-height img{ height: 480px !important; max-height: 480px !important; min-height: 480px !important; }
#img2img_sketch, #img2maskimg, #inpaint_sketch { overflow: overlay !important; resize: auto; background: var(--panel-background-fill); z-index: 5; }
.image-buttons button{ min-width: auto; }
.infotext { overflow-wrap: break-word; line-height: 1.5em; }
.infotext > p { padding-left: 1em; text-indent: -1em; white-space: pre-wrap; }
.infotext>p { padding-left: 1em; text-indent: -1em; white-space: pre-wrap; }
.tooltip { display: block; position: fixed; top: 1em; right: 1em; padding: 0.5em; background: var(--input-background-fill); color: var(--body-text-color); border: 1pt solid var(--button-primary-border-color);
width: 22em; min-height: 1.3em; font-size: 0.8em; transition: opacity 0.2s ease-in; pointer-events: none; opacity: 0; z-index: 999; }
.tooltip-show { opacity: 0.9; }
@@ -104,15 +109,15 @@ div#extras_scale_to_tab div.form{ flex-direction: row; }
/* settings */
#si-sparkline-memo, #si-sparkline-load { background-color: #111; }
#quicksettings { width: fit-content; }
#quicksettings > button { padding: 0 1em 0 0; align-self: end; margin-bottom: var(--text-sm); }
#quicksettings>button { padding: 0 1em 0 0; align-self: end; margin-bottom: var(--text-sm); }
#settings { display: flex; gap: var(--layout-gap); }
#settings div { border: none; gap: 0; margin: 0 0 var(--layout-gap) 0px; padding: 0; }
#settings > div.tab-content { flex: 10 0 75%; display: grid; }
#settings > div.tab-content > div { border: none; padding: 0; }
#settings > div.tab-content > div > div > div > div > div { flex-direction: unset; }
#settings > div.tab-nav { display: grid; grid-template-columns: repeat(auto-fill, .5em minmax(10em, 1fr)); flex: 1 0 auto; width: 12em; align-self: flex-start; gap: var(--spacing-xxl); }
#settings > div.tab-nav button { display: block; border: none; text-align: left; white-space: initial; padding: 0; }
#settings > div.tab-nav > #settings_show_all_pages { padding: var(--size-2) var(--size-4); }
#settings>div.tab-content { flex: 10 0 75%; display: grid; }
#settings>div.tab-content>div { border: none; padding: 0; }
#settings>div.tab-content>div>div>div>div>div { flex-direction: unset; }
#settings>div.tab-nav { display: grid; grid-template-columns: repeat(auto-fill, .5em minmax(10em, 1fr)); flex: 1 0 auto; width: 12em; align-self: flex-start; gap: var(--spacing-xxl); }
#settings>div.tab-nav button { display: block; border: none; text-align: left; white-space: initial; padding: 0; }
#settings>div.tab-nav>#settings_show_all_pages { padding: var(--size-2) var(--size-4); }
#settings .block.gradio-checkbox { margin: 0; width: auto; }
#settings .dirtyable { gap: .5em; }
#settings .dirtyable.hidden { display: none; }
@@ -146,8 +151,8 @@ div#extras_scale_to_tab div.form{ flex-direction: row; }
.modalControls span:hover, .modalControls span:focus { color: var(--highlight-color); filter: none; }
.lightboxModalPreviewZone { display: flex; width: 100%; height: 100%; }
.lightboxModalPreviewZone:focus-visible { outline: none; }
.lightboxModalPreviewZone > img { display: block; margin: auto; width: auto; }
.lightboxModalPreviewZone > img.modalImageFullscreen{ object-fit: contain; height: 100%; width: 100%; min-height: 0; background: transparent; }
.lightboxModalPreviewZone>img { display: block; margin: auto; width: auto; }
.lightboxModalPreviewZone>img.modalImageFullscreen{ object-fit: contain; height: 100%; width: 100%; min-height: 0; background: transparent; }
table.settings-value-table { background: white; border-collapse: collapse; margin: 1em; border: var(--spacing-sm) solid white; }
table.settings-value-table td { padding: 0.4em; border: 1px solid #ccc; max-width: 36em; }
.modalPrev, .modalNext { cursor: pointer; position: relative; z-index: 1; top: 0; width: auto; height: 100vh; line-height: 100vh; text-align: center; padding: 16px;
@@ -178,13 +183,13 @@ table.settings-value-table td { padding: 0.4em; border: 1px solid #ccc; max-widt
#extensions .date{ opacity: 0.85; font-size: 90%; }
/* extra networks */
.extra-networks > div { margin: 0; border-bottom: none !important; gap: 0.3em 0; }
.extra-networks>div { margin: 0; border-bottom: none !important; gap: 0.3em 0; }
.extra-networks .second-line { display: flex; width: -moz-available; width: -webkit-fill-available; gap: 0.3em; box-shadow: var(--input-shadow); }
.extra-networks .search { flex: 1; }
.extra-networks .description { flex: 3; }
.extra-networks .tab-nav > button { margin-right: 0; height: 24px; padding: 2px 4px 2px 4px; }
.extra-networks .tab-nav>button { margin-right: 0; height: 24px; padding: 2px 4px 2px 4px; }
.extra-networks .buttons { position: absolute; right: 0; margin: -4px; background: var(--background-color); }
.extra-networks .buttons > button { margin-left: -0.2em; height: 1.4em; color: var(--primary-300) !important; }
.extra-networks .buttons>button { margin-left: -0.2em; height: 1.4em; color: var(--primary-300) !important; }
.extra-networks .custom-button { width: 120px; width: 100%; background: none; justify-content: left; text-align: left; padding: 3px 3px 3px 12px; text-indent: -6px; box-shadow: none; line-break: auto; }
.extra-networks .custom-button:hover { background: var(--button-primary-background-fill) }
.extra-networks-tab { padding: 0 !important; }
@@ -200,18 +205,18 @@ table.settings-value-table td { padding: 0.4em; border: 1px solid #ccc; max-widt
.extra-network-cards .card:hover .overlay { background: rgba(0, 0, 0, 0.40); }
.extra-network-cards .card .overlay .tags { display: none; overflow-wrap: break-word; }
.extra-network-cards .card .overlay .tag { padding: 2px; margin: 2px; background: rgba(70, 70, 70, 0.60); font-size: var(--text-md); cursor: pointer; display: inline-block; }
.extra-network-cards .card .actions > span { padding: 4px; }
.extra-network-cards .card .actions > span:hover { color: var(--highlight-color); }
.extra-network-cards .card .actions>span { padding: 4px; }
.extra-network-cards .card .actions>span:hover { color: var(--highlight-color); }
.extra-network-cards .card:hover .actions { display: block; }
.extra-network-cards .card:hover .overlay .tags { display: block; }
.extra-network-cards .card .actions { font-size: 3em; display: none; text-align-last: right; cursor: pointer; font-variant: unicase; position: absolute; z-index: 100; right: 0; height: 0.7em; width: 100%; background: rgba(0, 0, 0, 0.40); }
.extra-network-cards .card-list { display: flex; margin: 0.3em; padding: 0.3em; background: var(--input-background-fill); cursor: pointer; border-radius: var(--button-large-radius); }
.extra-network-cards .card-list .tag { color: var(--primary-500); margin-left: 0.8em; }
.extra-details-close { position: fixed; top: 0.2em; right: 0.2em; z-index: 99; background: var(--button-secondary-background-fill) !important; }
#txt2img_description, #img2img_description { max-height: 63px; overflow-y: auto !important; }
#txt2img_description > label > textarea, #img2img_description > label > textarea { font-size: 0.9em }
#txt2img_description, #img2img_description, #control_description { max-height: 63px; overflow-y: auto !important; }
#txt2img_description>label>textarea, #img2img_description>label>textarea, #control_description>label>textarea { font-size: 0.9em }
#txt2img_extra_details > div, #img2img_extra_details > div { overflow-y: auto; min-height: 40vh; max-height: 80vh; align-self: flex-start; }
#txt2img_extra_details>div, #img2img_extra_details>div { overflow-y: auto; min-height: 40vh; max-height: 80vh; align-self: flex-start; }
#txt2img_extra_details, #img2img_extra_details { position: fixed; bottom: 50%; left: 50%; transform: translate(-50%, 50%); padding: 0.8em; border: var(--block-border-width) solid var(--highlight-color) !important;
z-index: 100; box-shadow: var(--button-shadow); }
#txt2img_extra_details td:first-child, #img2img_extra_details td:first-child { font-weight: bold; vertical-align: top; }
@@ -221,14 +226,14 @@ table.settings-value-table td { padding: 0.4em; border: 1px solid #ccc; max-widt
/* specific elements */
#modelmerger_interp_description { margin-top: 1em; margin-bottom: 1em; }
#scripts_alwayson_txt2img, #scripts_alwayson_img2img { padding: 0 }
#scripts_alwayson_txt2img > .label-wrap, #scripts_alwayson_img2img > .label-wrap { background: var(--input-background-fill); padding: 0; margin: 0; border-radius: var(--radius-lg); }
#scripts_alwayson_txt2img > .label-wrap > span, #scripts_alwayson_img2img > .label-wrap > span { padding: var(--spacing-xxl); }
#scripts_alwayson_txt2img>.label-wrap, #scripts_alwayson_img2img>.label-wrap { background: var(--input-background-fill); padding: 0; margin: 0; border-radius: var(--radius-lg); }
#scripts_alwayson_txt2img>.label-wrap>span, #scripts_alwayson_img2img>.label-wrap>span { padding: var(--spacing-xxl); }
#scripts_alwayson_txt2img div { max-width: var(--left-column); }
#script_txt2img_agent_scheduler { display: none; }
#refresh_tac_refreshTempFiles { display: none; }
#train_tab { flex-flow: row-reverse; }
#models_tab { flex-flow: row-reverse; }
#swap_axes > button { min-width: 100px; font-size: 1em; }
#swap_axes>button { min-width: 100px; font-size: 1em; }
#ui_defaults_review { margin: 1em; }
/* extras */
@@ -251,6 +256,17 @@ table.settings-value-table td { padding: 0.4em; border: 1px solid #ccc; max-widt
.nvml { position: fixed; bottom: 10px; right: 10px; background: var(--background-fill-primary); border: 1px solid var(--button-primary-border-color); padding: 6px; color: var(--button-primary-text-color);
font-size: 0.7em; z-index: 50; font-family: monospace; display: none; }
/* control */
#control_input_type { max-width: 18em }
#control_settings .small-accordion .form { min-width: 350px !important }
.control-button { min-height: 42px; max-height: 42px; line-height: 1em; }
.control-tabs>.tab-nav { margin-bottom: 0; margin-top: 0; }
.processor-settings { padding: 0 !important; max-width: 300px; }
.processor-group>div { flex-flow: wrap;gap: 1em; }
/* main info */
.main-info { font-weight: var(--section-header-text-weight); color: var(--body-text-color-subdued); padding: 1em !important; margin-top: 2em !important; line-height: var(--line-lg) !important; }
/* loader */
.splash { position: fixed; top: 0; left: 0; width: 100vw; height: 100vh; z-index: 1000; display: block; text-align: center; }
.motd { margin-top: 2em; color: var(--body-text-color-subdued); font-family: monospace; font-variant: all-petite-caps; }
@@ -280,76 +296,40 @@ table.settings-value-table td { padding: 0.4em; border: 1px solid #ccc; max-widt
--spacing-xxl: 6px;
}
/* Apply different styles for devices with coarse pointers dependant on screen resolution */
@media (hover: none) and (pointer: coarse) {
/* Do not affect displays larger than 1024px wide. */
@media (max-width: 1024px) {
/* Screens smaller than 400px wide */
@media (max-width: 399px) {
@media (hover: none) and (pointer: coarse) { /* Apply different styles for devices with coarse pointers dependant on screen resolution */
@media (max-width: 1024px) { /* Do not affect displays larger than 1024px wide. */
@media (max-width: 399px) { /* Screens smaller than 400px wide */
:root, .light, .dark { --left-column: 100%; }
/* maintain single column for from image operations on larger mobile devices */
#txt2img_results, #img2img_results, #extras_results { min-width: calc(min(320px, 100%)) !important;}
#txt2img_results, #img2img_results, #extras_results { min-width: calc(min(320px, 100%)) !important;} /* maintain single column for from image operations on larger mobile devices */
#txt2img_footer p { text-wrap: wrap; }
}
/* Screens larger than 400px wide */
@media (min-width: 400px) {
}
@media (min-width: 400px) { /* Screens larger than 400px wide */
:root, .light, .dark {--left-column: 50%;}
/* maintain side by side split on larger mobile displays for from text */
#txt2img_results, #extras_results, #txt2img_footer p {text-wrap: wrap; max-width: 100% !important; }
#txt2img_results, #extras_results, #txt2im g_footer p {text-wrap: wrap; max-width: 100% !important; } /* maintain side by side split on larger mobile displays for from text */
}
#scripts_alwayson_txt2img div, #scripts_alwayson_img2img div { max-width: 100%; }
#txt2img_prompt_container, #img2img_prompt_container { resize:vertical !important; }
/* make generate and enqueue buttons take up the entire width of their rows. */
#txt2img_generate_box, #txt2img_enqueue_wrapper { min-width: 100% !important;}
/*make interrogate buttons take up appropriate space. */
#img2img_toprow > div.gradio-column {flex-grow: 1 !important;}
#txt2img_prompt_container, #img2img_prompt_container, #control_prompt_container { resize:vertical !important; }
#txt2img_generate_box, #txt2img_enqueue_wrapper { min-width: 100% !important;} /* make generate and enqueue buttons take up the entire width of their rows. */
#img2img_toprow>div.gradio-column {flex-grow: 1 !important;} /*make interrogate buttons take up appropriate space. */
#img2img_actions_column {display: flex; min-width: fit-content !important; flex-direction: row;justify-content: space-evenly; align-items: center;}
#txt2img_generate_box, #img2img_generate_box, #txt2img_enqueue_wrapper,#img2img_enqueue_wrapper {display: flex;flex-direction: column;height: 4em !important;align-items: stretch;justify-content: space-evenly;}
/* maintain single column for from image operations on larger mobile devices */
#img2img_interface, #img2img_results, #img2img_footer p {text-wrap: wrap; min-width: 100% !important; max-width: 100% !important;}
/* fix inpaint image display being too large for mobile displays */
#img2img_sketch, #img2maskimg, #inpaint_sketch {display: flex; alignment-baseline:after-edge !important; overflow: auto !important; resize: none !important; }
#img2img_interface, #img2img_results, #img2img_footer p {text-wrap: wrap; min-width: 100% !important; max-width: 100% !important;} /* maintain single column for from image operations on larger mobile devices */
#img2img_sketch, #img2maskimg, #inpaint_sketch {display: flex; overflow: auto !important; resize: none !important; } /* fix inpaint image display being too large for mobile displays */
#img2maskimg canvas { width: auto !important; max-height: 100% !important; height: auto !important; }
/* fix from text/image UI elements to prevent them from moving around within the UI */
#txt2img_sampler, #txt2img_batch, #txt2img_seed_group, #txt2img_advanced, #txt2img_second_pass, #img2img_sampling_group, #img2img_resize_group, #img2img_batch_group, #img2img_seed_group, #img2img_denoise_group, #img2img_advanced_group { width: 100% !important; }
#img2img_resize_group .gradio-radio > div { display: flex; flex-direction: column; width: unset !important; }
#txt2img_sampler, #txt2img_batch, #txt2img_seed_group, #txt2img_advanced, #txt2img_second_pass, #img2img_sampling_group, #img2img_resize_group, #img2img_batch_group, #img2img_seed_group, #img2img_denoise_group, #img2img_advanced_group { width: 100% !important; } /* fix from text/image UI elements to prevent them from moving around within the UI */
#img2img_resize_group .gradio-radio>div { display: flex; flex-direction: column; width: unset !important; }
#inpaint_controls div {display:flex;flex-direction: row;}
#inpaint_controls .gradio-radio > div { display: flex; flex-direction: column !important; }
/* move image preview/output on models page to bottom of page */
#models_tab { flex-direction: column-reverse !important; }
/* fix settings for agent scheduler */
#enqueue_keyboard_shortcut_modifiers, #enqueue_keyboard_shortcut_key div { max-width: 40% !important;}
/* adjust width of certain settings item to allow aligning as row, but not have it go off the screen */
#settings { display: flex; flex-direction: row; flex-wrap: wrap; max-width: 100% !important; }
#settings div.tab-content > div > div > div { max-width: 80% !important;}
#inpaint_controls .gradio-radio>div { display: flex; flex-direction: column !important; }
#models_tab { flex-direction: column-reverse !important; } /* move image preview/output on models page to bottom of page */
#enqueue_keyboard_shortcut_modifiers, #enqueue_keyboard_shortcut_key div { max-width: 40% !important;} /* fix settings for agent scheduler */
#settings { display: flex; flex-direction: row; flex-wrap: wrap; max-width: 100% !important; } /* adjust width of certain settings item to allow aligning as row, but not have it go off the screen */
#settings div.tab-content>div>div>div { max-width: 80% !important;}
#settings div .gradio-radio { width: unset !important; }
/* enable scrolling on extensions tab */
#tab_extensions table { border-collapse: collapse; display: block; overflow-x:auto !important;}
/* increase scrollbar size to make it finger friendly */
::-webkit-scrollbar { width: 25px !important; height:25px; }
/* adjust dropdown size to make them easier to select individual items on mobile. */
.gradio-dropdown ul.options {max-height: 41vh !important; }
#tab_extensions table { border-collapse: collapse; display: block; overflow-x:auto !important;} /* enable scrolling on extensions tab */
::-webkit-scrollbar { width: 25px !important; height:25px; } /* increase scrollbar size to make it finger friendly */
.gradio-dropdown ul.options {max-height: 41vh !important; } /* adjust dropdown size to make them easier to select individual items on mobile. */
.gradio-dropdown ul.options li.item {height: 40px !important; display: flex; align-items: center;}
/* adjust slider input fields as they were too large for mobile devices. */
.gradio-slider input[type="number"] { width: 4em; font-size: 0.8rem; height: 16px; text-align: center; }
.gradio-slider input[type="number"] { width: 4em; font-size: 0.8rem; height: 16px; text-align: center; } /* adjust slider input fields as they were too large for mobile devices. */
#txt2img_settings .block .padded:not(.gradio-accordion) {padding: 0 !important;margin-right: 0; min-width: 100% !important; width:100% !important;}
}
}
}
javascript/settings.js
+37 -2
View File
@@ -108,7 +108,7 @@ onAfterUiUpdate(async () => {
const settingsSearch = gradioApp().querySelectorAll('#settings_search > label > textarea')[0];
settingsSearch.oninput = (e) => {
setTimeout(() => {
log('settingsSearch', e.target.value)
log('settingsSearch', e.target.value);
showAllSettings();
gradioApp().querySelectorAll('#tab_settings .tabitem').forEach((section) => {
section.querySelectorAll('.dirtyable').forEach((setting) => {
@@ -129,6 +129,40 @@ onOptionsChanged(() => {
});
});
async function initModels() {
const warn = () => `
<p style='color: white'>No models available</p>
- Select a model from reference list to download or<br>
- Set model path to a folder containing your models<br>
Current model path: ${opts.ckpt_dir}<br>
`;
const el = gradioApp().getElementById('main_info');
const en = gradioApp().getElementById('txt2img_extra_networks');
if (!el || !en) return;
const req = await fetch('/sdapi/v1/sd-models');
const res = req.ok ? await req.json() : [];
log('initModels', res.length);
const ready = () => `
<p style='color: white'>Ready</p>
${res.length} models available<br>
`;
el.innerHTML = res.length > 0 ? ready() : warn();
el.style.display = 'block';
setTimeout(() => el.style.display = 'none', res.length === 0 ? 30000 : 1500);
if (res.length === 0) {
if (en.classList.contains('hide')) gradioApp().getElementById('txt2img_extra_networks_btn').click();
const repeat = setInterval(() => {
const buttons = Array.from(gradioApp().querySelectorAll('#txt2img_model_subdirs > button')) || [];
const reference = buttons.find((b) => b.innerText === 'Reference');
if (reference) {
clearInterval(repeat);
reference.click();
log('enReferenceSelect');
}
}, 100);
}
}
function initSettings() {
if (settingsInitialized) return;
settingsInitialized = true;
@@ -138,7 +172,7 @@ function initSettings() {
const observer = new MutationObserver((mutations) => {
const showAllPages = gradioApp().getElementById('settings_show_all_pages');
if (showAllPages.style.display === 'none') return;
const mutation = (mut) => mut.type === 'attributes' && mut.attributeName === 'style'
const mutation = (mut) => mut.type === 'attributes' && mut.attributeName === 'style';
if (mutations.some(mutation)) showAllSettings();
});
const tabContentWrapper = document.createElement('div');
@@ -155,3 +189,4 @@ function initSettings() {
}
onUiLoaded(initSettings);
onUiLoaded(initModels);
javascript/timeless-beige.css
+297
View File
@@ -0,0 +1,297 @@
/* generic html tags */
:root, .light, .dark {
--font: 'system-ui', 'ui-sans-serif', 'system-ui', "Roboto", sans-serif;
--font-mono: 'ui-monospace', 'Consolas', monospace;
--font-size: 16px;
--primary-100: #212226; /* bg color*/
--primary-200: #17181b; /* drop down menu/ prompt window fill*/
--primary-300: #0a0c0e; /* black */
--primary-400: #2f3034; /* small buttons*/
--primary-500: #434242; /* main accent color retro beige*/
--primary-700: #e75d5d; /* light blue gray*/
--primary-800: #e75d5d; /* sat orange(hover accent)*/
--highlight-color: var(--primary-500);
--inactive-color: var(--primary--800);
--body-text-color: var(--neutral-100);
--body-text-color-subdued: var(--neutral-300);
--background-color: var(--primary-100);
--background-fill-primary: var(--input-background-fill);
--input-padding: 8px;
--input-background-fill: var(--primary-200);
--input-shadow: none;
--button-secondary-text-color: white;
--button-secondary-background-fill: var(--primary-400);
--button-secondary-background-fill-hover: var(--primary-700);
--block-title-text-color: var(--neutral-300);
--radius-sm: 1px;
--radius-lg: 6px;
--spacing-md: 4px;
--spacing-xxl: 8px;
--line-sm: 1.2em;
--line-md: 1.4em;
}
html { font-size: var(--font-size); }
body, button, input, select, textarea { font-family: var(--font);}
button { font-size: 1.2rem; max-width: 400px; }
img { background-color: var(--background-color); }
input[type=range] { height: var(--line-sm); appearance: none; margin-top: 0; min-width: 160px; background-color: var(--background-color); width: 100%; background: transparent; }
input[type=range]::-webkit-slider-runnable-track, input[type=range]::-moz-range-track { width: 100%; height: 6px; cursor: pointer; background: var(--primary-400); border-radius: var(--radius-lg); border: 0px solid #222222; }
input[type=range]::-webkit-slider-thumb, input[type=range]::-moz-range-thumb { border: 0px solid #000000; height: var(--line-sm); width: 8px; border-radius: var(--radius-lg); background: white; cursor: pointer; appearance: none; margin-top: 0px; }
input[type=range]::-moz-range-progress { background-color: var(--primary-500); height: 6px; border-radius: var(--radius-lg); }
::-webkit-scrollbar-track { background: #333333; }
::-webkit-scrollbar-thumb { background-color: var(--highlight-color); border-radius: var(--radius-lg); border-width: 0; box-shadow: 2px 2px 3px #111111; }
div.form { border-width: 0; box-shadow: none; background: transparent; overflow: visible; margin-bottom: 6px; }
div.compact { gap: 1em; }
/* gradio style classes */
fieldset .gr-block.gr-box, label.block span { padding: 0; margin-top: -4px; }
.border-2 { border-width: 0; }
.border-b-2 { border-bottom-width: 2px; border-color: var(--highlight-color) !important; padding-bottom: 2px; margin-bottom: 8px; }
.bg-white { color: lightyellow; background-color: var(--inactive-color); }
.gr-box { border-radius: var(--radius-sm) !important; background-color: #111111 !important; box-shadow: 2px 2px 3px #111111; border-width: 0; padding: 4px; margin: 12px 0px 12px 0px }
.gr-button { font-weight: normal; box-shadow: 2px 2px 3px #111111; font-size: 0.8rem; min-width: 32px; min-height: 32px; padding: 3px; margin: 3px; }
.gr-check-radio { background-color: var(--inactive-color); border-width: 0; border-radius: var(--radius-lg); box-shadow: 2px 2px 3px #111111; }
.gr-check-radio:checked { background-color: var(--highlight-color); }
.gr-compact { background-color: var(--background-color); }
.gr-form { border-width: 0; }
.gr-input { background-color: #333333 !important; padding: 4px; margin: 4px; }
.gr-input-label { color: lightyellow; border-width: 0; background: transparent; padding: 2px !important; }
.gr-panel { background-color: var(--background-color); }
.eta-bar { display: none !important }
svg.feather.feather-image, .feather .feather-image { display: none }
.gap-2 { padding-top: 8px; }
.gr-box > div > div > input.gr-text-input { right: 0; width: 4em; padding: 0; top: -12px; border: none; max-height: 20px; }
.output-html { line-height: 1.2rem; overflow-x: hidden; }
.output-html > div { margin-bottom: 8px; }
.overflow-hidden .flex .flex-col .relative col .gap-4 { min-width: var(--left-column); max-width: var(--left-column); } /* this is a problematic one */
.p-2 { padding: 0; }
.px-4 { padding-lefT: 1rem; padding-right: 1rem; }
.py-6 { padding-bottom: 0; }
.tabs { background-color: var(--background-color); }
.block.token-counter span { background-color: var(--input-background-fill) !important; box-shadow: 2px 2px 2px #111; border: none !important; font-size: 0.8rem; }
.tab-nav { zoom: 120%; margin-top: 10px; margin-bottom: 10px; border-bottom: 2px solid var(--highlight-color) !important; padding-bottom: 2px; }
div.tab-nav button.selected {background-color: var(--button-primary-background-fill);}
#settings div.tab-nav button.selected {background-color: var(--background-color); color: var(--primary-800); font-weight: bold;}
.label-wrap { background-color: #292b30; /* extension tab color*/ padding: 16px 8px 8px 8px; border-radius: var(--radius-lg); padding-left: 8px !important; }
.small-accordion .label-wrap { padding: 8px 0px 8px 0px; }
.small-accordion .label-wrap .icon { margin-right: 1em; }
.gradio-button.tool { border: none; box-shadow: none; border-radius: var(--radius-lg);}
button.selected {background: var(--button-primary-background-fill);}
.center.boundedheight.flex {background-color: var(--input-background-fill);}
.compact {border-radius: var(--border-radius-lg);}
#logMonitorData {background-color: var(--input-background-fill);}
#tab_extensions table td, #tab_extensions table th, #tab_config table td, #tab_config table th { border: none; padding: 0.5em; background-color: var(--primary-200); }
#tab_extensions table, #tab_config table { width: 96vw; }
#tab_extensions table input[type=checkbox] {appearance: none; border-radius: 0px;}
#tab_extensions button:hover { background-color: var(--button-secondary-background-fill-hover);}
/* automatic style classes */
.progressDiv { border-radius: var(--radius-sm) !important; position: fixed; top: 44px; right: 26px; max-width: 262px; height: 48px; z-index: 99; box-shadow: var(--button-shadow); }
.progressDiv .progress { border-radius: var(--radius-lg) !important; background: var(--highlight-color); line-height: 3rem; height: 48px; }
.gallery-item { box-shadow: none !important; }
.performance { color: #888; }
.extra-networks { border-left: 2px solid var(--highlight-color) !important; padding-left: 4px; }
.image-buttons { gap: 10px !important; justify-content: center; }
.image-buttons > button { max-width: 160px; }
.tooltip { background: var(--primary-800); color: white; border: none; border-radius: var(--radius-lg) }
#system_row > button, #settings_row > button, #config_row > button { max-width: 190px; }
/* gradio elements overrides */
#div.gradio-container { overflow-x: hidden; }
#img2img_label_copy_to_img2img { font-weight: normal; }
#txt2img_prompt, #txt2img_neg_prompt, #img2img_prompt, #img2img_neg_prompt { background-color: var(--background-color); box-shadow: 4px 4px 4px 0px #333333 !important; }
#txt2img_prompt > label > textarea, #txt2img_neg_prompt > label > textarea, #img2img_prompt > label > textarea, #img2img_neg_prompt > label > textarea { font-size: 1.1rem; }
#img2img_settings { min-width: calc(2 * var(--left-column)); max-width: calc(2 * var(--left-column)); background-color: #111111; padding-top: 16px; }
#interrogate, #deepbooru { margin: 0 0px 10px 0px; max-width: 80px; max-height: 80px; font-weight: normal; font-size: 0.95em; }
#quicksettings .gr-button-tool { font-size: 1.6rem; box-shadow: none; margin-top: -2px; height: 2.4em; }
#quicksettings button {padding: 0 0.5em 0.1em 0.5em;}
#open_folder_extras, #footer, #style_pos_col, #style_neg_col, #roll_col, #extras_upscaler_2, #extras_upscaler_2_visibility, #txt2img_seed_resize_from_w, #txt2img_seed_resize_from_h { display: none; }
#save-animation { border-radius: var(--radius-sm) !important; margin-bottom: 16px; background-color: #111111; }
#script_list { padding: 4px; margin-top: 16px; margin-bottom: 8px; }
#settings > div.flex-wrap { width: 15em; }
#txt2img_cfg_scale { min-width: 200px; }
#txt2img_checkboxes, #img2img_checkboxes { background-color: transparent; }
#txt2img_checkboxes, #img2img_checkboxes { margin-bottom: 0.2em; }
#txt2img_actions_column, #img2img_actions_column { flex-flow: wrap; justify-content: space-between; }
#txt2img_enqueue_wrapper, #img2img_enqueue_wrapper { min-width: unset; width: 48%; }
#txt2img_generate_box, #img2img_generate_box { min-width: unset; width: 48%; }
#extras_upscale { margin-top: 10px }
#txt2img_progress_row > div { min-width: var(--left-column); max-width: var(--left-column); }
#txt2img_settings { min-width: var(--left-column); max-width: var(--left-column); background-color: #111111; padding-top: 16px; }
#pnginfo_html2_info { margin-top: -18px; background-color: var(--input-background-fill); padding: var(--input-padding) }
#txt2img_tools, #img2img_tools { margin-top: -4px; margin-bottom: -4px; }
#txt2img_styles_row, #img2img_styles_row { margin-top: -6px; z-index: 200; }
/* based on gradio built-in dark theme */
:root, .light, .dark {
--body-background-fill: var(--background-color);
--color-accent-soft: var(--neutral-700);
--background-fill-secondary: none;
--border-color-accent: var(--background-color);
--border-color-primary: var(--background-color);
--link-text-color-active: var(--primary-500);
--link-text-color: var(--secondary-500);
--link-text-color-hover: var(--secondary-400);
--link-text-color-visited: var(--secondary-600);
--shadow-spread: 1px;
--block-background-fill: None;
--block-border-color: var(--border-color-primary);
--block_border_width: None;
--block-info-text-color: var(--body-text-color-subdued);
--block-label-background-fill: var(--background-fill-secondary);
--block-label-border-color: var(--border-color-primary);
--block_label_border_width: None;
--block-label-text-color: var(--neutral-200);
--block_shadow: None;
--block_title_background_fill: None;
--block_title_border_color: None;
--block_title_border_width: None;
--panel-background-fill: var(--background-fill-secondary);
--panel-border-color: var(--border-color-primary);
--panel_border_width: None;
--checkbox-background-color: var(--primary-400);
--checkbox-background-color-focus: var(--primary-700);
--checkbox-background-color-hover: var(--primary-700);
--checkbox-background-color-selected: var(--primary-500);
--checkbox-border-color: transparent;
--checkbox-border-color-focus: var(--primary-800);
--checkbox-border-color-hover: var(--primary-800);
--checkbox-border-color-selected: var(--primary-800);
--checkbox-border-width: var(--input-border-width);
--checkbox-label-background-fill: None;
--checkbox-label-background-fill-hover: None;
--checkbox-label-background-fill-selected: var(--checkbox-label-background-fill);
--checkbox-label-border-color: var(--border-color-primary);
--checkbox-label-border-color-hover: var(--checkbox-label-border-color);
--checkbox-label-border-width: var(--input-border-width);
--checkbox-label-text-color: var(--body-text-color);
--checkbox-label-text-color-selected: var(--checkbox-label-text-color);
--error-background-fill: var(--background-fill-primary);
--error-border-color: var(--border-color-primary);
--error-text-color: #f768b7; /*was ef4444*/
--input-background-fill-focus: var(--secondary-600);
--input-background-fill-hover: var(--input-background-fill);
--input-border-color: var(--background-color);
--input-border-color-focus: var(--primary-800);
--input-placeholder-color: var(--neutral-500);
--input-shadow-focus: None;
--loader_color: None;
--slider_color: None;
--stat-background-fill: linear-gradient(to right, var(--primary-400), var(--primary-800));
--table-border-color: var(--neutral-700);
--table-even-background-fill: var(--primary-300);
--table-odd-background-fill: var(--primary-200);
--table-row-focus: var(--color-accent-soft);
--button-border-width: var(--input-border-width);
--button-cancel-background-fill: linear-gradient(to bottom right, #dc2626, #b91c1c);
--button-cancel-background-fill-hover: linear-gradient(to bottom right, #dc2626, #dc2626);
--button-cancel-border-color: #dc2626;
--button-cancel-border-color-hover: var(--button-cancel-border-color);
--button-cancel-text-color: white;
--button-cancel-text-color-hover: var(--button-cancel-text-color);
--button-primary-background-fill: var(--primary-500);
--button-primary-background-fill-hover: var(--primary-800);
--button-primary-border-color: var(--primary-500);
--button-primary-border-color-hover: var(--button-primary-border-color);
--button-primary-text-color: white;
--button-primary-text-color-hover: var(--button-primary-text-color);
--button-secondary-border-color: var(--neutral-600);
--button-secondary-border-color-hover: var(--button-secondary-border-color);
--button-secondary-text-color-hover: var(--button-secondary-text-color);
--secondary-50: #eff6ff;
--secondary-100: #dbeafe;
--secondary-200: #bfdbfe;
--secondary-300: #93c5fd;
--secondary-400: #60a5fa;
--secondary-500: #3b82f6;
--secondary-600: #2563eb;
--secondary-700: #1d4ed8;
--secondary-800: #1e40af;
--secondary-900: #1e3a8a;
--secondary-950: #1d3660;
--neutral-50: #f0f0f0; /* */
--neutral-100: #e0dedc;/* majority of text (neutral gray yellow) */
--neutral-200: #d0d0d0;
--neutral-300: #9d9dab; /* top tab text (light accent) */
--neutral-400: #ffba85;/* tab title (light beige) */
--neutral-500: #484746; /* prompt text (desat accent)*/
--neutral-600: #605a54; /* tab outline color (accent color)*/
--neutral-700: #1b1c1e; /* small settings tab accent (dark)*/
--neutral-800: #e75d5d; /* bright orange accent */
--neutral-900: #111827;
--neutral-950: #0b0f19;
--radius-xxs: 0;
--radius-xs: 0;
--radius-md: 0;
--radius-xl: 0;
--radius-xxl: 0;
--body-text-size: var(--text-md);
--body-text-weight: 400;
--embed-radius: var(--radius-lg);
--color-accent: var(--primary-500);
--shadow-drop: 0;
--shadow-drop-lg: 0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1);
--shadow-inset: rgba(0,0,0,0.05) 0px 2px 4px 0px inset;
--block-border-width: 1px;
--block-info-text-size: var(--text-sm);
--block-info-text-weight: 400;
--block-label-border-width: 1px;
--block-label-margin: 0;
--block-label-padding: var(--spacing-sm) var(--spacing-lg);
--block-label-radius: calc(var(--radius-lg) - 1px) 0 calc(var(--radius-lg) - 1px) 0;
--block-label-right-radius: 0 calc(var(--radius-lg) - 1px) 0 calc(var(--radius-lg) - 1px);
--block-label-text-size: var(--text-sm);
--block-label-text-weight: 400;
--block-padding: var(--spacing-xl) calc(var(--spacing-xl) + 2px);
--block-radius: var(--radius-lg);
--block-shadow: var(--shadow-drop);
--block-title-background-fill: none;
--block-title-border-color: none;
--block-title-border-width: 0;
--block-title-padding: 0;
--block-title-radius: none;
--block-title-text-size: var(--text-md);
--block-title-text-weight: 400;
--container-radius: var(--radius-lg);
--form-gap-width: 1px;
--layout-gap: var(--spacing-xxl);
--panel-border-width: 0;
--section-header-text-size: var(--text-md);
--section-header-text-weight: 400;
--checkbox-border-radius: var(--radius-sm);
--checkbox-label-gap: 2px;
--checkbox-label-padding: var(--spacing-md);
--checkbox-label-shadow: var(--shadow-drop);
--checkbox-label-text-size: var(--text-md);
--checkbox-label-text-weight: 400;
--checkbox-check: url("data:image/svg+xml,%3csvg viewBox='0 0 16 16' fill='white' xmlns='http://www.w3.org/2000/svg'%3e%3cpath d='M12.207 4.793a1 1 0 010 1.414l-5 5a1 1 0 01-1.414 0l-2-2a1 1 0 011.414-1.414L6.5 9.086l4.293-4.293a1 1 0 011.414 0z'/%3e%3c/svg%3e");
--radio-circle: url("data:image/svg+xml,%3csvg viewBox='0 0 16 16' fill='white' xmlns='http://www.w3.org/2000/svg'%3e%3ccircle cx='8' cy='8' r='3'/%3e%3c/svg%3e");
--checkbox-shadow: var(--input-shadow);
--error-border-width: 1px;
--input-border-width: 1px;
--input-radius: var(--radius-lg);
--input-text-size: var(--text-md);
--input-text-weight: 400;
--loader-color: var(--color-accent);
--prose-text-size: var(--text-md);
--prose-text-weight: 400;
--prose-header-text-weight: 600;
--slider-color: ;
--table-radius: var(--radius-lg);
--button-large-padding: 2px 6px;
--button-large-radius: var(--radius-lg);
--button-large-text-size: var(--text-lg);
--button-large-text-weight: 400;
--button-shadow: none;
--button-shadow-active: none;
--button-shadow-hover: none;
--button-small-padding: var(--spacing-sm) calc(2 * var(--spacing-sm));
--button-small-radius: var(--radius-lg);
--button-small-text-size: var(--text-md);
--button-small-text-weight: 400;
--button-transition: none;
--size-9: 64px;
--size-14: 64px;
}
javascript/ui.js
+38 -3
View File
@@ -66,38 +66,54 @@ function extract_image_from_gallery(gallery) {
window.args_to_array = Array.from; // Compatibility with e.g. extensions that may expect this to be around
function switchToTab(tab) {
const tabs = Array.from(gradioApp().querySelectorAll('#tabs > .tab-nav > button'));
const btn = tabs?.find((t) => t.innerText === tab);
log('switchToTab', tab);
if (btn) btn.click();
}
function switch_to_txt2img(...args) {
gradioApp().querySelector('#tabs').querySelectorAll('button')[0].click();
switchToTab('Text');
return Array.from(arguments);
}
function switch_to_img2img_tab(no) {
gradioApp().querySelector('#tabs').querySelectorAll('button')[1].click();
switchToTab('Image');
gradioApp().getElementById('mode_img2img').querySelectorAll('button')[no].click();
}
function switch_to_img2img(...args) {
switchToTab('Image');
switch_to_img2img_tab(0);
return Array.from(arguments);
}
function switch_to_sketch(...args) {
switchToTab('Image');
switch_to_img2img_tab(1);
return Array.from(arguments);
}
function switch_to_inpaint(...args) {
switchToTab('Image');
switch_to_img2img_tab(2);
return Array.from(arguments);
}
function switch_to_inpaint_sketch(...args) {
switchToTab('Image');
switch_to_img2img_tab(3);
return Array.from(arguments);
}
function switch_to_extras(...args) {
gradioApp().querySelector('#tabs').querySelectorAll('button')[2].click();
switchToTab('Process');
return Array.from(arguments);
}
function switch_to_control(...args) {
switchToTab('Control');
return Array.from(arguments);
}
@@ -164,6 +180,17 @@ function submit_img2img(...args) {
return res;
}
function submit_control(...args) {
log('submitControl');
clearGallery('control');
const id = randomId();
requestProgress(id, null, gradioApp().getElementById('control_gallery'));
const res = create_submit_args(args);
res[0] = id;
res[1] = gradioApp().querySelector('#control-tabs > .tab-nav > .selected')?.innerText.toLowerCase() || ''; // selected tab name
return res;
}
function submit_postprocessing(...args) {
log('SubmitExtras');
clearGallery('extras');
@@ -211,6 +238,12 @@ function recalculate_prompts_inpaint(...args) {
return Array.from(arguments);
}
function recalculate_prompts_control(...args) {
recalculatePromptTokens('control_prompt');
recalculatePromptTokens('control_neg_prompt');
return Array.from(arguments);
}
function registerDragDrop() {
const qs = gradioApp().getElementById('quicksettings');
if (!qs) return;
@@ -279,6 +312,8 @@ onAfterUiUpdate(async () => {
registerTextarea('txt2img_neg_prompt', 'txt2img_negative_token_counter', 'txt2img_negative_token_button');
registerTextarea('img2img_prompt', 'img2img_token_counter', 'img2img_token_button');
registerTextarea('img2img_neg_prompt', 'img2img_negative_token_counter', 'img2img_negative_token_button');
registerTextarea('control_prompt', 'control_token_counter', 'control_token_button');
registerTextarea('control_neg_prompt', 'control_negative_token_counter', 'control_negative_token_button');
});
function update_txt2img_tokens(...args) {
launch.py
+10 -1
View File
@@ -47,6 +47,16 @@ def get_custom_args():
if current != default:
custom[arg] = getattr(args, arg)
installer.log.info(f'Command line args: {sys.argv[1:]} {installer.print_dict(custom)}')
if os.environ.get('SD_ENV_DEBUG', None) is not None:
env = os.environ.copy()
if 'PATH' in env:
del env['PATH']
if 'PS1' in env:
del env['PS1']
installer.log.trace(f'Environment: {installer.print_dict(env)}')
else:
env = [f'{k}={v}' for k, v in os.environ.items() if k.startswith('SD_')]
installer.log.debug(f'Env flags: {env}')
@lru_cache()
@@ -205,7 +215,6 @@ if __name__ == "__main__":
installer.log.info('Startup: standard')
installer.install_requirements()
installer.install_packages()
installer.install_repositories()
installer.install_submodules()
init_paths()
installer.install_extensions()
models/Reference/amused--amused-256.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

models/Reference/amused--amused-512.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

models/Reference/damo-vilab--text-to-video-ms-1.7b.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

models/Reference/dreamshaperXL_turboDpmppSDE.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

models/Reference/dreamshaper_8.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 54 KiB

models/Reference/juggernautXL_v7Rundiffusion.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

models/Reference/juggernaut_reborn.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

models/Reference/playgroundai--playground-v1.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

models/Reference/playgroundai--playground-v2-1024px-aesthetic.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

models/Reference/playgroundai--playground-v2-256px-base.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 100 KiB

models/Reference/playgroundai--playground-v2-512px-base.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 100 KiB

models/Reference/salesforce--blipdiffusion.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 46 KiB

models/Reference/segmind--Segmind-Vega.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 76 KiB

models/Reference/stabilityai--stable-video-diffusion-img2vid-xt.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 342 KiB

After

Width:  |  Height:  |  Size: 55 KiB

models/Reference/stabilityai--stable-video-diffusion-img2vid.jpg
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 342 KiB

After

Width:  |  Height:  |  Size: 56 KiB

modules/api/api.py
+37 -9
View File
@@ -13,7 +13,7 @@ import requests
import piexif
import piexif.helper
import gradio as gr
from modules import errors, shared, sd_samplers, deepbooru, sd_hijack, images, scripts, ui, postprocessing
from modules import errors, shared, sd_samplers, deepbooru, sd_hijack, images, scripts, ui, postprocessing, script_callbacks, generation_parameters_copypaste
from modules.sd_vae import vae_dict
from modules.api import models
from modules.processing import StableDiffusionProcessingTxt2Img, StableDiffusionProcessingImg2Img, process_images
@@ -133,7 +133,7 @@ class Api:
self.add_api_route("/sdapi/v1/embeddings", self.get_embeddings, methods=["GET"], response_model=models.EmbeddingsResponse)
self.add_api_route("/sdapi/v1/refresh-checkpoints", self.refresh_checkpoints, methods=["POST"])
self.add_api_route("/sdapi/v1/sd-vae", self.get_sd_vaes, methods=["GET"], response_model=List[models.SDVaeItem])
self.add_api_route("/sdapi/v1/refresh-vaes", self.refresh_vaes, methods=["POST"])
self.add_api_route("/sdapi/v1/refresh-vae", self.refresh_vaes, methods=["POST"])
self.add_api_route("/sdapi/v1/create/embedding", self.create_embedding, methods=["POST"], response_model=models.CreateResponse)
self.add_api_route("/sdapi/v1/create/hypernetwork", self.create_hypernetwork, methods=["POST"], response_model=models.CreateResponse)
self.add_api_route("/sdapi/v1/preprocess", self.preprocess, methods=["POST"], response_model=models.PreprocessResponse)
@@ -145,6 +145,7 @@ class Api:
self.add_api_route("/sdapi/v1/reload-checkpoint", self.reloadapi, methods=["POST"])
self.add_api_route("/sdapi/v1/scripts", self.get_scripts_list, methods=["GET"], response_model=models.ScriptsList)
self.add_api_route("/sdapi/v1/script-info", self.get_script_info, methods=["GET"], response_model=List[models.ScriptInfo])
self.add_api_route("/sdapi/v1/extensions", self.get_extensions_list, methods=["GET"], response_model=List[models.ExtensionItem])
self.add_api_route("/sdapi/v1/log", self.get_log_buffer, methods=["GET"], response_model=List)
self.add_api_route("/sdapi/v1/start", self.session_start, methods=["GET"])
self.add_api_route("/sdapi/v1/motd", self.get_motd, methods=["GET"], response_model=str)
@@ -287,14 +288,14 @@ class Api:
p.scripts = script_runner
p.outpath_grids = shared.opts.outdir_grids or shared.opts.outdir_txt2img_grids
p.outpath_samples = shared.opts.outdir_samples or shared.opts.outdir_txt2img_samples
shared.state.begin('api-txt2img')
shared.state.begin('api-txt2img', api=True)
script_args = self.init_script_args(p, txt2imgreq, self.default_script_arg_txt2img, selectable_scripts, selectable_script_idx, script_runner)
if selectable_scripts is not None:
processed = scripts.scripts_txt2img.run(p, *script_args) # Need to pass args as list here
else:
p.script_args = tuple(script_args) # Need to pass args as tuple here
processed = process_images(p)
shared.state.end()
shared.state.end(api=False)
b64images = list(map(encode_pil_to_base64, processed.images)) if send_images else []
return models.TextToImageResponse(images=b64images, parameters=vars(txt2imgreq), info=processed.js())
@@ -335,14 +336,14 @@ class Api:
p.scripts = script_runner
p.outpath_grids = shared.opts.outdir_img2img_grids
p.outpath_samples = shared.opts.outdir_img2img_samples
shared.state.begin('api-img2img')
shared.state.begin('api-img2img', api=True)
script_args = self.init_script_args(p, img2imgreq, self.default_script_arg_img2img, selectable_scripts, selectable_script_idx, script_runner)
if selectable_scripts is not None:
processed = scripts.scripts_img2img.run(p, *script_args) # Need to pass args as list here
else:
p.script_args = tuple(script_args) # Need to pass args as tuple here
processed = process_images(p)
shared.state.end()
shared.state.end(api=False)
b64images = list(map(encode_pil_to_base64, processed.images)) if send_images else []
if not img2imgreq.include_init_images:
@@ -368,14 +369,22 @@ class Api:
def pnginfoapi(self, req: models.PNGInfoRequest):
if not req.image.strip():
return models.PNGInfoResponse(info="")
image = decode_base64_to_image(req.image.strip())
if image is None:
return models.PNGInfoResponse(info="")
geninfo, items = images.read_info_from_image(image)
if geninfo is None:
geninfo = ""
items = {**{'parameters': geninfo}, **items}
return models.PNGInfoResponse(info=geninfo, items=items)
if items and items['parameters']:
del items['parameters']
params = generation_parameters_copypaste.parse_generation_parameters(geninfo)
script_callbacks.infotext_pasted_callback(geninfo, params)
return models.PNGInfoResponse(info=geninfo, items=items, parameters=params)
def progressapi(self, req: models.ProgressRequest = Depends()):
if shared.state.job_count == 0:
@@ -464,7 +473,7 @@ class Api:
return [{"name": upscaler.name, "model_name": upscaler.scaler.model_name, "model_path": upscaler.data_path, "model_url": None, "scale": upscaler.scale} for upscaler in shared.sd_upscalers]
def get_sd_models(self):
return [{"title": x.title, "name": x.name, "filename": x.filename, "type": x.type, "hash": x.shorthash, "sha256": x.sha256, "config": find_checkpoint_config_near_filename(x)} for x in checkpoints_list.values()]
return [{"title": x.title, "model_name": x.name, "filename": x.filename, "type": x.type, "hash": x.shorthash, "sha256": x.sha256, "config": find_checkpoint_config_near_filename(x)} for x in checkpoints_list.values()]
def get_hypernetworks(self):
return [{"name": name, "path": shared.hypernetworks[name]} for name in shared.hypernetworks]
@@ -640,6 +649,25 @@ class Api:
cuda = { 'error': f'{err}' }
return models.MemoryResponse(ram = ram, cuda = cuda)
def get_extensions_list(self):
from modules import extensions
extensions.list_extensions()
ext_list = []
for ext in extensions.extensions:
ext: extensions.Extension
ext.read_info()
if ext.remote is not None:
ext_list.append({
"name": ext.name,
"remote": ext.remote,
"branch": ext.branch,
"commit_hash":ext.commit_hash,
"commit_date":ext.commit_date,
"version":ext.version,
"enabled":ext.enabled
})
return ext_list
def launch(self):
config = {
"listen": shared.cmd_opts.listen,
modules/api/models.py
+13 -6
View File
@@ -170,7 +170,8 @@ class PNGInfoRequest(BaseModel):
class PNGInfoResponse(BaseModel):
info: str = Field(title="Image info", description="A string with the parameters used to generate the image")
items: dict = Field(title="Items", description="An object containing all the info the image had")
items: dict = Field(title="Items", description="A dictionary containing all the other fields the image had")
parameters: dict = Field(title="Parameters", description="A dictionary with parsed generation info fields")
class LogRequest(BaseModel):
lines: int = Field(default=100, title="Lines", description="How many lines to return")
@@ -209,7 +210,7 @@ for key, metadata in shared.opts.data_labels.items():
if metadata is not None:
fields.update({key: (Optional[optType], Field(
default=metadata.default ,description=metadata.label))})
default=metadata.default, description=metadata.label))})
else:
fields.update({key: (Optional[optType], Field())})
@@ -245,7 +246,7 @@ class UpscalerItem(BaseModel):
class SDModelItem(BaseModel):
title: str = Field(title="Title")
name: str = Field(title="Model Name")
model_name: str = Field(title="Model Name")
filename: str = Field(title="Filename")
type: str = Field(title="Model type")
sha256: Optional[str] = Field(title="SHA256 hash")
@@ -286,7 +287,6 @@ class ExtraNetworkItem(BaseModel):
# metadata: Optional[Any] = Field(title="Metadata")
# local: Optional[str] = Field(title="Local")
class ArtistItem(BaseModel):
name: str = Field(title="Name")
score: float = Field(title="Score")
@@ -311,7 +311,6 @@ class ScriptsList(BaseModel):
txt2img: list = Field(default=None, title="Txt2img", description="Titles of scripts (txt2img)")
img2img: list = Field(default=None, title="Img2img", description="Titles of scripts (img2img)")
class ScriptArg(BaseModel):
label: str = Field(default=None, title="Label", description="Name of the argument in UI")
value: Optional[Any] = Field(default=None, title="Value", description="Default value of the argument")
@@ -320,9 +319,17 @@ class ScriptArg(BaseModel):
step: Optional[Any] = Field(default=None, title="Minimum", description="Step for changing value of the argumentin UI")
choices: Optional[Any] = Field(default=None, title="Choices", description="Possible values for the argument")
class ScriptInfo(BaseModel):
name: str = Field(default=None, title="Name", description="Script name")
is_alwayson: bool = Field(default=None, title="IsAlwayson", description="Flag specifying whether this script is an alwayson script")
is_img2img: bool = Field(default=None, title="IsImg2img", description="Flag specifying whether this script is an img2img script")
args: List[ScriptArg] = Field(title="Arguments", description="List of script's arguments")
class ExtensionItem(BaseModel):
name: str = Field(title="Name", description="Extension name")
remote: str = Field(title="Remote", description="Extension Repository URL")
branch: str = Field(title="Branch", description="Extension Repository Branch")
commit_hash: str = Field(title="Commit Hash", description="Extension Repository Commit Hash")
version: str = Field(title="Version", description="Extension Version")
commit_date: str = Field(title="Commit Date", description="Extension Repository Commit Date")
enabled: bool = Field(title="Enabled", description="Flag specifying whether this extension is enabled")
modules/call_queue.py
+6 -2
View File
@@ -76,7 +76,11 @@ def wrap_gradio_call(func, extra_outputs=None, add_stats=False, name=None):
if not shared.mem_mon.disabled:
vram = {k: -(v//-(1024*1024)) for k, v in shared.mem_mon.read().items()}
if vram.get('active_peak', 0) > 0:
vram_html = f" | <p class='vram'>GPU active {max(vram['active_peak'], vram['reserved_peak'])} MB reserved {vram['reserved']} | used {vram['used']} MB free {vram['free']} MB total {vram['total']} MB | retries {vram['retries']} oom {vram['oom']}</p>"
res[-1] += f"<div class='performance'><p class='time'>Time: {elapsed_text}</p>{vram_html}</div>"
vram_html = " | <p class='vram'>"
vram_html += f"GPU active {max(vram['active_peak'], vram['reserved_peak'])} MB reserved {vram['reserved']} | used {vram['used']} MB free {vram['free']} MB total {vram['total']} MB"
vram_html += f" | retries {vram['retries']} oom {vram['oom']}" if vram.get('retries', 0) > 0 or vram.get('oom', 0) > 0 else ''
vram_html += "</p>"
if isinstance(res, list):
res[-1] += f"<div class='performance'><p class='time'>Time: {elapsed_text}</p>{vram_html}</div>"
return tuple(res)
return f
modules/control/proc/__init__.py
View File
modules/control/proc/canny.py
+36
View File
@@ -0,0 +1,36 @@
import warnings
import cv2
import numpy as np
from PIL import Image
from modules.control.util import HWC3, resize_image
class CannyDetector:
def __call__(self, input_image=None, low_threshold=100, high_threshold=200, detect_resolution=512, image_resolution=512, output_type=None, **kwargs):
if "img" in kwargs:
warnings.warn("img is deprecated, please use `input_image=...` instead.", DeprecationWarning)
input_image = kwargs.pop("img")
if input_image is None:
raise ValueError("input_image must be defined.")
if not isinstance(input_image, np.ndarray):
input_image = np.array(input_image, dtype=np.uint8)
output_type = output_type or "pil"
else:
output_type = output_type or "np"
input_image = HWC3(input_image)
input_image = resize_image(input_image, detect_resolution)
detected_map = cv2.Canny(input_image, low_threshold, high_threshold)
detected_map = HWC3(detected_map)
img = resize_image(input_image, image_resolution)
H, W, _C = img.shape
detected_map = cv2.resize(detected_map, (W, H), interpolation=cv2.INTER_LINEAR)
if output_type == "pil":
detected_map = Image.fromarray(detected_map)
detected_map = detected_map.convert('L')
return detected_map
modules/control/proc/dwpose/__init__.py
+90
View File
@@ -0,0 +1,90 @@
# Openpose
# Original from CMU https://github.com/CMU-Perceptual-Computing-Lab/openpose
# 2nd Edited by https://github.com/Hzzone/pytorch-openpose
# 3rd Edited by ControlNet
# 4th Edited by ControlNet (added face and correct hands)
import os
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"
import cv2
import numpy as np
from PIL import Image
from modules.control.util import HWC3, resize_image
from .draw import draw_bodypose, draw_handpose, draw_facepose
def draw_pose(pose, H, W):
bodies = pose['bodies']
faces = pose['faces']
hands = pose['hands']
candidate = bodies['candidate']
subset = bodies['subset']
canvas = np.zeros(shape=(H, W, 3), dtype=np.uint8)
canvas = draw_bodypose(canvas, candidate, subset)
canvas = draw_handpose(canvas, hands)
canvas = draw_facepose(canvas, faces)
return canvas
class DWposeDetector:
def __init__(self, det_config=None, det_ckpt=None, pose_config=None, pose_ckpt=None, device="cpu"):
from .wholebody import Wholebody
self.pose_estimation = Wholebody(det_config, det_ckpt, pose_config, pose_ckpt, device)
def to(self, device):
self.pose_estimation.to(device)
return self
def __call__(self, input_image, detect_resolution=512, image_resolution=512, output_type="pil", min_confidence=0.3, **kwargs):
input_image = cv2.cvtColor(np.array(input_image, dtype=np.uint8), cv2.COLOR_RGB2BGR)
input_image = HWC3(input_image)
input_image = resize_image(input_image, detect_resolution)
H, W, _C = input_image.shape
candidate, subset = self.pose_estimation(input_image)
if candidate is None:
return Image.fromarray(input_image)
nums, _keys, locs = candidate.shape
candidate[..., 0] /= float(W)
candidate[..., 1] /= float(H)
body = candidate[:,:18].copy()
body = body.reshape(nums*18, locs)
score = subset[:,:18]
for i in range(len(score)):
for j in range(len(score[i])):
if score[i][j] > min_confidence:
score[i][j] = int(18*i+j)
else:
score[i][j] = -1
un_visible = subset < min_confidence
candidate[un_visible] = -1
_foot = candidate[:,18:24]
faces = candidate[:,24:92]
hands = candidate[:,92:113]
hands = np.vstack([hands, candidate[:,113:]])
bodies = dict(candidate=body, subset=score)
pose = dict(bodies=bodies, hands=hands, faces=faces)
detected_map = draw_pose(pose, H, W)
detected_map = HWC3(detected_map)
img = resize_image(input_image, image_resolution)
H, W, _C = img.shape
detected_map = cv2.resize(detected_map, (W, H), interpolation=cv2.INTER_LINEAR)
if output_type == "pil":
detected_map = Image.fromarray(detected_map)
return detected_map
modules/control/proc/dwpose/config/dwpose-l_384x288.py
+257
View File
@@ -0,0 +1,257 @@
# runtime
max_epochs = 270
stage2_num_epochs = 30
base_lr = 4e-3
train_cfg = dict(max_epochs=max_epochs, val_interval=10)
randomness = dict(seed=21)
# optimizer
optim_wrapper = dict(
type='OptimWrapper',
optimizer=dict(type='AdamW', lr=base_lr, weight_decay=0.05),
paramwise_cfg=dict(
norm_decay_mult=0, bias_decay_mult=0, bypass_duplicate=True))
# learning rate
param_scheduler = [
dict(
type='LinearLR',
start_factor=1.0e-5,
by_epoch=False,
begin=0,
end=1000),
dict(
# use cosine lr from 150 to 300 epoch
type='CosineAnnealingLR',
eta_min=base_lr * 0.05,
begin=max_epochs // 2,
end=max_epochs,
T_max=max_epochs // 2,
by_epoch=True,
convert_to_iter_based=True),
]
# automatically scaling LR based on the actual training batch size
auto_scale_lr = dict(base_batch_size=512)
# codec settings
codec = dict(
type='SimCCLabel',
input_size=(288, 384),
sigma=(6., 6.93),
simcc_split_ratio=2.0,
normalize=False,
use_dark=False)
# model settings
model = dict(
type='TopdownPoseEstimator',
data_preprocessor=dict(
type='PoseDataPreprocessor',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
bgr_to_rgb=True),
backbone=dict(
_scope_='mmdet',
type='CSPNeXt',
arch='P5',
expand_ratio=0.5,
deepen_factor=1.,
widen_factor=1.,
out_indices=(4, ),
channel_attention=True,
norm_cfg=dict(type='SyncBN'),
act_cfg=dict(type='SiLU'),
init_cfg=dict(
type='Pretrained',
prefix='backbone.',
checkpoint='https://download.openmmlab.com/mmpose/v1/projects/'
'rtmpose/cspnext-l_udp-aic-coco_210e-256x192-273b7631_20230130.pth'
)),
head=dict(
type='RTMCCHead',
in_channels=1024,
out_channels=133,
input_size=codec['input_size'],
in_featuremap_size=(9, 12),
simcc_split_ratio=codec['simcc_split_ratio'],
final_layer_kernel_size=7,
gau_cfg=dict(
hidden_dims=256,
s=128,
expansion_factor=2,
dropout_rate=0.,
drop_path=0.,
act_fn='SiLU',
use_rel_bias=False,
pos_enc=False),
loss=dict(
type='KLDiscretLoss',
use_target_weight=True,
beta=10.,
label_softmax=True),
decoder=codec),
test_cfg=dict(flip_test=True, ))
# base dataset settings
dataset_type = 'CocoWholeBodyDataset'
data_mode = 'topdown'
data_root = '/data/'
backend_args = dict(backend='local')
# backend_args = dict(
# backend='petrel',
# path_mapping=dict({
# f'{data_root}': 's3://openmmlab/datasets/detection/coco/',
# f'{data_root}': 's3://openmmlab/datasets/detection/coco/'
# }))
# pipelines
train_pipeline = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='RandomFlip', direction='horizontal'),
dict(type='RandomHalfBody'),
dict(
type='RandomBBoxTransform', scale_factor=[0.6, 1.4], rotate_factor=80),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='mmdet.YOLOXHSVRandomAug'),
dict(
type='Albumentation',
transforms=[
dict(type='Blur', p=0.1),
dict(type='MedianBlur', p=0.1),
dict(
type='CoarseDropout',
max_holes=1,
max_height=0.4,
max_width=0.4,
min_holes=1,
min_height=0.2,
min_width=0.2,
p=1.0),
]),
dict(type='GenerateTarget', encoder=codec),
dict(type='PackPoseInputs')
]
val_pipeline = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='PackPoseInputs')
]
train_pipeline_stage2 = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='RandomFlip', direction='horizontal'),
dict(type='RandomHalfBody'),
dict(
type='RandomBBoxTransform',
shift_factor=0.,
scale_factor=[0.75, 1.25],
rotate_factor=60),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='mmdet.YOLOXHSVRandomAug'),
dict(
type='Albumentation',
transforms=[
dict(type='Blur', p=0.1),
dict(type='MedianBlur', p=0.1),
dict(
type='CoarseDropout',
max_holes=1,
max_height=0.4,
max_width=0.4,
min_holes=1,
min_height=0.2,
min_width=0.2,
p=0.5),
]),
dict(type='GenerateTarget', encoder=codec),
dict(type='PackPoseInputs')
]
datasets = []
dataset_coco=dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='coco/annotations/coco_wholebody_train_v1.0.json',
data_prefix=dict(img='coco/train2017/'),
pipeline=[],
)
datasets.append(dataset_coco)
scene = ['Magic_show', 'Entertainment', 'ConductMusic', 'Online_class',
'TalkShow', 'Speech', 'Fitness', 'Interview', 'Olympic', 'TVShow',
'Singing', 'SignLanguage', 'Movie', 'LiveVlog', 'VideoConference']
for i in range(len(scene)):
datasets.append(
dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='UBody/annotations/'+scene[i]+'/keypoint_annotation.json',
data_prefix=dict(img='UBody/images/'+scene[i]+'/'),
pipeline=[],
)
)
# data loaders
train_dataloader = dict(
batch_size=32,
num_workers=10,
persistent_workers=True,
sampler=dict(type='DefaultSampler', shuffle=True),
dataset=dict(
type='CombinedDataset',
metainfo=dict(from_file='configs/_base_/datasets/coco_wholebody.py'),
datasets=datasets,
pipeline=train_pipeline,
test_mode=False,
))
val_dataloader = dict(
batch_size=32,
num_workers=10,
persistent_workers=True,
drop_last=False,
sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
dataset=dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='coco/annotations/coco_wholebody_val_v1.0.json',
bbox_file=f'{data_root}coco/person_detection_results/'
'COCO_val2017_detections_AP_H_56_person.json',
data_prefix=dict(img='coco/val2017/'),
test_mode=True,
pipeline=val_pipeline,
))
test_dataloader = val_dataloader
# hooks
default_hooks = dict(
checkpoint=dict(
save_best='coco-wholebody/AP', rule='greater', max_keep_ckpts=1))
custom_hooks = [
dict(
type='EMAHook',
ema_type='ExpMomentumEMA',
momentum=0.0002,
update_buffers=True,
priority=49),
dict(
type='mmdet.PipelineSwitchHook',
switch_epoch=max_epochs - stage2_num_epochs,
switch_pipeline=train_pipeline_stage2)
]
# evaluators
val_evaluator = dict(
type='CocoWholeBodyMetric',
ann_file=data_root + 'coco/annotations/coco_wholebody_val_v1.0.json')
test_evaluator = val_evaluator
modules/control/proc/dwpose/config/rtmpose-l_8xb32-270e_coco-ubody-wholebody-384x288.py
+259
View File
@@ -0,0 +1,259 @@
# _base_ = ['../../../_base_/default_runtime.py']
# runtime
max_epochs = 270
stage2_num_epochs = 30
base_lr = 4e-3
train_cfg = dict(max_epochs=max_epochs, val_interval=10)
randomness = dict(seed=21)
# optimizer
optim_wrapper = dict(
type='OptimWrapper',
optimizer=dict(type='AdamW', lr=base_lr, weight_decay=0.05),
paramwise_cfg=dict(
norm_decay_mult=0, bias_decay_mult=0, bypass_duplicate=True))
# learning rate
param_scheduler = [
dict(
type='LinearLR',
start_factor=1.0e-5,
by_epoch=False,
begin=0,
end=1000),
dict(
# use cosine lr from 150 to 300 epoch
type='CosineAnnealingLR',
eta_min=base_lr * 0.05,
begin=max_epochs // 2,
end=max_epochs,
T_max=max_epochs // 2,
by_epoch=True,
convert_to_iter_based=True),
]
# automatically scaling LR based on the actual training batch size
auto_scale_lr = dict(base_batch_size=512)
# codec settings
codec = dict(
type='SimCCLabel',
input_size=(288, 384),
sigma=(6., 6.93),
simcc_split_ratio=2.0,
normalize=False,
use_dark=False)
# model settings
model = dict(
type='TopdownPoseEstimator',
data_preprocessor=dict(
type='PoseDataPreprocessor',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
bgr_to_rgb=True),
backbone=dict(
_scope_='mmdet',
type='CSPNeXt',
arch='P5',
expand_ratio=0.5,
deepen_factor=1.,
widen_factor=1.,
out_indices=(4, ),
channel_attention=True,
norm_cfg=dict(type='SyncBN'),
act_cfg=dict(type='SiLU'),
init_cfg=dict(
type='Pretrained',
prefix='backbone.',
checkpoint='https://download.openmmlab.com/mmpose/v1/projects/'
'rtmpose/cspnext-l_udp-aic-coco_210e-256x192-273b7631_20230130.pth'
)),
head=dict(
type='RTMCCHead',
in_channels=1024,
out_channels=133,
input_size=codec['input_size'],
in_featuremap_size=(9, 12),
simcc_split_ratio=codec['simcc_split_ratio'],
final_layer_kernel_size=7,
gau_cfg=dict(
hidden_dims=256,
s=128,
expansion_factor=2,
dropout_rate=0.,
drop_path=0.,
act_fn='SiLU',
use_rel_bias=False,
pos_enc=False),
loss=dict(
type='KLDiscretLoss',
use_target_weight=True,
beta=10.,
label_softmax=True),
decoder=codec),
test_cfg=dict(flip_test=True, ))
# base dataset settings
dataset_type = 'CocoWholeBodyDataset'
data_mode = 'topdown'
data_root = 'data/'
backend_args = dict(backend='local')
# backend_args = dict(
# backend='petrel',
# path_mapping=dict({
# f'{data_root}': 's3://openmmlab/datasets/detection/coco/',
# f'{data_root}': 's3://openmmlab/datasets/detection/coco/'
# }))
# pipelines
train_pipeline = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='RandomFlip', direction='horizontal'),
dict(type='RandomHalfBody'),
dict(
type='RandomBBoxTransform', scale_factor=[0.6, 1.4], rotate_factor=80),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='mmdet.YOLOXHSVRandomAug'),
dict(
type='Albumentation',
transforms=[
dict(type='Blur', p=0.1),
dict(type='MedianBlur', p=0.1),
dict(
type='CoarseDropout',
max_holes=1,
max_height=0.4,
max_width=0.4,
min_holes=1,
min_height=0.2,
min_width=0.2,
p=1.0),
]),
dict(type='GenerateTarget', encoder=codec),
dict(type='PackPoseInputs')
]
val_pipeline = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='PackPoseInputs')
]
train_pipeline_stage2 = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='RandomFlip', direction='horizontal'),
dict(type='RandomHalfBody'),
dict(
type='RandomBBoxTransform',
shift_factor=0.,
scale_factor=[0.75, 1.25],
rotate_factor=60),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='mmdet.YOLOXHSVRandomAug'),
dict(
type='Albumentation',
transforms=[
dict(type='Blur', p=0.1),
dict(type='MedianBlur', p=0.1),
dict(
type='CoarseDropout',
max_holes=1,
max_height=0.4,
max_width=0.4,
min_holes=1,
min_height=0.2,
min_width=0.2,
p=0.5),
]),
dict(type='GenerateTarget', encoder=codec),
dict(type='PackPoseInputs')
]
datasets = []
dataset_coco=dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='coco/annotations/coco_wholebody_train_v1.0.json',
data_prefix=dict(img='coco/train2017/'),
pipeline=[],
)
datasets.append(dataset_coco)
scene = ['Magic_show', 'Entertainment', 'ConductMusic', 'Online_class',
'TalkShow', 'Speech', 'Fitness', 'Interview', 'Olympic', 'TVShow',
'Singing', 'SignLanguage', 'Movie', 'LiveVlog', 'VideoConference']
for i in range(len(scene)):
datasets.append(
dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='UBody/annotations/'+scene[i]+'/keypoint_annotation.json',
data_prefix=dict(img='UBody/images/'+scene[i]+'/'),
pipeline=[],
)
)
# data loaders
train_dataloader = dict(
batch_size=32,
num_workers=10,
persistent_workers=True,
sampler=dict(type='DefaultSampler', shuffle=True),
dataset=dict(
type='CombinedDataset',
metainfo=dict(from_file='configs/_base_/datasets/coco_wholebody.py'),
datasets=datasets,
pipeline=train_pipeline,
test_mode=False,
))
val_dataloader = dict(
batch_size=32,
num_workers=10,
persistent_workers=True,
drop_last=False,
sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
dataset=dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='coco/annotations/coco_wholebody_val_v1.0.json',
bbox_file=f'{data_root}coco/person_detection_results/'
'COCO_val2017_detections_AP_H_56_person.json',
data_prefix=dict(img='coco/val2017/'),
test_mode=True,
pipeline=val_pipeline,
))
test_dataloader = val_dataloader
# hooks
default_hooks = dict(
checkpoint=dict(
save_best='coco-wholebody/AP', rule='greater', max_keep_ckpts=1))
custom_hooks = [
dict(
type='EMAHook',
ema_type='ExpMomentumEMA',
momentum=0.0002,
update_buffers=True,
priority=49),
dict(
type='mmdet.PipelineSwitchHook',
switch_epoch=max_epochs - stage2_num_epochs,
switch_pipeline=train_pipeline_stage2)
]
# evaluators
val_evaluator = dict(
type='CocoWholeBodyMetric',
ann_file=data_root + 'coco/annotations/coco_wholebody_val_v1.0.json')
test_evaluator = val_evaluator
modules/control/proc/dwpose/config/rtmpose-m_8xb64-270e_coco-ubody-wholebody-256x192.py
+259
View File
@@ -0,0 +1,259 @@
# _base_ = ['../../../_base_/default_runtime.py']
# runtime
max_epochs = 270
stage2_num_epochs = 30
base_lr = 4e-3
train_cfg = dict(max_epochs=max_epochs, val_interval=10)
randomness = dict(seed=21)
# optimizer
optim_wrapper = dict(
type='OptimWrapper',
optimizer=dict(type='AdamW', lr=base_lr, weight_decay=0.05),
paramwise_cfg=dict(
norm_decay_mult=0, bias_decay_mult=0, bypass_duplicate=True))
# learning rate
param_scheduler = [
dict(
type='LinearLR',
start_factor=1.0e-5,
by_epoch=False,
begin=0,
end=1000),
dict(
# use cosine lr from 150 to 300 epoch
type='CosineAnnealingLR',
eta_min=base_lr * 0.05,
begin=max_epochs // 2,
end=max_epochs,
T_max=max_epochs // 2,
by_epoch=True,
convert_to_iter_based=True),
]
# automatically scaling LR based on the actual training batch size
auto_scale_lr = dict(base_batch_size=512)
# codec settings
codec = dict(
type='SimCCLabel',
input_size=(192, 256),
sigma=(4.9, 5.66),
simcc_split_ratio=2.0,
normalize=False,
use_dark=False)
# model settings
model = dict(
type='TopdownPoseEstimator',
data_preprocessor=dict(
type='PoseDataPreprocessor',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
bgr_to_rgb=True),
backbone=dict(
_scope_='mmdet',
type='CSPNeXt',
arch='P5',
expand_ratio=0.5,
deepen_factor=0.67,
widen_factor=0.75,
out_indices=(4, ),
channel_attention=True,
norm_cfg=dict(type='SyncBN'),
act_cfg=dict(type='SiLU'),
init_cfg=dict(
type='Pretrained',
prefix='backbone.',
checkpoint='https://download.openmmlab.com/mmpose/v1/projects/'
'rtmpose/cspnext-m_udp-aic-coco_210e-256x192-f2f7d6f6_20230130.pth'
)),
head=dict(
type='RTMCCHead',
in_channels=768,
out_channels=133,
input_size=codec['input_size'],
in_featuremap_size=(6, 8),
simcc_split_ratio=codec['simcc_split_ratio'],
final_layer_kernel_size=7,
gau_cfg=dict(
hidden_dims=256,
s=128,
expansion_factor=2,
dropout_rate=0.,
drop_path=0.,
act_fn='SiLU',
use_rel_bias=False,
pos_enc=False),
loss=dict(
type='KLDiscretLoss',
use_target_weight=True,
beta=10.,
label_softmax=True),
decoder=codec),
test_cfg=dict(flip_test=True, ))
# base dataset settings
dataset_type = 'CocoWholeBodyDataset'
data_mode = 'topdown'
data_root = 'data/'
backend_args = dict(backend='local')
# backend_args = dict(
# backend='petrel',
# path_mapping=dict({
# f'{data_root}': 's3://openmmlab/datasets/detection/coco/',
# f'{data_root}': 's3://openmmlab/datasets/detection/coco/'
# }))
# pipelines
train_pipeline = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='RandomFlip', direction='horizontal'),
dict(type='RandomHalfBody'),
dict(
type='RandomBBoxTransform', scale_factor=[0.6, 1.4], rotate_factor=80),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='mmdet.YOLOXHSVRandomAug'),
dict(
type='Albumentation',
transforms=[
dict(type='Blur', p=0.1),
dict(type='MedianBlur', p=0.1),
dict(
type='CoarseDropout',
max_holes=1,
max_height=0.4,
max_width=0.4,
min_holes=1,
min_height=0.2,
min_width=0.2,
p=1.0),
]),
dict(type='GenerateTarget', encoder=codec),
dict(type='PackPoseInputs')
]
val_pipeline = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='PackPoseInputs')
]
train_pipeline_stage2 = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='RandomFlip', direction='horizontal'),
dict(type='RandomHalfBody'),
dict(
type='RandomBBoxTransform',
shift_factor=0.,
scale_factor=[0.75, 1.25],
rotate_factor=60),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='mmdet.YOLOXHSVRandomAug'),
dict(
type='Albumentation',
transforms=[
dict(type='Blur', p=0.1),
dict(type='MedianBlur', p=0.1),
dict(
type='CoarseDropout',
max_holes=1,
max_height=0.4,
max_width=0.4,
min_holes=1,
min_height=0.2,
min_width=0.2,
p=0.5),
]),
dict(type='GenerateTarget', encoder=codec),
dict(type='PackPoseInputs')
]
datasets = []
dataset_coco=dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='coco/annotations/coco_wholebody_train_v1.0.json',
data_prefix=dict(img='coco/train2017/'),
pipeline=[],
)
datasets.append(dataset_coco)
scene = ['Magic_show', 'Entertainment', 'ConductMusic', 'Online_class',
'TalkShow', 'Speech', 'Fitness', 'Interview', 'Olympic', 'TVShow',
'Singing', 'SignLanguage', 'Movie', 'LiveVlog', 'VideoConference']
for i in range(len(scene)):
datasets.append(
dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='UBody/annotations/'+scene[i]+'/keypoint_annotation.json',
data_prefix=dict(img='UBody/images/'+scene[i]+'/'),
pipeline=[],
)
)
# data loaders
train_dataloader = dict(
batch_size=64,
num_workers=10,
persistent_workers=True,
sampler=dict(type='DefaultSampler', shuffle=True),
dataset=dict(
type='CombinedDataset',
metainfo=dict(from_file='configs/_base_/datasets/coco_wholebody.py'),
datasets=datasets,
pipeline=train_pipeline,
test_mode=False,
))
val_dataloader = dict(
batch_size=32,
num_workers=10,
persistent_workers=True,
drop_last=False,
sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
dataset=dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='coco/annotations/coco_wholebody_val_v1.0.json',
bbox_file=f'{data_root}coco/person_detection_results/'
'COCO_val2017_detections_AP_H_56_person.json',
data_prefix=dict(img='coco/val2017/'),
test_mode=True,
pipeline=val_pipeline,
))
test_dataloader = val_dataloader
# hooks
default_hooks = dict(
checkpoint=dict(
save_best='coco-wholebody/AP', rule='greater', max_keep_ckpts=1))
custom_hooks = [
dict(
type='EMAHook',
ema_type='ExpMomentumEMA',
momentum=0.0002,
update_buffers=True,
priority=49),
dict(
type='mmdet.PipelineSwitchHook',
switch_epoch=max_epochs - stage2_num_epochs,
switch_pipeline=train_pipeline_stage2)
]
# evaluators
val_evaluator = dict(
type='CocoWholeBodyMetric',
ann_file=data_root + 'coco/annotations/coco_wholebody_val_v1.0.json')
test_evaluator = val_evaluator
modules/control/proc/dwpose/config/rtmpose-t_8xb64-270e_coco-ubody-wholebody-256x192.py
+259
View File
@@ -0,0 +1,259 @@
# _base_ = ['../../../_base_/default_runtime.py']
# runtime
max_epochs = 270
stage2_num_epochs = 30
base_lr = 4e-3
train_cfg = dict(max_epochs=max_epochs, val_interval=10)
randomness = dict(seed=21)
# optimizer
optim_wrapper = dict(
type='OptimWrapper',
optimizer=dict(type='AdamW', lr=base_lr, weight_decay=0.05),
paramwise_cfg=dict(
norm_decay_mult=0, bias_decay_mult=0, bypass_duplicate=True))
# learning rate
param_scheduler = [
dict(
type='LinearLR',
start_factor=1.0e-5,
by_epoch=False,
begin=0,
end=1000),
dict(
# use cosine lr from 150 to 300 epoch
type='CosineAnnealingLR',
eta_min=base_lr * 0.05,
begin=max_epochs // 2,
end=max_epochs,
T_max=max_epochs // 2,
by_epoch=True,
convert_to_iter_based=True),
]
# automatically scaling LR based on the actual training batch size
auto_scale_lr = dict(base_batch_size=512)
# codec settings
codec = dict(
type='SimCCLabel',
input_size=(192, 256),
sigma=(4.9, 5.66),
simcc_split_ratio=2.0,
normalize=False,
use_dark=False)
# model settings
model = dict(
type='TopdownPoseEstimator',
data_preprocessor=dict(
type='PoseDataPreprocessor',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
bgr_to_rgb=True),
backbone=dict(
_scope_='mmdet',
type='CSPNeXt',
arch='P5',
expand_ratio=0.5,
deepen_factor=0.167,
widen_factor=0.375,
out_indices=(4, ),
channel_attention=True,
norm_cfg=dict(type='SyncBN'),
act_cfg=dict(type='SiLU'),
init_cfg=dict(
type='Pretrained',
prefix='backbone.',
checkpoint='https://download.openmmlab.com/mmpose/v1/projects/'
'rtmpose/cspnext-tiny_udp-aic-coco_210e-256x192-cbed682d_20230130.pth'
)),
head=dict(
type='RTMCCHead',
in_channels=384,
out_channels=133,
input_size=codec['input_size'],
in_featuremap_size=(6, 8),
simcc_split_ratio=codec['simcc_split_ratio'],
final_layer_kernel_size=7,
gau_cfg=dict(
hidden_dims=256,
s=128,
expansion_factor=2,
dropout_rate=0.,
drop_path=0.,
act_fn='SiLU',
use_rel_bias=False,
pos_enc=False),
loss=dict(
type='KLDiscretLoss',
use_target_weight=True,
beta=10.,
label_softmax=True),
decoder=codec),
test_cfg=dict(flip_test=True, ))
# base dataset settings
dataset_type = 'CocoWholeBodyDataset'
data_mode = 'topdown'
data_root = 'data/'
backend_args = dict(backend='local')
# backend_args = dict(
# backend='petrel',
# path_mapping=dict({
# f'{data_root}': 's3://openmmlab/datasets/detection/coco/',
# f'{data_root}': 's3://openmmlab/datasets/detection/coco/'
# }))
# pipelines
train_pipeline = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='RandomFlip', direction='horizontal'),
dict(type='RandomHalfBody'),
dict(
type='RandomBBoxTransform', scale_factor=[0.6, 1.4], rotate_factor=80),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='mmdet.YOLOXHSVRandomAug'),
dict(
type='Albumentation',
transforms=[
dict(type='Blur', p=0.1),
dict(type='MedianBlur', p=0.1),
dict(
type='CoarseDropout',
max_holes=1,
max_height=0.4,
max_width=0.4,
min_holes=1,
min_height=0.2,
min_width=0.2,
p=1.0),
]),
dict(type='GenerateTarget', encoder=codec),
dict(type='PackPoseInputs')
]
val_pipeline = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='PackPoseInputs')
]
train_pipeline_stage2 = [
dict(type='LoadImage', backend_args=backend_args),
dict(type='GetBBoxCenterScale'),
dict(type='RandomFlip', direction='horizontal'),
dict(type='RandomHalfBody'),
dict(
type='RandomBBoxTransform',
shift_factor=0.,
scale_factor=[0.75, 1.25],
rotate_factor=60),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='mmdet.YOLOXHSVRandomAug'),
dict(
type='Albumentation',
transforms=[
dict(type='Blur', p=0.1),
dict(type='MedianBlur', p=0.1),
dict(
type='CoarseDropout',
max_holes=1,
max_height=0.4,
max_width=0.4,
min_holes=1,
min_height=0.2,
min_width=0.2,
p=0.5),
]),
dict(type='GenerateTarget', encoder=codec),
dict(type='PackPoseInputs')
]
datasets = []
dataset_coco=dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='coco/annotations/coco_wholebody_train_v1.0.json',
data_prefix=dict(img='coco/train2017/'),
pipeline=[],
)
datasets.append(dataset_coco)
scene = ['Magic_show', 'Entertainment', 'ConductMusic', 'Online_class',
'TalkShow', 'Speech', 'Fitness', 'Interview', 'Olympic', 'TVShow',
'Singing', 'SignLanguage', 'Movie', 'LiveVlog', 'VideoConference']
for i in range(len(scene)):
datasets.append(
dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='UBody/annotations/'+scene[i]+'/keypoint_annotation.json',
data_prefix=dict(img='UBody/images/'+scene[i]+'/'),
pipeline=[],
)
)
# data loaders
train_dataloader = dict(
batch_size=64,
num_workers=10,
persistent_workers=True,
sampler=dict(type='DefaultSampler', shuffle=True),
dataset=dict(
type='CombinedDataset',
metainfo=dict(from_file='configs/_base_/datasets/coco_wholebody.py'),
datasets=datasets,
pipeline=train_pipeline,
test_mode=False,
))
val_dataloader = dict(
batch_size=32,
num_workers=10,
persistent_workers=True,
drop_last=False,
sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
dataset=dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='coco/annotations/coco_wholebody_val_v1.0.json',
bbox_file=f'{data_root}coco/person_detection_results/'
'COCO_val2017_detections_AP_H_56_person.json',
data_prefix=dict(img='coco/val2017/'),
test_mode=True,
pipeline=val_pipeline,
))
test_dataloader = val_dataloader
# hooks
default_hooks = dict(
checkpoint=dict(
save_best='coco-wholebody/AP', rule='greater', max_keep_ckpts=1))
custom_hooks = [
dict(
type='EMAHook',
ema_type='ExpMomentumEMA',
momentum=0.0002,
update_buffers=True,
priority=49),
dict(
type='mmdet.PipelineSwitchHook',
switch_epoch=max_epochs - stage2_num_epochs,
switch_pipeline=train_pipeline_stage2)
]
# evaluators
val_evaluator = dict(
type='CocoWholeBodyMetric',
ann_file=data_root + 'coco/annotations/coco_wholebody_val_v1.0.json')
test_evaluator = val_evaluator
modules/control/proc/dwpose/config/yolox_l_8xb8-300e_coco.py
+245
View File
@@ -0,0 +1,245 @@
img_scale = (640, 640) # width, height
# model settings
model = dict(
type='YOLOX',
data_preprocessor=dict(
type='DetDataPreprocessor',
pad_size_divisor=32,
batch_augments=[
dict(
type='BatchSyncRandomResize',
random_size_range=(480, 800),
size_divisor=32,
interval=10)
]),
backbone=dict(
type='CSPDarknet',
deepen_factor=1.0,
widen_factor=1.0,
out_indices=(2, 3, 4),
use_depthwise=False,
spp_kernal_sizes=(5, 9, 13),
norm_cfg=dict(type='BN', momentum=0.03, eps=0.001),
act_cfg=dict(type='Swish'),
),
neck=dict(
type='YOLOXPAFPN',
in_channels=[256, 512, 1024],
out_channels=256,
num_csp_blocks=3,
use_depthwise=False,
upsample_cfg=dict(scale_factor=2, mode='nearest'),
norm_cfg=dict(type='BN', momentum=0.03, eps=0.001),
act_cfg=dict(type='Swish')),
bbox_head=dict(
type='YOLOXHead',
num_classes=80,
in_channels=256,
feat_channels=256,
stacked_convs=2,
strides=(8, 16, 32),
use_depthwise=False,
norm_cfg=dict(type='BN', momentum=0.03, eps=0.001),
act_cfg=dict(type='Swish'),
loss_cls=dict(
type='CrossEntropyLoss',
use_sigmoid=True,
reduction='sum',
loss_weight=1.0),
loss_bbox=dict(
type='IoULoss',
mode='square',
eps=1e-16,
reduction='sum',
loss_weight=5.0),
loss_obj=dict(
type='CrossEntropyLoss',
use_sigmoid=True,
reduction='sum',
loss_weight=1.0),
loss_l1=dict(type='L1Loss', reduction='sum', loss_weight=1.0)),
train_cfg=dict(assigner=dict(type='SimOTAAssigner', center_radius=2.5)),
# In order to align the source code, the threshold of the val phase is
# 0.01, and the threshold of the test phase is 0.001.
test_cfg=dict(score_thr=0.01, nms=dict(type='nms', iou_threshold=0.65)))
# dataset settings
data_root = 'data/coco/'
dataset_type = 'CocoDataset'
# Example to use different file client
# Method 1: simply set the data root and let the file I/O module
# automatically infer from prefix (not support LMDB and Memcache yet)
# data_root = 's3://openmmlab/datasets/detection/coco/'
# Method 2: Use `backend_args`, `file_client_args` in versions before 3.0.0rc6
# backend_args = dict(
# backend='petrel',
# path_mapping=dict({
# './data/': 's3://openmmlab/datasets/detection/',
# 'data/': 's3://openmmlab/datasets/detection/'
# }))
backend_args = None
train_pipeline = [
dict(type='Mosaic', img_scale=img_scale, pad_val=114.0),
dict(
type='RandomAffine',
scaling_ratio_range=(0.1, 2),
# img_scale is (width, height)
border=(-img_scale[0] // 2, -img_scale[1] // 2)),
dict(
type='MixUp',
img_scale=img_scale,
ratio_range=(0.8, 1.6),
pad_val=114.0),
dict(type='YOLOXHSVRandomAug'),
dict(type='RandomFlip', prob=0.5),
# According to the official implementation, multi-scale
# training is not considered here but in the
# 'mmdet/models/detectors/yolox.py'.
# Resize and Pad are for the last 15 epochs when Mosaic,
# RandomAffine, and MixUp are closed by YOLOXModeSwitchHook.
dict(type='Resize', scale=img_scale, keep_ratio=True),
dict(
type='Pad',
pad_to_square=True,
# If the image is three-channel, the pad value needs
# to be set separately for each channel.
pad_val=dict(img=(114.0, 114.0, 114.0))),
dict(type='FilterAnnotations', min_gt_bbox_wh=(1, 1), keep_empty=False),
dict(type='PackDetInputs')
]
train_dataset = dict(
# use MultiImageMixDataset wrapper to support mosaic and mixup
type='MultiImageMixDataset',
dataset=dict(
type=dataset_type,
data_root=data_root,
ann_file='annotations/instances_train2017.json',
data_prefix=dict(img='train2017/'),
pipeline=[
dict(type='LoadImageFromFile', backend_args=backend_args),
dict(type='LoadAnnotations', with_bbox=True)
],
filter_cfg=dict(filter_empty_gt=False, min_size=32),
backend_args=backend_args),
pipeline=train_pipeline)
test_pipeline = [
dict(type='LoadImageFromFile', backend_args=backend_args),
dict(type='Resize', scale=img_scale, keep_ratio=True),
dict(
type='Pad',
pad_to_square=True,
pad_val=dict(img=(114.0, 114.0, 114.0))),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='PackDetInputs',
meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
'scale_factor'))
]
train_dataloader = dict(
batch_size=8,
num_workers=4,
persistent_workers=True,
sampler=dict(type='DefaultSampler', shuffle=True),
dataset=train_dataset)
val_dataloader = dict(
batch_size=8,
num_workers=4,
persistent_workers=True,
drop_last=False,
sampler=dict(type='DefaultSampler', shuffle=False),
dataset=dict(
type=dataset_type,
data_root=data_root,
ann_file='annotations/instances_val2017.json',
data_prefix=dict(img='val2017/'),
test_mode=True,
pipeline=test_pipeline,
backend_args=backend_args))
test_dataloader = val_dataloader
val_evaluator = dict(
type='CocoMetric',
ann_file=data_root + 'annotations/instances_val2017.json',
metric='bbox',
backend_args=backend_args)
test_evaluator = val_evaluator
# training settings
max_epochs = 300
num_last_epochs = 15
interval = 10
train_cfg = dict(max_epochs=max_epochs, val_interval=interval)
# optimizer
# default 8 gpu
base_lr = 0.01
optim_wrapper = dict(
type='OptimWrapper',
optimizer=dict(
type='SGD', lr=base_lr, momentum=0.9, weight_decay=5e-4,
nesterov=True),
paramwise_cfg=dict(norm_decay_mult=0., bias_decay_mult=0.))
# learning rate
param_scheduler = [
dict(
# use quadratic formula to warm up 5 epochs
# and lr is updated by iteration
# TODO: fix default scope in get function
type='mmdet.QuadraticWarmupLR',
by_epoch=True,
begin=0,
end=5,
convert_to_iter_based=True),
dict(
# use cosine lr from 5 to 285 epoch
type='CosineAnnealingLR',
eta_min=base_lr * 0.05,
begin=5,
T_max=max_epochs - num_last_epochs,
end=max_epochs - num_last_epochs,
by_epoch=True,
convert_to_iter_based=True),
dict(
# use fixed lr during last 15 epochs
type='ConstantLR',
by_epoch=True,
factor=1,
begin=max_epochs - num_last_epochs,
end=max_epochs,
)
]
default_hooks = dict(
checkpoint=dict(
interval=interval,
max_keep_ckpts=3 # only keep latest 3 checkpoints
))
custom_hooks = [
dict(
type='YOLOXModeSwitchHook',
num_last_epochs=num_last_epochs,
priority=48),
dict(type='SyncNormHook', priority=48),
dict(
type='EMAHook',
ema_type='ExpMomentumEMA',
momentum=0.0001,
update_buffers=True,
priority=49)
]
# NOTE: `auto_scale_lr` is for automatically scaling LR,
# USER SHOULD NOT CHANGE ITS VALUES.
# base_batch_size = (8 GPUs) x (8 samples per GPU)
auto_scale_lr = dict(base_batch_size=64)
modules/control/proc/dwpose/draw.py
+307
View File
@@ -0,0 +1,307 @@
import math
import numpy as np
import cv2
eps = 0.01
def smart_resize(x, s):
Ht, Wt = s
if x.ndim == 2:
Ho, Wo = x.shape
Co = 1
else:
Ho, Wo, Co = x.shape
if Co == 3 or Co == 1:
k = float(Ht + Wt) / float(Ho + Wo)
return cv2.resize(x, (int(Wt), int(Ht)), interpolation=cv2.INTER_AREA if k < 1 else cv2.INTER_LANCZOS4)
else:
return np.stack([smart_resize(x[:, :, i], s) for i in range(Co)], axis=2)
def smart_resize_k(x, fx, fy):
if x.ndim == 2:
Ho, Wo = x.shape
Co = 1
else:
Ho, Wo, Co = x.shape
Ht, Wt = Ho * fy, Wo * fx
if Co == 3 or Co == 1:
k = float(Ht + Wt) / float(Ho + Wo)
return cv2.resize(x, (int(Wt), int(Ht)), interpolation=cv2.INTER_AREA if k < 1 else cv2.INTER_LANCZOS4)
else:
return np.stack([smart_resize_k(x[:, :, i], fx, fy) for i in range(Co)], axis=2)
def padRightDownCorner(img, stride, padValue):
h = img.shape[0]
w = img.shape[1]
pad = 4 * [None]
pad[0] = 0 # up
pad[1] = 0 # left
pad[2] = 0 if (h % stride == 0) else stride - (h % stride) # down
pad[3] = 0 if (w % stride == 0) else stride - (w % stride) # right
img_padded = img
pad_up = np.tile(img_padded[0:1, :, :]*0 + padValue, (pad[0], 1, 1))
img_padded = np.concatenate((pad_up, img_padded), axis=0)
pad_left = np.tile(img_padded[:, 0:1, :]*0 + padValue, (1, pad[1], 1))
img_padded = np.concatenate((pad_left, img_padded), axis=1)
pad_down = np.tile(img_padded[-2:-1, :, :]*0 + padValue, (pad[2], 1, 1))
img_padded = np.concatenate((img_padded, pad_down), axis=0)
pad_right = np.tile(img_padded[:, -2:-1, :]*0 + padValue, (1, pad[3], 1))
img_padded = np.concatenate((img_padded, pad_right), axis=1)
return img_padded, pad
def transfer(model, model_weights):
transfered_model_weights = {}
for weights_name in model.state_dict().keys():
transfered_model_weights[weights_name] = model_weights['.'.join(weights_name.split('.')[1:])]
return transfered_model_weights
def draw_bodypose(canvas, candidate, subset):
H, W, _C = canvas.shape
candidate = np.array(candidate)
subset = np.array(subset)
stickwidth = 4
limbSeq = [[2, 3], [2, 6], [3, 4], [4, 5], [6, 7], [7, 8], [2, 9], [9, 10], \
[10, 11], [2, 12], [12, 13], [13, 14], [2, 1], [1, 15], [15, 17], \
[1, 16], [16, 18], [3, 17], [6, 18]]
colors = [[255, 0, 0], [255, 85, 0], [255, 170, 0], [255, 255, 0], [170, 255, 0], [85, 255, 0], [0, 255, 0], \
[0, 255, 85], [0, 255, 170], [0, 255, 255], [0, 170, 255], [0, 85, 255], [0, 0, 255], [85, 0, 255], \
[170, 0, 255], [255, 0, 255], [255, 0, 170], [255, 0, 85]]
for i in range(17):
for n in range(len(subset)):
index = subset[n][np.array(limbSeq[i]) - 1]
if -1 in index:
continue
Y = candidate[index.astype(int), 0] * float(W)
X = candidate[index.astype(int), 1] * float(H)
mX = np.mean(X)
mY = np.mean(Y)
length = ((X[0] - X[1]) ** 2 + (Y[0] - Y[1]) ** 2) ** 0.5
angle = math.degrees(math.atan2(X[0] - X[1], Y[0] - Y[1]))
polygon = cv2.ellipse2Poly((int(mY), int(mX)), (int(length / 2), stickwidth), int(angle), 0, 360, 1)
cv2.fillConvexPoly(canvas, polygon, colors[i])
canvas = (canvas * 0.6).astype(np.uint8)
for i in range(18):
for n in range(len(subset)):
index = int(subset[n][i])
if index == -1:
continue
x, y = candidate[index][0:2]
x = int(x * W)
y = int(y * H)
cv2.circle(canvas, (int(x), int(y)), 4, colors[i], thickness=-1)
return canvas
def draw_handpose(canvas, all_hand_peaks):
import matplotlib as mpl
H, W, _C = canvas.shape
edges = [[0, 1], [1, 2], [2, 3], [3, 4], [0, 5], [5, 6], [6, 7], [7, 8], [0, 9], [9, 10], \
[10, 11], [11, 12], [0, 13], [13, 14], [14, 15], [15, 16], [0, 17], [17, 18], [18, 19], [19, 20]]
# (person_number*2, 21, 2)
for i in range(len(all_hand_peaks)):
peaks = all_hand_peaks[i]
peaks = np.array(peaks)
for ie, e in enumerate(edges):
x1, y1 = peaks[e[0]]
x2, y2 = peaks[e[1]]
x1 = int(x1 * W)
y1 = int(y1 * H)
x2 = int(x2 * W)
y2 = int(y2 * H)
if x1 > eps and y1 > eps and x2 > eps and y2 > eps:
cv2.line(canvas, (x1, y1), (x2, y2), mpl.colors.hsv_to_rgb([ie / float(len(edges)), 1.0, 1.0]) * 255, thickness=2)
for _, keyponit in enumerate(peaks):
x, y = keyponit
x = int(x * W)
y = int(y * H)
if x > eps and y > eps:
cv2.circle(canvas, (x, y), 4, (0, 0, 255), thickness=-1)
return canvas
def draw_facepose(canvas, all_lmks):
H, W, _C = canvas.shape
for lmks in all_lmks:
lmks = np.array(lmks)
for lmk in lmks:
x, y = lmk
x = int(x * W)
y = int(y * H)
if x > eps and y > eps:
cv2.circle(canvas, (x, y), 3, (255, 255, 255), thickness=-1)
return canvas
# detect hand according to body pose keypoints
# please refer to https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/src/openpose/hand/handDetector.cpp
def handDetect(candidate, subset, oriImg):
# right hand: wrist 4, elbow 3, shoulder 2
# left hand: wrist 7, elbow 6, shoulder 5
ratioWristElbow = 0.33
detect_result = []
image_height, image_width = oriImg.shape[0:2]
for person in subset.astype(int):
# if any of three not detected
has_left = np.sum(person[[5, 6, 7]] == -1) == 0
has_right = np.sum(person[[2, 3, 4]] == -1) == 0
if not (has_left or has_right):
continue
hands = []
#left hand
if has_left:
left_shoulder_index, left_elbow_index, left_wrist_index = person[[5, 6, 7]]
x1, y1 = candidate[left_shoulder_index][:2]
x2, y2 = candidate[left_elbow_index][:2]
x3, y3 = candidate[left_wrist_index][:2]
hands.append([x1, y1, x2, y2, x3, y3, True])
# right hand
if has_right:
right_shoulder_index, right_elbow_index, right_wrist_index = person[[2, 3, 4]]
x1, y1 = candidate[right_shoulder_index][:2]
x2, y2 = candidate[right_elbow_index][:2]
x3, y3 = candidate[right_wrist_index][:2]
hands.append([x1, y1, x2, y2, x3, y3, False])
for x1, y1, x2, y2, x3, y3, is_left in hands:
# pos_hand = pos_wrist + ratio * (pos_wrist - pos_elbox) = (1 + ratio) * pos_wrist - ratio * pos_elbox
# handRectangle.x = posePtr[wrist*3] + ratioWristElbow * (posePtr[wrist*3] - posePtr[elbow*3]);
# handRectangle.y = posePtr[wrist*3+1] + ratioWristElbow * (posePtr[wrist*3+1] - posePtr[elbow*3+1]);
# const auto distanceWristElbow = getDistance(poseKeypoints, person, wrist, elbow);
# const auto distanceElbowShoulder = getDistance(poseKeypoints, person, elbow, shoulder);
# handRectangle.width = 1.5f * fastMax(distanceWristElbow, 0.9f * distanceElbowShoulder);
x = x3 + ratioWristElbow * (x3 - x2)
y = y3 + ratioWristElbow * (y3 - y2)
distanceWristElbow = math.sqrt((x3 - x2) ** 2 + (y3 - y2) ** 2)
distanceElbowShoulder = math.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2)
width = 1.5 * max(distanceWristElbow, 0.9 * distanceElbowShoulder)
# x-y refers to the center --> offset to topLeft point
# handRectangle.x -= handRectangle.width / 2.f;
# handRectangle.y -= handRectangle.height / 2.f;
x -= width / 2
y -= width / 2 # width = height
# overflow the image
if x < 0:
x = 0
if y < 0:
y = 0
width1 = width
width2 = width
if x + width > image_width:
width1 = image_width - x
if y + width > image_height:
width2 = image_height - y
width = min(width1, width2)
# the max hand box value is 20 pixels
if width >= 20:
detect_result.append([int(x), int(y), int(width), is_left])
'''
return value: [[x, y, w, True if left hand else False]].
width=height since the network require squared input.
x, y is the coordinate of top left
'''
return detect_result
# Written by Lvmin
def faceDetect(candidate, subset, oriImg):
# left right eye ear 14 15 16 17
detect_result = []
image_height, image_width = oriImg.shape[0:2]
for person in subset.astype(int):
has_head = person[0] > -1
if not has_head:
continue
has_left_eye = person[14] > -1
has_right_eye = person[15] > -1
has_left_ear = person[16] > -1
has_right_ear = person[17] > -1
if not (has_left_eye or has_right_eye or has_left_ear or has_right_ear):
continue
head, left_eye, right_eye, left_ear, right_ear = person[[0, 14, 15, 16, 17]]
width = 0.0
x0, y0 = candidate[head][:2]
if has_left_eye:
x1, y1 = candidate[left_eye][:2]
d = max(abs(x0 - x1), abs(y0 - y1))
width = max(width, d * 3.0)
if has_right_eye:
x1, y1 = candidate[right_eye][:2]
d = max(abs(x0 - x1), abs(y0 - y1))
width = max(width, d * 3.0)
if has_left_ear:
x1, y1 = candidate[left_ear][:2]
d = max(abs(x0 - x1), abs(y0 - y1))
width = max(width, d * 1.5)
if has_right_ear:
x1, y1 = candidate[right_ear][:2]
d = max(abs(x0 - x1), abs(y0 - y1))
width = max(width, d * 1.5)
x, y = x0, y0
x -= width
y -= width
if x < 0:
x = 0
if y < 0:
y = 0
width1 = width * 2
width2 = width * 2
if x + width > image_width:
width1 = image_width - x
if y + width > image_height:
width2 = image_height - y
width = min(width1, width2)
if width >= 20:
detect_result.append([int(x), int(y), int(width)])
return detect_result
# get max index of 2d array
def npmax(array):
arrayindex = array.argmax(1)
arrayvalue = array.max(1)
i = arrayvalue.argmax()
j = arrayindex[i]
return i, j
modules/control/proc/dwpose/wholebody.py
+111
View File
@@ -0,0 +1,111 @@
# Copyright (c) OpenMMLab. All rights reserved.
import os
import numpy as np
from modules.shared import log
mmok = True
try:
import mmcv # pylint: disable=unused-import
except ImportError as e:
mmok = False
log.error(f"Control processor DWPose: {e}")
try:
from mmpose.apis import inference_topdown
from mmpose.apis import init_model as init_pose_estimator
from mmpose.evaluation.functional import nms
from mmpose.utils import adapt_mmdet_pipeline
from mmpose.structures import merge_data_samples
except ImportError as e:
mmok = False
log.error(f"Control processor DWPose: {e}")
try:
from mmdet.apis import inference_detector, init_detector
except ImportError as e:
mmok = False
log.error(f"Control processor DWPose: {e}")
def inference_detector(*args, **kwargs):
return lambda *args, **kwargs: None
if not mmok:
log.error('Control processor DWPose: OpenMMLab is not installed')
class Wholebody:
def __init__(self, det_config=None, det_ckpt=None, pose_config=None, pose_ckpt=None, device="cpu"):
if not mmok:
self.detector = lambda *args, **kwargs: None
return None
prefix = os.path.dirname(__file__)
if det_config is None:
det_config = "config/yolox_l_8xb8-300e_coco.py"
if pose_config is None:
pose_config = "config/dwpose-l_384x288.py"
if not det_config.startswith('prefix'):
det_config = os.path.join(prefix, det_config)
if not pose_config.startswith('prefix'):
pose_config = os.path.join(prefix, pose_config)
if det_ckpt is None:
det_ckpt = 'https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_l_8x8_300e_coco/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth'
if pose_ckpt is None:
pose_ckpt = "https://huggingface.co/wanghaofan/dw-ll_ucoco_384/resolve/main/dw-ll_ucoco_384.pth"
# build detector
self.detector = init_detector(det_config, det_ckpt, device=device)
self.detector.cfg = adapt_mmdet_pipeline(self.detector.cfg)
# build pose estimator
self.pose_estimator = init_pose_estimator(
pose_config,
pose_ckpt,
device=device)
def to(self, device):
self.detector.to(device)
self.pose_estimator.to(device)
return self
def __call__(self, oriImg):
if not mmok:
return None, None
# predict bbox
det_result = inference_detector(self.detector, oriImg)
pred_instance = det_result.pred_instances.cpu().numpy()
bboxes = np.concatenate((pred_instance.bboxes, pred_instance.scores[:, None]), axis=1)
bboxes = bboxes[np.logical_and(pred_instance.labels == 0, pred_instance.scores > 0.5)]
# set NMS threshold
bboxes = bboxes[nms(bboxes, 0.7), :4]
# predict keypoints
if len(bboxes) == 0:
pose_results = inference_topdown(self.pose_estimator, oriImg)
else:
pose_results = inference_topdown(self.pose_estimator, oriImg, bboxes)
preds = merge_data_samples(pose_results)
preds = preds.pred_instances
# preds = pose_results[0].pred_instances
keypoints = preds.get('transformed_keypoints', preds.keypoints)
if 'keypoint_scores' in preds:
scores = preds.keypoint_scores
else:
scores = np.ones(keypoints.shape[:-1])
if 'keypoints_visible' in preds:
visible = preds.keypoints_visible
else:
visible = np.ones(keypoints.shape[:-1])
keypoints_info = np.concatenate(
(keypoints, scores[..., None], visible[..., None]),
axis=-1)
# compute neck joint
neck = np.mean(keypoints_info[:, [5, 6]], axis=1)
# neck score when visualizing pred
neck[:, 2:4] = np.logical_and(
keypoints_info[:, 5, 2:4] > 0.3,
keypoints_info[:, 6, 2:4] > 0.3).astype(int)
new_keypoints_info = np.insert(
keypoints_info, 17, neck, axis=1)
mmpose_idx = [17, 6, 8, 10, 7, 9, 12, 14, 16, 13, 15, 2, 1, 4, 3]
openpose_idx = [1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17]
new_keypoints_info[:, openpose_idx] = new_keypoints_info[:, mmpose_idx]
keypoints_info = new_keypoints_info
keypoints, scores, visible = keypoints_info[..., :2], keypoints_info[..., 2], keypoints_info[..., 3]
return keypoints, scores
modules/control/proc/edge.py
+64
View File
@@ -0,0 +1,64 @@
import warnings
import cv2
import numpy as np
from PIL import Image
from modules.control.util import HWC3, resize_image
ed = None
"""
PFmode: bool
EdgeDetectionOperator: int
GradientThresholdValue: int
AnchorThresholdValue: int
ScanInterval: int
MinPathLength: int
Sigma: float
SumFlag: bool
NFAValidation: bool
MinLineLength: int
MaxDistanceBetweenTwoLines: float
LineFitErrorThreshold: float
MaxErrorThreshold: float
"""
class EdgeDetector:
def __call__(self, input_image=None, pf=True, mode='edge', detect_resolution=512, image_resolution=512, output_type=None, **kwargs):
global ed # pylint: disable=global-statement
if ed is None:
ed = cv2.ximgproc.createEdgeDrawing()
params = cv2.ximgproc.EdgeDrawing.Params()
params.PFmode = pf
ed.setParams(params)
if "img" in kwargs:
warnings.warn("img is deprecated, please use `input_image=...` instead.", DeprecationWarning)
input_image = kwargs.pop("img")
if input_image is None:
raise ValueError("input_image must be defined.")
if not isinstance(input_image, np.ndarray):
input_image = np.array(input_image, dtype=np.uint8)
output_type = output_type or "pil"
else:
output_type = output_type or "np"
input_image = HWC3(input_image)
input_image = resize_image(input_image, detect_resolution)
img_gray = cv2.cvtColor(input_image, cv2.COLOR_BGR2GRAY)
edges = ed.detectEdges(img_gray)
if mode == 'edge':
edge_map = ed.getEdgeImage(edges)
else:
edge_map = ed.getGradientImage(edges)
edge_map = np.expand_dims(edge_map, axis=2)
edge_map = cv2.cvtColor(edge_map, cv2.COLOR_GRAY2BGR).astype(np.uint8)
edge_map = HWC3(edge_map)
img = resize_image(input_image, image_resolution)
H, W, _C = img.shape
edge_map = cv2.resize(edge_map, (W, H), interpolation=cv2.INTER_LINEAR)
if output_type == "pil":
edge_map = Image.fromarray(edge_map)
edge_map = edge_map.convert('L')
return edge_map
modules/control/proc/hed.py
+128
View File
@@ -0,0 +1,128 @@
# This is an improved version and model of HED edge detection with Apache License, Version 2.0.
# Please use this implementation in your products
# This implementation may produce slightly different results from Saining Xie's official implementations,
# but it generates smoother edges and is more suitable for ControlNet as well as other image-to-image translations.
# Different from official models and other implementations, this is an RGB-input model (rather than BGR)
# and in this way it works better for gradio's RGB protocol
import os
import warnings
import cv2
import numpy as np
import torch
from einops import rearrange
from huggingface_hub import hf_hub_download
from PIL import Image
from modules.control.util import HWC3, nms, resize_image, safe_step
class DoubleConvBlock(torch.nn.Module): # pylint: disable=abstract-method
def __init__(self, input_channel, output_channel, layer_number):
super().__init__()
self.convs = torch.nn.Sequential()
self.convs.append(torch.nn.Conv2d(in_channels=input_channel, out_channels=output_channel, kernel_size=(3, 3), stride=(1, 1), padding=1))
for _i in range(1, layer_number):
self.convs.append(torch.nn.Conv2d(in_channels=output_channel, out_channels=output_channel, kernel_size=(3, 3), stride=(1, 1), padding=1))
self.projection = torch.nn.Conv2d(in_channels=output_channel, out_channels=1, kernel_size=(1, 1), stride=(1, 1), padding=0)
def __call__(self, x, down_sampling=False):
h = x
if down_sampling:
h = torch.nn.functional.max_pool2d(h, kernel_size=(2, 2), stride=(2, 2))
for conv in self.convs:
h = conv(h)
h = torch.nn.functional.relu(h)
return h, self.projection(h)
class ControlNetHED_Apache2(torch.nn.Module): # pylint: disable=abstract-method
def __init__(self):
super().__init__()
self.norm = torch.nn.Parameter(torch.zeros(size=(1, 3, 1, 1)))
self.block1 = DoubleConvBlock(input_channel=3, output_channel=64, layer_number=2)
self.block2 = DoubleConvBlock(input_channel=64, output_channel=128, layer_number=2)
self.block3 = DoubleConvBlock(input_channel=128, output_channel=256, layer_number=3)
self.block4 = DoubleConvBlock(input_channel=256, output_channel=512, layer_number=3)
self.block5 = DoubleConvBlock(input_channel=512, output_channel=512, layer_number=3)
def __call__(self, x):
h = x - self.norm
h, projection1 = self.block1(h)
h, projection2 = self.block2(h, down_sampling=True)
h, projection3 = self.block3(h, down_sampling=True)
h, projection4 = self.block4(h, down_sampling=True)
h, projection5 = self.block5(h, down_sampling=True)
return projection1, projection2, projection3, projection4, projection5
class HEDdetector:
def __init__(self, netNetwork):
self.netNetwork = netNetwork
@classmethod
def from_pretrained(cls, pretrained_model_or_path, filename=None, cache_dir=None):
filename = filename or "ControlNetHED.pth"
if os.path.isdir(pretrained_model_or_path):
model_path = os.path.join(pretrained_model_or_path, filename)
else:
model_path = hf_hub_download(pretrained_model_or_path, filename, cache_dir=cache_dir)
netNetwork = ControlNetHED_Apache2()
netNetwork.load_state_dict(torch.load(model_path, map_location='cpu'))
netNetwork.float().eval()
return cls(netNetwork)
def to(self, device):
self.netNetwork.to(device)
return self
def __call__(self, input_image, detect_resolution=512, image_resolution=512, safe=False, output_type="pil", scribble=False, **kwargs):
if "return_pil" in kwargs:
warnings.warn("return_pil is deprecated. Use output_type instead.", DeprecationWarning)
output_type = "pil" if kwargs["return_pil"] else "np"
if type(output_type) is bool:
warnings.warn("Passing `True` or `False` to `output_type` is deprecated and will raise an error in future versions")
if output_type:
output_type = "pil"
device = next(iter(self.netNetwork.parameters())).device
if not isinstance(input_image, np.ndarray):
input_image = np.array(input_image, dtype=np.uint8)
input_image = HWC3(input_image)
input_image = resize_image(input_image, detect_resolution)
assert input_image.ndim == 3
H, W, _C = input_image.shape
image_hed = torch.from_numpy(input_image.copy()).float().to(device)
image_hed = rearrange(image_hed, 'h w c -> 1 c h w')
edges = self.netNetwork(image_hed)
edges = [e.detach().cpu().numpy().astype(np.float32)[0, 0] for e in edges]
edges = [cv2.resize(e, (W, H), interpolation=cv2.INTER_LINEAR) for e in edges]
edges = np.stack(edges, axis=2)
edge = 1 / (1 + np.exp(-np.mean(edges, axis=2).astype(np.float64)))
if safe:
edge = safe_step(edge)
edge = (edge * 255.0).clip(0, 255).astype(np.uint8)
detected_map = edge
detected_map = HWC3(detected_map)
img = resize_image(input_image, image_resolution)
H, W, _C = img.shape
detected_map = cv2.resize(detected_map, (W, H), interpolation=cv2.INTER_LINEAR)
if scribble:
detected_map = nms(detected_map, 127, 3.0)
detected_map = cv2.GaussianBlur(detected_map, (0, 0), 3.0)
detected_map[detected_map > 4] = 255
detected_map[detected_map < 255] = 0
if output_type == "pil":
detected_map = Image.fromarray(detected_map)
return detected_map
modules/control/proc/leres/__init__.py
+108
View File
@@ -0,0 +1,108 @@
import os
import cv2
import numpy as np
import torch
from huggingface_hub import hf_hub_download
from PIL import Image
from modules.control.util import HWC3, resize_image
from .leres.depthmap import estimateboost, estimateleres
from .leres.multi_depth_model_woauxi import RelDepthModel
from .leres.net_tools import strip_prefix_if_present
from .pix2pix.models.pix2pix4depth_model import Pix2Pix4DepthModel
from .pix2pix.options.test_options import TestOptions
class LeresDetector:
def __init__(self, model, pix2pixmodel):
self.model = model
self.pix2pixmodel = pix2pixmodel
@classmethod
def from_pretrained(cls, pretrained_model_or_path, filename=None, pix2pix_filename=None, cache_dir=None):
filename = filename or "res101.pth"
pix2pix_filename = pix2pix_filename or "latest_net_G.pth"
if os.path.isdir(pretrained_model_or_path):
model_path = os.path.join(pretrained_model_or_path, filename)
else:
model_path = hf_hub_download(pretrained_model_or_path, filename, cache_dir=cache_dir)
checkpoint = torch.load(model_path, map_location=torch.device('cpu'))
model = RelDepthModel(backbone='resnext101')
model.load_state_dict(strip_prefix_if_present(checkpoint['depth_model'], "module."), strict=True)
del checkpoint
if os.path.isdir(pretrained_model_or_path):
model_path = os.path.join(pretrained_model_or_path, pix2pix_filename)
else:
model_path = hf_hub_download(pretrained_model_or_path, pix2pix_filename, cache_dir=cache_dir)
opt = TestOptions().parse()
if not torch.cuda.is_available():
opt.gpu_ids = [] # cpu mode
pix2pixmodel = Pix2Pix4DepthModel(opt)
pix2pixmodel.save_dir = os.path.dirname(model_path)
pix2pixmodel.load_networks('latest')
pix2pixmodel.eval()
return cls(model, pix2pixmodel)
def to(self, device):
self.model.to(device)
return self
def __call__(self, input_image, thr_a=0, thr_b=0, boost=False, detect_resolution=512, image_resolution=512, output_type="pil"):
# device = next(iter(self.model.parameters())).device
if not isinstance(input_image, np.ndarray):
input_image = np.array(input_image, dtype=np.uint8)
input_image = HWC3(input_image)
input_image = resize_image(input_image, detect_resolution)
assert input_image.ndim == 3
height, width, _dim = input_image.shape
if boost:
depth = estimateboost(input_image, self.model, 0, self.pix2pixmodel, max(width, height))
else:
depth = estimateleres(input_image, self.model, width, height)
numbytes=2
depth_min = depth.min()
depth_max = depth.max()
max_val = (2**(8*numbytes))-1
# check output before normalizing and mapping to 16 bit
if depth_max - depth_min > np.finfo("float").eps:
out = max_val * (depth - depth_min) / (depth_max - depth_min)
else:
out = np.zeros(depth.shape)
# single channel, 16 bit image
depth_image = out.astype("uint16")
# convert to uint8
depth_image = cv2.convertScaleAbs(depth_image, alpha=255.0/65535.0)
# remove near
if thr_a != 0:
thr_a = thr_a/100*255
depth_image = cv2.threshold(depth_image, thr_a, 255, cv2.THRESH_TOZERO)[1]
# invert image
depth_image = cv2.bitwise_not(depth_image)
# remove bg
if thr_b != 0:
thr_b = thr_b/100*255
depth_image = cv2.threshold(depth_image, thr_b, 255, cv2.THRESH_TOZERO)[1]
detected_map = depth_image
detected_map = HWC3(detected_map)
img = resize_image(input_image, image_resolution)
H, W, _C = img.shape
detected_map = cv2.resize(detected_map, (W, H), interpolation=cv2.INTER_LINEAR)
if output_type == "pil":
detected_map = Image.fromarray(detected_map)
return detected_map
modules/control/proc/leres/leres/LICENSE
+23
View File
@@ -0,0 +1,23 @@
https://github.com/thygate/stable-diffusion-webui-depthmap-script
MIT License
Copyright (c) 2023 Bob Thiry
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
modules/control/proc/leres/leres/Resnet.py
+199
View File
@@ -0,0 +1,199 @@
import torch.nn as nn
import torch.nn as NN
__all__ = ['ResNet', 'resnet18', 'resnet34', 'resnet50', 'resnet101',
'resnet152']
model_urls = {
'resnet18': 'https://download.pytorch.org/models/resnet18-5c106cde.pth',
'resnet34': 'https://download.pytorch.org/models/resnet34-333f7ec4.pth',
'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth',
'resnet101': 'https://download.pytorch.org/models/resnet101-5d3b4d8f.pth',
'resnet152': 'https://download.pytorch.org/models/resnet152-b121ed2d.pth',
}
def conv3x3(in_planes, out_planes, stride=1):
"""3x3 convolution with padding"""
return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
padding=1, bias=False)
class BasicBlock(nn.Module):
expansion = 1
def __init__(self, inplanes, planes, stride=1, downsample=None):
super(BasicBlock, self).__init__()
self.conv1 = conv3x3(inplanes, planes, stride)
self.bn1 = NN.BatchNorm2d(planes) #NN.BatchNorm2d
self.relu = nn.ReLU(inplace=True)
self.conv2 = conv3x3(planes, planes)
self.bn2 = NN.BatchNorm2d(planes) #NN.BatchNorm2d
self.downsample = downsample
self.stride = stride
def forward(self, x):
residual = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
if self.downsample is not None:
residual = self.downsample(x)
out += residual
out = self.relu(out)
return out
class Bottleneck(nn.Module):
expansion = 4
def __init__(self, inplanes, planes, stride=1, downsample=None):
super(Bottleneck, self).__init__()
self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)
self.bn1 = NN.BatchNorm2d(planes) #NN.BatchNorm2d
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride,
padding=1, bias=False)
self.bn2 = NN.BatchNorm2d(planes) #NN.BatchNorm2d
self.conv3 = nn.Conv2d(planes, planes * self.expansion, kernel_size=1, bias=False)
self.bn3 = NN.BatchNorm2d(planes * self.expansion) #NN.BatchNorm2d
self.relu = nn.ReLU(inplace=True)
self.downsample = downsample
self.stride = stride
def forward(self, x):
residual = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
out = self.relu(out)
out = self.conv3(out)
out = self.bn3(out)
if self.downsample is not None:
residual = self.downsample(x)
out += residual
out = self.relu(out)
return out
class ResNet(nn.Module):
def __init__(self, block, layers, num_classes=1000):
self.inplanes = 64
super(ResNet, self).__init__()
self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,
bias=False)
self.bn1 = NN.BatchNorm2d(64) #NN.BatchNorm2d
self.relu = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.layer1 = self._make_layer(block, 64, layers[0])
self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
#self.avgpool = nn.AvgPool2d(7, stride=1)
#self.fc = nn.Linear(512 * block.expansion, num_classes)
for m in self.modules():
if isinstance(m, nn.Conv2d):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
elif isinstance(m, nn.BatchNorm2d):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
def _make_layer(self, block, planes, blocks, stride=1):
downsample = None
if stride != 1 or self.inplanes != planes * block.expansion:
downsample = nn.Sequential(
nn.Conv2d(self.inplanes, planes * block.expansion,
kernel_size=1, stride=stride, bias=False),
NN.BatchNorm2d(planes * block.expansion), #NN.BatchNorm2d
)
layers = []
layers.append(block(self.inplanes, planes, stride, downsample))
self.inplanes = planes * block.expansion
for _i in range(1, blocks):
layers.append(block(self.inplanes, planes))
return nn.Sequential(*layers)
def forward(self, x):
features = []
x = self.conv1(x)
x = self.bn1(x)
x = self.relu(x)
x = self.maxpool(x)
x = self.layer1(x)
features.append(x)
x = self.layer2(x)
features.append(x)
x = self.layer3(x)
features.append(x)
x = self.layer4(x)
features.append(x)
return features
def resnet18(pretrained=True, **kwargs):
"""Constructs a ResNet-18 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = ResNet(BasicBlock, [2, 2, 2, 2], **kwargs)
return model
def resnet34(pretrained=True, **kwargs):
"""Constructs a ResNet-34 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = ResNet(BasicBlock, [3, 4, 6, 3], **kwargs)
return model
def resnet50(pretrained=True, **kwargs):
"""Constructs a ResNet-50 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs)
return model
def resnet101(pretrained=True, **kwargs):
"""Constructs a ResNet-101 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = ResNet(Bottleneck, [3, 4, 23, 3], **kwargs)
return model
def resnet152(pretrained=True, **kwargs):
"""Constructs a ResNet-152 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = ResNet(Bottleneck, [3, 8, 36, 3], **kwargs)
return model
modules/control/proc/leres/leres/Resnext_torch.py
+237
View File
@@ -0,0 +1,237 @@
#!/usr/bin/env python
# coding: utf-8
import torch.nn as nn
try:
from urllib import urlretrieve
except ImportError:
from urllib.request import urlretrieve
__all__ = ['resnext101_32x8d']
model_urls = {
'resnext50_32x4d': 'https://download.pytorch.org/models/resnext50_32x4d-7cdf4587.pth',
'resnext101_32x8d': 'https://download.pytorch.org/models/resnext101_32x8d-8ba56ff5.pth',
}
def conv3x3(in_planes, out_planes, stride=1, groups=1, dilation=1):
"""3x3 convolution with padding"""
return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
padding=dilation, groups=groups, bias=False, dilation=dilation)
def conv1x1(in_planes, out_planes, stride=1):
"""1x1 convolution"""
return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False)
class BasicBlock(nn.Module):
expansion = 1
def __init__(self, inplanes, planes, stride=1, downsample=None, groups=1,
base_width=64, dilation=1, norm_layer=None):
super(BasicBlock, self).__init__()
if norm_layer is None:
norm_layer = nn.BatchNorm2d
if groups != 1 or base_width != 64:
raise ValueError('BasicBlock only supports groups=1 and base_width=64')
if dilation > 1:
raise NotImplementedError("Dilation > 1 not supported in BasicBlock")
# Both self.conv1 and self.downsample layers downsample the input when stride != 1
self.conv1 = conv3x3(inplanes, planes, stride)
self.bn1 = norm_layer(planes)
self.relu = nn.ReLU(inplace=True)
self.conv2 = conv3x3(planes, planes)
self.bn2 = norm_layer(planes)
self.downsample = downsample
self.stride = stride
def forward(self, x):
identity = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
if self.downsample is not None:
identity = self.downsample(x)
out += identity
out = self.relu(out)
return out
class Bottleneck(nn.Module):
# Bottleneck in torchvision places the stride for downsampling at 3x3 convolution(self.conv2)
# while original implementation places the stride at the first 1x1 convolution(self.conv1)
# according to "Deep residual learning for image recognition"https://arxiv.org/abs/1512.03385.
# This variant is also known as ResNet V1.5 and improves accuracy according to
# https://ngc.nvidia.com/catalog/model-scripts/nvidia:resnet_50_v1_5_for_pytorch.
expansion = 4
def __init__(self, inplanes, planes, stride=1, downsample=None, groups=1,
base_width=64, dilation=1, norm_layer=None):
super(Bottleneck, self).__init__()
if norm_layer is None:
norm_layer = nn.BatchNorm2d
width = int(planes * (base_width / 64.)) * groups
# Both self.conv2 and self.downsample layers downsample the input when stride != 1
self.conv1 = conv1x1(inplanes, width)
self.bn1 = norm_layer(width)
self.conv2 = conv3x3(width, width, stride, groups, dilation)
self.bn2 = norm_layer(width)
self.conv3 = conv1x1(width, planes * self.expansion)
self.bn3 = norm_layer(planes * self.expansion)
self.relu = nn.ReLU(inplace=True)
self.downsample = downsample
self.stride = stride
def forward(self, x):
identity = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
out = self.relu(out)
out = self.conv3(out)
out = self.bn3(out)
if self.downsample is not None:
identity = self.downsample(x)
out += identity
out = self.relu(out)
return out
class ResNet(nn.Module):
def __init__(self, block, layers, num_classes=1000, zero_init_residual=False,
groups=1, width_per_group=64, replace_stride_with_dilation=None,
norm_layer=None):
super(ResNet, self).__init__()
if norm_layer is None:
norm_layer = nn.BatchNorm2d
self._norm_layer = norm_layer
self.inplanes = 64
self.dilation = 1
if replace_stride_with_dilation is None:
# each element in the tuple indicates if we should replace
# the 2x2 stride with a dilated convolution instead
replace_stride_with_dilation = [False, False, False]
if len(replace_stride_with_dilation) != 3:
raise ValueError("replace_stride_with_dilation should be None "
"or a 3-element tuple, got {}".format(replace_stride_with_dilation))
self.groups = groups
self.base_width = width_per_group
self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3,
bias=False)
self.bn1 = norm_layer(self.inplanes)
self.relu = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.layer1 = self._make_layer(block, 64, layers[0])
self.layer2 = self._make_layer(block, 128, layers[1], stride=2,
dilate=replace_stride_with_dilation[0])
self.layer3 = self._make_layer(block, 256, layers[2], stride=2,
dilate=replace_stride_with_dilation[1])
self.layer4 = self._make_layer(block, 512, layers[3], stride=2,
dilate=replace_stride_with_dilation[2])
#self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
#self.fc = nn.Linear(512 * block.expansion, num_classes)
for m in self.modules():
if isinstance(m, nn.Conv2d):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
# Zero-initialize the last BN in each residual branch,
# so that the residual branch starts with zeros, and each residual block behaves like an identity.
# This improves the model by 0.2~0.3% according to https://arxiv.org/abs/1706.02677
if zero_init_residual:
for m in self.modules():
if isinstance(m, Bottleneck):
nn.init.constant_(m.bn3.weight, 0)
elif isinstance(m, BasicBlock):
nn.init.constant_(m.bn2.weight, 0)
def _make_layer(self, block, planes, blocks, stride=1, dilate=False):
norm_layer = self._norm_layer
downsample = None
previous_dilation = self.dilation
if dilate:
self.dilation *= stride
stride = 1
if stride != 1 or self.inplanes != planes * block.expansion:
downsample = nn.Sequential(
conv1x1(self.inplanes, planes * block.expansion, stride),
norm_layer(planes * block.expansion),
)
layers = []
layers.append(block(self.inplanes, planes, stride, downsample, self.groups,
self.base_width, previous_dilation, norm_layer))
self.inplanes = planes * block.expansion
for _ in range(1, blocks):
layers.append(block(self.inplanes, planes, groups=self.groups,
base_width=self.base_width, dilation=self.dilation,
norm_layer=norm_layer))
return nn.Sequential(*layers)
def _forward_impl(self, x):
# See note [TorchScript super()]
features = []
x = self.conv1(x)
x = self.bn1(x)
x = self.relu(x)
x = self.maxpool(x)
x = self.layer1(x)
features.append(x)
x = self.layer2(x)
features.append(x)
x = self.layer3(x)
features.append(x)
x = self.layer4(x)
features.append(x)
#x = self.avgpool(x)
#x = torch.flatten(x, 1)
#x = self.fc(x)
return features
def forward(self, x):
return self._forward_impl(x)
def resnext101_32x8d(pretrained=True, **kwargs):
"""Constructs a ResNet-152 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
kwargs['groups'] = 32
kwargs['width_per_group'] = 8
model = ResNet(Bottleneck, [3, 4, 23, 3], **kwargs)
return model
modules/control/proc/leres/leres/__init__.py
View File
modules/control/proc/leres/leres/depthmap.py
+546
View File
@@ -0,0 +1,546 @@
# Author: thygate
# https://github.com/thygate/stable-diffusion-webui-depthmap-script
import gc
from operator import getitem
import cv2
import numpy as np
import skimage.measure
import torch
from torchvision.transforms import transforms
from modules.control.util import torch_gc
whole_size_threshold = 1600 # R_max from the paper
pix2pixsize = 1024
def scale_torch(img):
"""
Scale the image and output it in torch.tensor.
:param img: input rgb is in shape [H, W, C], input depth/disp is in shape [H, W]
:param scale: the scale factor. float
:return: img. [C, H, W]
"""
if len(img.shape) == 2:
img = img[np.newaxis, :, :]
if img.shape[2] == 3:
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.485, 0.456, 0.406) , (0.229, 0.224, 0.225) )])
img = transform(img.astype(np.float32))
else:
img = img.astype(np.float32)
img = torch.from_numpy(img)
return img
def estimateleres(img, model, w, h):
device = next(iter(model.parameters())).device
# leres transform input
rgb_c = img[:, :, ::-1].copy()
A_resize = cv2.resize(rgb_c, (w, h))
img_torch = scale_torch(A_resize)[None, :, :, :]
# compute
img_torch = img_torch.to(device)
prediction = model.depth_model(img_torch)
prediction = prediction.squeeze().cpu().numpy()
prediction = cv2.resize(prediction, (img.shape[1], img.shape[0]), interpolation=cv2.INTER_CUBIC)
return prediction
def generatemask(size):
# Generates a Guassian mask
mask = np.zeros(size, dtype=np.float32)
sigma = int(size[0]/16)
k_size = int(2 * np.ceil(2 * int(size[0]/16)) + 1)
mask[int(0.15*size[0]):size[0] - int(0.15*size[0]), int(0.15*size[1]): size[1] - int(0.15*size[1])] = 1
mask = cv2.GaussianBlur(mask, (int(k_size), int(k_size)), sigma)
mask = (mask - mask.min()) / (mask.max() - mask.min())
mask = mask.astype(np.float32)
return mask
def resizewithpool(img, size):
i_size = img.shape[0]
n = int(np.floor(i_size/size))
out = skimage.measure.block_reduce(img, (n, n), np.max)
return out
def rgb2gray(rgb):
# Converts rgb to gray
return np.dot(rgb[..., :3], [0.2989, 0.5870, 0.1140])
def calculateprocessingres(img, basesize, confidence=0.1, scale_threshold=3, whole_size_threshold=3000):
# Returns the R_x resolution described in section 5 of the main paper.
# Parameters:
# img :input rgb image
# basesize : size the dilation kernel which is equal to receptive field of the network.
# confidence: value of x in R_x; allowed percentage of pixels that are not getting any contextual cue.
# scale_threshold: maximum allowed upscaling on the input image ; it has been set to 3.
# whole_size_threshold: maximum allowed resolution. (R_max from section 6 of the main paper)
# Returns:
# outputsize_scale*speed_scale :The computed R_x resolution
# patch_scale: K parameter from section 6 of the paper
# speed scale parameter is to process every image in a smaller size to accelerate the R_x resolution search
speed_scale = 32
image_dim = int(min(img.shape[0:2]))
gray = rgb2gray(img)
grad = np.abs(cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=3)) + np.abs(cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=3))
grad = cv2.resize(grad, (image_dim, image_dim), cv2.INTER_AREA)
# thresholding the gradient map to generate the edge-map as a proxy of the contextual cues
m = grad.min()
M = grad.max()
middle = m + (0.4 * (M - m))
grad[grad < middle] = 0
grad[grad >= middle] = 1
# dilation kernel with size of the receptive field
kernel = np.ones((int(basesize/speed_scale), int(basesize/speed_scale)), float)
# dilation kernel with size of the a quarter of receptive field used to compute k
# as described in section 6 of main paper
kernel2 = np.ones((int(basesize / (4*speed_scale)), int(basesize / (4*speed_scale))), float)
# Output resolution limit set by the whole_size_threshold and scale_threshold.
threshold = min(whole_size_threshold, scale_threshold * max(img.shape[:2]))
outputsize_scale = basesize / speed_scale
for p_size in range(int(basesize/speed_scale), int(threshold/speed_scale), int(basesize / (2*speed_scale))):
grad_resized = resizewithpool(grad, p_size)
grad_resized = cv2.resize(grad_resized, (p_size, p_size), cv2.INTER_NEAREST)
grad_resized[grad_resized >= 0.5] = 1
grad_resized[grad_resized < 0.5] = 0
dilated = cv2.dilate(grad_resized, kernel, iterations=1)
meanvalue = (1-dilated).mean()
if meanvalue > confidence:
break
else:
outputsize_scale = p_size
grad_region = cv2.dilate(grad_resized, kernel2, iterations=1)
patch_scale = grad_region.mean()
return int(outputsize_scale*speed_scale), patch_scale
# Generate a double-input depth estimation
def doubleestimate(img, size1, size2, pix2pixsize, model, net_type, pix2pixmodel):
# Generate the low resolution estimation
estimate1 = singleestimate(img, size1, model, net_type)
# Resize to the inference size of merge network.
estimate1 = cv2.resize(estimate1, (pix2pixsize, pix2pixsize), interpolation=cv2.INTER_CUBIC)
# Generate the high resolution estimation
estimate2 = singleestimate(img, size2, model, net_type)
# Resize to the inference size of merge network.
estimate2 = cv2.resize(estimate2, (pix2pixsize, pix2pixsize), interpolation=cv2.INTER_CUBIC)
# Inference on the merge model
pix2pixmodel.set_input(estimate1, estimate2)
pix2pixmodel.test()
visuals = pix2pixmodel.get_current_visuals()
prediction_mapped = visuals['fake_B']
prediction_mapped = (prediction_mapped+1)/2
prediction_mapped = (prediction_mapped - torch.min(prediction_mapped)) / (
torch.max(prediction_mapped) - torch.min(prediction_mapped))
prediction_mapped = prediction_mapped.squeeze().cpu().numpy()
return prediction_mapped
# Generate a single-input depth estimation
def singleestimate(img, msize, model, net_type):
# if net_type == 0:
return estimateleres(img, model, msize, msize)
# else:
# return estimatemidasBoost(img, model, msize, msize)
def applyGridpatch(blsize, stride, img, box):
# Extract a simple grid patch.
counter1 = 0
patch_bound_list = {}
for k in range(blsize, img.shape[1] - blsize, stride):
for j in range(blsize, img.shape[0] - blsize, stride):
patch_bound_list[str(counter1)] = {}
patchbounds = [j - blsize, k - blsize, j - blsize + 2 * blsize, k - blsize + 2 * blsize]
patch_bound = [box[0] + patchbounds[1], box[1] + patchbounds[0], patchbounds[3] - patchbounds[1],
patchbounds[2] - patchbounds[0]]
patch_bound_list[str(counter1)]['rect'] = patch_bound
patch_bound_list[str(counter1)]['size'] = patch_bound[2]
counter1 = counter1 + 1
return patch_bound_list
# Generating local patches to perform the local refinement described in section 6 of the main paper.
def generatepatchs(img, base_size):
# Compute the gradients as a proxy of the contextual cues.
img_gray = rgb2gray(img)
whole_grad = np.abs(cv2.Sobel(img_gray, cv2.CV_64F, 0, 1, ksize=3)) +\
np.abs(cv2.Sobel(img_gray, cv2.CV_64F, 1, 0, ksize=3))
threshold = whole_grad[whole_grad > 0].mean()
whole_grad[whole_grad < threshold] = 0
# We use the integral image to speed-up the evaluation of the amount of gradients for each patch.
gf = whole_grad.sum()/len(whole_grad.reshape(-1))
grad_integral_image = cv2.integral(whole_grad)
# Variables are selected such that the initial patch size would be the receptive field size
# and the stride is set to 1/3 of the receptive field size.
blsize = int(round(base_size/2))
stride = int(round(blsize*0.75))
# Get initial Grid
patch_bound_list = applyGridpatch(blsize, stride, img, [0, 0, 0, 0])
# Refine initial Grid of patches by discarding the flat (in terms of gradients of the rgb image) ones. Refine
# each patch size to ensure that there will be enough depth cues for the network to generate a consistent depth map.
patch_bound_list = adaptiveselection(grad_integral_image, patch_bound_list, gf)
# Sort the patch list to make sure the merging operation will be done with the correct order: starting from biggest
# patch
patchset = sorted(patch_bound_list.items(), key=lambda x: getitem(x[1], 'size'), reverse=True)
return patchset
def getGF_fromintegral(integralimage, rect):
# Computes the gradient density of a given patch from the gradient integral image.
x1 = rect[1]
x2 = rect[1]+rect[3]
y1 = rect[0]
y2 = rect[0]+rect[2]
value = integralimage[x2, y2]-integralimage[x1, y2]-integralimage[x2, y1]+integralimage[x1, y1]
return value
# Adaptively select patches
def adaptiveselection(integral_grad, patch_bound_list, gf):
patchlist = {}
count = 0
height, width = integral_grad.shape
search_step = int(32/factor)
# Go through all patches
for c in range(len(patch_bound_list)):
# Get patch
bbox = patch_bound_list[str(c)]['rect']
# Compute the amount of gradients present in the patch from the integral image.
cgf = getGF_fromintegral(integral_grad, bbox)/(bbox[2]*bbox[3])
# Check if patching is beneficial by comparing the gradient density of the patch to
# the gradient density of the whole image
if cgf >= gf:
bbox_test = bbox.copy()
patchlist[str(count)] = {}
# Enlarge each patch until the gradient density of the patch is equal
# to the whole image gradient density
while True:
bbox_test[0] = bbox_test[0] - int(search_step/2)
bbox_test[1] = bbox_test[1] - int(search_step/2)
bbox_test[2] = bbox_test[2] + search_step
bbox_test[3] = bbox_test[3] + search_step
# Check if we are still within the image
if bbox_test[0] < 0 or bbox_test[1] < 0 or bbox_test[1] + bbox_test[3] >= height \
or bbox_test[0] + bbox_test[2] >= width:
break
# Compare gradient density
cgf = getGF_fromintegral(integral_grad, bbox_test)/(bbox_test[2]*bbox_test[3])
if cgf < gf:
break
bbox = bbox_test.copy()
# Add patch to selected patches
patchlist[str(count)]['rect'] = bbox
patchlist[str(count)]['size'] = bbox[2]
count = count + 1
# Return selected patches
return patchlist
def impatch(image, rect):
# Extract the given patch pixels from a given image.
w1 = rect[0]
h1 = rect[1]
w2 = w1 + rect[2]
h2 = h1 + rect[3]
image_patch = image[h1:h2, w1:w2]
return image_patch
class ImageandPatchs:
def __init__(self, root_dir, name, patchsinfo, rgb_image, scale=1):
self.root_dir = root_dir
self.patchsinfo = patchsinfo
self.name = name
self.patchs = patchsinfo
self.scale = scale
self.rgb_image = cv2.resize(rgb_image, (round(rgb_image.shape[1]*scale), round(rgb_image.shape[0]*scale)),
interpolation=cv2.INTER_CUBIC)
self.do_have_estimate = False
self.estimation_updated_image = None
self.estimation_base_image = None
def __len__(self):
return len(self.patchs)
def set_base_estimate(self, est):
self.estimation_base_image = est
if self.estimation_updated_image is not None:
self.do_have_estimate = True
def set_updated_estimate(self, est):
self.estimation_updated_image = est
if self.estimation_base_image is not None:
self.do_have_estimate = True
def __getitem__(self, index):
patch_id = int(self.patchs[index][0])
rect = np.array(self.patchs[index][1]['rect'])
msize = self.patchs[index][1]['size']
## applying scale to rect:
rect = np.round(rect * self.scale)
rect = rect.astype('int')
msize = round(msize * self.scale)
patch_rgb = impatch(self.rgb_image, rect)
if self.do_have_estimate:
patch_whole_estimate_base = impatch(self.estimation_base_image, rect)
patch_whole_estimate_updated = impatch(self.estimation_updated_image, rect)
return {'patch_rgb': patch_rgb, 'patch_whole_estimate_base': patch_whole_estimate_base,
'patch_whole_estimate_updated': patch_whole_estimate_updated, 'rect': rect,
'size': msize, 'id': patch_id}
else:
return {'patch_rgb': patch_rgb, 'rect': rect, 'size': msize, 'id': patch_id}
def print_options(self, opt):
"""Print and save options
It will print both current options and default values(if different).
It will save options into a text file / [checkpoints_dir] / opt.txt
"""
message = ''
message += '----------------- Options ---------------\n'
for k, v in sorted(vars(opt).items()):
comment = ''
default = self.parser.get_default(k)
if v != default:
comment = '\t[default: %s]' % str(default)
message += '{:>25}: {:<30}{}\n'.format(str(k), str(v), comment)
message += '----------------- End -------------------'
print(message)
# save to the disk
"""
expr_dir = os.path.join(opt.checkpoints_dir, opt.name)
util.mkdirs(expr_dir)
file_name = os.path.join(expr_dir, '{}_opt.txt'.format(opt.phase))
with open(file_name, 'wt') as opt_file:
opt_file.write(message)
opt_file.write('\n')
"""
def parse(self):
"""Parse our options, create checkpoints directory suffix, and set up gpu device."""
opt = self.gather_options()
opt.isTrain = self.isTrain # train or test
# process opt.suffix
if opt.suffix:
suffix = ('_' + opt.suffix.format(**vars(opt))) if opt.suffix != '' else ''
opt.name = opt.name + suffix
#self.print_options(opt)
# set gpu ids
str_ids = opt.gpu_ids.split(',')
opt.gpu_ids = []
for str_id in str_ids:
id = int(str_id)
if id >= 0:
opt.gpu_ids.append(id)
#if len(opt.gpu_ids) > 0:
# torch.cuda.set_device(opt.gpu_ids[0])
self.opt = opt
return self.opt
def estimateboost(img, model, model_type, pix2pixmodel, max_res=512, depthmap_script_boost_rmax=None):
global whole_size_threshold
# get settings
if depthmap_script_boost_rmax:
whole_size_threshold = depthmap_script_boost_rmax
if model_type == 0: #leres
net_receptive_field_size = 448
patch_netsize = 2 * net_receptive_field_size
elif model_type == 1: #dpt_beit_large_512
net_receptive_field_size = 512
patch_netsize = 2 * net_receptive_field_size
else: #other midas
net_receptive_field_size = 384
patch_netsize = 2 * net_receptive_field_size
gc.collect()
torch_gc()
# Generate mask used to smoothly blend the local pathc estimations to the base estimate.
# It is arbitrarily large to avoid artifacts during rescaling for each crop.
mask_org = generatemask((3000, 3000))
mask = mask_org.copy()
# Value x of R_x defined in the section 5 of the main paper.
r_threshold_value = 0.2
#if R0:
# r_threshold_value = 0
input_resolution = img.shape
scale_threshold = 3 # Allows up-scaling with a scale up to 3
# Find the best input resolution R-x. The resolution search described in section 5-double estimation of the main paper and section B of the
# supplementary material.
whole_image_optimal_size, patch_scale = calculateprocessingres(img, net_receptive_field_size, r_threshold_value, scale_threshold, whole_size_threshold)
# print('wholeImage being processed in :', whole_image_optimal_size)
# Generate the base estimate using the double estimation.
whole_estimate = doubleestimate(img, net_receptive_field_size, whole_image_optimal_size, pix2pixsize, model, model_type, pix2pixmodel)
# Compute the multiplier described in section 6 of the main paper to make sure our initial patch can select
# small high-density regions of the image.
global factor
factor = max(min(1, 4 * patch_scale * whole_image_optimal_size / whole_size_threshold), 0.2)
# print('Adjust factor is:', 1/factor)
# Check if Local boosting is beneficial.
if max_res < whole_image_optimal_size:
# print("No Local boosting. Specified Max Res is smaller than R20, Returning doubleestimate result")
return cv2.resize(whole_estimate, (input_resolution[1], input_resolution[0]), interpolation=cv2.INTER_CUBIC)
# Compute the default target resolution.
if img.shape[0] > img.shape[1]:
a = 2 * whole_image_optimal_size
b = round(2 * whole_image_optimal_size * img.shape[1] / img.shape[0])
else:
a = round(2 * whole_image_optimal_size * img.shape[0] / img.shape[1])
b = 2 * whole_image_optimal_size
b = int(round(b / factor))
a = int(round(a / factor))
"""
# recompute a, b and saturate to max res.
if max(a,b) > max_res:
print('Default Res is higher than max-res: Reducing final resolution')
if img.shape[0] > img.shape[1]:
a = max_res
b = round(max_res * img.shape[1] / img.shape[0])
else:
a = round(max_res * img.shape[0] / img.shape[1])
b = max_res
b = int(b)
a = int(a)
"""
img = cv2.resize(img, (b, a), interpolation=cv2.INTER_CUBIC)
# Extract selected patches for local refinement
base_size = net_receptive_field_size * 2
patchset = generatepatchs(img, base_size)
# print('Target resolution: ', img.shape)
# Computing a scale in case user prompted to generate the results as the same resolution of the input.
# Notice that our method output resolution is independent of the input resolution and this parameter will only
# enable a scaling operation during the local patch merge implementation to generate results with the same resolution
# as the input.
"""
if output_resolution == 1:
mergein_scale = input_resolution[0] / img.shape[0]
print('Dynamicly change merged-in resolution; scale:', mergein_scale)
else:
mergein_scale = 1
"""
# always rescale to input res for now
mergein_scale = input_resolution[0] / img.shape[0]
imageandpatchs = ImageandPatchs('', '', patchset, img, mergein_scale)
whole_estimate_resized = cv2.resize(whole_estimate, (round(img.shape[1]*mergein_scale),
round(img.shape[0]*mergein_scale)), interpolation=cv2.INTER_CUBIC)
imageandpatchs.set_base_estimate(whole_estimate_resized.copy())
imageandpatchs.set_updated_estimate(whole_estimate_resized.copy())
print('Resulting depthmap resolution will be :', whole_estimate_resized.shape[:2])
print('Patches to process: '+str(len(imageandpatchs)))
# Enumerate through all patches, generate their estimations and refining the base estimate.
for patch_ind in range(len(imageandpatchs)):
# Get patch information
patch = imageandpatchs[patch_ind] # patch object
patch_rgb = patch['patch_rgb'] # rgb patch
patch_whole_estimate_base = patch['patch_whole_estimate_base'] # corresponding patch from base
rect = patch['rect'] # patch size and location
patch['id'] # patch ID
org_size = patch_whole_estimate_base.shape # the original size from the unscaled input
print('\t Processing patch', patch_ind, '/', len(imageandpatchs)-1, '|', rect)
# We apply double estimation for patches. The high resolution value is fixed to twice the receptive
# field size of the network for patches to accelerate the process.
patch_estimation = doubleestimate(patch_rgb, net_receptive_field_size, patch_netsize, pix2pixsize, model, model_type, pix2pixmodel)
patch_estimation = cv2.resize(patch_estimation, (pix2pixsize, pix2pixsize), interpolation=cv2.INTER_CUBIC)
patch_whole_estimate_base = cv2.resize(patch_whole_estimate_base, (pix2pixsize, pix2pixsize), interpolation=cv2.INTER_CUBIC)
# Merging the patch estimation into the base estimate using our merge network:
# We feed the patch estimation and the same region from the updated base estimate to the merge network
# to generate the target estimate for the corresponding region.
pix2pixmodel.set_input(patch_whole_estimate_base, patch_estimation)
# Run merging network
pix2pixmodel.test()
visuals = pix2pixmodel.get_current_visuals()
prediction_mapped = visuals['fake_B']
prediction_mapped = (prediction_mapped+1)/2
prediction_mapped = prediction_mapped.squeeze().cpu().numpy()
mapped = prediction_mapped
# We use a simple linear polynomial to make sure the result of the merge network would match the values of
# base estimate
p_coef = np.polyfit(mapped.reshape(-1), patch_whole_estimate_base.reshape(-1), deg=1)
merged = np.polyval(p_coef, mapped.reshape(-1)).reshape(mapped.shape)
merged = cv2.resize(merged, (org_size[1],org_size[0]), interpolation=cv2.INTER_CUBIC)
# Get patch size and location
w1 = rect[0]
h1 = rect[1]
w2 = w1 + rect[2]
h2 = h1 + rect[3]
# To speed up the implementation, we only generate the Gaussian mask once with a sufficiently large size
# and resize it to our needed size while merging the patches.
if mask.shape != org_size:
mask = cv2.resize(mask_org, (org_size[1],org_size[0]), interpolation=cv2.INTER_LINEAR)
tobemergedto = imageandpatchs.estimation_updated_image
# Update the whole estimation:
# We use a simple Gaussian mask to blend the merged patch region with the base estimate to ensure seamless
# blending at the boundaries of the patch region.
tobemergedto[h1:h2, w1:w2] = np.multiply(tobemergedto[h1:h2, w1:w2], 1 - mask) + np.multiply(merged, mask)
imageandpatchs.set_updated_estimate(tobemergedto)
# output
return cv2.resize(imageandpatchs.estimation_updated_image, (input_resolution[1], input_resolution[0]), interpolation=cv2.INTER_CUBIC)
modules/control/proc/leres/leres/multi_depth_model_woauxi.py
+34
View File
@@ -0,0 +1,34 @@
import torch
import torch.nn as nn
from . import network_auxi as network
from .net_tools import get_func
class RelDepthModel(nn.Module):
def __init__(self, backbone='resnet50'):
super(RelDepthModel, self).__init__()
if backbone == 'resnet50':
encoder = 'resnet50_stride32'
elif backbone == 'resnext101':
encoder = 'resnext101_stride32x8d'
self.depth_model = DepthModel(encoder)
def inference(self, rgb):
input = rgb.to(self.depth_model.device)
depth = self.depth_model(input)
#pred_depth_out = depth - depth.min() + 0.01
return depth #pred_depth_out
class DepthModel(nn.Module):
def __init__(self, encoder):
super(DepthModel, self).__init__()
backbone = network.__name__.split('.')[-1] + '.' + encoder
self.encoder_modules = get_func(backbone)()
self.decoder_modules = network.Decoder()
def forward(self, x):
lateral_out = self.encoder_modules(x)
out_logit = self.decoder_modules(lateral_out)
return out_logit
modules/control/proc/leres/leres/net_tools.py
+54
View File
@@ -0,0 +1,54 @@
import os
from collections import OrderedDict
import importlib
import torch
def get_func(func_name):
"""Helper to return a function object by name. func_name must identify a
function in this module or the path to a function relative to the base
'modeling' module.
"""
if func_name == '':
return None
try:
parts = func_name.split('.')
# Refers to a function in this module
if len(parts) == 1:
return globals()[parts[0]]
# Otherwise, assume we're referencing a module under modeling
module_name = 'modules.control.proc.leres.leres.' + '.'.join(parts[:-1])
module = importlib.import_module(module_name)
return getattr(module, parts[-1])
except Exception:
print('Failed to find function: %s', func_name)
raise
def load_ckpt(args, depth_model, shift_model, focal_model):
"""
Load checkpoint.
"""
if os.path.isfile(args.load_ckpt):
print("loading checkpoint %s" % args.load_ckpt)
checkpoint = torch.load(args.load_ckpt)
if shift_model is not None:
shift_model.load_state_dict(strip_prefix_if_present(checkpoint['shift_model'], 'module.'),
strict=True)
if focal_model is not None:
focal_model.load_state_dict(strip_prefix_if_present(checkpoint['focal_model'], 'module.'),
strict=True)
depth_model.load_state_dict(strip_prefix_if_present(checkpoint['depth_model'], "module."),
strict=True)
del checkpoint
if torch.cuda.is_available():
torch.cuda.empty_cache()
def strip_prefix_if_present(state_dict, prefix):
keys = sorted(state_dict.keys())
if not all(key.startswith(prefix) for key in keys):
return state_dict
stripped_state_dict = OrderedDict()
for key, value in state_dict.items():
stripped_state_dict[key.replace(prefix, "")] = value
return stripped_state_dict
modules/control/proc/leres/leres/network_auxi.py
+419
View File
@@ -0,0 +1,419 @@
import torch
import torch.nn as nn
import torch.nn.init as init
from . import Resnet, Resnext_torch
def resnet50_stride32():
return DepthNet(backbone='resnet', depth=50, upfactors=[2, 2, 2, 2])
def resnext101_stride32x8d():
return DepthNet(backbone='resnext101_32x8d', depth=101, upfactors=[2, 2, 2, 2])
class Decoder(nn.Module):
def __init__(self):
super(Decoder, self).__init__()
self.inchannels = [256, 512, 1024, 2048]
self.midchannels = [256, 256, 256, 512]
self.upfactors = [2,2,2,2]
self.outchannels = 1
self.conv = FTB(inchannels=self.inchannels[3], midchannels=self.midchannels[3])
self.conv1 = nn.Conv2d(in_channels=self.midchannels[3], out_channels=self.midchannels[2], kernel_size=3, padding=1, stride=1, bias=True)
self.upsample = nn.Upsample(scale_factor=self.upfactors[3], mode='bilinear', align_corners=True)
self.ffm2 = FFM(inchannels=self.inchannels[2], midchannels=self.midchannels[2], outchannels = self.midchannels[2], upfactor=self.upfactors[2])
self.ffm1 = FFM(inchannels=self.inchannels[1], midchannels=self.midchannels[1], outchannels = self.midchannels[1], upfactor=self.upfactors[1])
self.ffm0 = FFM(inchannels=self.inchannels[0], midchannels=self.midchannels[0], outchannels = self.midchannels[0], upfactor=self.upfactors[0])
self.outconv = AO(inchannels=self.midchannels[0], outchannels=self.outchannels, upfactor=2)
self._init_params()
def _init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): #NN.BatchNorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
def forward(self, features):
x_32x = self.conv(features[3]) # 1/32
x_32 = self.conv1(x_32x)
x_16 = self.upsample(x_32) # 1/16
x_8 = self.ffm2(features[2], x_16) # 1/8
x_4 = self.ffm1(features[1], x_8) # 1/4
x_2 = self.ffm0(features[0], x_4) # 1/2
#-----------------------------------------
x = self.outconv(x_2) # original size
return x
class DepthNet(nn.Module):
__factory = {
18: Resnet.resnet18,
34: Resnet.resnet34,
50: Resnet.resnet50,
101: Resnet.resnet101,
152: Resnet.resnet152
}
def __init__(self,
backbone='resnet',
depth=50,
upfactors=None):
if upfactors is None:
upfactors = [2, 2, 2, 2]
super(DepthNet, self).__init__()
self.backbone = backbone
self.depth = depth
self.pretrained = False
self.inchannels = [256, 512, 1024, 2048]
self.midchannels = [256, 256, 256, 512]
self.upfactors = upfactors
self.outchannels = 1
# Build model
if self.backbone == 'resnet':
if self.depth not in DepthNet.__factory:
raise KeyError("Unsupported depth:", self.depth)
self.encoder = DepthNet.__factory[depth](pretrained=self.pretrained)
elif self.backbone == 'resnext101_32x8d':
self.encoder = Resnext_torch.resnext101_32x8d(pretrained=self.pretrained)
else:
self.encoder = Resnext_torch.resnext101(pretrained=self.pretrained)
def forward(self, x):
x = self.encoder(x) # 1/32, 1/16, 1/8, 1/4
return x
class FTB(nn.Module):
def __init__(self, inchannels, midchannels=512):
super(FTB, self).__init__()
self.in1 = inchannels
self.mid = midchannels
self.conv1 = nn.Conv2d(in_channels=self.in1, out_channels=self.mid, kernel_size=3, padding=1, stride=1,
bias=True)
# NN.BatchNorm2d
self.conv_branch = nn.Sequential(nn.ReLU(inplace=True), \
nn.Conv2d(in_channels=self.mid, out_channels=self.mid, kernel_size=3,
padding=1, stride=1, bias=True), \
nn.BatchNorm2d(num_features=self.mid), \
nn.ReLU(inplace=True), \
nn.Conv2d(in_channels=self.mid, out_channels=self.mid, kernel_size=3,
padding=1, stride=1, bias=True))
self.relu = nn.ReLU(inplace=True)
self.init_params()
def forward(self, x):
x = self.conv1(x)
x = x + self.conv_branch(x)
x = self.relu(x)
return x
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.BatchNorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
class ATA(nn.Module):
def __init__(self, inchannels, reduction=8):
super(ATA, self).__init__()
self.inchannels = inchannels
self.avg_pool = nn.AdaptiveAvgPool2d(1)
self.fc = nn.Sequential(nn.Linear(self.inchannels * 2, self.inchannels // reduction),
nn.ReLU(inplace=True),
nn.Linear(self.inchannels // reduction, self.inchannels),
nn.Sigmoid())
self.init_params()
def forward(self, low_x, high_x):
n, c, _, _ = low_x.size()
x = torch.cat([low_x, high_x], 1)
x = self.avg_pool(x)
x = x.view(n, -1)
x = self.fc(x).view(n, c, 1, 1)
x = low_x * x + high_x
return x
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
# init.normal(m.weight, std=0.01)
init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
# init.normal_(m.weight, std=0.01)
init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.BatchNorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
class FFM(nn.Module):
def __init__(self, inchannels, midchannels, outchannels, upfactor=2):
super(FFM, self).__init__()
self.inchannels = inchannels
self.midchannels = midchannels
self.outchannels = outchannels
self.upfactor = upfactor
self.ftb1 = FTB(inchannels=self.inchannels, midchannels=self.midchannels)
# self.ata = ATA(inchannels = self.midchannels)
self.ftb2 = FTB(inchannels=self.midchannels, midchannels=self.outchannels)
self.upsample = nn.Upsample(scale_factor=self.upfactor, mode='bilinear', align_corners=True)
self.init_params()
def forward(self, low_x, high_x):
x = self.ftb1(low_x)
x = x + high_x
x = self.ftb2(x)
x = self.upsample(x)
return x
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.Batchnorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
class AO(nn.Module):
# Adaptive output module
def __init__(self, inchannels, outchannels, upfactor=2):
super(AO, self).__init__()
self.inchannels = inchannels
self.outchannels = outchannels
self.upfactor = upfactor
self.adapt_conv = nn.Sequential(
nn.Conv2d(in_channels=self.inchannels, out_channels=self.inchannels // 2, kernel_size=3, padding=1,
stride=1, bias=True), \
nn.BatchNorm2d(num_features=self.inchannels // 2), \
nn.ReLU(inplace=True), \
nn.Conv2d(in_channels=self.inchannels // 2, out_channels=self.outchannels, kernel_size=3, padding=1,
stride=1, bias=True), \
nn.Upsample(scale_factor=self.upfactor, mode='bilinear', align_corners=True))
self.init_params()
def forward(self, x):
x = self.adapt_conv(x)
return x
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.Batchnorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
# ==============================================================================================================
class ResidualConv(nn.Module):
def __init__(self, inchannels):
super(ResidualConv, self).__init__()
# NN.BatchNorm2d
self.conv = nn.Sequential(
# nn.BatchNorm2d(num_features=inchannels),
nn.ReLU(inplace=False),
# nn.Conv2d(in_channels=inchannels, out_channels=inchannels, kernel_size=3, padding=1, stride=1, groups=inchannels,bias=True),
# nn.Conv2d(in_channels=inchannels, out_channels=inchannels, kernel_size=1, padding=0, stride=1, groups=1,bias=True)
nn.Conv2d(in_channels=inchannels, out_channels=inchannels / 2, kernel_size=3, padding=1, stride=1,
bias=False),
nn.BatchNorm2d(num_features=inchannels / 2),
nn.ReLU(inplace=False),
nn.Conv2d(in_channels=inchannels / 2, out_channels=inchannels, kernel_size=3, padding=1, stride=1,
bias=False)
)
self.init_params()
def forward(self, x):
x = self.conv(x) + x
return x
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.BatchNorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
class FeatureFusion(nn.Module):
def __init__(self, inchannels, outchannels):
super(FeatureFusion, self).__init__()
self.conv = ResidualConv(inchannels=inchannels)
# NN.BatchNorm2d
self.up = nn.Sequential(ResidualConv(inchannels=inchannels),
nn.ConvTranspose2d(in_channels=inchannels, out_channels=outchannels, kernel_size=3,
stride=2, padding=1, output_padding=1),
nn.BatchNorm2d(num_features=outchannels),
nn.ReLU(inplace=True))
def forward(self, lowfeat, highfeat):
return self.up(highfeat + self.conv(lowfeat))
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.BatchNorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
class SenceUnderstand(nn.Module):
def __init__(self, channels):
super(SenceUnderstand, self).__init__()
self.channels = channels
self.conv1 = nn.Sequential(nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1),
nn.ReLU(inplace=True))
self.pool = nn.AdaptiveAvgPool2d(8)
self.fc = nn.Sequential(nn.Linear(512 * 8 * 8, self.channels),
nn.ReLU(inplace=True))
self.conv2 = nn.Sequential(
nn.Conv2d(in_channels=self.channels, out_channels=self.channels, kernel_size=1, padding=0),
nn.ReLU(inplace=True))
self.initial_params()
def forward(self, x):
n, c, h, w = x.size()
x = self.conv1(x)
x = self.pool(x)
x = x.view(n, -1)
x = self.fc(x)
x = x.view(n, self.channels, 1, 1)
x = self.conv2(x)
x = x.repeat(1, 1, h, w)
return x
def initial_params(self, dev=0.01):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# print torch.sum(m.weight)
m.weight.data.normal_(0, dev)
if m.bias is not None:
m.bias.data.fill_(0)
elif isinstance(m, nn.ConvTranspose2d):
# print torch.sum(m.weight)
m.weight.data.normal_(0, dev)
if m.bias is not None:
m.bias.data.fill_(0)
elif isinstance(m, nn.Linear):
m.weight.data.normal_(0, dev)
if __name__ == '__main__':
net = DepthNet(depth=50, pretrained=True)
print(net)
inputs = torch.ones(4,3,128,128)
out = net(inputs)
print(out.size())
modules/control/proc/leres/pix2pix/LICENSE
+19
View File
@@ -0,0 +1,19 @@
https://github.com/compphoto/BoostingMonocularDepth
Copyright 2021, Seyed Mahdi Hosseini Miangoleh, Sebastian Dille, Computational Photography Laboratory. All rights reserved.
This software is for academic use only. A redistribution of this
software, with or without modifications, has to be for academic
use only, while giving the appropriate credit to the original
authors of the software. The methods implemented as a part of
this software may be covered under patents or patent applications.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR ''AS IS'' AND ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
modules/control/proc/leres/pix2pix/__init__.py
View File
modules/control/proc/leres/pix2pix/models/__init__.py
+67
View File
@@ -0,0 +1,67 @@
"""This package contains modules related to objective functions, optimizations, and network architectures.
To add a custom model class called 'dummy', you need to add a file called 'dummy_model.py' and define a subclass DummyModel inherited from BaseModel.
You need to implement the following five functions:
-- <__init__>: initialize the class; first call BaseModel.__init__(self, opt).
-- <set_input>: unpack data from dataset and apply preprocessing.
-- <forward>: produce intermediate results.
-- <optimize_parameters>: calculate loss, gradients, and update network weights.
-- <modify_commandline_options>: (optionally) add model-specific options and set default options.
In the function <__init__>, you need to define four lists:
-- self.loss_names (str list): specify the training losses that you want to plot and save.
-- self.model_names (str list): define networks used in our training.
-- self.visual_names (str list): specify the images that you want to display and save.
-- self.optimizers (optimizer list): define and initialize optimizers. You can define one optimizer for each network. If two networks are updated at the same time, you can use itertools.chain to group them. See cycle_gan_model.py for an usage.
Now you can use the model class by specifying flag '--model dummy'.
See our template model class 'template_model.py' for more details.
"""
import importlib
from .base_model import BaseModel
def find_model_using_name(model_name):
"""Import the module "models/[model_name]_model.py".
In the file, the class called DatasetNameModel() will
be instantiated. It has to be a subclass of BaseModel,
and it is case-insensitive.
"""
model_filename = "modules.control.proc.leres.pix2pix.models." + model_name + "_model"
modellib = importlib.import_module(model_filename)
model = None
target_model_name = model_name.replace('_', '') + 'model'
for name, cls in modellib.__dict__.items():
if name.lower() == target_model_name.lower() \
and issubclass(cls, BaseModel):
model = cls
if model is None:
print("In %s.py, there should be a subclass of BaseModel with class name that matches %s in lowercase." % (model_filename, target_model_name))
exit(0)
return model
def get_option_setter(model_name):
"""Return the static method <modify_commandline_options> of the model class."""
model_class = find_model_using_name(model_name)
return model_class.modify_commandline_options
def create_model(opt):
"""Create a model given the option.
This function warps the class CustomDatasetDataLoader.
This is the main interface between this package and 'train.py'/'test.py'
Example:
>>> from models import create_model
>>> model = create_model(opt)
"""
model = find_model_using_name(opt.model)
instance = model(opt)
print("model [%s] was created" % type(instance).__name__)
return instance
modules/control/proc/leres/pix2pix/models/base_model.py
+242
View File
@@ -0,0 +1,242 @@
import gc
import os
from abc import ABC, abstractmethod
from collections import OrderedDict
import torch
from modules.control.util import torch_gc
from . import networks
class BaseModel(ABC):
"""This class is an abstract base class (ABC) for models.
To create a subclass, you need to implement the following five functions:
-- <__init__>: initialize the class; first call BaseModel.__init__(self, opt).
-- <set_input>: unpack data from dataset and apply preprocessing.
-- <forward>: produce intermediate results.
-- <optimize_parameters>: calculate losses, gradients, and update network weights.
-- <modify_commandline_options>: (optionally) add model-specific options and set default options.
"""
def __init__(self, opt):
"""Initialize the BaseModel class.
Parameters:
opt (Option class)-- stores all the experiment flags; needs to be a subclass of BaseOptions
When creating your custom class, you need to implement your own initialization.
In this function, you should first call <BaseModel.__init__(self, opt)>
Then, you need to define four lists:
-- self.loss_names (str list): specify the training losses that you want to plot and save.
-- self.model_names (str list): define networks used in our training.
-- self.visual_names (str list): specify the images that you want to display and save.
-- self.optimizers (optimizer list): define and initialize optimizers. You can define one optimizer for each network. If two networks are updated at the same time, you can use itertools.chain to group them. See cycle_gan_model.py for an example.
"""
self.opt = opt
self.gpu_ids = opt.gpu_ids
self.isTrain = opt.isTrain
self.device = torch.device('cuda:{}'.format(self.gpu_ids[0])) if self.gpu_ids else torch.device('cpu') # get device name: CPU or GPU
self.save_dir = os.path.join(opt.checkpoints_dir, opt.name) # save all the checkpoints to save_dir
if opt.preprocess != 'scale_width': # with [scale_width], input images might have different sizes, which hurts the performance of cudnn.benchmark.
torch.backends.cudnn.benchmark = True
self.loss_names = []
self.model_names = []
self.visual_names = []
self.optimizers = []
self.image_paths = []
self.metric = 0 # used for learning rate policy 'plateau'
@staticmethod
def modify_commandline_options(parser, is_train):
"""Add new model-specific options, and rewrite default values for existing options.
Parameters:
parser -- original option parser
is_train (bool) -- whether training phase or test phase. You can use this flag to add training-specific or test-specific options.
Returns:
the modified parser.
"""
return parser
@abstractmethod
def set_input(self, input):
"""Unpack input data from the dataloader and perform necessary pre-processing steps.
Parameters:
input (dict): includes the data itself and its metadata information.
"""
pass
@abstractmethod
def forward(self):
"""Run forward pass; called by both functions <optimize_parameters> and <test>."""
pass
@abstractmethod
def optimize_parameters(self):
"""Calculate losses, gradients, and update network weights; called in every training iteration"""
pass
def setup(self, opt):
"""Load and print networks; create schedulers
Parameters:
opt (Option class) -- stores all the experiment flags; needs to be a subclass of BaseOptions
"""
if self.isTrain:
self.schedulers = [networks.get_scheduler(optimizer, opt) for optimizer in self.optimizers]
if not self.isTrain or opt.continue_train:
load_suffix = 'iter_%d' % opt.load_iter if opt.load_iter > 0 else opt.epoch
self.load_networks(load_suffix)
self.print_networks(opt.verbose)
def eval(self):
"""Make models eval mode during test time"""
for name in self.model_names:
if isinstance(name, str):
net = getattr(self, 'net' + name)
net.eval()
def test(self):
"""Forward function used in test time.
It also calls <compute_visuals> to produce additional visualization results
"""
self.forward()
self.compute_visuals()
def compute_visuals(self): # noqa
"""Calculate additional output images for visdom and HTML visualization"""
pass
def get_image_paths(self):
""" Return image paths that are used to load current data"""
return self.image_paths
def update_learning_rate(self):
"""Update learning rates for all the networks; called at the end of every epoch"""
old_lr = self.optimizers[0].param_groups[0]['lr']
for scheduler in self.schedulers:
if self.opt.lr_policy == 'plateau':
scheduler.step(self.metric)
else:
scheduler.step()
lr = self.optimizers[0].param_groups[0]['lr']
print('learning rate %.7f -> %.7f' % (old_lr, lr))
def get_current_visuals(self):
"""Return visualization images. train.py will display these images with visdom, and save the images to a HTML"""
visual_ret = OrderedDict()
for name in self.visual_names:
if isinstance(name, str):
visual_ret[name] = getattr(self, name)
return visual_ret
def get_current_losses(self):
"""Return traning losses / errors. train.py will print out these errors on console, and save them to a file"""
errors_ret = OrderedDict()
for name in self.loss_names:
if isinstance(name, str):
errors_ret[name] = float(getattr(self, 'loss_' + name)) # float(...) works for both scalar tensor and float number
return errors_ret
def save_networks(self, epoch):
"""Save all the networks to the disk.
Parameters:
epoch (int) -- current epoch; used in the file name '%s_net_%s.pth' % (epoch, name)
"""
for name in self.model_names:
if isinstance(name, str):
save_filename = '%s_net_%s.pth' % (epoch, name)
save_path = os.path.join(self.save_dir, save_filename)
net = getattr(self, 'net' + name)
if len(self.gpu_ids) > 0 and torch.cuda.is_available():
torch.save(net.module.cpu().state_dict(), save_path)
net.cuda(self.gpu_ids[0])
else:
torch.save(net.cpu().state_dict(), save_path)
def unload_network(self, name):
"""Unload network and gc.
"""
if isinstance(name, str):
net = getattr(self, 'net' + name)
del net
gc.collect()
torch_gc()
return None
def __patch_instance_norm_state_dict(self, state_dict, module, keys, i=0):
"""Fix InstanceNorm checkpoints incompatibility (prior to 0.4)"""
key = keys[i]
if i + 1 == len(keys): # at the end, pointing to a parameter/buffer
if module.__class__.__name__.startswith('InstanceNorm') and \
(key == 'running_mean' or key == 'running_var'):
if getattr(module, key) is None:
state_dict.pop('.'.join(keys))
if module.__class__.__name__.startswith('InstanceNorm') and \
(key == 'num_batches_tracked'):
state_dict.pop('.'.join(keys))
else:
self.__patch_instance_norm_state_dict(state_dict, getattr(module, key), keys, i + 1)
def load_networks(self, epoch):
"""Load all the networks from the disk.
Parameters:
epoch (int) -- current epoch; used in the file name '%s_net_%s.pth' % (epoch, name)
"""
for name in self.model_names:
if isinstance(name, str):
load_filename = '%s_net_%s.pth' % (epoch, name)
load_path = os.path.join(self.save_dir, load_filename)
net = getattr(self, 'net' + name)
if isinstance(net, torch.nn.DataParallel):
net = net.module
# print('Loading depth boost model from %s' % load_path)
# if you are using PyTorch newer than 0.4 (e.g., built from
# GitHub source), you can remove str() on self.device
state_dict = torch.load(load_path, map_location=str(self.device))
if hasattr(state_dict, '_metadata'):
del state_dict._metadata
# patch InstanceNorm checkpoints prior to 0.4
for key in list(state_dict.keys()): # need to copy keys here because we mutate in loop
self.__patch_instance_norm_state_dict(state_dict, net, key.split('.'))
net.load_state_dict(state_dict)
def print_networks(self, verbose):
"""Print the total number of parameters in the network and (if verbose) network architecture
Parameters:
verbose (bool) -- if verbose: print the network architecture
"""
print('---------- Networks initialized -------------')
for name in self.model_names:
if isinstance(name, str):
net = getattr(self, 'net' + name)
num_params = 0
for param in net.parameters():
num_params += param.numel()
if verbose:
print(net)
print('[Network %s] Total number of parameters : %.3f M' % (name, num_params / 1e6))
print('-----------------------------------------------')
def set_requires_grad(self, nets, requires_grad=False):
"""Set requies_grad=Fasle for all the networks to avoid unnecessary computations
Parameters:
nets (network list) -- a list of networks
requires_grad (bool) -- whether the networks require gradients or not
"""
if not isinstance(nets, list):
nets = [nets]
for net in nets:
if net is not None:
for param in net.parameters():
param.requires_grad = requires_grad

Some files were not shown because too many files have changed in this diff Show More