Merge branch 'master' into ipex-native-win

CHANGELOG.md

@@ -1,5 +1,51 @@
 # Change Log for SD.Next
 
+## Update for 2023-08-14
+
+- general:
+  - update all metadata saved with images  
+    see <https://github.com/vladmandic/automatic/wiki/Metadata> for details  
+    (work-in-progress)
+  - improved **amd** installer with support for **navi 2x & 3x** and **rocm 5.4/5.5/5.6**  
+    thanks @evshiron  
+  - fix img2img resizing (applies to original, diffusers, hires)  
+- diffusers:
+  - enable batch img2img workflows
+- original:
+  - new samplers: **dpm++ 3M sde** (standard and karras variations)  
+    enable in *settings -> samplers -> show samplers*
+
+## Update for 2023-08-11
+
+This is a big one that's been cooking in `dev` for a while now, but finally ready for release...
+
+- diffusers:
+  - **pipeline autodetect**
+    if pipeline is set to autodetect (default for new installs), app will try to autodetect pipeline based on selected model  
+    this should reduce user errors such as loading **sd-xl** model when **sd** pipeline is selected  
+  - **quick vae decode** as alternative to full vae decode which is very resource intensive  
+    quick decode is based on `taesd` and produces lower quality, but its great for tests or grids as it runs much faster and uses far less vram  
+    disabled by default, selectable in *txt2img/img2img -> advanced -> full quality*  
+  - **prompt attention** for sd and sd-xl  
+    supports both `full parser` and native `compel`  
+    thanks @ai-casanova  
+  - advanced **lora load/apply** methods  
+    in addition to standard lora loading that was recently added to sd-xl using diffusers, now we have  
+    - **sequential apply** (load & apply multiple loras in sequential manner) and  
+    - **merge and apply** (load multiple loras and merge before applying to model)  
+    see *settings -> diffusers -> lora methods*  
+    thanks @hameerabbasi and @ai-casanova  
+  - **sd-xl vae** from safetensors now applies correct config  
+    result is that 3rd party vaes can be used without washed out colors  
+  - options for optimized memory handling for lower memory usage  
+    see *settings -> diffusers*
+- general:
+  - new **civitai model search and download**  
+    native support for civitai, integrated into ui as *models -> civitai*  
+  - updated requirements  
+    this time its a bigger change so upgrade may take longer to install new requirements
+  - improved **extra networks** performance with large number of networks
+
 ## Update for 2023-08-05
 
 Another minor update, but it unlocks some cool new items...
@@ -11,7 +57,6 @@ Another minor update, but it unlocks some cool new items...
   - new torch 2.0 with ipex (intel arc)  
   - additional callbacks for extensions  
     enables latest comfyui extension  
-  - update requirements  
 
 ## Update for 2023-07-30
 

README.md

@@ -71,15 +71,15 @@ Additional models will be added as they become available and there is public int
 - *Intel Arc* GPUs using *Intel OneAPI* **Ipex/XPU** libraries  
 - *Apple M1/M2* on *OSX* using built-in support in Torch with **MPS** optimizations
 
-## [Installation Instructions](https://github.com/vladmandic/automatic/wiki/Installation)
+## Install & Run
 
-### Common Problems
-
-- [Common Installation Errors ](https://github.com/vladmandic/automatic/discussions/1627)  
-- [Q&A Discussions](https://github.com/vladmandic/automatic/discussions/1011)
+- [Step-by-step install guide](https://github.com/vladmandic/automatic/wiki/Installation)
+- [Advanced install notes](https://github.com/vladmandic/automatic/wiki/Advanced-Install)
 
 ### Installation Notes
 
+- [Common installation errors](https://github.com/vladmandic/automatic/discussions/1627)  
+- [FAQ](https://github.com/vladmandic/automatic/discussions/1011)
 - Server can run without virtual environment,  
   but it is recommended to use it to avoid library version conflicts with other applications  
 - **nVidia/CUDA** and **AMD/ROCm** are auto-detected if present and available,  
@@ -87,6 +87,10 @@ Additional models will be added as they become available and there is public int
   as installer will assume CPU-only environment  
 - Full startup sequence is logged in `sdnext.log`, so if you encounter any issues, please check it first  
 
+### Run
+
+Once SD.Next is installed, simply run `webui.bat` (*Windows*) or `webui.sh` (*Linux or MacOS*)
+
 Below is partial list of all available parameters, run `webui --help` for the full list:
 
     Setup options:

TODO.md

@@ -26,6 +26,7 @@ Stuff to be added, in no particular order...
   - Port `p.all_hr_prompts`
   - Import core repos to reduce dependencies
   - Update `gradio`
+  - Parse StabilityAI `modelspec` metadata
 - Non-technical:
   - Create additional themes
   - Update Wiki
@@ -41,6 +42,7 @@ Stuff to be added, in no particular order...
   - Style editor (use json format instead of csv)
   - Profile manager (for config.json and ui-config.json)
   - Multi-user support
+  - Add [SAG](https://huggingface.co/docs/diffusers/v0.19.3/en/api/pipelines/self_attention_guidance),(https://github.com/ashen-sensored/sd_webui_SAG)
   - Image phash and hdash using `imagehash`
   - Model merge using `git-rebasin`
   - Enable refiner-style workflow for `ldm` backend

cli/idle.py

@@ -0,0 +1,60 @@
+#!/usr/bin/env python
+
+import os
+import time
+import datetime
+import logging
+import urllib3
+import requests
+
+class Dot(dict):
+    __getattr__ = dict.get
+    __setattr__ = dict.__setitem__
+    __delattr__ = dict.__delitem__
+
+opts = Dot({
+    "timeout": 3600,
+    "frequency": 60,
+    "action": "sudo shutdown now",
+    "url": "https://127.0.0.1:7860",
+    "user": "",
+    "password": "",
+})
+
+log_format = '%(asctime)s %(levelname)s: %(message)s'
+logging.basicConfig(level = logging.INFO, format = log_format)
+log = logging.getLogger("sd")
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+status = None
+
+def progress():
+    auth = requests.auth.HTTPBasicAuth(opts.user, opts.password) if opts.user is not None and len(opts.user) > 0 and opts.password is not None and len(opts.password) > 0 else None
+    req = requests.get(f'{opts.url}/sdapi/v1/progress?skip_current_image=true', verify=False, auth=auth, timeout=60)
+    if req.status_code != 200:
+        log.error({ 'url': req.url, 'request': req.status_code, 'reason': req.reason })
+        return status
+    else:
+        res = Dot(req.json())
+        log.debug({ 'url': req.url, 'request': req.status_code, 'result': res })
+        return res
+
+log.info(f'sdnext monitor started: {opts}')
+while True:
+    try:
+        status = progress()
+        state = status.get('state', {})
+        last_job = state.get('job_timestamp', None)
+        if last_job is None:
+            log.warning(f'sdnext montoring cannot get last job info: {status}')
+        else:
+            last_job = datetime.datetime.strptime(last_job, "%Y%m%d%H%M%S")
+            elapsed = datetime.datetime.now() - last_job
+            timeout = round(opts.timeout - elapsed.total_seconds())
+            log.info(f'sdnext: last_job={last_job} elapsed={elapsed} timeout={timeout}')
+            if timeout < 0:
+                log.warning(f'sdnext reached: timeout={opts.timeout} action={opts.action}')
+                os.system(opts.action)
+    except Exception as e:
+        log.error(f'sdnext monitor error: {e}')
+    finally:
+        time.sleep(opts.frequency)

cli/simple-img2img.py

@@ -1,19 +1,24 @@
 #!/usr/bin/env python
+import os
 import io
 import sys
 import base64
 import logging
 import requests
+import urllib3
 from PIL import Image
 
+sd_url = os.environ.get('SDAPI_URL', "http://127.0.0.1:7860")
+sd_username = os.environ.get('SDAPI_USR', None)
+sd_password = os.environ.get('SDAPI_PWD', None)
+
 logging.basicConfig(level = logging.INFO, format = '%(asctime)s %(levelname)s: %(message)s')
 log = logging.getLogger(__name__)
-sd_url = "http://127.0.0.1:7860"
 options = {
     "init_images": [],
     "prompt": "city at night",
     "negative_prompt": "foggy, blurry",
-    "steps": 1,
+    "steps": 20,
     "batch_size": 1,
     "n_iter": 1,
     "seed": -1,
@@ -24,9 +29,17 @@ options = {
     "save_images": False,
     "send_images": True,
 }
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+
+
+def auth():
+    if sd_username is not None and sd_password is not None:
+        return requests.auth.HTTPBasicAuth(sd_username, sd_password)
+    return None
+
 
 def post(endpoint: str, dct: dict = None):
-    req = requests.post(f'{sd_url}{endpoint}', json = dct, timeout=300)
+    req = requests.post(f'{sd_url}{endpoint}', json = dct, timeout=300, verify=False, auth=auth())
     if req.status_code != 200:
         return { 'error': req.status_code, 'reason': req.reason, 'url': req.url }
     else:
@@ -44,7 +57,8 @@ def encode(f):
 
 def generate(num: int = 0):
     log.info(f'sending generate request: {num+1} {options}')
-    options['init_images'] = [encode('../html/logo.png')]
+    options['init_images'] = [encode('html/logo-dark.png')]
+    options['batch_size'] = len(options['init_images'])
     data = post('/sdapi/v1/img2img', options)
     if 'images' in data:
         for i in range(len(data['images'])):

cli/simple-txt2img.py

@@ -5,6 +5,7 @@ import sys
 import base64
 import logging
 import requests
+import urllib3
 from PIL import Image
 
 sd_url = os.environ.get('SDAPI_URL', "http://127.0.0.1:7860")
@@ -13,6 +14,7 @@ sd_password = os.environ.get('SDAPI_PWD', None)
 
 logging.basicConfig(level = logging.INFO, format = '%(asctime)s %(levelname)s: %(message)s')
 log = logging.getLogger(__name__)
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 
 filename='/tmp/simple-txt2img.jpg'
 model = None # desired model name, will be set if not none
@@ -31,6 +33,7 @@ options = {
     "send_images": True,
 }
 
+
 def auth():
     if sd_username is not None and sd_password is not None:
         return requests.auth.HTTPBasicAuth(sd_username, sd_password)

cli/train.py

@@ -375,7 +375,7 @@ def check_versions():
     log.info('checking accelerate')
     error = False
     import accelerate
-    if accelerate.__version__ != '0.19.0':
+    if accelerate.__version__ != '0.20.3':
         log.error(f'invalid accelerate version: accelerate=0.19.0 found={accelerate.__version__}')
         error = True
     log.info('checking diffusers')
@@ -384,7 +384,7 @@ def check_versions():
         log.error(f'invalid diffusers version: diffusers=0.10.2 found={diffusers.__version__}')
         error = True
     if error:
-        log.info('> pip install accelerate==0.19.0 diffusers==0.10.2')
+        log.info('> pip install accelerate==0.20.3 diffusers==0.10.2')
         exit(1)
 
 

configs/sd_xl_base.yaml

@@ -0,0 +1,98 @@
+model:
+  target: sgm.models.diffusion.DiffusionEngine
+  params:
+    scale_factor: 0.13025
+    disable_first_stage_autocast: True
+
+    denoiser_config:
+      target: sgm.modules.diffusionmodules.denoiser.DiscreteDenoiser
+      params:
+        num_idx: 1000
+
+        weighting_config:
+          target: sgm.modules.diffusionmodules.denoiser_weighting.EpsWeighting
+        scaling_config:
+          target: sgm.modules.diffusionmodules.denoiser_scaling.EpsScaling
+        discretization_config:
+          target: sgm.modules.diffusionmodules.discretizer.LegacyDDPMDiscretization
+
+    network_config:
+      target: sgm.modules.diffusionmodules.openaimodel.UNetModel
+      params:
+        adm_in_channels: 2816
+        num_classes: sequential
+        use_checkpoint: True
+        in_channels: 4
+        out_channels: 4
+        model_channels: 320
+        attention_resolutions: [4, 2]
+        num_res_blocks: 2
+        channel_mult: [1, 2, 4]
+        num_head_channels: 64
+        use_spatial_transformer: True
+        use_linear_in_transformer: True
+        transformer_depth: [1, 2, 10]  # note: the first is unused (due to attn_res starting at 2) 32, 16, 8 --> 64, 32, 16
+        context_dim: 2048
+        spatial_transformer_attn_type: softmax-xformers
+        legacy: False
+
+    conditioner_config:
+      target: sgm.modules.GeneralConditioner
+      params:
+        emb_models:
+          # crossattn cond
+          - is_trainable: False
+            input_key: txt
+            target: sgm.modules.encoders.modules.FrozenCLIPEmbedder
+            params:
+              layer: hidden
+              layer_idx: 11
+          # crossattn and vector cond
+          - is_trainable: False
+            input_key: txt
+            target: sgm.modules.encoders.modules.FrozenOpenCLIPEmbedder2
+            params:
+              arch: ViT-bigG-14
+              version: laion2b_s39b_b160k
+              freeze: True
+              layer: penultimate
+              always_return_pooled: True
+              legacy: False
+          # vector cond
+          - is_trainable: False
+            input_key: original_size_as_tuple
+            target: sgm.modules.encoders.modules.ConcatTimestepEmbedderND
+            params:
+              outdim: 256  # multiplied by two
+          # vector cond
+          - is_trainable: False
+            input_key: crop_coords_top_left
+            target: sgm.modules.encoders.modules.ConcatTimestepEmbedderND
+            params:
+              outdim: 256  # multiplied by two
+          # vector cond
+          - is_trainable: False
+            input_key: target_size_as_tuple
+            target: sgm.modules.encoders.modules.ConcatTimestepEmbedderND
+            params:
+              outdim: 256  # multiplied by two
+
+    first_stage_config:
+      target: sgm.models.autoencoder.AutoencoderKLInferenceWrapper
+      params:
+        embed_dim: 4
+        monitor: val/rec_loss
+        ddconfig:
+          attn_type: vanilla-xformers
+          double_z: true
+          z_channels: 4
+          resolution: 256
+          in_channels: 3
+          out_ch: 3
+          ch: 128
+          ch_mult: [1, 2, 4, 4]
+          num_res_blocks: 2
+          attn_resolutions: []
+          dropout: 0.0
+        lossconfig:
+          target: torch.nn.Identity

configs/sd_xl_refiner.yaml

@@ -0,0 +1,91 @@
+model:
+  target: sgm.models.diffusion.DiffusionEngine
+  params:
+    scale_factor: 0.13025
+    disable_first_stage_autocast: True
+
+    denoiser_config:
+      target: sgm.modules.diffusionmodules.denoiser.DiscreteDenoiser
+      params:
+        num_idx: 1000
+
+        weighting_config:
+          target: sgm.modules.diffusionmodules.denoiser_weighting.EpsWeighting
+        scaling_config:
+          target: sgm.modules.diffusionmodules.denoiser_scaling.EpsScaling
+        discretization_config:
+          target: sgm.modules.diffusionmodules.discretizer.LegacyDDPMDiscretization
+
+    network_config:
+      target: sgm.modules.diffusionmodules.openaimodel.UNetModel
+      params:
+        adm_in_channels: 2560
+        num_classes: sequential
+        use_checkpoint: True
+        in_channels: 4
+        out_channels: 4
+        model_channels: 384
+        attention_resolutions: [4, 2]
+        num_res_blocks: 2
+        channel_mult: [1, 2, 4, 4]
+        num_head_channels: 64
+        use_spatial_transformer: True
+        use_linear_in_transformer: True
+        transformer_depth: 4
+        context_dim: [1280, 1280, 1280, 1280]  # 1280
+        spatial_transformer_attn_type: softmax-xformers
+        legacy: False
+
+    conditioner_config:
+      target: sgm.modules.GeneralConditioner
+      params:
+        emb_models:
+          # crossattn and vector cond
+          - is_trainable: False
+            input_key: txt
+            target: sgm.modules.encoders.modules.FrozenOpenCLIPEmbedder2
+            params:
+              arch: ViT-bigG-14
+              version: laion2b_s39b_b160k
+              legacy: False
+              freeze: True
+              layer: penultimate
+              always_return_pooled: True
+          # vector cond
+          - is_trainable: False
+            input_key: original_size_as_tuple
+            target: sgm.modules.encoders.modules.ConcatTimestepEmbedderND
+            params:
+              outdim: 256  # multiplied by two
+          # vector cond
+          - is_trainable: False
+            input_key: crop_coords_top_left
+            target: sgm.modules.encoders.modules.ConcatTimestepEmbedderND
+            params:
+              outdim: 256  # multiplied by two
+          # vector cond
+          - is_trainable: False
+            input_key: aesthetic_score
+            target: sgm.modules.encoders.modules.ConcatTimestepEmbedderND
+            params:
+              outdim: 256  # multiplied by one
+
+    first_stage_config:
+      target: sgm.models.autoencoder.AutoencoderKLInferenceWrapper
+      params:
+        embed_dim: 4
+        monitor: val/rec_loss
+        ddconfig:
+          attn_type: vanilla-xformers
+          double_z: true
+          z_channels: 4
+          resolution: 256
+          in_channels: 3
+          out_ch: 3
+          ch: 128
+          ch_mult: [1, 2, 4, 4]
+          num_res_blocks: 2
+          attn_resolutions: []
+          dropout: 0.0
+        lossconfig:
+          target: torch.nn.Identity

html/locale_en.json

@@ -562,21 +562,20 @@
   {"id":"","label":"Token merging ratio","localized":"","hint":"Enable redundant token merging via tomesd for speed and memory improvements, 0=disabled"},
   {"id":"","label":"Token merging ratio for img2img","localized":"","hint":"Enable redundant token merging for img2img via tomesd for speed and memory improvements, 0=disabled"},
   {"id":"","label":"Token merging ratio for hires pass","localized":"","hint":"Enable redundant token merging for hires pass via tomesd for speed and memory improvements, 0=disabled"},
-  {"id":"","label":"Diffusers allow loading from safetensors files","localized":"","hint":"Allow loading of safetensors files as diffuser models"},
   {"id":"","label":"Select diffuser pipeline when loading from safetensors","localized":"","hint":""},
   {"id":"","label":"Move base model to CPU when using refiner","localized":"","hint":""},
   {"id":"","label":"Move refiner model to CPU when not in use","localized":"","hint":""},
   {"id":"","label":"Move UNet to CPU while VAE decoding","localized":"","hint":""},
   {"id":"","label":"Use model EMA weights when possible","localized":"","hint":""},
   {"id":"","label":"Generator device","localized":"","hint":""},
-  {"id":"","label":"Enable sequential CPU offload","localized":"","hint":"Reduces GPU memory usage by transferring weights to the CPU. Increases inference time approximately 10%. Use with Enable Attention slicing for minimal memory consumption"},
-  {"id":"","label":"Enable model CPU offload","localized":"","hint":"Transferring of entire models to the CPU, negligible impact on inference time while still providing some memory savings. Use with Enable Attention slicing for additional memory savings"},
-  {"id":"","label":"Enable VAE slicing","localized":"","hint":"Decodes batch latents one image at a time with limited VRAM. Small performance boost in VAE decode on multi-image batches. Use with Enable Attention slicing"},
-  {"id":"","label":"Enable VAE tiling","localized":"","hint":"Divide large images into overlapping tiles with limited VRAM. Might result in a minor increase in processing time. Use with Enable Attention Slicing"},
-  {"id":"","label":"Enable attention slicing","localized":"","hint":"Performs attention computation in steps instead of all at once. 10% slower inference times. Greatly reduces memory usage. Best used, period"},
+  {"id":"","label":"Enable sequential CPU offload","localized":"","hint":"Reduces GPU memory usage by transferring weights to the CPU. Increases inference time approximately 10%"},
+  {"id":"","label":"Enable model CPU offload","localized":"","hint":"Transferring of entire models to the CPU, negligible impact on inference time while still providing some memory savings"},
+  {"id":"","label":"Enable VAE slicing","localized":"","hint":"Decodes batch latents one image at a time with limited VRAM. Small performance boost in VAE decode on multi-image batches"},
+  {"id":"","label":"Enable VAE tiling","localized":"","hint":"Divide large images into overlapping tiles with limited VRAM. Results in a minor increase in processing time"},
+  {"id":"","label":"Enable attention slicing","localized":"","hint":"Performs attention computation in steps instead of all at once. Slower inference times, but greatly reduced memory usage"},
   {"id":"","label":"Diffusers model loading variant","localized":"","hint":""},
-  {"id":"","label":"Diffusers VAE loading variant","localized":"","hint":""}
-
+  {"id":"","label":"Diffusers VAE loading variant","localized":"","hint":""},
+  {"id":"","label":"Diffusers LoRA loading variant","localized":"","hint":"'sequential apply' loads and applies each LoRA in order of appearance, 'merge and apply' loads all LoRAs and merges them in-memory before applying to model, 'diffusers default' uses single LoRA loading method"}
 ],
 "scripts": [
   {"id":"","label":"Script","localized":"","hint":""},

html/locale_ko.json

@@ -562,7 +562,6 @@
   {"id":"","label":"Token merging ratio","localized":"토큰 병합 비율","hint":"속도와 메모리 절감을 위해 tomesd를 사용해 토큰 병합을 활성화한다. (0이면 비활성화)"},
   {"id":"","label":"Token merging ratio for img2img","localized":"이미지➠이미지 토큰 병합 비율","hint":"속도와 메모리 절감을 위해 이미지➠이미지에서 tomesd를 사용해 토큰 병합을 활성화한다. (0이면 비활성화)"},
   {"id":"","label":"Token merging ratio for hires pass","localized":"텍스트➠이미지 업스케일링(Hires fix) 토큰 병합 비율","hint":"속도와 메모리 절감을 위해 Hires fix에서 tomesd를 사용해 토큰 병합을 활성화한다. (0이면 비활성화)"},
-  {"id":"","label":"Diffusers allow loading from safetensors files","localized":"safetensors 파일에서 로드 허용","hint":"safetensors 파일을 Diffusers 모델로 로드할 수 있게 한다."},
   {"id":"","label":"Select diffuser pipeline when loading from safetensors","localized":"safetensors 파일에서 로드할 때 사용할 파이프라인 선택","hint":""},
   {"id":"","label":"Move base model to CPU when using refiner","localized":"리파이너를 사용 중일 때 base 모델을 CPU로 이동","hint":""},
   {"id":"","label":"Move refiner model to CPU when not in use","localized":"사용 중이지 않을 때 리파이너 모델을 CPU로 이동","hint":""},

installer.py

@@ -81,7 +81,7 @@ def setup_logging():
     }))
     logging.basicConfig(level=logging.ERROR, format='%(asctime)s | %(name)s | %(levelname)s | %(module)s | %(message)s', handlers=[logging.NullHandler()]) # redirect default logger to null
     pretty_install(console=console)
-    traceback_install(console=console, extra_lines=1, width=console.width, word_wrap=False, indent_guides=False, suppress=[])
+    traceback_install(console=console, extra_lines=1, max_frames=10, width=console.width, word_wrap=False, indent_guides=False, suppress=[])
     while log.hasHandlers() and len(log.handlers) > 0:
         log.removeHandler(log.handlers[0])
 
@@ -166,8 +166,9 @@ def pip(arg: str, ignore: bool = False, quiet: bool = False):
     arg = arg.replace('>=', '==')
     if not quiet:
         log.info(f'Installing package: {arg.replace("install", "").replace("--upgrade", "").replace("--no-deps", "").replace("--force", "").replace("  ", " ").strip()}')
-    log.debug(f"Running pip: {arg}")
-    result = subprocess.run(f'"{sys.executable}" -m pip {arg}', shell=True, check=False, env=os.environ, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    env_args = os.environ.get("PIP_EXTRA_ARGS", "")
+    log.debug(f"Running pip: {arg} {env_args}")
+    result = subprocess.run(f'"{sys.executable}" -m pip {arg} {env_args}', shell=True, check=False, env=os.environ, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
     txt = result.stdout.decode(encoding="utf8", errors="ignore")
     if len(result.stderr) > 0:
         txt += ('\n' if len(txt) > 0 else '') + result.stderr.decode(encoding="utf8", errors="ignore")
@@ -339,7 +340,56 @@ def check_torch():
         log.info('AMD ROCm toolkit detected')
         os.environ.setdefault('PYTORCH_HIP_ALLOC_CONF', 'garbage_collection_threshold:0.8,max_split_size_mb:512')
         os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow-rocm')
-        torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/rocm5.4.2')
+
+        try:
+            command = subprocess.run('rocm_agent_enumerator', shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            amd_gpus = command.stdout.decode(encoding="utf8", errors="ignore").split('\n')
+            amd_gpus = [x for x in amd_gpus if x and x != 'gfx000']
+            log.debug(f'ROCm agents detected: {amd_gpus}')
+        except Exception as e:
+            log.debug(f'Run rocm_agent_enumerator failed: {e}')
+            amd_gpus = []
+
+        # use the first available amd gpu by default
+        hip_visible_devices = []
+        for idx, gpu in enumerate(amd_gpus):
+            if gpu in ['gfx1100', 'gfx1101', 'gfx1102']:
+                hip_visible_devices.append((idx, gpu, 'navi3x'))
+                break
+            # experimental navi 2x support
+            if gpu in ['gfx1030', 'gfx1031', 'gfx1032', 'gfx1034']:
+                hip_visible_devices.append((idx, gpu, 'navi2x'))
+                break
+        if len(hip_visible_devices) > 0:
+            idx, gpu, arch = hip_visible_devices[0]
+            log.debug(f'ROCm agent used by default: idx={idx} gpu={gpu} arch={arch}')
+
+            os.environ.setdefault('HIP_VISIBLE_DEVICES', str(idx))
+            if arch == 'navi3x':
+                os.environ.setdefault('HSA_OVERRIDE_GFX_VERSION', '11.0.0')
+                # do not use tensorflow-rocm for navi 3x
+                if os.environ.get('TENSORFLOW_PACKAGE') == 'tensorflow-rocm':
+                    os.environ['TENSORFLOW_PACKAGE'] = 'tensorflow==2.13.0'
+            elif arch == 'navi2x':
+                os.environ.setdefault('HSA_OVERRIDE_GFX_VERSION', '10.3.0')
+            else:
+                log.debug(f'HSA_OVERRIDE_GFX_VERSION auto config is skipped for {gpu}')
+
+        try:
+            command = subprocess.run('hipconfig --version', shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            major_ver, minor_ver, *_ = command.stdout.decode(encoding="utf8", errors="ignore").split('.')
+            rocm_ver = f'{major_ver}.{minor_ver}'
+            log.debug(f'ROCm version detected: {rocm_ver}')
+        except Exception as e:
+            log.debug(f'Run hipconfig failed: {e}')
+            rocm_ver = None
+
+        if rocm_ver in ['5.5', '5.6']:
+            # install torch nightly via torchvision to avoid wasting bandwidth when torchvision depends on torch from yesterday
+            torch_command = os.environ.get('TORCH_COMMAND', f'torchvision --pre --index-url https://download.pytorch.org/whl/nightly/rocm{rocm_ver}')
+        else:
+            torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/rocm5.4.2')
+
         xformers_package = os.environ.get('XFORMERS_PACKAGE', 'none')
     elif allow_ipex and (args.use_ipex or shutil.which('sycl-ls') is not None or shutil.which('sycl-ls.exe') is not None or os.environ.get('ONEAPI_ROOT') is not None or os.path.exists('/opt/intel/oneapi') or os.path.exists("C:/Program Files (x86)/Intel/oneAPI") or os.path.exists("C:/oneAPI")):
         args.use_ipex = True # pylint: disable=attribute-defined-outside-init
@@ -349,10 +399,10 @@ def check_torch():
         os.environ.setdefault('NEOReadDebugKeys', '1')
         os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
         if "linux" in sys.platform:
-            torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.0.1a0 torchvision==0.15.2a0 intel_extension_for_pytorch==2.0.110+xpu -f https://developer.intel.com/ipex-whl-stable-xpu')
+            torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.0.1a0 torchvision==0.15.2a0 intel_extension_for_pytorch==2.0.110+xpu openvino==2023.1.0.dev20230728 -f https://developer.intel.com/ipex-whl-stable-xpu')
             os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow==2.13.0 intel-extension-for-tensorflow[gpu]')
         else:
-            torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.0.0a0 torchvision==0.15.2a0 intel_extension_for_pytorch==2.0.110+gitba7f6c1 -f https://developer.intel.com/ipex-whl-stable-xpu')
+            torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.0.0a0 torchvision==0.15.1 intel_extension_for_pytorch==2.0.110+gitba7f6c1 openvino==2023.1.0.dev20230728 -f https://developer.intel.com/ipex-whl-stable-xpu')
     else:
         machine = platform.machine()
         if sys.platform == 'darwin':
@@ -408,7 +458,7 @@ def check_torch():
     try:
         if 'xformers' in xformers_package:
             install(f'--no-deps {xformers_package}', ignore=True)
-        else:
+        elif not args.experimental:
             x = pkg_resources.working_set.by_key.get('xformers', None)
             if x is not None:
                 log.warning(f'Not used, uninstalling: {x}')
@@ -453,8 +503,17 @@ def install_packages():
     install(invisiblewatermark_package, 'invisible-watermark')
     install('onnxruntime==1.15.1', 'onnxruntime', ignore=True)
     install('pi-heif', 'pi_heif', ignore=True)
-    tensorflow_package = os.environ.get('TENSORFLOW_PACKAGE', 'tensorflow==2.12.0')
+    install('git+https://github.com/damian0815/compel', 'compel', ignore=True)
+    tensorflow_package = os.environ.get('TENSORFLOW_PACKAGE', 'tensorflow==2.13.0')
     install(tensorflow_package, 'tensorflow', ignore=True)
+    bitsandbytes_package = os.environ.get('BITSANDBYTES_PACKAGE', None)
+    if bitsandbytes_package is not None:
+        install(bitsandbytes_package, 'bitsandbytes', ignore=True)
+    elif not args.experimental:
+        bitsandbytes_package = pkg_resources.working_set.by_key.get('bitsandbytes', None)
+        if bitsandbytes_package is not None:
+            log.warning(f'Not used, uninstalling: {bitsandbytes_package}')
+            pip('uninstall bitsandbytes --yes --quiet', ignore=True, quiet=True)
     if args.profile:
         print_profile(pr, 'Packages')
 
@@ -478,7 +537,7 @@ def install_repositories():
     clone(taming_transformers_repo, d('taming-transformers'), taming_transformers_commit)
     k_diffusion_repo = os.environ.get('K_DIFFUSION_REPO', 'https://github.com/crowsonkb/k-diffusion.git')
     # k_diffusion_commit = os.environ.get('K_DIFFUSION_COMMIT_HASH', "b43db16749d51055f813255eea2fdf1def801919")
-    k_diffusion_commit = os.environ.get('K_DIFFUSION_COMMIT_HASH', None)
+    k_diffusion_commit = os.environ.get('K_DIFFUSION_COMMIT_HASH', 'ab527a9')
     clone(k_diffusion_repo, d('k-diffusion'), k_diffusion_commit)
     codeformer_repo = os.environ.get('CODEFORMER_REPO', 'https://github.com/sczhou/CodeFormer.git')
     # codeformer_commit = os.environ.get('CODEFORMER_COMMIT_HASH', "c5b4593074ba6214284d6acd5f1719b6c5d739af")

launch.py

@@ -169,6 +169,7 @@ if __name__ == "__main__":
     if installer.check_timestamp():
         installer.log.info('No changes detected: Quick launch active')
         installer.install_requirements()
+        installer.install_packages()
         installer.check_extensions()
     else:
         installer.install_requirements()

modules/devices.py

@@ -40,7 +40,7 @@ def get_cuda_device_string():
 
 
 def get_optimal_device_name():
-    if cuda_ok or backend == 'ipex' or backend == 'directml':
+    if cuda_ok or backend == 'directml':
         return get_cuda_device_string()
     if has_mps():
         return "mps"
@@ -67,7 +67,7 @@ def torch_gc(force=False):
         previous_oom = oom
         shared.log.warning(f'GPU out-of-memory error: {mem}')
     if used > 95:
-        shared.log.warning(f'GPU high memory utilization: {used}% {mem}')
+        shared.log.info(f'GPU high memory utilization: {used}% {mem}')
         force = True
         if backend == "directml":
             practical_used = round(100 * torch.cuda.memory_allocated() / (1 << 30) / gpu.get('total', 1))
@@ -76,7 +76,7 @@ def torch_gc(force=False):
     if shared.opts.disable_gc and not force:
         return
     collected = gc.collect()
-    if cuda_ok or backend == 'ipex':
+    if cuda_ok:
         try:
             with torch.cuda.device(get_cuda_device_string()):
                 torch.cuda.empty_cache()
@@ -182,7 +182,7 @@ elif sys.platform == 'darwin':
 else:
     backend = 'cpu'
 
-cuda_ok = torch.cuda.is_available() and not backend == 'ipex'
+cuda_ok = torch.cuda.is_available()
 cpu = torch.device("cpu")
 device = device_interrogate = device_gfpgan = device_esrgan = device_codeformer = None
 dtype = torch.float16
@@ -221,8 +221,6 @@ def autocast(disable=False):
         return contextlib.nullcontext()
     if shared.cmd_opts.use_directml:
         return torch.dml.amp.autocast(dtype)
-    if backend == 'ipex':
-        return torch.xpu.amp.autocast(enabled=True, dtype=dtype)
     if cuda_ok:
         return torch.autocast("cuda")
     else:
@@ -234,8 +232,6 @@ def without_autocast(disable=False):
         return contextlib.nullcontext()
     if shared.cmd_opts.use_directml:
         return torch.dml.amp.autocast(enabled=False) if torch.is_autocast_enabled() else contextlib.nullcontext() # pylint: disable=unexpected-keyword-arg
-    if backend == 'ipex':
-        return torch.xpu.amp.autocast(enabled=False) if torch.is_autocast_enabled() else contextlib.nullcontext()
     if cuda_ok:
         return torch.autocast("cuda", enabled=False) if torch.is_autocast_enabled() else contextlib.nullcontext()
     else:

modules/dml/backend.py

@@ -7,7 +7,6 @@ import modules.dml.amp as amp
 from .utils import rDevice, get_device
 from .device import device
 from .device_properties import DeviceProperties
-from .memory import MemoryProvider
 
 def amd_mem_get_info(device: Optional[rDevice]=None) -> tuple[int, int]:
     from .memory_amd import AMDMemoryProvider
@@ -29,7 +28,7 @@ class DirectML:
     is_autocast_enabled = False
     autocast_gpu_dtype = torch.float16
 
-    memory_provider: Optional[MemoryProvider] = None
+    memory_provider = None
 
     def is_available() -> bool:
         return torch_directml.is_available()

modules/errors.py

@@ -36,7 +36,7 @@ def print_error_explanation(message):
 
 def display(e: Exception, task, suppress=[]): # noqa: B006
     log.error(f"{task or 'error'}: {type(e).__name__}")
-    console.print_exception(show_locals=False, max_frames=5, extra_lines=1, suppress=suppress, theme="ansi_dark", word_wrap=False, width=min([console.width, 200]))
+    console.print_exception(show_locals=False, max_frames=10, extra_lines=1, suppress=suppress, theme="ansi_dark", word_wrap=False, width=min([console.width, 200]))
 
 
 def display_once(e: Exception, task):

modules/img2img.py

@@ -74,13 +74,13 @@ def process_batch(p, input_files, input_dir, output_dir, inpaint_mask_dir, args)
         shared.log.debug(f'Processed: {len(image_files)} Memory: {memory_stats()} batch')
 
 
-def img2img(id_task: str, mode: int, prompt: str, negative_prompt: str, prompt_styles, init_img, sketch, init_img_with_mask, inpaint_color_sketch, inpaint_color_sketch_orig, init_img_inpaint, init_mask_inpaint, steps: int, sampler_index: int, latent_index: int, mask_blur: int, mask_alpha: float, inpainting_fill: int, restore_faces: bool, tiling: bool, n_iter: int, batch_size: int, cfg_scale: float, image_cfg_scale: float, diffusers_guidance_rescale: float, refiner_start: float, clip_skip: int, denoising_strength: float, seed: int, subseed: int, subseed_strength: float, seed_resize_from_h: int, seed_resize_from_w: int, selected_scale_tab: int, height: int, width: int, scale_by: float, resize_mode: int, inpaint_full_res: bool, inpaint_full_res_padding: int, inpainting_mask_invert: int, img2img_batch_files: list, img2img_batch_input_dir: str, img2img_batch_output_dir: str, img2img_batch_inpaint_mask_dir: str, override_settings_texts, *args): # pylint: disable=unused-argument
+def img2img(id_task: str, mode: int, prompt: str, negative_prompt: str, prompt_styles, init_img, sketch, init_img_with_mask, inpaint_color_sketch, inpaint_color_sketch_orig, init_img_inpaint, init_mask_inpaint, steps: int, sampler_index: int, latent_index: int, mask_blur: int, mask_alpha: float, inpainting_fill: int, full_quality: bool, restore_faces: bool, tiling: bool, n_iter: int, batch_size: int, cfg_scale: float, image_cfg_scale: float, diffusers_guidance_rescale: float, refiner_start: float, clip_skip: int, denoising_strength: float, seed: int, subseed: int, subseed_strength: float, seed_resize_from_h: int, seed_resize_from_w: int, selected_scale_tab: int, height: int, width: int, scale_by: float, resize_mode: int, inpaint_full_res: bool, inpaint_full_res_padding: int, inpainting_mask_invert: int, img2img_batch_files: list, img2img_batch_input_dir: str, img2img_batch_output_dir: str, img2img_batch_inpaint_mask_dir: str, override_settings_texts, *args): # pylint: disable=unused-argument
 
     if shared.sd_model is None:
         shared.log.warning('Model not loaded')
         return [], '', '', 'Error: model not loaded'
 
-    shared.log.debug(f'img2img: id_task={id_task}|mode={mode}|prompt={prompt}|negative_prompt={negative_prompt}|prompt_styles={prompt_styles}|init_img={init_img}|sketch={sketch}|init_img_with_mask={init_img_with_mask}|inpaint_color_sketch={inpaint_color_sketch}|inpaint_color_sketch_orig={inpaint_color_sketch_orig}|init_img_inpaint={init_img_inpaint}|init_mask_inpaint={init_mask_inpaint}|steps={steps}|sampler_index={sampler_index}|latent_index={latent_index}|mask_blur={mask_blur}|mask_alpha={mask_alpha}|inpainting_fill={inpainting_fill}|restore_faces={restore_faces}|tiling={tiling}|n_iter={n_iter}|batch_size={batch_size}|cfg_scale={cfg_scale}|image_cfg_scale={image_cfg_scale}|clip_skip={clip_skip}|denoising_strength={denoising_strength}|seed={seed}|subseed{subseed}|subseed_strength={subseed_strength}|seed_resize_from_h={seed_resize_from_h}|seed_resize_from_w={seed_resize_from_w}|selected_scale_tab={selected_scale_tab}|height={height}|width={width}|scale_by={scale_by}|resize_mode={resize_mode}|inpaint_full_res={inpaint_full_res}|inpaint_full_res_padding={inpaint_full_res_padding}|inpainting_mask_invert={inpainting_mask_invert}|img2img_batch_files={img2img_batch_files}|img2img_batch_input_dir={img2img_batch_input_dir}|img2img_batch_output_dir={img2img_batch_output_dir}|img2img_batch_inpaint_mask_dir={img2img_batch_inpaint_mask_dir}|override_settings_texts={override_settings_texts}|args={args}')
+    shared.log.debug(f'img2img: id_task={id_task}|mode={mode}|prompt={prompt}|negative_prompt={negative_prompt}|prompt_styles={prompt_styles}|init_img={init_img}|sketch={sketch}|init_img_with_mask={init_img_with_mask}|inpaint_color_sketch={inpaint_color_sketch}|inpaint_color_sketch_orig={inpaint_color_sketch_orig}|init_img_inpaint={init_img_inpaint}|init_mask_inpaint={init_mask_inpaint}|steps={steps}|sampler_index={sampler_index}|latent_index={latent_index}|mask_blur={mask_blur}|mask_alpha={mask_alpha}|inpainting_fill={inpainting_fill}|full_quality={full_quality}|restore_faces={restore_faces}|tiling={tiling}|n_iter={n_iter}|batch_size={batch_size}|cfg_scale={cfg_scale}|image_cfg_scale={image_cfg_scale}|clip_skip={clip_skip}|denoising_strength={denoising_strength}|seed={seed}|subseed{subseed}|subseed_strength={subseed_strength}|seed_resize_from_h={seed_resize_from_h}|seed_resize_from_w={seed_resize_from_w}|selected_scale_tab={selected_scale_tab}|height={height}|width={width}|scale_by={scale_by}|resize_mode={resize_mode}|inpaint_full_res={inpaint_full_res}|inpaint_full_res_padding={inpaint_full_res_padding}|inpainting_mask_invert={inpainting_mask_invert}|img2img_batch_files={img2img_batch_files}|img2img_batch_input_dir={img2img_batch_input_dir}|img2img_batch_output_dir={img2img_batch_output_dir}|img2img_batch_inpaint_mask_dir={img2img_batch_inpaint_mask_dir}|override_settings_texts={override_settings_texts}|args={args}')
 
     if init_img is None:
         shared.log.debug('Init image not set')
@@ -158,6 +158,7 @@ def img2img(id_task: str, mode: int, prompt: str, negative_prompt: str, prompt_s
         clip_skip=clip_skip,
         width=width,
         height=height,
+        full_quality=full_quality,
         restore_faces=restore_faces,
         tiling=tiling,
         init_images=[image],
@@ -176,6 +177,7 @@ def img2img(id_task: str, mode: int, prompt: str, negative_prompt: str, prompt_s
     )
     p.scripts = modules.scripts.scripts_img2img
     p.script_args = args
+    p.extra_generation_params['Resize mode'] = resize_mode
     if mask:
         p.extra_generation_params["Mask blur"] = mask_blur
     if is_batch:

modules/ipex_specific/__init__.py

@@ -38,8 +38,73 @@ def ipex_init():
     torch.cuda.FloatTensor = torch.xpu.FloatTensor
     torch.Tensor.cuda = torch.Tensor.xpu
     torch.Tensor.is_cuda = torch.Tensor.is_xpu
+    torch.cuda._initialization_lock = torch.xpu.lazy_init._initialization_lock
+    torch.cuda._initialized = torch.xpu.lazy_init._initialized
+    torch.cuda._lazy_seed_tracker = torch.xpu.lazy_init._lazy_seed_tracker
+    torch.cuda._queued_calls = torch.xpu.lazy_init._queued_calls
+    torch.cuda._tls = torch.xpu.lazy_init._tls
+    torch.cuda.threading = torch.xpu.lazy_init.threading
+    torch.cuda.traceback = torch.xpu.lazy_init.traceback
+    torch.cuda.Optional = torch.xpu.Optional
+    torch.cuda.__cached__ = torch.xpu.__cached__
+    torch.cuda.__loader__ = torch.xpu.__loader__
+    torch.cuda.ComplexFloatStorage = torch.xpu.ComplexFloatStorage
+    torch.cuda.Tuple = torch.xpu.Tuple
+    torch.cuda.streams = torch.xpu.streams
+    torch.cuda._lazy_new = torch.xpu._lazy_new
+    torch.cuda.FloatStorage = torch.xpu.FloatStorage
+    torch.cuda.Any = torch.xpu.Any
+    torch.cuda.__doc__ = torch.xpu.__doc__
+    torch.cuda.default_generators = torch.xpu.default_generators
+    torch.cuda.HalfTensor = torch.xpu.HalfTensor
+    torch.cuda._get_device_index = torch.xpu._get_device_index
+    torch.cuda.__path__ = torch.xpu.__path__
+    torch.cuda.Device = torch.xpu.Device
+    torch.cuda.IntTensor = torch.xpu.IntTensor
+    torch.cuda.ByteStorage = torch.xpu.ByteStorage
+    torch.cuda.set_stream = torch.xpu.set_stream
+    torch.cuda.BoolStorage = torch.xpu.BoolStorage
+    torch.cuda.get_device_capability = torch.xpu.get_device_capability
+    torch.cuda.os = torch.xpu.os
+    torch.cuda.torch = torch.xpu.torch
+    torch.cuda.BFloat16Storage = torch.xpu.BFloat16Storage
+    torch.cuda.Union = torch.xpu.Union
+    torch.cuda.DoubleTensor = torch.xpu.DoubleTensor
+    torch.cuda.ShortTensor = torch.xpu.ShortTensor
+    torch.cuda.LongTensor = torch.xpu.LongTensor
+    torch.cuda.IntStorage = torch.xpu.IntStorage
+    torch.cuda.LongStorage = torch.xpu.LongStorage
+    torch.cuda.__annotations__ = torch.xpu.__annotations__
+    torch.cuda.__package__ = torch.xpu.__package__
+    torch.cuda.__builtins__ = torch.xpu.__builtins__
+    torch.cuda.CharTensor = torch.xpu.CharTensor
+    torch.cuda.List = torch.xpu.List
+    torch.cuda._lazy_init = torch.xpu._lazy_init
+    torch.cuda.BFloat16Tensor = torch.xpu.BFloat16Tensor
+    torch.cuda.DoubleStorage = torch.xpu.DoubleStorage
+    torch.cuda.ByteTensor = torch.xpu.ByteTensor
+    torch.cuda.StreamContext = torch.xpu.StreamContext
+    torch.cuda.ComplexDoubleStorage = torch.xpu.ComplexDoubleStorage
+    torch.cuda.ShortStorage = torch.xpu.ShortStorage
+    torch.cuda._lazy_call = torch.xpu._lazy_call
+    torch.cuda.HalfStorage = torch.xpu.HalfStorage
+    torch.cuda.random = torch.xpu.random
+    torch.cuda._device = torch.xpu._device
+    torch.cuda.classproperty = torch.xpu.classproperty
+    torch.cuda.__name__ = torch.xpu.__name__
+    torch.cuda._device_t = torch.xpu._device_t
+    torch.cuda.warnings = torch.xpu.warnings
+    torch.cuda.__spec__ = torch.xpu.__spec__
+    torch.cuda.BoolTensor = torch.xpu.BoolTensor
+    torch.cuda.CharStorage = torch.xpu.CharStorage
+    torch.cuda.__file__ = torch.xpu.__file__
+    torch.cuda._is_in_bad_fork = torch.xpu.lazy_init._is_in_bad_fork
+    #torch.cuda.is_current_stream_capturing = torch.xpu.is_current_stream_capturing
 
     #Memory:
+    torch.cuda.memory = torch.xpu.memory
+    if 'linux' in sys.platform and "WSL2" in os.popen("uname -a").read():
+        torch.xpu.empty_cache = lambda: None
     torch.cuda.empty_cache = torch.xpu.empty_cache
     torch.cuda.memory_stats = torch.xpu.memory_stats
     torch.cuda.memory_summary = torch.xpu.memory_summary
@@ -47,8 +112,12 @@ def ipex_init():
     torch.cuda.memory_allocated = torch.xpu.memory_allocated
     torch.cuda.max_memory_allocated = torch.xpu.max_memory_allocated
     torch.cuda.memory_reserved = torch.xpu.memory_reserved
+    torch.cuda.memory_cached = torch.xpu.memory_reserved
     torch.cuda.max_memory_reserved = torch.xpu.max_memory_reserved
+    torch.cuda.max_memory_cached = torch.xpu.max_memory_reserved
     torch.cuda.reset_peak_memory_stats = torch.xpu.reset_peak_memory_stats
+    torch.cuda.reset_max_memory_cached = torch.xpu.reset_peak_memory_stats
+    torch.cuda.reset_max_memory_allocated = torch.xpu.reset_peak_memory_stats
     torch.cuda.memory_stats_as_nested_dict = torch.xpu.memory_stats_as_nested_dict
     torch.cuda.reset_accumulated_memory_stats = torch.xpu.reset_accumulated_memory_stats
 
@@ -63,7 +132,11 @@ def ipex_init():
     torch.cuda.seed_all = torch.xpu.seed_all
     torch.cuda.initial_seed = torch.xpu.initial_seed
 
-    #Training:
+    #AMP:
+    torch.cuda.amp = torch.xpu.amp
+    if not hasattr(torch.cuda.amp, "common"):
+        torch.cuda.amp.common = contextlib.nullcontext()
+    torch.cuda.amp.common.amp_definitely_not_available = lambda: False
     try:
         torch.cuda.amp.GradScaler = torch.xpu.amp.GradScaler
     except Exception:
@@ -77,10 +150,12 @@ def ipex_init():
     #Fix functions with ipex:
     torch.cuda.mem_get_info = lambda device=None: [(torch.xpu.get_device_properties(device).total_memory - torch.xpu.memory_allocated(device)), torch.xpu.get_device_properties(device).total_memory]
     torch._utils._get_available_device_type = lambda: "xpu" # pylint: disable=protected-access
-    if 'linux' in sys.platform:
-        torch.xpu.empty_cache = torch.xpu.empty_cache if "WSL2" not in os.popen("uname -a").read() else lambda: None
-    torch.cuda.get_device_properties.major = 2023
-    torch.cuda.get_device_properties.minor = 2
+    torch.has_cuda = True
+    torch.cuda.has_half = True
+    torch.cuda.is_bf16_supported = True
+    #torch.version.cuda = "11.7" #Breaks System Info
+    torch.cuda.get_device_properties.major = 11
+    torch.cuda.get_device_properties.minor = 7
     torch.backends.cuda.sdp_kernel = return_null_context
     torch.nn.DataParallel = DummyDataParallel
     torch.cuda.ipc_collect = lambda: None
@@ -88,3 +163,7 @@ def ipex_init():
 
     ipex_hijacks()
     ipex_diffusers()
+    try:
+        from .openvino import openvino_fx
+    except Exception:
+        pass

modules/ipex_specific/hijacks.py

@@ -1,6 +1,6 @@
 import torch
 import intel_extension_for_pytorch as ipex
-from modules import shared
+from modules import devices
 from modules.sd_hijack_utils import CondFunc
 
 def ipex_no_cuda(orig_func, *args, **kwargs): # pylint: disable=redefined-outer-name
@@ -8,22 +8,74 @@ def ipex_no_cuda(orig_func, *args, **kwargs): # pylint: disable=redefined-outer-
     orig_func(*args, **kwargs)
     torch.cuda.is_available = torch.xpu.is_available
 
+#Autocast
+original_autocast = torch.autocast
+def ipex_autocast(*args, **kwargs):
+    if args[0] == "cuda":
+        if "dtype" in kwargs:
+            return original_autocast("xpu", *args[1:], **kwargs)
+        else:
+            return original_autocast("xpu", *args[1:], dtype=devices.dtype, **kwargs)
+    else:
+        return original_autocast(*args, **kwargs)
+
+#Diffusers BF16:
+original_linear_forward = torch.nn.modules.Linear.forward
+def linear_forward(self, input):
+    if input.dtype != self.weight.data.dtype:
+        return original_linear_forward(self, input.to(self.weight.data.dtype))
+    else:
+        return original_linear_forward(self, input)
+
+#Embedding BF16
+original_torch_cat = torch.cat
+def torch_cat(input, *args, **kwargs):
+    if len(input) == 3 and (input[0].dtype != input[1].dtype or input[2].dtype != input[1].dtype):
+        return original_torch_cat([input[0].to(input[1].dtype), input[1], input[2].to(input[1].dtype)], *args, **kwargs)
+    else:
+        return original_torch_cat(input, *args, **kwargs)
+
+original_conv2d = torch.nn.functional.conv2d
+#Diffusers BF16:
+def conv2d(input, weight, *args, **kwargs):
+    if input.dtype != weight.data.dtype:
+        return original_conv2d(input.to(weight.data.dtype), weight, *args, **kwargs)
+    else:
+        return original_conv2d(input, weight, *args, **kwargs)
+
+original_interpolate = torch.nn.functional.interpolate
+#Latent antialias:
+def interpolate(input, size=None, scale_factor=None, mode='nearest', align_corners=None, recompute_scale_factor=None, antialias=False):
+    if antialias:
+        return original_interpolate(input.to("cpu"), size=size, scale_factor=scale_factor, mode=mode,
+        align_corners=align_corners, recompute_scale_factor=recompute_scale_factor, antialias=antialias).to(devices.device)
+    else:
+        return original_interpolate(input, size=size, scale_factor=scale_factor, mode=mode,
+        align_corners=align_corners, recompute_scale_factor=recompute_scale_factor, antialias=antialias)
+
 def ipex_hijacks():
     #Libraries that blindly uses cuda:
     #Adetailer:
     CondFunc('torch.Tensor.to',
-        lambda orig_func, self, device=None, *args, **kwargs: orig_func(self, shared.device, *args, **kwargs),
+        lambda orig_func, self, device=None, *args, **kwargs: orig_func(self, devices.device, *args, **kwargs),
         lambda orig_func, self, device=None, *args, **kwargs: (type(device) is torch.device and device.type == "cuda") or (type(device) is str and "cuda" in device))
     CondFunc('torch.empty',
-        lambda orig_func, *args, device=None, **kwargs: orig_func(*args, device=shared.device, **kwargs),
+        lambda orig_func, *args, device=None, **kwargs: orig_func(*args, device=devices.device, **kwargs),
         lambda orig_func, *args, device=None, **kwargs: (type(device) is torch.device and device.type == "cuda") or (type(device) is str and "cuda" in device))
     #ControlNet depth_leres
     CondFunc('torch.load',
-        lambda orig_func, *args, map_location=None, **kwargs: orig_func(*args, shared.device, **kwargs),
+        lambda orig_func, *args, map_location=None, **kwargs: orig_func(*args, devices.device, **kwargs),
         lambda orig_func, *args, map_location=None, **kwargs: (map_location is None) or (type(map_location) is torch.device and map_location.type == "cuda") or (type(map_location) is str and "cuda" in map_location))
     #Diffusers Model CPU Offload:
     CondFunc('torch.randn',
-        lambda orig_func, *args, device=None, **kwargs: orig_func(*args, device=shared.device, **kwargs),
+        lambda orig_func, *args, device=None, **kwargs: orig_func(*args, device=devices.device, **kwargs),
+        lambda orig_func, *args, device=None, **kwargs: (type(device) is torch.device and device.type == "cuda") or (type(device) is str and "cuda" in device))
+    #Other:
+    CondFunc('torch.ones',
+        lambda orig_func, *args, device=None, **kwargs: orig_func(*args, device=devices.device, **kwargs),
+        lambda orig_func, *args, device=None, **kwargs: (type(device) is torch.device and device.type == "cuda") or (type(device) is str and "cuda" in device))
+    CondFunc('torch.zeros',
+        lambda orig_func, *args, device=None, **kwargs: orig_func(*args, device=devices.device, **kwargs),
         lambda orig_func, *args, device=None, **kwargs: (type(device) is torch.device and device.type == "cuda") or (type(device) is str and "cuda" in device))
 
     #Broken functions when torch.cuda.is_available is True:
@@ -33,13 +85,10 @@ def ipex_hijacks():
         lambda orig_func, *args, **kwargs: True)
 
     #Functions with dtype errors:
+    #Original backend:
     CondFunc('torch.nn.modules.GroupNorm.forward',
         lambda orig_func, self, input: orig_func(self, input.to(self.weight.data.dtype)),
         lambda orig_func, self, input: input.dtype != self.weight.data.dtype)
-    #FP32:
-    CondFunc('torch.nn.modules.Linear.forward',
-        lambda orig_func, self, input: orig_func(self, input.to(self.weight.data.dtype)),
-        lambda orig_func, self, input: input.dtype != self.weight.data.dtype)
     #Embedding FP32:
     CondFunc('torch.bmm',
         lambda orig_func, input, mat2, *args, **kwargs: orig_func(input, mat2.to(input.dtype), *args, **kwargs),
@@ -50,28 +99,16 @@ def ipex_hijacks():
         orig_func(input.to(weight.data.dtype), normalized_shape, weight, *args, **kwargs),
         lambda orig_func, input, normalized_shape=None, weight=None, *args, **kwargs:
         input.dtype != weight.data.dtype and weight is not None)
-    #Embedding BF16
-    CondFunc('torch.cat',
-        lambda orig_func, input, *args, **kwargs: orig_func([input[0].to(input[1].dtype), input[1], input[2].to(input[1].dtype)], *args, **kwargs),
-        lambda orig_func, input, *args, **kwargs: len(input) == 3 and (input[0].dtype != input[1].dtype or input[2].dtype != input[1].dtype))
-    #Diffusers BF16:
-    CondFunc('torch.nn.functional.conv2d',
-        lambda orig_func, input, weight, *args, **kwargs: orig_func(input.to(weight.data.dtype), weight, *args, **kwargs),
-        lambda orig_func, input, weight, *args, **kwargs: input.dtype != weight.data.dtype)
 
     #Functions that does not work with the XPU:
     #UniPC:
     CondFunc('torch.linalg.solve',
-        lambda orig_func, A, B, *args, **kwargs: orig_func(A.to("cpu"), B.to("cpu"), *args, **kwargs).to(shared.device),
+        lambda orig_func, A, B, *args, **kwargs: orig_func(A.to("cpu"), B.to("cpu"), *args, **kwargs).to(devices.device),
         lambda orig_func, A, B, *args, **kwargs: A.device != torch.device("cpu") or B.device != torch.device("cpu"))
     #SDE Samplers:
     CondFunc('torch.Generator',
         lambda orig_func, device: torch.xpu.Generator(device),
         lambda orig_func, device: device != torch.device("cpu") and device != "cpu")
-    #Latent antialias:
-    CondFunc('torch.nn.functional.interpolate',
-        lambda orig_func, input, *args, **kwargs: orig_func(input.to("cpu"), *args, **kwargs).to(shared.device),
-        lambda orig_func, input, size=None, scale_factor=None, mode='nearest', align_corners=None, recompute_scale_factor=None, antialias=False: antialias)
     #Diffusers Float64 (ARC GPUs doesn't support double or Float64):
     if not torch.xpu.has_fp64_dtype():
         CondFunc('torch.from_numpy',
@@ -80,12 +117,19 @@ def ipex_hijacks():
     #ControlNet and TiledVAE:
     CondFunc('torch.batch_norm',
         lambda orig_func, input, weight, bias, *args, **kwargs: orig_func(input,
-        weight if weight is not None else torch.ones(input.size()[1], device=shared.device),
-        bias if bias is not None else torch.zeros(input.size()[1], device=shared.device), *args, **kwargs),
+        weight if weight is not None else torch.ones(input.size()[1], device=devices.device),
+        bias if bias is not None else torch.zeros(input.size()[1], device=devices.device), *args, **kwargs),
         lambda orig_func, input, *args, **kwargs: input.device != torch.device("cpu"))
     #ControlNet
     CondFunc('torch.instance_norm',
         lambda orig_func, input, weight, bias, *args, **kwargs: orig_func(input,
-        weight if weight is not None else torch.ones(input.size()[1], device=shared.device),
-        bias if bias is not None else torch.zeros(input.size()[1], device=shared.device), *args, **kwargs),
+        weight if weight is not None else torch.ones(input.size()[1], device=devices.device),
+        bias if bias is not None else torch.zeros(input.size()[1], device=devices.device), *args, **kwargs),
         lambda orig_func, input, *args, **kwargs: input.device != torch.device("cpu"))
+
+    #Functions that make compile mad with CondFunc:
+    torch.autocast = ipex_autocast
+    torch.nn.modules.Linear.forward = linear_forward
+    torch.cat = torch_cat
+    torch.nn.functional.conv2d = conv2d
+    torch.nn.functional.interpolate = interpolate

modules/ipex_specific/openvino.py

@@ -0,0 +1,25 @@
+import os
+import torch
+import intel_extension_for_pytorch as ipex
+from openvino.frontend.pytorch.torchdynamo.execute import execute
+from openvino.frontend.pytorch.torchdynamo.partition import Partitioner
+from torch._dynamo.backends.common import fake_tensor_unsupported
+from torch._dynamo.backends.registry import register_backend
+from torch.fx.experimental.proxy_tensor import make_fx
+
+@register_backend
+@fake_tensor_unsupported
+def openvino_fx(subgraph, example_inputs):
+    if os.getenv("OPENVINO_TORCH_BACKEND_DEVICE") is None:
+        os.environ.setdefault("OPENVINO_TORCH_BACKEND_DEVICE", "GPU")
+
+    model = make_fx(subgraph)(*example_inputs)
+    with torch.no_grad():
+        model.eval()
+    partitioner = Partitioner()
+    compiled_model = partitioner.make_partitions(model)
+
+    def _call(*args):
+        res = execute(compiled_model, *args, executor="openvino")
+        return res
+    return _call

modules/lora_diffusers.py

@@ -1,35 +1,522 @@
 import diffusers
-from modules import shared
+import diffusers.models.lora as diffusers_lora
+# from modules import shared
+import modules.shared as shared
+
 
 lora_state = { # TODO Lora state for Diffusers
-    'multiplier': 1.0,
+    'multiplier': [],
     'active': False,
     'loaded': 0,
+    'all_loras': []
 }
-
 def unload_diffusers_lora():
     try:
         pipe = shared.sd_model
-        pipe.unload_lora_weights()
+        if shared.opts.diffusers_lora_loader == "diffusers default":
+            pipe.unload_lora_weights()
+            pipe._remove_text_encoder_monkey_patch() # pylint: disable=W0212
+            proc_cls_name = next(iter(pipe.unet.attn_processors.values())).__class__.__name__
+            non_lora_proc_cls = getattr(diffusers.models.attention_processor, proc_cls_name)#[len("LORA"):])
+            pipe.unet.set_attn_processor(non_lora_proc_cls())
+            # shared.log.debug('Diffusers LoRA unloaded')
+        else:
+            lora_state['all_loras'].reverse()
+            lora_state['multiplier'].reverse()
+            for i, lora_network in enumerate(lora_state['all_loras']):
+                if shared.opts.diffusers_lora_loader == "merge and apply":
+                    lora_network.restore_from(multiplier=lora_state['multiplier'][i])
+                if shared.opts.diffusers_lora_loader == "sequential apply":
+                    lora_network.unapply_to()
         lora_state['active'] = False
         lora_state['loaded'] = 0
-        pipe._remove_text_encoder_monkey_patch() # pylint: disable=W0212
-        proc_cls_name = next(iter(pipe.unet.attn_processors.values())).__class__.__name__
-        non_lora_proc_cls = getattr(diffusers.models.attention_processor, proc_cls_name[len("LORA"):])
-        pipe.unet.set_attn_processor(non_lora_proc_cls())
-        # shared.log.debug('Diffusers LoRA unloaded')
-    except Exception:
-        pass
+        lora_state['all_loras'] = []
+        lora_state['multiplier'] = []
+
+    except Exception as e:
+        shared.log.error(f"Diffusers LoRA unloading failed: {e}")
 
 
 def load_diffusers_lora(name, lora, strength = 1.0):
     try:
         pipe = shared.sd_model
-        pipe.load_lora_weights(lora.filename, cache_dir=shared.opts.diffusers_dir, local_files_only=True, lora_scale=strength)
         lora_state['active'] = True
         lora_state['loaded'] += 1
-        lora_state['multiplier'] = strength
-        # pipe.unet.load_attn_procs("pcuenq/pokemon-lora")
-        shared.log.info(f"Diffusers LoRA loaded: {name} {lora_state['multiplier']}")
+        lora_state['multiplier'].append(strength)
+        if shared.opts.diffusers_lora_loader == "diffusers default":
+            pipe.load_lora_weights(lora.filename, cache_dir=shared.opts.diffusers_dir, local_files_only=True, lora_scale=strength)
+            shared.log.info(f"LoRA loaded: {name} {lora_state['multiplier']}")
+        else:
+            from safetensors.torch import load_file
+            lora_sd = load_file(lora.filename)
+            if "XL" in pipe.__class__.__name__:
+                text_encoders = [pipe.text_encoder, pipe.text_encoder_2]
+            else:
+                text_encoders = pipe.text_encoder
+            lora_network: LoRANetwork = create_network_from_weights(text_encoders, pipe.unet, lora_sd, multiplier=strength)
+            lora_network.load_state_dict(lora_sd)
+            if shared.opts.diffusers_lora_loader == "merge and apply":
+                lora_network.merge_to(multiplier=strength)
+            if shared.opts.diffusers_lora_loader == "sequential apply":
+                lora_network.to(shared.device, dtype=pipe.unet.dtype)
+                lora_network.apply_to(multiplier=strength)
+            lora_state['all_loras'].append(lora_network)
+            shared.log.info(f"LoRA loaded: {name}:{strength} loader={shared.opts.diffusers_lora_loader}")
     except Exception as e:
         shared.log.error(f"Diffusers LoRA loading failed: {name} {e}")
+
+
+# Diffusersで動くLoRA。このファイル単独で完結する。
+# LoRA module for Diffusers. This file works independently.
+import bisect
+import math
+from typing import Any, Dict, List, Mapping, Optional, Union
+from diffusers import UNet2DConditionModel
+from tqdm import tqdm
+from transformers import CLIPTextModel
+import torch
+
+
+def make_unet_conversion_map() -> Dict[str, str]:
+    unet_conversion_map_layer = []
+
+    for i in range(3):  # num_blocks is 3 in sdxl
+        # loop over downblocks/upblocks
+        for j in range(2):
+            # loop over resnets/attentions for downblocks
+            hf_down_res_prefix = f"down_blocks.{i}.resnets.{j}."
+            sd_down_res_prefix = f"input_blocks.{3*i + j + 1}.0."
+            unet_conversion_map_layer.append((sd_down_res_prefix, hf_down_res_prefix))
+
+            if i < 3:
+                # no attention layers in down_blocks.3
+                hf_down_atn_prefix = f"down_blocks.{i}.attentions.{j}."
+                sd_down_atn_prefix = f"input_blocks.{3*i + j + 1}.1."
+                unet_conversion_map_layer.append((sd_down_atn_prefix, hf_down_atn_prefix))
+
+        for j in range(3):
+            # loop over resnets/attentions for upblocks
+            hf_up_res_prefix = f"up_blocks.{i}.resnets.{j}."
+            sd_up_res_prefix = f"output_blocks.{3*i + j}.0."
+            unet_conversion_map_layer.append((sd_up_res_prefix, hf_up_res_prefix))
+
+            # if i > 0: commentout for sdxl
+            # no attention layers in up_blocks.0
+            hf_up_atn_prefix = f"up_blocks.{i}.attentions.{j}."
+            sd_up_atn_prefix = f"output_blocks.{3*i + j}.1."
+            unet_conversion_map_layer.append((sd_up_atn_prefix, hf_up_atn_prefix))
+
+        if i < 3:
+            # no downsample in down_blocks.3
+            hf_downsample_prefix = f"down_blocks.{i}.downsamplers.0.conv."
+            sd_downsample_prefix = f"input_blocks.{3*(i+1)}.0.op."
+            unet_conversion_map_layer.append((sd_downsample_prefix, hf_downsample_prefix))
+
+            # no upsample in up_blocks.3
+            hf_upsample_prefix = f"up_blocks.{i}.upsamplers.0."
+            sd_upsample_prefix = f"output_blocks.{3*i + 2}.{2}."  # change for sdxl
+            unet_conversion_map_layer.append((sd_upsample_prefix, hf_upsample_prefix))
+
+    hf_mid_atn_prefix = "mid_block.attentions.0."
+    sd_mid_atn_prefix = "middle_block.1."
+    unet_conversion_map_layer.append((sd_mid_atn_prefix, hf_mid_atn_prefix))
+
+    for j in range(2):
+        hf_mid_res_prefix = f"mid_block.resnets.{j}."
+        sd_mid_res_prefix = f"middle_block.{2*j}."
+        unet_conversion_map_layer.append((sd_mid_res_prefix, hf_mid_res_prefix))
+
+    unet_conversion_map_resnet = [
+        # (stable-diffusion, HF Diffusers)
+        ("in_layers.0.", "norm1."),
+        ("in_layers.2.", "conv1."),
+        ("out_layers.0.", "norm2."),
+        ("out_layers.3.", "conv2."),
+        ("emb_layers.1.", "time_emb_proj."),
+        ("skip_connection.", "conv_shortcut."),
+    ]
+
+    unet_conversion_map = []
+    for sd, hf in unet_conversion_map_layer:
+        if "resnets" in hf:
+            for sd_res, hf_res in unet_conversion_map_resnet:
+                unet_conversion_map.append((sd + sd_res, hf + hf_res))
+        else:
+            unet_conversion_map.append((sd, hf))
+
+    for j in range(2):
+        hf_time_embed_prefix = f"time_embedding.linear_{j+1}."
+        sd_time_embed_prefix = f"time_embed.{j*2}."
+        unet_conversion_map.append((sd_time_embed_prefix, hf_time_embed_prefix))
+
+    for j in range(2):
+        hf_label_embed_prefix = f"add_embedding.linear_{j+1}."
+        sd_label_embed_prefix = f"label_emb.0.{j*2}."
+        unet_conversion_map.append((sd_label_embed_prefix, hf_label_embed_prefix))
+
+    unet_conversion_map.append(("input_blocks.0.0.", "conv_in."))
+    unet_conversion_map.append(("out.0.", "conv_norm_out."))
+    unet_conversion_map.append(("out.2.", "conv_out."))
+
+    sd_hf_conversion_map = {sd.replace(".", "_")[:-1]: hf.replace(".", "_")[:-1] for sd, hf in unet_conversion_map}
+    return sd_hf_conversion_map
+
+
+UNET_CONVERSION_MAP = make_unet_conversion_map()
+
+
+class LoRAModule(torch.nn.Module):
+    """
+    replaces forward method of the original Linear, instead of replacing the original Linear module.
+    """
+
+    def __init__(
+        self,
+        lora_name,
+        org_module: torch.nn.Module,
+        multiplier=1.0,
+        lora_dim=4,
+        alpha=1,
+    ):
+        """if alpha == 0 or None, alpha is rank (no scaling)."""
+        super().__init__()
+        self.lora_name = lora_name
+
+        if isinstance(org_module, diffusers_lora.LoRACompatibleConv): #Modified to support Diffusers>=0.19.2
+            in_dim = org_module.in_channels
+            out_dim = org_module.out_channels
+        else:
+            in_dim = org_module.in_features
+            out_dim = org_module.out_features
+
+        self.lora_dim = lora_dim
+
+        if isinstance(org_module, diffusers_lora.LoRACompatibleConv): #Modified to support Diffusers>=0.19.2
+            kernel_size = org_module.kernel_size
+            stride = org_module.stride
+            padding = org_module.padding
+            self.lora_down = torch.nn.Conv2d(in_dim, self.lora_dim, kernel_size, stride, padding, bias=False)
+            self.lora_up = torch.nn.Conv2d(self.lora_dim, out_dim, (1, 1), (1, 1), bias=False)
+        else:
+            self.lora_down = torch.nn.Linear(in_dim, self.lora_dim, bias=False)
+            self.lora_up = torch.nn.Linear(self.lora_dim, out_dim, bias=False)
+
+        if isinstance(alpha, torch.Tensor):
+            alpha = alpha.detach().float().numpy()  # without casting, bf16 causes error
+        alpha = self.lora_dim if alpha is None or alpha == 0 else alpha
+        self.scale = alpha / self.lora_dim
+        self.register_buffer("alpha", torch.tensor(alpha))  # 勾配計算に含めない / not included in gradient calculation
+
+        # same as microsoft's
+        torch.nn.init.kaiming_uniform_(self.lora_down.weight, a=math.sqrt(5))
+        torch.nn.init.zeros_(self.lora_up.weight)
+
+        self.multiplier = multiplier
+        self.org_module = [org_module]
+        self.enabled = True
+        self.network: LoRANetwork = None
+        self.org_forward = None
+
+    # override org_module's forward method
+    def apply_to(self, multiplier=None):
+        if multiplier is not None:
+            self.multiplier = multiplier
+        if self.org_forward is None:
+            self.org_forward = self.org_module[0].forward
+            self.org_module[0].forward = self.forward
+
+    # restore org_module's forward method
+    def unapply_to(self):
+        if self.org_forward is not None:
+            self.org_module[0].forward = self.org_forward
+
+    # forward with lora
+    def forward(self, x):
+        if not self.enabled:
+            return self.org_forward(x)
+        return self.org_forward(x) + self.lora_up(self.lora_down(x)) * self.multiplier * self.scale
+
+    def set_network(self, network):
+        self.network = network
+
+    # merge lora weight to org weight
+    def merge_to(self, multiplier=1.0):
+        # get lora weight
+        lora_weight = self.get_weight(multiplier)
+
+        # get org weight
+        org_sd = self.org_module[0].state_dict()
+        org_weight = org_sd["weight"]
+        weight = org_weight + lora_weight.to(org_weight.device, dtype=org_weight.dtype)
+
+        # set weight to org_module
+        org_sd["weight"] = weight
+        self.org_module[0].load_state_dict(org_sd)
+
+    # restore org weight from lora weight
+    def restore_from(self, multiplier=1.0):
+        # get lora weight
+        lora_weight = self.get_weight(multiplier)
+
+        # get org weight
+        org_sd = self.org_module[0].state_dict()
+        org_weight = org_sd["weight"]
+        weight = org_weight - lora_weight.to(org_weight.device, dtype=org_weight.dtype)
+
+        # set weight to org_module
+        org_sd["weight"] = weight
+        self.org_module[0].load_state_dict(org_sd)
+
+    # return lora weight
+    def get_weight(self, multiplier=None):
+        if multiplier is None:
+            multiplier = self.multiplier
+
+        # get up/down weight from module
+        up_weight = self.lora_up.weight.to(torch.float)
+        down_weight = self.lora_down.weight.to(torch.float)
+
+        # pre-calculated weight
+        if len(down_weight.size()) == 2:
+            # linear
+            weight = self.multiplier * (up_weight @ down_weight) * self.scale
+        elif down_weight.size()[2:4] == (1, 1):
+            # conv2d 1x1
+            weight = (
+                self.multiplier
+                * (up_weight.squeeze(3).squeeze(2) @ down_weight.squeeze(3).squeeze(2)).unsqueeze(2).unsqueeze(3)
+                * self.scale
+            )
+        else:
+            # conv2d 3x3
+            conved = torch.nn.functional.conv2d(down_weight.permute(1, 0, 2, 3), up_weight).permute(1, 0, 2, 3)
+            weight = self.multiplier * conved * self.scale
+
+        return weight
+
+
+# Create network from weights for inference, weights are not loaded here
+def create_network_from_weights(
+    text_encoder: Union[CLIPTextModel, List[CLIPTextModel]], unet: UNet2DConditionModel, weights_sd: Dict, multiplier: float = 1.0
+):
+    # get dim/alpha mapping
+    modules_dim = {}
+    modules_alpha = {}
+    for key, value in weights_sd.items():
+        if "." not in key:
+            continue
+
+        lora_name = key.split(".")[0]
+        if "alpha" in key:
+            modules_alpha[lora_name] = value
+        elif "lora_down" in key:
+            dim = value.size()[0]
+            modules_dim[lora_name] = dim
+            # print(lora_name, value.size(), dim)
+
+    # support old LoRA without alpha
+    for key in modules_dim.keys():
+        if key not in modules_alpha:
+            modules_alpha[key] = modules_dim[key]
+
+    return LoRANetwork(text_encoder, unet, multiplier=multiplier, modules_dim=modules_dim, modules_alpha=modules_alpha)
+
+
+def merge_lora_weights(pipe, weights_sd: Dict, multiplier: float = 1.0):
+    text_encoders = [pipe.text_encoder, pipe.text_encoder_2] if hasattr(pipe, "text_encoder_2") else [pipe.text_encoder]
+    unet = pipe.unet
+
+    lora_network = create_network_from_weights(text_encoders, unet, weights_sd, multiplier=multiplier)
+    lora_network.load_state_dict(weights_sd)
+    lora_network.merge_to(multiplier=multiplier)
+
+
+# block weightや学習に対応しない簡易版 / simple version without block weight and training
+class LoRANetwork(torch.nn.Module): # pylint: disable=abstract-method
+    UNET_TARGET_REPLACE_MODULE = ["Transformer2DModel"]
+    UNET_TARGET_REPLACE_MODULE_CONV2D_3X3 = ["ResnetBlock2D", "Downsample2D", "Upsample2D"]
+    TEXT_ENCODER_TARGET_REPLACE_MODULE = ["CLIPAttention", "CLIPMLP"]
+    LORA_PREFIX_UNET = "lora_unet"
+    LORA_PREFIX_TEXT_ENCODER = "lora_te"
+
+    # SDXL: must starts with LORA_PREFIX_TEXT_ENCODER
+    LORA_PREFIX_TEXT_ENCODER1 = "lora_te1"
+    LORA_PREFIX_TEXT_ENCODER2 = "lora_te2"
+
+    def __init__(
+        self,
+        text_encoder: Union[List[CLIPTextModel], CLIPTextModel],
+        unet: UNet2DConditionModel,
+        multiplier: float = 1.0,
+        modules_dim: Optional[Dict[str, int]] = None,
+        modules_alpha: Optional[Dict[str, int]] = None,
+        varbose: Optional[bool] = False, # pylint: disable=unused-argument
+    ) -> None:
+        super().__init__()
+        self.multiplier = multiplier
+
+        # shared.log.debug("create LoRA network from weights")
+
+        # convert SDXL Stability AI's U-Net modules to Diffusers
+        converted = self.convert_unet_modules(modules_dim, modules_alpha)
+        if converted:
+            shared.log.debug(f"LoRA convert: modules={converted} SDXL SAI/SGM to Diffusers")
+
+        # create module instances
+        def create_modules(
+            is_unet: bool,
+            text_encoder_idx: Optional[int],  # None, 1, 2
+            root_module: torch.nn.Module,
+            target_replace_modules: List[torch.nn.Module],
+        ) -> List[LoRAModule]:
+            prefix = (
+                self.LORA_PREFIX_UNET
+                if is_unet
+                else (
+                    self.LORA_PREFIX_TEXT_ENCODER
+                    if text_encoder_idx is None
+                    else (self.LORA_PREFIX_TEXT_ENCODER1 if text_encoder_idx == 1 else self.LORA_PREFIX_TEXT_ENCODER2)
+                )
+            )
+            loras = []
+            skipped = []
+            for name, module in root_module.named_modules():
+                if module.__class__.__name__ in target_replace_modules:
+                    for child_name, child_module in module.named_modules():
+                        is_linear = isinstance(child_module, (torch.nn.Linear, diffusers_lora.LoRACompatibleLinear))  #Modified to support Diffusers>=0.19.2
+                        is_conv2d = isinstance(child_module, (torch.nn.Conv2d, diffusers_lora.LoRACompatibleConv))  #Modified to support Diffusers>=0.19.2
+
+                        if is_linear or is_conv2d:
+                            lora_name = prefix + "." + name + "." + child_name
+                            lora_name = lora_name.replace(".", "_")
+
+                            if lora_name not in modules_dim:
+                                # print(f"skipped {lora_name} (not found in modules_dim)")
+                                skipped.append(lora_name)
+                                continue
+
+                            dim = modules_dim[lora_name]
+                            alpha = modules_alpha[lora_name]
+                            lora = LoRAModule(
+                                lora_name,
+                                child_module,
+                                self.multiplier,
+                                dim,
+                                alpha,
+                            )
+                            loras.append(lora)
+            return loras, skipped
+
+        text_encoders = text_encoder if type(text_encoder) == list else [text_encoder]
+
+        # create LoRA for text encoder
+        # 毎回すべてのモジュールを作るのは無駄なので要検討 / it is wasteful to create all modules every time, need to consider
+        self.text_encoder_loras: List[LoRAModule] = []
+        skipped_te = []
+        for i, text_encoder in enumerate(text_encoders):
+            if len(text_encoders) > 1:
+                index = i + 1
+            else:
+                index = None
+
+            text_encoder_loras, skipped = create_modules(False, index, text_encoder, LoRANetwork.TEXT_ENCODER_TARGET_REPLACE_MODULE)
+            self.text_encoder_loras.extend(text_encoder_loras)
+            skipped_te += skipped
+
+        # extend U-Net target modules to include Conv2d 3x3
+        target_modules = LoRANetwork.UNET_TARGET_REPLACE_MODULE + LoRANetwork.UNET_TARGET_REPLACE_MODULE_CONV2D_3X3
+
+        self.unet_loras: List[LoRAModule]
+        self.unet_loras, skipped_un = create_modules(True, None, unet, target_modules)
+        shared.log.debug(f"LoRA modules loaded/skipped: te={len(self.text_encoder_loras)}/{len(skipped_te)} unet={len(self.unet_loras)}/skip={len(skipped_un)}")
+
+        # assertion
+        names = set()
+        for lora in self.text_encoder_loras + self.unet_loras:
+            names.add(lora.lora_name)
+        for lora_name in modules_dim.keys():
+            assert lora_name in names, f"{lora_name} is not found in created LoRA modules."
+
+        # make to work load_state_dict
+        for lora in self.text_encoder_loras + self.unet_loras:
+            self.add_module(lora.lora_name, lora)
+
+    # SDXL: convert SDXL Stability AI's U-Net modules to Diffusers
+    def convert_unet_modules(self, modules_dim, modules_alpha):
+        converted_count = 0
+        not_converted_count = 0
+
+        map_keys = list(UNET_CONVERSION_MAP.keys())
+        map_keys.sort()
+
+        for key in list(modules_dim.keys()):
+            if key.startswith(LoRANetwork.LORA_PREFIX_UNET + "_"):
+                search_key = key.replace(LoRANetwork.LORA_PREFIX_UNET + "_", "")
+                position = bisect.bisect_right(map_keys, search_key)
+                map_key = map_keys[position - 1]
+                if search_key.startswith(map_key):
+                    new_key = key.replace(map_key, UNET_CONVERSION_MAP[map_key])
+                    modules_dim[new_key] = modules_dim[key]
+                    modules_alpha[new_key] = modules_alpha[key]
+                    del modules_dim[key]
+                    del modules_alpha[key]
+                    converted_count += 1
+                else:
+                    not_converted_count += 1
+        assert (
+            converted_count == 0 or not_converted_count == 0
+        ), f"some modules are not converted: {converted_count} converted, {not_converted_count} not converted"
+        return converted_count
+
+    def set_multiplier(self, multiplier):
+        self.multiplier = multiplier
+        for lora in self.text_encoder_loras + self.unet_loras:
+            lora.multiplier = self.multiplier
+
+    def apply_to(self, multiplier=1.0, apply_text_encoder=True, apply_unet=True):
+        if apply_text_encoder:
+            # shared.log.debug("LoRA apply for text encoder")
+            for lora in self.text_encoder_loras:
+                lora.apply_to(multiplier)
+        if apply_unet:
+            # shared.log.debug("LoRA apply for U-Net")
+            for lora in self.unet_loras:
+                lora.apply_to(multiplier)
+
+    def unapply_to(self):
+        for lora in self.text_encoder_loras + self.unet_loras:
+            lora.unapply_to()
+
+    def merge_to(self, multiplier=1.0):
+        # shared.log.debug("LoRA merge weights for text encoder")
+        for lora in tqdm(self.text_encoder_loras + self.unet_loras):
+            lora.merge_to(multiplier)
+
+    def restore_from(self, multiplier=1.0):
+        # shared.log.debug("LoRA restore weights")
+        for lora in tqdm(self.text_encoder_loras + self.unet_loras):
+            lora.restore_from(multiplier)
+
+    def load_state_dict(self, state_dict: Mapping[str, Any], strict: bool = True):
+        # convert SDXL Stability AI's state dict to Diffusers' based state dict
+        map_keys = list(UNET_CONVERSION_MAP.keys())  # prefix of U-Net modules
+        map_keys.sort()
+        for key in list(state_dict.keys()):
+            if key.startswith(LoRANetwork.LORA_PREFIX_UNET + "_"):
+                search_key = key.replace(LoRANetwork.LORA_PREFIX_UNET + "_", "")
+                position = bisect.bisect_right(map_keys, search_key)
+                map_key = map_keys[position - 1]
+                if search_key.startswith(map_key):
+                    new_key = key.replace(map_key, UNET_CONVERSION_MAP[map_key])
+                    state_dict[new_key] = state_dict[key]
+                    del state_dict[key]
+
+        # in case of V2, some weights have different shape, so we need to convert them
+        # because V2 LoRA is based on U-Net created by use_linear_projection=False
+        my_state_dict = self.state_dict()
+        for key in state_dict.keys():
+            if state_dict[key].size() != my_state_dict[key].size():
+                # print(f"convert {key} from {state_dict[key].size()} to {my_state_dict[key].size()}")
+                state_dict[key] = state_dict[key].view(my_state_dict[key].size())
+
+        return super().load_state_dict(state_dict, strict)

modules/modelloader.py

@@ -3,7 +3,6 @@ import shutil
 import importlib
 from typing import Dict
 from urllib.parse import urlparse
-
 from modules import shared
 from modules.upscaler import Upscaler, UpscalerLanczos, UpscalerNearest, UpscalerNone
 from modules.paths import script_path, models_path
@@ -11,10 +10,54 @@ from modules.paths import script_path, models_path
 diffuser_repos = []
 
 
+def download_civit_model(model_url: str, model_name: str, model_path: str, preview):
+    model_file = os.path.join(shared.opts.ckpt_dir, model_path, model_name)
+    res = f'CivitAI download: name={model_name} url={model_url} path={model_path}'
+    if os.path.isfile(model_file):
+        res += ' already exists'
+        shared.log.warning(res)
+        return res
+    import requests
+    import rich.progress as p
+
+    req = requests.get(model_url, stream=True, timeout=30)
+    total_size = int(req.headers.get('content-length', 0))
+    block_size = 16384 # 16KB blocks
+    written = 0
+    shared.state.begin()
+    shared.state.job = 'downloload model'
+    try:
+        with open(model_file, 'wb') as f:
+            with p.Progress(p.TextColumn('[cyan]{task.description}'), p.DownloadColumn(), p.BarColumn(), p.TaskProgressColumn(), p.TimeRemainingColumn(), p.TimeElapsedColumn(), p.TransferSpeedColumn()) as progress:
+                task = progress.add_task(description="Download starting", total=total_size)
+                # for data in tqdm(req.iter_content(block_size), total=total_size//1024, unit='KB', unit_scale=False):
+                for data in req.iter_content(block_size):
+                    written = written + len(data)
+                    f.write(data)
+                    progress.update(task, advance=block_size, description="Downloading")
+        if written < 1024 * 1024 * 1024: # min threshold
+            os.remove(model_file)
+            raise ValueError(f'removed invalid download: bytes={written}')
+        if preview is not None:
+            preview_file = os.path.splitext(model_file)[0] + '.jpg'
+            preview.save(preview_file)
+            res += f' preview={preview_file}'
+    except Exception as e:
+        shared.log.error(f'CivitAI download error: name={model_name} url={model_url} path={model_path} {e}')
+    if total_size == written:
+        shared.log.info(f'{res} size={total_size}')
+    else:
+        shared.log.error(f'{res} size={total_size} written={written}')
+    shared.state.end()
+    return res
+
+
 def download_diffusers_model(hub_id: str, cache_dir: str = None, download_config: Dict[str, str] = None, token = None, variant = None, revision = None, mirror = None):
     from diffusers import DiffusionPipeline
     import huggingface_hub as hf
 
+    shared.state.begin()
+    shared.state.job = 'downloload model'
     if download_config is None:
         download_config = {
             "force_download": False,
@@ -47,6 +90,7 @@ def download_diffusers_model(hub_id: str, cache_dir: str = None, download_config
         with open(os.path.join(download_dir, "hidden"), "w", encoding="utf-8") as f:
             f.write("True")
     shared.writefile(model_info_dict, os.path.join(pipeline_dir, "model_info.json"))
+    shared.state.end()
     return pipeline_dir
 
 

modules/postprocessing.py

@@ -76,6 +76,7 @@ def run_postprocessing(extras_mode, image, image_folder: List[tempfile.NamedTemp
             images.save_image(pp.image, path=outpath, basename=basename, seed=None, prompt=None, extension=ext or opts.samples_format, info=infotext, short_filename=True, no_prompt=True, grid=False, pnginfo_section_name="extras", existing_info=pp.image.info, forced_filename=None)
         if extras_mode != 2 or show_extras_results:
             outputs.append(pp.image)
+        image.close()
 
     devices.torch_gc()
     return outputs, infotext, params

modules/processing.py

@@ -86,7 +86,7 @@ class StableDiffusionProcessing:
     """
     The first set of paramaters: sd_models -> do_not_reload_embeddings represent the minimum required to create a StableDiffusionProcessing
     """
-    def __init__(self, sd_model=None, outpath_samples=None, outpath_grids=None, prompt: str = "", styles: List[str] = None, seed: int = -1, subseed: int = -1, subseed_strength: float = 0, seed_resize_from_h: int = -1, seed_resize_from_w: int = -1, seed_enable_extras: bool = True, sampler_name: str = None, latent_sampler: str = None, batch_size: int = 1, n_iter: int = 1, steps: int = 50, cfg_scale: float = 7.0, image_cfg_scale: float = None, clip_skip: int = 1, width: int = 512, height: int = 512, restore_faces: bool = False, tiling: bool = False, do_not_save_samples: bool = False, do_not_save_grid: bool = False, extra_generation_params: Dict[Any, Any] = None, overlay_images: Any = None, negative_prompt: str = None, eta: float = None, do_not_reload_embeddings: bool = False, denoising_strength: float = 0, diffusers_guidance_rescale: float = 0.7, ddim_discretize: str = None, s_min_uncond: float = 0.0, s_churn: float = 0.0, s_tmax: float = None, s_tmin: float = 0.0, s_noise: float = 1.0, override_settings: Dict[str, Any] = None, override_settings_restore_afterwards: bool = True, sampler_index: int = None, script_args: list = None): # pylint: disable=unused-argument
+    def __init__(self, sd_model=None, outpath_samples=None, outpath_grids=None, prompt: str = "", styles: List[str] = None, seed: int = -1, subseed: int = -1, subseed_strength: float = 0, seed_resize_from_h: int = -1, seed_resize_from_w: int = -1, seed_enable_extras: bool = True, sampler_name: str = None, latent_sampler: str = None, batch_size: int = 1, n_iter: int = 1, steps: int = 50, cfg_scale: float = 7.0, image_cfg_scale: float = None, clip_skip: int = 1, width: int = 512, height: int = 512, full_quality: bool = True, restore_faces: bool = False, tiling: bool = False, do_not_save_samples: bool = False, do_not_save_grid: bool = False, extra_generation_params: Dict[Any, Any] = None, overlay_images: Any = None, negative_prompt: str = None, eta: float = None, do_not_reload_embeddings: bool = False, denoising_strength: float = 0, diffusers_guidance_rescale: float = 0.7, ddim_discretize: str = None, s_min_uncond: float = 0.0, s_churn: float = 0.0, s_tmax: float = None, s_tmin: float = 0.0, s_noise: float = 1.0, override_settings: Dict[str, Any] = None, override_settings_restore_afterwards: bool = True, sampler_index: int = None, script_args: list = None): # pylint: disable=unused-argument
 
         self.outpath_samples: str = outpath_samples
         self.outpath_grids: str = outpath_grids
@@ -110,6 +110,7 @@ class StableDiffusionProcessing:
         self.diffusers_guidance_rescale = diffusers_guidance_rescale
         self.width: int = width
         self.height: int = height
+        self.full_quality: bool = full_quality
         self.restore_faces: bool = restore_faces
         self.tiling: bool = tiling
         self.do_not_save_samples: bool = do_not_save_samples
@@ -444,12 +445,13 @@ def fix_seed(p):
     p.subseed = get_fixed_seed(p.subseed)
 
 
-def create_infotext(p: StableDiffusionProcessing, all_prompts, all_seeds, all_subseeds, comments=None, iteration=0, position_in_batch=0, index=None, all_negative_prompts=None): # pylint: disable=unused-argument
+def create_infotext(p: StableDiffusionProcessing, all_prompts, all_seeds, all_subseeds, comments=None, iteration=0, position_in_batch=0, index=None, all_negative_prompts=None):
     if index is None:
         index = position_in_batch + iteration * p.batch_size
-
     if all_negative_prompts is None:
         all_negative_prompts = p.all_negative_prompts
+    vae = (None if not shared.opts.add_model_name_to_info or sd_vae.loaded_vae_file is None else os.path.splitext(os.path.basename(sd_vae.loaded_vae_file))[0]) if p.full_quality else 'TAESD'
+    comment = ', '.join(comments) if comments is not None and type(comments) is list else None
 
     generation_params = {
         "Steps": p.steps,
@@ -457,14 +459,15 @@ def create_infotext(p: StableDiffusionProcessing, all_prompts, all_seeds, all_su
         "Sampler": p.sampler_name,
         "CFG scale": p.cfg_scale,
         "Size": f"{p.width}x{p.height}",
+        "Batch": f'{p.n_iter}x{p.batch_size}' if p.n_iter > 1 or p.batch_size > 1 else None,
         "Parser": shared.opts.prompt_attention,
-        "Model": None if not shared.opts.add_model_name_to_info or not shared.sd_model.sd_checkpoint_info.model_name else shared.sd_model.sd_checkpoint_info.model_name.replace(',', '').replace(':', ''),
-        "Model hash": getattr(p, 'sd_model_hash', None if not shared.opts.add_model_hash_to_info or not shared.sd_model.sd_model_hash else shared.sd_model.sd_model_hash),
-        "Refiner": None if not shared.opts.add_model_name_to_info or not shared.sd_refiner or not shared.sd_refiner.sd_checkpoint_info.model_name else shared.sd_refiner.sd_checkpoint_info.model_name.replace(',', '').replace(':', ''),
-        "VAE": None if not shared.opts.add_model_name_to_info or sd_vae.loaded_vae_file is None else os.path.splitext(os.path.basename(sd_vae.loaded_vae_file))[0],
+        "Model": None if (not shared.opts.add_model_name_to_info) or (not shared.sd_model.sd_checkpoint_info.model_name) else shared.sd_model.sd_checkpoint_info.model_name.replace(',', '').replace(':', ''),
+        "Model hash": getattr(p, 'sd_model_hash', None if (not shared.opts.add_model_hash_to_info) or (not shared.sd_model.sd_model_hash) else shared.sd_model.sd_model_hash),
+        "Refiner": None if (not shared.opts.add_model_name_to_info) or (not shared.sd_refiner) or (not shared.sd_refiner.sd_checkpoint_info.model_name) else shared.sd_refiner.sd_checkpoint_info.model_name.replace(',', '').replace(':', ''),
+        "VAE": vae,
         # subseed
         "Variation seed": None if p.subseed_strength == 0 else all_subseeds[index],
-        "Variation seed strength": None if p.subseed_strength == 0 else p.subseed_strength,
+        "Variation strength": None if p.subseed_strength == 0 else p.subseed_strength,
         # seed resize
         "Seed resize from": None if p.seed_resize_from_w == 0 or p.seed_resize_from_h == 0 else f"{p.seed_resize_from_w}x{p.seed_resize_from_h}",
         "Init image hash": getattr(p, 'init_img_hash', None),
@@ -473,18 +476,19 @@ def create_infotext(p: StableDiffusionProcessing, all_prompts, all_seeds, all_su
         "Clip skip": p.clip_skip if p.clip_skip > 1 else None,
         # ensd
         "ENSD": shared.opts.eta_noise_seed_delta if shared.opts.eta_noise_seed_delta != 0 and sd_samplers_common.is_sampler_using_eta_noise_seed_delta(p) else None,
-        # enable_hr
-        "Latent sampler": p.latent_sampler if p.enable_hr else None,
-        "Image CFG scale": p.image_cfg_scale if p.enable_hr else None,
-        "Denoising strength": p.denoising_strength if p.enable_hr else None,
-        "Refiner start": p.refiner_start if p.enable_hr else None,
-        "Secondary steps": p.hr_second_pass_steps if p.enable_hr else None,
-        # restore_faces
+        # restore_faces, tiling
         "Face restoration": shared.opts.face_restoration_model if p.restore_faces else None,
+        "Tiling": p.tiling if p.tiling else None,
+        # enable_hr
+        "Prompt2": p.refiner_prompt if p.enable_hr and len(p.refiner_prompt) > 0 else None,
+        "Negative2": p.refiner_negative if p.enable_hr and len(p.refiner_negative) > 0 else None,
+        "Latent sampler": p.latent_sampler if p.enable_hr and p.latent_sampler != p.sampler_name else None,
+        "Denoising strength": p.denoising_strength if p.enable_hr else None,
         # sdnext
+        "Backend": 'Diffusers' if shared.backend == shared.Backend.DIFFUSERS else 'Original',
         "Version": git_commit,
-        "Pipeline": 'Diffusers' if shared.backend == shared.Backend.DIFFUSERS else 'Original',
-        "Operations": ', '.join(list(set(p.ops))) if len(p.ops) > 0 else None
+        "Comment": comment,
+        "Operations": ', '.join(list(set(p.ops))) if len(p.ops) > 0 else None,
     }
     token_merging_ratio = p.get_token_merging_ratio()
     token_merging_ratio_hr = p.get_token_merging_ratio(for_hr=True) if p.enable_hr else None
@@ -1053,6 +1057,7 @@ class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
         if add_color_corrections:
             self.color_corrections = []
         imgs = []
+        unprocessed = []
         for img in self.init_images:
             # Save init image
             if shared.opts.save_init_img:
@@ -1072,7 +1077,9 @@ class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
             if crop_region is not None:
                 image = image.crop(crop_region)
                 image = images.resize_image(3, image, self.width, self.height)
-            self.init_images = image # assign early for diffusers
+            if shared.backend == shared.Backend.DIFFUSERS:
+                unprocessed.append(image)
+            self.init_images = [image] # assign early for diffusers
             if image_mask is not None:
                 if self.inpainting_fill != 1:
                     image = masking.fill(image, latent_mask)
@@ -1081,6 +1088,7 @@ class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
             image = np.array(image).astype(np.float32) / 255.0
             image = np.moveaxis(image, 2, 0)
             imgs.append(image)
+        self.init_images = unprocessed if shared.backend == shared.Backend.DIFFUSERS else imgs
         if len(imgs) == 1:
             batch_images = np.expand_dims(imgs[0], axis=0).repeat(self.batch_size, axis=0)
             if self.overlay_images is not None:

modules/processing_diffusers.py

@@ -1,12 +1,16 @@
 import inspect
+import typing
 import torch
 import modules.devices as devices
 import modules.shared as shared
 import modules.sd_samplers as sd_samplers
 import modules.sd_models as sd_models
+import modules.sd_vae as sd_vae
+import modules.taesd.sd_vae_taesd as sd_vae_taesd
 import modules.images as images
 from modules.lora_diffusers import lora_state, unload_diffusers_lora
 from modules.processing import StableDiffusionProcessing
+import modules.prompt_parser_diffusers as prompt_parser_diffusers
 
 
 try:
@@ -15,16 +19,6 @@ except Exception as ex:
     shared.log.error(f'Failed to import diffusers: {ex}')
 
 
-def encode_prompt(encoder, prompt):
-    cfg = encoder.config
-    # TODO implement similar hijack for diffusers text encoder but following diffusers pipeline.encode_prompt concepts
-    # from modules import sd_hijack_clip
-    # model.text_encoder = sd_hijack_clip.FrozenCLIPEmbedderWithCustomWords(model.text_encoder, None)
-    shared.log.debug(f'Diffuser encoder: {encoder.__class__.__name__} dict={getattr(cfg, "vocab_size", None)} layers={getattr(cfg, "num_hidden_layers", None)} tokens={getattr(cfg, "max_position_embeddings", None)}')
-    embeds = prompt
-    return embeds
-
-
 def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_prompts):
     results = []
 
@@ -35,7 +29,10 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
 
     def vae_decode(latents, model, output_type='np'):
         if hasattr(model, 'vae') and torch.is_tensor(latents):
-            shared.log.debug(f'Diffusers VAE decode: name={model.vae.config.get("_name_or_path", "default")} dtype={model.vae.dtype} upcast={model.vae.config.get("force_upcast", None)}')
+            if latents.shape[0] == 0:
+                shared.log.error(f'VAE nothing to decode: {latents.shape}')
+                return []
+            shared.log.debug(f'Diffusers VAE decode: name={sd_vae.loaded_vae_file} dtype={model.vae.dtype} upcast={model.vae.config.get("force_upcast", None)} images={latents.shape[0]}')
             if shared.opts.diffusers_move_unet and not model.has_accelerate:
                 shared.log.debug('Diffusers: Moving UNet to CPU')
                 unet_device = model.unet.device
@@ -50,26 +47,69 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         else:
             return latents
 
+    def taesd_vae_decode(latents, model, output_type='np'):
+        shared.log.debug('Diffusers VAE decode: name=TAESD')
+        decoded = torch.zeros((len(latents), 3, p.height, p.width), dtype=devices.dtype_vae, device=devices.device)
+        for i in range(len(output.images)):
+            decoded[i] = (sd_vae_taesd.decode(latents[i]) * 2.0) - 1.0
+        imgs = model.image_processor.postprocess(decoded, output_type=output_type)
+        return imgs
 
-    def set_pipeline_args(model, prompt, negative_prompt, **kwargs):
+    def fix_prompts(prompts, negative_prompts, prompts_2, negative_prompts_2):
+        if type(prompts) is str:
+            prompts = [prompts]
+        if type(negative_prompts) is str:
+            negative_prompts = [negative_prompts]
+        while len(negative_prompts) < len(prompts):
+            negative_prompts.append(negative_prompts[-1])
+        if type(prompts_2) is str:
+            prompts_2 = [prompts_2]
+        if type(prompts_2) is list:
+            while len(prompts_2) < len(prompts):
+                prompts_2.append(prompts_2[-1])
+        if type(negative_prompts_2) is str:
+            negative_prompts_2 = [negative_prompts_2]
+        if type(negative_prompts_2) is list:
+            while len(negative_prompts_2) < len(prompts_2):
+                negative_prompts_2.append(negative_prompts_2[-1])
+        return prompts, negative_prompts, prompts_2, negative_prompts_2
+
+    def set_pipeline_args(model, prompts: list, negative_prompts: list, prompts_2: typing.Optional[list]=None, negative_prompts_2: typing.Optional[list]=None, is_refiner: bool=False, **kwargs):
         args = {}
         pipeline = model
         signature = inspect.signature(type(pipeline).__call__)
         possible = signature.parameters.keys()
         generator_device = devices.cpu if shared.opts.diffusers_generator_device == "cpu" else shared.device
         generator = [torch.Generator(generator_device).manual_seed(s) for s in seeds]
+        prompt_embed = None
+        pooled = None
+        negative_embed = None
+        negative_pooled = None
+        prompts, negative_prompts, prompts_2, negative_prompts_2 = fix_prompts(prompts, negative_prompts, prompts_2, negative_prompts_2)
+        if shared.opts.data['prompt_attention'] in {'Compel parser', 'Full parser'}:
+            prompt_embed, pooled, negative_embed, negative_pooled = prompt_parser_diffusers.compel_encode_prompts(model,
+                                                                                                                  prompts,
+                                                                                                                  negative_prompts,
+                                                                                                                  prompts_2,
+                                                                                                                  negative_prompts_2,
+                                                                                                                  is_refiner,
+                                                                                                                  kwargs.pop("clip_skip", None))
         if 'prompt' in possible:
-            if hasattr(model, 'text_encoder') and 'prompt_embeds' in possible:
-                # args['prompt_embeds'] = encode_prompt(model, prompt)
-                args['prompt'] = prompt
+            if hasattr(model, 'text_encoder') and 'prompt_embeds' in possible and prompt_embed is not None:
+                args['prompt_embeds'] = prompt_embed
+                if shared.sd_model_type == "sdxl":
+                    args['pooled_prompt_embeds'] = pooled
+                    args['prompt_2'] = None #Cannot pass prompts when passing embeds
             else:
-                args['prompt'] = prompt
+                args['prompt'] = prompts
         if 'negative_prompt' in possible:
-            if hasattr(model, 'text_encoder') and 'negative_prompt_embeds' in possible:
-                # args['negative_prompt_embeds'] = encode_prompt(model, negative_prompt)
-                args['negative_prompt'] = negative_prompt
+            if hasattr(model, 'text_encoder') and 'negative_prompt_embeds' in possible and negative_embed is not None:
+                args['negative_prompt_embeds'] = negative_embed
+                if shared.sd_model_type == "sdxl":
+                    args['negative_pooled_prompt_embeds'] = negative_pooled
+                    args['negative_prompt_2'] = None
             else:
-                args['negative_prompt'] = negative_prompt
+                args['negative_prompt'] = negative_prompts
         if 'num_inference_steps' in possible:
             args['num_inference_steps'] = p.steps
         if 'guidance_scale' in possible:
@@ -82,8 +122,8 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
             args['callback_steps'] = 1
         if 'callback' in possible:
             args['callback'] = diffusers_callback
-        if 'cross_attention_kwargs' in possible and lora_state['active']:
-            args['cross_attention_kwargs'] = { 'scale': lora_state['multiplier']}
+        if 'cross_attention_kwargs' in possible and lora_state['active'] and shared.opts.diffusers_lora_loader == "diffusers default":
+            args['cross_attention_kwargs'] = { 'scale': lora_state['multiplier'][0]}
         for arg in kwargs:
             if arg in possible:
                 args[arg] = kwargs[arg]
@@ -102,6 +142,14 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
             clean['prompt'] = len(clean['prompt'])
         if 'negative_prompt' in clean:
             clean['negative_prompt'] = len(clean['negative_prompt'])
+        if 'prompt_embeds' in clean:
+            clean['prompt_embeds'] = clean['prompt_embeds'].shape
+        if 'pooled_prompt_embeds' in clean:
+            clean['pooled_prompt_embeds'] = clean['pooled_prompt_embeds'].shape
+        if 'negative_prompt_embeds' in clean:
+            clean['negative_prompt_embeds'] = clean['negative_prompt_embeds'].shape
+        if 'negative_pooled_prompt_embeds' in clean:
+            clean['negative_pooled_prompt_embeds'] = clean['negative_pooled_prompt_embeds'].shape
         clean['generator'] = generator_device
         shared.log.debug(f'Diffuser pipeline: {pipeline.__class__.__name__} task={sd_models.get_diffusers_task(model)} set={clean}')
         return args
@@ -112,8 +160,19 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         if sampler is None:
             sampler = sd_samplers.all_samplers_map.get("UniPC")
         sd_samplers.create_sampler(sampler.name, shared.sd_model) # TODO(Patrick): For wrapped pipelines this is currently a no-op
+        sampler_options = f'type:{shared.opts.schedulers_prediction_type} ' if shared.opts.schedulers_prediction_type != 'default' else ''
+        sampler_options += 'no_karras ' if not shared.opts.schedulers_use_karras else ''
+        sampler_options += 'no_low_order' if not shared.opts.schedulers_use_loworder else ''
+        sampler_options += 'dynamic_thresholding' if shared.opts.schedulers_use_thresholding else ''
+        sampler_options += f'solver:{shared.opts.schedulers_dpm_solver}' if shared.opts.schedulers_dpm_solver != 'sde-dpmsolver++' else ''
+        sampler_options += f'beta:{shared.opts.schedulers_beta_schedule}:{shared.opts.schedulers_beta_start}:{shared.opts.schedulers_beta_end}' if shared.opts.schedulers_beta_schedule != 'default' else ''
+        p.extra_generation_params['Sampler options'] = sampler_options if len(sampler_options) > 0 else None
+        p.extra_generation_params['Pipeline'] = shared.sd_model.__class__.__name__
 
     cross_attention_kwargs={}
+    if len(getattr(p, 'init_images', [])) > 0:
+        while len(p.init_images) < len(prompts):
+            p.init_images.append(p.init_images[-1])
     if lora_state['active']:
         cross_attention_kwargs['scale'] = lora_state['multiplier']
     task_specific_kwargs={}
@@ -127,11 +186,8 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         p.ops.append('inpaint')
         task_specific_kwargs = {"image": p.init_images, "mask_image": p.mask, "strength": p.denoising_strength, "height": p.height, "width": p.width}
 
-    # TODO diffusers use transformers for prompt parsing
-    # from modules.prompt_parser import parse_prompt_attention
-    # parsed_prompt = [parse_prompt_attention(prompt) for prompt in prompts]
-
     if shared.state.interrupted or shared.state.skipped:
+        unload_diffusers_lora()
         return results
 
     if shared.opts.diffusers_move_base and not shared.sd_model.has_accelerate:
@@ -140,26 +196,31 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
     refiner_enabled = shared.sd_refiner is not None and p.enable_hr
     pipe_args = set_pipeline_args(
         model=shared.sd_model,
-        prompt=prompts,
-        negative_prompt=negative_prompts,
-        prompt_2=[p.refiner_prompt] if len(p.refiner_prompt) > 0 else prompts,
-        negative_prompt_2=[p.refiner_negative] if len(p.refiner_negative) > 0 else negative_prompts,
+        prompts=prompts,
+        negative_prompts=negative_prompts,
+        prompts_2=[p.refiner_prompt] if len(p.refiner_prompt) > 0 else prompts,
+        negative_prompts_2=[p.refiner_negative] if len(p.refiner_negative) > 0 else negative_prompts,
         eta=shared.opts.eta_ddim,
         guidance_rescale=p.diffusers_guidance_rescale,
         denoising_start=0 if refiner_enabled and p.refiner_start > 0 and p.refiner_start < 1 else None,
         denoising_end=p.refiner_start if refiner_enabled and p.refiner_start > 0 and p.refiner_start < 1 else None,
         output_type='latent' if hasattr(shared.sd_model, 'vae') else 'np',
+        is_refiner=False,
+        clip_skip=p.clip_skip,
         **task_specific_kwargs
     )
+    p.extra_generation_params['CFG rescale'] = p.diffusers_guidance_rescale
+    p.extra_generation_params["Eta DDIM"] = shared.opts.eta_ddim if shared.opts.eta_ddim is not None and shared.opts.eta_ddim > 0 else None
     output = shared.sd_model(**pipe_args) # pylint: disable=not-callable
-
     if shared.state.interrupted or shared.state.skipped:
+        unload_diffusers_lora()
         return results
 
     if shared.sd_refiner is None or not p.enable_hr:
-        output.images = vae_decode(output.images, shared.sd_model)
+        output.images = vae_decode(output.images, shared.sd_model) if p.full_quality else taesd_vae_decode(output.images, shared.sd_model)
 
     if lora_state['active']:
+        p.extra_generation_params['Lora method'] = shared.opts.diffusers_lora_loader
         unload_diffusers_lora()
 
     if refiner_enabled:
@@ -191,8 +252,8 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         for i in range(len(output.images)):
             pipe_args = set_pipeline_args(
                 model=shared.sd_refiner,
-                prompt=[p.refiner_prompt] if len(p.refiner_prompt) > 0 else prompts[i],
-                negative_prompt=[p.refiner_negative] if len(p.refiner_negative) > 0 else negative_prompts[i],
+                prompts=[p.refiner_prompt] if len(p.refiner_prompt) > 0 else prompts[i],
+                negative_prompts=[p.refiner_negative] if len(p.refiner_negative) > 0 else negative_prompts[i],
                 num_inference_steps=p.hr_second_pass_steps,
                 eta=shared.opts.eta_ddim,
                 strength=p.denoising_strength,
@@ -202,8 +263,14 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
                 denoising_end=1 if p.refiner_start > 0 and p.refiner_start < 1 else None,
                 image=output.images[i],
                 output_type='latent' if hasattr(shared.sd_refiner, 'vae') else 'np',
+                is_refiner=True,
+                clip_skip=p.clip_skip,
             )
             refiner_output = shared.sd_refiner(**pipe_args) # pylint: disable=not-callable
+            p.extra_generation_params['Refiner CFG scale'] = p.image_cfg_scale if p.image_cfg_scale is not None else None
+            p.extra_generation_params['Refiner start'] = p.refiner_start
+            p.extra_generation_params["Hires steps"] = p.hr_second_pass_steps
+
             if not shared.state.interrupted and not shared.state.skipped:
                 refiner_images = vae_decode(refiner_output.images, shared.sd_refiner)
                 results.append(refiner_images[0])

modules/prompt_parser_diffusers.py

@@ -0,0 +1,131 @@
+import os
+import typing
+import torch
+import diffusers
+from compel import Compel, ReturnedEmbeddingsType
+import modules.shared as shared
+import modules.prompt_parser as prompt_parser
+
+debug_output = os.environ.get('SD_PROMPT_DEBUG', None)
+debug = shared.log.info if debug_output is not None else lambda *args, **kwargs: None
+
+
+def convert_to_compel(prompt: str):
+    if prompt is None:
+        return None
+    all_schedules = prompt_parser.get_learned_conditioning_prompt_schedules([prompt], 100)[0]
+    output_list = prompt_parser.parse_prompt_attention(all_schedules[0][1])
+    converted_prompt = []
+    for subprompt, weight in output_list:
+        if subprompt != " ":
+            if weight == 1:
+                converted_prompt.append(subprompt)
+            else:
+                converted_prompt.append(f"({subprompt}){weight}")
+    converted_prompt = " ".join(converted_prompt)
+    return converted_prompt
+
+
+CLIP_SKIP_MAPPING = {
+    None: ReturnedEmbeddingsType.LAST_HIDDEN_STATES_NORMALIZED,
+    1: ReturnedEmbeddingsType.LAST_HIDDEN_STATES_NORMALIZED,
+    2: ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NORMALIZED,
+}
+
+
+def compel_encode_prompts(
+    pipeline: diffusers.StableDiffusionXLPipeline | diffusers.StableDiffusionPipeline,
+    prompts: list,
+    negative_prompts: list,
+    prompts_2: typing.Optional[list] = None,
+    negative_prompts_2: typing.Optional[list] = None,
+    is_refiner: bool = None,
+    clip_skip: typing.Optional[int] = None,
+):
+    prompt_embeds = []
+    positive_pooleds = []
+    negative_embeds = []
+    negative_pooleds = []
+    for i in range(len(prompts)):
+        prompt_embed, positive_pooled, negative_embed, negative_pooled = compel_encode_prompt(pipeline,
+                                                                                              prompts[i],
+                                                                                              negative_prompts[i],
+                                                                                              prompts_2[i] if prompts_2 is not None else None,
+                                                                                              negative_prompts_2[i] if negative_prompts_2 is not None else None,
+                                                                                              is_refiner, clip_skip)
+        prompt_embeds.append(prompt_embed)
+        positive_pooleds.append(positive_pooled)
+        negative_embeds.append(negative_embed)
+        negative_pooleds.append(negative_pooled)
+
+    prompt_embeds = torch.cat(prompt_embeds, dim=0)
+    negative_embeds = torch.cat(negative_embeds, dim=0)
+    if shared.sd_model_type == "sdxl":
+        positive_pooleds = torch.cat(positive_pooleds, dim=0)
+        negative_pooleds = torch.cat(negative_pooleds, dim=0)
+    return prompt_embeds, positive_pooleds, negative_embeds, negative_pooleds
+
+
+def compel_encode_prompt(
+    pipeline: diffusers.StableDiffusionXLPipeline | diffusers.StableDiffusionPipeline,
+    prompt: str,
+    negative_prompt: str,
+    prompt_2: typing.Optional[str] = None,
+    negative_prompt_2: typing.Optional[str] = None,
+    is_refiner: bool = None,
+    clip_skip: typing.Optional[int] = None,
+):
+    if shared.sd_model_type not in {"sd", "sdxl"}:
+        shared.log.warning(f"Prompt parser: Compel not supported: {type(pipeline).__name__}")
+        return (None, None, None, None)
+
+    if shared.sd_model_type == "sdxl":
+        embedding_type = ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED
+        if clip_skip is not None and clip_skip > 1:
+            shared.log.warning(f"Prompt parser SDXL unsupported: clip_skip={clip_skip}")
+    else:
+        embedding_type = CLIP_SKIP_MAPPING.get(clip_skip, ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NORMALIZED)
+        if clip_skip not in CLIP_SKIP_MAPPING:
+            shared.log.warning(f"Prompt parser unsupported: clip_skip={clip_skip} expected={set(CLIP_SKIP_MAPPING.keys())}")
+
+    if shared.opts.data["prompt_attention"] != "Compel parser":
+        prompt = convert_to_compel(prompt)
+        negative_prompt = convert_to_compel(negative_prompt)
+        prompt_2 = convert_to_compel(prompt_2)
+        negative_prompt_2 = convert_to_compel(negative_prompt_2)
+
+    compel_te1 = Compel(
+        tokenizer=pipeline.tokenizer,
+        text_encoder=pipeline.text_encoder,
+        returned_embeddings_type=embedding_type,
+        requires_pooled=False,
+        # truncate_long_prompts=False,
+        device=shared.device
+    )
+
+    if shared.sd_model_type == "sdxl":
+        compel_te2 = Compel(
+            tokenizer=pipeline.tokenizer_2,
+            text_encoder=pipeline.text_encoder_2,
+            returned_embeddings_type=embedding_type,
+            requires_pooled=True,
+            device=shared.device
+        )
+        if not is_refiner:
+            positive_te1 = compel_te1(prompt)
+            positive_te2, positive_pooled = compel_te2(prompt_2)
+            positive = torch.cat((positive_te1, positive_te2), dim=-1)
+            negative_te1 = compel_te1(negative_prompt)
+            negative_te2, negative_pooled = compel_te2(negative_prompt_2)
+            negative = torch.cat((negative_te1, negative_te2), dim=-1)
+        else:
+            positive, positive_pooled = compel_te2(prompt)
+            negative, negative_pooled = compel_te2(negative_prompt)
+        parsed = compel_te1.parse_prompt_string(prompt)
+        debug(f"Prompt parser Compel: {parsed}")
+        [prompt_embed, negative_embed] = compel_te2.pad_conditioning_tensors_to_same_length([positive, negative])
+        return prompt_embed, positive_pooled, negative_embed, negative_pooled
+
+    positive, negative = compel_te1(prompt), compel_te1(negative_prompt)
+    [prompt_embed, negative_embed] = compel_te1.pad_conditioning_tensors_to_same_length([positive, negative])
+    return prompt_embed, None, negative_embed, None

modules/sd_models.py

@@ -136,12 +136,9 @@ def list_models():
     checkpoints_list.clear()
     checkpoint_aliases.clear()
     ext_filter=[".safetensors"] if shared.opts.sd_disable_ckpt else [".ckpt", ".safetensors"]
-    model_list = []
-    if shared.backend == shared.Backend.ORIGINAL or shared.opts.diffusers_allow_safetensors:
-        model_list += modelloader.load_models(model_path=model_path, model_url=None, command_path=shared.opts.ckpt_dir, ext_filter=ext_filter, download_name=None, ext_blacklist=[".vae.ckpt", ".vae.safetensors"])
+    model_list = modelloader.load_models(model_path=model_path, model_url=None, command_path=shared.opts.ckpt_dir, ext_filter=ext_filter, download_name=None, ext_blacklist=[".vae.ckpt", ".vae.safetensors"])
     if shared.backend == shared.Backend.DIFFUSERS:
         model_list += modelloader.load_diffusers_models(model_path=os.path.join(models_path, 'Diffusers'), command_path=shared.opts.diffusers_dir)
-
     for filename in sorted(model_list, key=str.lower):
         checkpoint_info = CheckpointInfo(filename)
         if checkpoint_info.name is not None:
@@ -522,6 +519,7 @@ class ModelData:
 
 model_data = ModelData()
 
+
 def change_backend():
     shared.log.info(f'Pipeline changed: {shared.backend}')
     unload_model_weights()
@@ -533,6 +531,64 @@ def change_backend():
     refresh_vae_list()
 
 
+def detect_pipeline(f: str, op: str = 'model'):
+    if not f.endswith('.safetensors'):
+        return None, None
+    guess = shared.opts.diffusers_pipeline
+    if guess == 'Autodetect':
+        try:
+            size = round(os.path.getsize(f) / 1024 / 1024 / 1024, 2)
+            if size < 1:
+                shared.log.warning(f'Model size smaller than expected: {f} size={size} GB')
+            elif size < 5:
+                guess = 'Stable Diffusion'
+            elif size < 6:
+                if op == 'model':
+                    shared.log.warning(f'Model detected as SD-XL refiner model, but attempting to load a base model: {f} size={size} GB')
+                else:
+                    guess = 'Stable Diffusion XL'
+            elif size < 7:
+                if op == 'refiner':
+                    shared.log.warning(f'Model size matches SD-XL base model, but attempting to load a refiner model: {f} size={size} GB')
+                else:
+                    guess = 'Stable Diffusion XL'
+            else:
+                shared.log.error(f'Diffusers autodetect failed, set diffuser pipeline manually: {f}')
+                return None, None
+            shared.log.debug(f'Diffusers autodetect {op}: {f} pipeline={guess} size={size} GB')
+        except Exception as e:
+            shared.log.error(f'Error detecting diffusers pipeline: model={f} {e}')
+            return None, None
+    if guess == shared.pipelines[1]:
+        pipeline = diffusers.StableDiffusionPipeline
+    elif guess == shared.pipelines[2]:
+        pipeline = diffusers.StableDiffusionXLPipeline
+    elif guess == shared.pipelines[3]:
+        pipeline = diffusers.KandinskyPipeline
+    elif guess == shared.pipelines[4]:
+        pipeline = diffusers.KandinskyV22Pipeline
+    elif guess == shared.pipelines[5]:
+        pipeline = diffusers.IFPipeline
+    elif guess == shared.pipelines[6]:
+        pipeline = diffusers.ShapEPipeline
+    elif guess == shared.pipelines[7]:
+        pipeline = diffusers.StableDiffusionImg2ImgPipeline
+    elif guess == shared.pipelines[8]:
+        pipeline = diffusers.StableDiffusionXLImg2ImgPipeline
+    elif guess == shared.pipelines[9]:
+        pipeline = diffusers.KandinskyImg2ImgPipeline
+    elif guess == shared.pipelines[10]:
+        pipeline = diffusers.KandinskyV22Img2ImgPipeline
+    elif guess == shared.pipelines[11]:
+        pipeline = diffusers.IFImg2ImgPipeline
+    elif guess == shared.pipelines[12]:
+        pipeline = diffusers.ShapEImg2ImgPipeline
+    else:
+        shared.log.error(f'Diffusers unknown pipeline: {guess}')
+        pipeline = None, None
+    return pipeline, guess
+
+
 def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=None, op='model'): # pylint: disable=unused-argument
     import torch # pylint: disable=reimported,redefined-outer-name
     if timer is None:
@@ -570,7 +626,7 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
     sd_model = None
 
     try:
-        if shared.cmd_opts.ckpt is not None and model_data.initial: # initial load\
+        if shared.cmd_opts.ckpt is not None and model_data.initial: # initial load
             ckpt_basename = os.path.basename(shared.cmd_opts.ckpt)
             model_name = modelloader.find_diffuser(ckpt_basename)
             if model_name is not None:
@@ -593,51 +649,26 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
 
             devices.set_cuda_params()
             vae = None
+            sd_vae.loaded_vae_file = None
             if op == 'model' or op == 'refiner':
                 vae_file, vae_source = sd_vae.resolve_vae(checkpoint_info.filename)
-                vae = sd_vae.load_vae_diffusers(None, vae_file, vae_source)
+                vae = sd_vae.load_vae_diffusers(checkpoint_info.path, vae_file, vae_source)
                 if vae is not None:
                     diffusers_load_config["vae"] = vae
 
             shared.log.info(f'Loading diffuser {op}: {checkpoint_info.filename}')
             if not os.path.isfile(checkpoint_info.path):
                 try:
-                    shared.log.debug(f'Diffusers load {op} config: {diffusers_load_config}')
+                    # shared.log.debug(f'Diffusers load {op} config: {diffusers_load_config}')
                     sd_model = diffusers.DiffusionPipeline.from_pretrained(checkpoint_info.path, **diffusers_load_config)
                 except Exception as e:
                     shared.log.error(f'Diffusers {op} failed loading model: {checkpoint_info.path} {e}')
             else:
                 diffusers_load_config["local_files_only "] = True
                 diffusers_load_config["extract_ema"] = shared.opts.diffusers_extract_ema
-                try:
-                    if shared.opts.diffusers_pipeline == shared.pipelines[0]:
-                        pipeline = diffusers.StableDiffusionPipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[1]:
-                        pipeline = diffusers.StableDiffusionXLPipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[2]:
-                        pipeline = diffusers.KandinskyPipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[3]:
-                        pipeline = diffusers.KandinskyV22Pipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[4]:
-                        pipeline = diffusers.IFPipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[5]:
-                        pipeline = diffusers.ShapEPipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[6]:
-                        pipeline = diffusers.StableDiffusionImg2ImgPipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[7]:
-                        pipeline = diffusers.StableDiffusionXLImg2ImgPipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[8]:
-                        pipeline = diffusers.KandinskyImg2ImgPipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[9]:
-                        pipeline = diffusers.KandinskyV22Img2ImgPipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[10]:
-                        pipeline = diffusers.IFImg2ImgPipeline
-                    elif shared.opts.diffusers_pipeline == shared.pipelines[11]:
-                        pipeline = diffusers.ShapEImg2ImgPipeline
-                    else:
-                        shared.log.error(f'Diffusers {op} unknown pipeline: {shared.opts.diffusers_pipeline}')
-                except Exception as e:
-                    shared.log.error(f'Diffusers {op} failed initializing pipeline: {shared.opts.diffusers_pipeline} {e}')
+                pipeline, _model_type = detect_pipeline(checkpoint_info.path, op)
+                if pipeline is None:
+                    shared.log.error(f'Diffusers {op} pipeline not initialized: {shared.opts.diffusers_pipeline}')
                     return
                 try:
                     if hasattr(pipeline, 'from_single_file'):
@@ -697,6 +728,8 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
             else:
                 sd_model.disable_attention_slicing()
         if hasattr(sd_model, "vae"):
+            if vae is not None:
+                sd_model.vae = vae
             if shared.opts.diffusers_vae_upcast != 'default':
                 if shared.opts.diffusers_vae_upcast == 'true':
                     sd_model.vae.config["force_upcast"] = True
@@ -704,7 +737,7 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
                 else:
                     sd_model.vae.config["force_upcast"] = False
                     sd_model.vae.config.force_upcast = False
-            shared.log.debug(f'Diffusers {op} VAE: name={sd_model.vae.config.get("_name_or_path", "default")} upcast={sd_model.vae.config.get("force_upcast", None)}')
+            shared.log.debug(f'Diffusers {op} VAE: name={sd_vae.loaded_vae_file} upcast={sd_model.vae.config.get("force_upcast", None)}')
         if shared.opts.cross_attention_optimization == "xFormers" and hasattr(sd_model, 'enable_xformers_memory_efficient_attention'):
             sd_model.enable_xformers_memory_efficient_attention()
         if shared.opts.opt_channelslast:
@@ -767,8 +800,7 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
         if op == 'refiner' and shared.opts.diffusers_move_refiner and not sd_model.has_accelerate:
             shared.log.debug('Moving refiner model to CPU')
             sd_model.to(devices.cpu)
-        elif not sd_model.has_accelerate:
-            # In offload modes, accelerate will move models around.
+        elif not sd_model.has_accelerate: # In offload modes, accelerate will move models around
             sd_model.to(devices.device)
         if op == 'refiner' and base_sent_to_cpu:
             shared.log.debug('Moving base model back to GPU')
@@ -811,7 +843,6 @@ def set_diffuser_pipe(pipe, new_pipe_type):
         new_pipe = diffusers.AutoPipelineForImage2Image.from_pipe(pipe)
     elif new_pipe_type == DiffusersTaskType.INPAINTING:
         new_pipe = diffusers.AutoPipelineForInpainting.from_pipe(pipe)
-
     if pipe.__class__ == new_pipe.__class__:
         return
 
@@ -843,7 +874,6 @@ def get_diffusers_task(pipe: diffusers.DiffusionPipeline) -> DiffusersTaskType:
         return DiffusersTaskType.IMAGE_2_IMAGE
     elif pipe.__class__ in diffusers.pipelines.auto_pipeline.AUTO_INPAINT_PIPELINES_MAPPING.values():
         return DiffusersTaskType.INPAINTING
-
     return DiffusersTaskType.TEXT_2_IMAGE
 
 
@@ -997,24 +1027,38 @@ def reload_model_weights(sd_model=None, info=None, reuse_dict=False, op='model')
     shared.log.info(f"Weights loaded in {timer.summary()}")
 
 
+def disable_offload(sd_model):
+    from accelerate.hooks import remove_hook_from_module
+    if not sd_model.has_accelerate:
+        return
+    for _name, model in sd_model.components.items():
+        if not isinstance(model, torch.nn.Module):
+            continue
+        remove_hook_from_module(model, recurse=True)
+
+
 def unload_model_weights(op='model'):
     from modules import sd_hijack
     if op == 'model' or op == 'dict':
         if model_data.sd_model:
-            if not model_data.sd_model.has_accelerate:
-                model_data.sd_model.to(devices.cpu)
             if shared.backend == shared.Backend.ORIGINAL:
+                model_data.sd_model.to(devices.cpu)
                 sd_hijack.model_hijack.undo_hijack(model_data.sd_model)
+            else:
+                disable_offload(model_data.sd_model)
+                model_data.sd_model.to('meta')
             model_data.sd_model = None
-            shared.log.debug(f'Weights unloaded {op}: {memory_stats()}')
+            shared.log.debug(f'Unload weights {op}: {memory_stats()}')
     else:
         if model_data.sd_refiner:
-            if not model_data.sd_refiner.has_accelerate:
-                model_data.sd_refiner.to(devices.cpu)
             if shared.backend == shared.Backend.ORIGINAL:
+                model_data.sd_model.to(devices.cpu)
                 sd_hijack.model_hijack.undo_hijack(model_data.sd_refiner)
+            else:
+                disable_offload(model_data.sd_model)
+                model_data.sd_refiner.to('meta')
             model_data.sd_refiner = None
-            shared.log.debug(f'Weights unloaded {op}: {memory_stats()}')
+            shared.log.debug(f'Unload weights {op}: {memory_stats()}')
     devices.torch_gc(force=True)
 
 

modules/sd_samplers_kdiffusion.py

@@ -19,8 +19,10 @@ samplers_k_diffusion = [
     ('DPM++ 2M Karras', 'sample_dpmpp_2m', ['k_dpmpp_2m_ka'], {'scheduler': 'karras'}),
     ('DPM++ SDE', 'sample_dpmpp_sde', ['k_dpmpp_sde'], {"second_order": True, "brownian_noise": True}),
     ('DPM++ SDE Karras', 'sample_dpmpp_sde', ['k_dpmpp_sde_ka'], {'scheduler': 'karras', "second_order": True, "brownian_noise": True}),
-    ('DPM++ 2M SDE', 'sample_dpmpp_2m_sde', ['k_dpmpp_2m_sde_ka'], {"brownian_noise": True, 'discard_next_to_last_sigma': True}),
+    ('DPM++ 2M SDE', 'sample_dpmpp_2m_sde', ['k_dpmpp_2m_sde'], {"brownian_noise": True, 'discard_next_to_last_sigma': True}),
     ('DPM++ 2M SDE Karras', 'sample_dpmpp_2m_sde', ['k_dpmpp_2m_sde_ka'], {'scheduler': 'karras', "brownian_noise": True, 'discard_next_to_last_sigma': True}),
+    ('DPM++ 3M SDE', 'sample_dpmpp_3m_sde', ['k_dpmpp_3m_sde'], {"brownian_noise": True, 'discard_next_to_last_sigma': True}),
+    ('DPM++ 3M SDE Karras', 'sample_dpmpp_3m_sde', ['k_dpmpp_3m_sde_ka'], {'scheduler': 'karras', "brownian_noise": True, 'discard_next_to_last_sigma': True}),
     ('DPM fast', 'sample_dpm_fast', ['k_dpm_fast'], {"uses_ensd": True}),
     ('DPM adaptive', 'sample_dpm_adaptive', ['k_dpm_ad'], {"uses_ensd": True}),
     ('DPM2', 'sample_dpm_2', ['k_dpm_2'], {'discard_next_to_last_sigma': True}),

modules/sd_vae.py

@@ -3,7 +3,7 @@ import collections
 import glob
 from copy import deepcopy
 import torch
-from modules import shared, paths, devices, script_callbacks, sd_models
+from modules import shared, paths, paths_internal, devices, script_callbacks, sd_models
 
 
 vae_ignore_keys = {"model_ema.decay", "model_ema.num_updates"}
@@ -169,7 +169,7 @@ def load_vae(model, vae_file=None, vae_source="from unknown source"):
     loaded_vae_file = vae_file
 
 
-def load_vae_diffusers(_model, vae_file=None, vae_source="from unknown source"):
+def load_vae_diffusers(model_file, vae_file=None, vae_source="from unknown source"):
     if vae_file is None:
         return None
     if not os.path.exists(vae_file):
@@ -196,14 +196,14 @@ def load_vae_diffusers(_model, vae_file=None, vae_source="from unknown source"):
     try:
         import diffusers
         if os.path.isfile(vae_file):
-            if shared.opts.diffusers_pipeline == "Stable Diffusion XL":
-                # load_config passed to from_single_file doesn't apply
-                # from_single_file by default downloads VAE1.5 config
-                shared.log.warning("Using SDXL VAE loaded from singular file will result in low contrast images.")
-            vae = diffusers.AutoencoderKL.from_single_file(vae_file)
+            _pipeline, model_type = sd_models.detect_pipeline(model_file, 'vae')
+            diffusers_load_config = { "config_file":  paths_internal.sd_default_config if model_type != 'Stable Diffusion XL' else os.path.join(paths_internal.sd_configs_path, 'sd_xl_base.yaml')}
+            vae = diffusers.AutoencoderKL.from_single_file(vae_file, **diffusers_load_config)
             vae = vae.to(devices.dtype_vae)
         else:
             vae = diffusers.AutoencoderKL.from_pretrained(vae_file, **diffusers_load_config)
+        global loaded_vae_file # pylint: disable=global-statement
+        loaded_vae_file = os.path.basename(vae_file)
         # shared.log.debug(f'Diffusers VAE config: {vae.config}')
         return vae
     except Exception as e:

modules/shared.py

@@ -41,6 +41,7 @@ loaded_hypernetworks = []
 gradio_theme = gr.themes.Base()
 settings_components = None
 pipelines = [
+    'Autodetect',
     'Stable Diffusion', 'Stable Diffusion XL', 'Kandinsky V1', 'Kandinsky V2', 'DeepFloyd IF', 'Shap-E',
     'Stable Diffusion Img2Img', 'Stable Diffusion XL Img2Img', 'Kandinsky V1 Img2Img', 'Kandinsky V2 Img2Img', 'DeepFloyd IF Img2Img', 'Shap-E Img2Img'
 ]
@@ -383,7 +384,7 @@ options_templates.update(options_section(('cuda', "Compute Settings"), {
     # "cuda_allow_tf32": OptionInfo(True, "Allow TF32 math ops"),
     # "cuda_allow_tf16_reduced": OptionInfo(True, "Allow TF16 reduced precision math ops"),
     "cuda_compile": OptionInfo(False, "Enable model compile (experimental)"),
-    "cuda_compile_backend": OptionInfo("none", "Model compile backend (experimental)", gr.Radio, lambda: {"choices": ['none', 'inductor', 'cudagraphs', 'aot_ts_nvfuser', 'hidet', 'ipex']}),
+    "cuda_compile_backend": OptionInfo("none", "Model compile backend (experimental)", gr.Radio, lambda: {"choices": ['none', 'inductor', 'cudagraphs', 'aot_ts_nvfuser', 'hidet', 'ipex', 'openvino_fx']}),
     "cuda_compile_mode": OptionInfo("default", "Model compile mode (experimental)", gr.Radio, lambda: {"choices": ['default', 'reduce-overhead', 'max-autotune']}),
     "cuda_compile_fullgraph": OptionInfo(False, "Model compile fullgraph"),
     "cuda_compile_verbose": OptionInfo(False, "Model compile verbose mode"),
@@ -394,11 +395,10 @@ options_templates.update(options_section(('cuda', "Compute Settings"), {
 }))
 
 options_templates.update(options_section(('diffusers', "Diffusers Settings"), {
-    "diffusers_allow_safetensors": OptionInfo(True, 'Diffusers allow loading from safetensors files'),
     "diffusers_pipeline": OptionInfo(pipelines[0], 'Diffusers pipeline', gr.Dropdown, lambda: {"choices": pipelines}),
     "diffusers_move_base": OptionInfo(False, "Move base model to CPU when using refiner"),
+    "diffusers_move_unet": OptionInfo(False, "Move base model to CPU when using VAE"),
     "diffusers_move_refiner": OptionInfo(True, "Move refiner model to CPU when not in use"),
-    "diffusers_move_unet": OptionInfo(False, "Move UNet to CPU while VAE decoding"),
     "diffusers_extract_ema": OptionInfo(True, "Use model EMA weights when possible"),
     "diffusers_generator_device": OptionInfo("default", "Generator device", gr.Radio, lambda: {"choices": ["default", "cpu"]}),
     "diffusers_seq_cpu_offload": OptionInfo(False, "Enable sequential CPU offload"),
@@ -409,6 +409,7 @@ options_templates.update(options_section(('diffusers', "Diffusers Settings"), {
     "diffusers_attention_slicing": OptionInfo(False, "Enable attention slicing"),
     "diffusers_model_load_variant": OptionInfo("default", "Diffusers model loading variant", gr.Radio, lambda: {"choices": ['default', 'fp32', 'fp16']}),
     "diffusers_vae_load_variant": OptionInfo("default", "Diffusers VAE loading variant", gr.Radio, lambda: {"choices": ['default', 'fp32', 'fp16']}),
+    "diffusers_lora_loader": OptionInfo("sequential apply", "Diffusers LoRA loading variant", gr.Radio, lambda: {"choices": ['sequential apply', 'merge and apply', 'diffusers default']}),
     # "diffusers_force_zeros": OptionInfo(False, "Force zeros for prompts when empty"),
     # "diffusers_aesthetics_score": OptionInfo(6.0, "Require aesthetic score", gr.Slider, {"minimum": 0, "maximum": 10, "step": 0.1}),
 }))

modules/textual_inversion/image_embedding.py

@@ -113,7 +113,6 @@ def extract_image_data_embed(image):
     outarr = crop_black(np.array(image.convert('RGB').getdata()).reshape(image.size[1], image.size[0], d).astype(np.uint8)) & 0x0F
     black_cols = np.where(np.sum(outarr, axis=(0, 2)) == 0)
     if black_cols[0].shape[0] < 2:
-        print('No Image data blocks found.')
         return None
 
     data_block_lower = outarr[:, :black_cols[0].min(), :].astype(np.uint8)

modules/txt2img.py

@@ -5,9 +5,9 @@ from modules.ui import plaintext_to_html
 from modules.memstats import memory_stats
 
 
-def txt2img(id_task: str, prompt: str, negative_prompt: str, prompt_styles, steps: int, sampler_index: int, latent_index: int, restore_faces: bool, tiling: bool, n_iter: int, batch_size: int, cfg_scale: float, image_cfg_scale: float, diffusers_guidance_rescale: float, clip_skip: int, seed: int, subseed: int, subseed_strength: float, seed_resize_from_h: int, seed_resize_from_w: int, height: int, width: int, enable_hr: bool, denoising_strength: float, hr_scale: float, hr_upscaler: str, hr_second_pass_steps: int, hr_resize_x: int, hr_resize_y: int, refiner_start: int, refiner_prompt: str, refiner_negative: str, override_settings_texts, *args): # pylint: disable=unused-argument
+def txt2img(id_task: str, prompt: str, negative_prompt: str, prompt_styles, steps: int, sampler_index: int, latent_index: int, full_quality: bool, restore_faces: bool, tiling: bool, n_iter: int, batch_size: int, cfg_scale: float, image_cfg_scale: float, diffusers_guidance_rescale: float, clip_skip: int, seed: int, subseed: int, subseed_strength: float, seed_resize_from_h: int, seed_resize_from_w: int, height: int, width: int, enable_hr: bool, denoising_strength: float, hr_scale: float, hr_upscaler: str, hr_second_pass_steps: int, hr_resize_x: int, hr_resize_y: int, refiner_start: int, refiner_prompt: str, refiner_negative: str, override_settings_texts, *args): # pylint: disable=unused-argument
 
-    shared.log.debug(f'txt2img: id_task={id_task}|prompt={prompt}|negative_prompt={negative_prompt}|prompt_styles={prompt_styles}|steps={steps}|sampler_index={sampler_index}|latent_index={latent_index}|restore_faces={restore_faces}|tiling={tiling}|n_iter={n_iter}|batch_size={batch_size}|cfg_scale={cfg_scale}|clip_skip={clip_skip}|seed={seed}|subseed={subseed}|subseed_strength={subseed_strength}|seed_resize_from_h={seed_resize_from_h}|seed_resize_from_w={seed_resize_from_w}||height={height}|width={width}|enable_hr={enable_hr}|denoising_strength={denoising_strength}|hr_scale={hr_scale}|hr_upscaler={hr_upscaler}|hr_second_pass_steps={hr_second_pass_steps}|hr_resize_x={hr_resize_x}|hr_resize_y={hr_resize_y}|image_cfg_scale={image_cfg_scale}|diffusers_guidance_rescale={diffusers_guidance_rescale}|refiner_start={refiner_start}||refiner_prompt={refiner_prompt}|refiner_negative={refiner_negative}|override_settings_texts={override_settings_texts}|args={args}')
+    shared.log.debug(f'txt2img: id_task={id_task}|prompt={prompt}|negative_prompt={negative_prompt}|prompt_styles={prompt_styles}|steps={steps}|sampler_index={sampler_index}|latent_index={latent_index}|full_quality={full_quality}|restore_faces={restore_faces}|tiling={tiling}|n_iter={n_iter}|batch_size={batch_size}|cfg_scale={cfg_scale}|clip_skip={clip_skip}|seed={seed}|subseed={subseed}|subseed_strength={subseed_strength}|seed_resize_from_h={seed_resize_from_h}|seed_resize_from_w={seed_resize_from_w}||height={height}|width={width}|enable_hr={enable_hr}|denoising_strength={denoising_strength}|hr_scale={hr_scale}|hr_upscaler={hr_upscaler}|hr_second_pass_steps={hr_second_pass_steps}|hr_resize_x={hr_resize_x}|hr_resize_y={hr_resize_y}|image_cfg_scale={image_cfg_scale}|diffusers_guidance_rescale={diffusers_guidance_rescale}|refiner_start={refiner_start}||refiner_prompt={refiner_prompt}|refiner_negative={refiner_negative}|override_settings_texts={override_settings_texts}|args={args}')
 
     if shared.sd_model is None:
         shared.log.warning('Model not loaded')
@@ -43,6 +43,7 @@ def txt2img(id_task: str, prompt: str, negative_prompt: str, prompt_styles, step
         clip_skip=clip_skip,
         width=width,
         height=height,
+        full_quality=full_quality,
         restore_faces=restore_faces,
         tiling=tiling,
         enable_hr=enable_hr,

modules/ui.py

@@ -379,6 +379,7 @@ def create_ui(startup_timer = None):
                         cfg_scale = gr.Slider(minimum=1.0, maximum=30.0, step=0.1, label='CFG Scale', value=6.0, elem_id="txt2img_cfg_scale")
                         clip_skip = gr.Slider(label='CLIP skip', value=1, minimum=1, maximum=14, step=1, elem_id='txt2img_clip_skip', interactive=True)
                     with FormRow(elem_classes="checkboxes-row", variant="compact"):
+                        full_quality = gr.Checkbox(label='Full quality', value=True, elem_id="txt2img_full_quality")
                         restore_faces = gr.Checkbox(label='Face restore', value=False, visible=len(modules.shared.face_restorers) > 1, elem_id="txt2img_restore_faces")
                         tiling = gr.Checkbox(label='Tiling', value=False, elem_id="txt2img_tiling")
 
@@ -435,7 +436,7 @@ def create_ui(startup_timer = None):
                     txt2img_prompt_styles,
                     steps,
                     sampler_index, latent_index,
-                    restore_faces, tiling,
+                    full_quality, restore_faces, tiling,
                     batch_count, batch_size,
                     cfg_scale, image_cfg_scale,
                     diffusers_guidance_rescale,
@@ -484,6 +485,7 @@ def create_ui(startup_timer = None):
                 (latent_index, "Latent sampler"),
                 (denoising_strength, "Denoising strength"),
                 (refiner_start, "Refiner start"),
+                (full_quality, "Full quality"),
                 (restore_faces, "Face restoration"),
                 (batch_size, "Batch size"),
                 (batch_count, "Batch count"),
@@ -669,6 +671,7 @@ def create_ui(startup_timer = None):
                         clip_skip = gr.Slider(label='CLIP skip', value=1, minimum=1, maximum=4, step=1, elem_id='img2img_clip_skip', interactive=True)
                         diffusers_guidance_rescale = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label='Guidance Rescale', value=0.7, elem_id="txt2img_image_cfg_rescale")
                     with FormRow(elem_classes="img2img_checkboxes_row", variant="compact"):
+                        full_quality = gr.Checkbox(label='Full quality', value=True, elem_id="img2img_full_quality")
                         restore_faces = gr.Checkbox(label='Restore faces', value=False, visible=len(modules.shared.face_restorers) > 1, elem_id="img2img_restore_faces")
                         tiling = gr.Checkbox(label='Tiling', value=False, elem_id="img2img_tiling")
 
@@ -733,7 +736,7 @@ def create_ui(startup_timer = None):
                     sampler_index, latent_index,
                     mask_blur, mask_alpha,
                     inpainting_fill,
-                    restore_faces, tiling,
+                    full_quality, restore_faces, tiling,
                     batch_count, batch_size,
                     cfg_scale, image_cfg_scale,
                     diffusers_guidance_rescale,

modules/ui_extensions.py

@@ -151,6 +151,8 @@ def install_extension_from_url(dirname, url, branch_name, search_text, sort_colu
     normalized_url = normalize_git_url(url)
     assert len([x for x in extensions.extensions if normalize_git_url(x.remote) == normalized_url]) == 0, 'Extension with this URL is already installed'
     tmpdir = os.path.join(paths.data_path, "tmp", dirname)
+    if url.endswith('.git'):
+        url = url.replace('.git', '')
     try:
         shutil.rmtree(tmpdir, True)
         if not branch_name:
@@ -175,6 +177,8 @@ def install_extension_from_url(dirname, url, branch_name, search_text, sort_colu
         run_extension_installer(target_dir)
         extensions.list_extensions()
         return [refresh_extensions_list_from_data(search_text, sort_column), html.escape(f"Extension installed: {target_dir} | Restart required")]
+    except Exception as e:
+        shared.log.error(f'Error installing extension: {url} {e}')
     finally:
         shutil.rmtree(tmpdir, True)
 

modules/ui_extra_networks.py

@@ -140,7 +140,7 @@ class ExtraNetworksPage:
                 continue
             try:
                 img = Image.open(f)
-                if img.width > 1024 or img.height > 1024:
+                if img.width > 1024 or img.height > 1024 or os.path.getsize(f) > 70000:
                     img = img.convert('RGB')
                     img.thumbnail((512, 512), Image.HAMMING)
                     img.save(fn)

modules/ui_models.py

@@ -17,6 +17,7 @@ def create_ui():
         with gr.Column(elem_id='models_output_container', scale=1):
             # models_output = gr.Text(elem_id="models_output", value="", show_label=False)
             gr.HTML(elem_id="models_progress", value="")
+            models_image = gr.Image(elem_id="models_image", show_label=False, interactive=False, type='pil')
             models_outcome = gr.HTML(elem_id="models_error", value="")
 
         with gr.Column(elem_id='models_input_container', scale=3):
@@ -238,5 +239,134 @@ def create_ui():
                 hf_results.select(fn=hf_select, inputs=[hf_results], outputs=[hf_selected])
                 hf_download_model_btn.click(fn=hf_download_model, inputs=[hf_selected, hf_token, hf_variant, hf_revision, hf_mirror], outputs=[models_outcome])
 
-            # with gr.Tab(label="CivitAI"):
-            #    pass
+            with gr.Tab(label="CivitAI"):
+                data = []
+
+                def civit_search(name, tag, model_type):
+                    import requests
+                    headers = { 'Content-type': 'application/json' }
+                    url = 'https://civitai.com/api/v1/models?limit=25&types=Checkpoint&Sort=Newest'
+                    if name is not None and len(name) > 0:
+                        url += f'&query={name}'
+                    if tag is not None and len(tag) > 0:
+                        url += f'&tag={tag}'
+                    r = requests.get(url, timeout=60, headers=headers)
+                    log.debug(f'CivitAI search: name={name} tag={tag} status={r.status_code}')
+                    if r.status_code != 200:
+                        return [], [], []
+                    body = r.json()
+                    nonlocal data
+                    data = body.get('items', [])
+                    data1 = []
+                    for model in data:
+                        found = 0
+                        for variant in model['modelVersions']:
+                            if model_type == 'SD 1.5':
+                                if 'SD 1.' in variant['baseModel']:
+                                    found += 1
+                            if model_type == 'SD XL':
+                                if 'SDXL' in variant['baseModel']:
+                                    found += 1
+                            else:
+                                if 'SD 1.' not in variant['baseModel'] and 'SDXL' not in variant['baseModel']:
+                                    found += 1
+                        if found > 0:
+                            data1.append([
+                                model['id'],
+                                model['name'],
+                                ', '.join(model['tags']),
+                                model['stats']['downloadCount'],
+                                model['stats']['rating']
+                            ])
+                    return data1, [], []
+
+                def civit_select1(evt: gr.SelectData, in_data):
+                    model_id = in_data[evt.index[0]][0]
+                    data2 = []
+                    preview_img = None
+                    for model in data:
+                        if model['id'] == model_id:
+                            for d in model['modelVersions']:
+                                if d.get('images') is not None and len(d['images']) > 0 and len(d['images'][0]['url']) > 0:
+                                    preview_img = d['images'][0]['url']
+                                data2.append([
+                                    d['id'],
+                                    d['modelId'],
+                                    d['name'],
+                                    d['baseModel'],
+                                    d['createdAt'],
+                                ])
+                    log.debug(f'CivitAI select: model={in_data[evt.index[0]]} versions={len(data2)}')
+                    return data2, preview_img
+
+                def civit_select2(evt: gr.SelectData, in_data):
+                    variant_id = in_data[evt.index[0]][0]
+                    model_id = in_data[evt.index[0]][1]
+                    data3 = []
+                    for model in data:
+                        if model['id'] == model_id:
+                            for variant in model['modelVersions']:
+                                if variant['id'] == variant_id:
+                                    for f in variant['files']:
+                                        data3.append([
+                                            f['name'],
+                                            round(f['sizeKB']),
+                                            json.dumps(f['metadata']),
+                                            f['downloadUrl'],
+                                        ])
+                    log.debug(f'CivitAI select: model={in_data[evt.index[0]]} files={len(data3)}')
+                    return data3
+
+                def civit_select3(evt: gr.SelectData, in_data):
+                    log.debug(f'CivitAI select: variant={in_data[evt.index[0]]}')
+                    return in_data[evt.index[0]][3], in_data[evt.index[0]][0], gr.update(interactive=True)
+
+                def civit_download_model(model_url: str, model_name: str, model_path: str, image_url: str):
+                    if model_url is None or len(model_url) == 0:
+                        return 'No model selected'
+                    try:
+                        from modules.modelloader import download_civit_model
+                        res = download_civit_model(model_url, model_name, model_path, image_url)
+                    except Exception as e:
+                        res = f"CivitAI model downloaded error: model={model_url} {e}"
+                        log.error(res)
+                        return res
+                    from modules.sd_models import list_models # pylint: disable=W0621
+                    list_models()
+                    return res
+
+                with gr.Row():
+                    with gr.Column(scale=1):
+                        civit_model_type = gr.Dropdown(label='Model type', choices=['SD 1.5', 'SD XL', 'Other'], value='SD 1.5')
+                    with gr.Column(scale=15):
+                        with gr.Row():
+                            civit_search_text = gr.Textbox('', label = 'Seach models', placeholder='keyword')
+                            civit_search_tag = gr.Textbox('', label = '', placeholder='tags')
+                            civit_search_btn = ToolButton(value="🔍", label="Search", interactive=False)
+                with gr.Row():
+                    civit_download_model_btn = gr.Button(value="Download model", variant='primary')
+                with gr.Row():
+                    civit_name = gr.Textbox('', label = 'Model name', placeholder='select model from search results', visible=True)
+                    civit_selected = gr.Textbox('', label = 'Model URL', placeholder='select model from search results', visible=True)
+                    civit_path = gr.Textbox('', label = 'Download path', placeholder='optional subfolder path where to save model', visible=True)
+                with gr.Row():
+                    with gr.Column():
+                        civit_headers2 = ['ID', 'ModelID', 'Name', 'Base', 'Created', 'Preview']
+                        civit_types2 = ['number', 'number', 'str', 'str', 'date', 'str']
+                        civit_results2 = gr.DataFrame([], label = 'Model versions', show_label = True, interactive = False, wrap = True, overflow_row_behaviour = 'paginate', max_rows = 10, headers = civit_headers2, datatype = civit_types2, type='array')
+                    with gr.Column():
+                        civit_headers3 = ['Name', 'Size', 'Metadata', 'URL']
+                        civit_types3 = ['str', 'number', 'str', 'str']
+                        civit_results3 = gr.DataFrame([], label = 'Model variants', show_label = True, interactive = False, wrap = True, overflow_row_behaviour = 'paginate', max_rows = 10, headers = civit_headers3, datatype = civit_types3, type='array')
+                with gr.Row():
+                    civit_headers1 = ['ID', 'Name', 'Tags', 'Downloads', 'Rating']
+                    civit_types1 = ['number', 'str', 'str', 'number', 'number']
+                    civit_results1 = gr.DataFrame([], label = 'Search results', show_label = True, interactive = False, wrap = True, overflow_row_behaviour = 'paginate', max_rows = 10, headers = civit_headers1, datatype = civit_types1, type='array')
+
+                civit_search_text.submit(fn=civit_search, inputs=[civit_search_text, civit_search_tag, civit_model_type], outputs=[civit_results1, civit_results2, civit_results3])
+                civit_search_tag.submit(fn=civit_search, inputs=[civit_search_text, civit_search_tag, civit_model_type], outputs=[civit_results1, civit_results2, civit_results3])
+                civit_search_btn.click(fn=civit_search, inputs=[civit_search_text, civit_search_tag, civit_model_type], outputs=[civit_results1, civit_results2, civit_results3])
+                civit_results1.select(fn=civit_select1, inputs=[civit_results1], outputs=[civit_results2, models_image])
+                civit_results2.select(fn=civit_select2, inputs=[civit_results2], outputs=[civit_results3])
+                civit_results3.select(fn=civit_select3, inputs=[civit_results3], outputs=[civit_selected, civit_name, civit_search_btn])
+                civit_download_model_btn.click(fn=civit_download_model, inputs=[civit_selected, civit_name, civit_path, models_image], outputs=[models_outcome])

pyproject.toml

@@ -47,6 +47,7 @@ ignore = [
   "B905", # Without explicit scrict
   "C408", # Rewrite as a literal
   "E402", # Module level import not at top of file
+  "E721", # Do not compare types, use `isinstance()`
   "F401", # Imported but unused
   "EXE001", # Shebang present
   "ISC003", # Implicit string concatenation

requirements.txt

@@ -4,7 +4,6 @@ aiohttp
 anyio
 appdirs
 astunparse
-bitsandbytes
 blendmodes
 clean-fid
 easydev
@@ -41,7 +40,6 @@ voluptuous
 yapf
 scikit-image
 basicsr
-compel
 fasteners
 typing-extensions==4.7.1
 antlr4-python3-runtime==4.9.3
@@ -54,8 +52,8 @@ einops==0.4.1
 gradio==3.32.0
 huggingface_hub==0.16.4
 numexpr==2.8.4
-numpy==1.23.5
-numba==0.57.0
+numpy==1.24.4
+numba==0.57.1
 pandas==1.5.3
 protobuf==3.20.3
 pytorch_lightning==1.9.4

scripts/prompts_from_file.py

@@ -2,7 +2,6 @@ import copy
 import random
 import shlex
 import gradio as gr
-from PIL import Image
 import modules.scripts as scripts
 from modules import sd_samplers, errors
 from modules.processing import Processed, process_images

webui.py

@@ -9,6 +9,7 @@ import logging
 import warnings
 import importlib
 from threading import Thread
+import urllib3
 from modules import timer, errors, paths # pylint: disable=unused-import
 
 startup_timer = timer.Timer()
@@ -20,6 +21,7 @@ try:
     import intel_extension_for_pytorch as ipex # pylint: disable=import-error, unused-import
 except Exception:
     pass
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 import torchvision # pylint: disable=W0611,C0411
 import pytorch_lightning # pytorch_lightning should be imported after torch, but it re-enables warnings on import so import once to disable them # pylint: disable=W0611,C0411
 if ".dev" in torch.__version__ or "+git" in torch.__version__:

webui.sh

@@ -44,8 +44,8 @@ do
     esac
 done
 
-# Do not run as root
-if [[ $(id -u) -eq 0 && can_run_as_root -eq 0 ]]
+# Do not run as root unless inside a Docker container
+if [[ $(id -u) -eq 0 && can_run_as_root -eq 0 && ! -f /.dockerenv ]]
 then
     echo "Cannot run as root"
     exit 1