Merge branch 'dev' into patch-1

CHANGELOG.md

@@ -1,5 +1,78 @@
 # Change Log for SD.Next
 
+## TODO
+
+- StableDiffusion 3
+
+## Update for 2024-06-08
+
+*Note*: New features require `diffusers==0.29.0.dev`
+
+### New Models
+
+- [Tenecent HunyuanDiT](https://github.com/Tencent/HunyuanDiT) bilingual english/chinese diffusion transformer model  
+  note: this is a very large model at ~17GB, but can be used with less VRAM using model offloading  
+  simply select from networks -> models -> reference, model will be auto-downloaded on first use  
+
+### New Functionality
+
+- [MuLan](https://github.com/mulanai/MuLan) Multi-langunage prompts
+  write your prompts forin ~110 auto-detected languages!  
+  compatible with *SD15* and *SDXL*  
+  enable in scripts -> MuLan and set encoder to `InternVL-14B-224px` encoder  
+  *note*: right now this is more of a proof-of-concept before smaller and/or quantized models are released  
+  model will be auto-downloaded on first use: note its huge size of 27GB  
+  even executing it in FP16 will require ~16GB of VRAM for text encoder alone  
+  examples:  
+  - English: photo of a beautiful woman wearing a white bikini on a beach with a city skyline in the background
+  - Croatian: fotografija lijepe žene u bijelom bikiniju na plaži s gradskim obzorom u pozadini
+  - Italian: Foto di una bella donna che indossa un bikini bianco su una spiaggia con lo skyline di una città sullo sfondo
+  - Spanish: Foto de una hermosa mujer con un bikini blanco en una playa con un horizonte de la ciudad en el fondo
+  - German: Foto einer schönen Frau in einem weißen Bikini an einem Strand mit einer Skyline der Stadt im Hintergrund
+  - Arabic: صورة لامرأة جميلة ترتدي بيكيني أبيض على شاطئ مع أفق المدينة في الخلفية
+  - Japanese: 街のスカイラインを背景にビーチで白いビキニを着た美しい女性の写真
+  - Chinese: 一个美丽的女人在海滩上穿着白色比基尼的照片, 背景是城市天际线
+  - Korean: 도시의 스카이라인을 배경으로 해변에서 흰색 비키니를 입은 아름 다운 여성의 사진
+- [T-Gate](https://github.com/HaozheLiu-ST/T-GATE) Speed up generations by gating at which step cross-attention is no longer needed  
+  enable via scripts -> t-gate  
+  compatible with *SD15*  
+- **PCM LoRAs** allow for fast denoising using less steps with standard *SD15* and *SDXL* models  
+  download from <https://huggingface.co/Kijai/converted_pcm_loras_fp16/tree/main>
+- [ByteDance ResAdapter](https://github.com/bytedance/res-adapter) resolution-free model adapter  
+  allows to use resolutions from 0.5 to 2.0 of original model resolution, compatible with *SD15* and *SDXL*
+  enable via scripts -> resadapter and select desired model
+- **Kohya HiRes Fix** allows for higher resolution generation using standard *SD15* models  
+  enable via scripts -> kohya-hires-fix  
+  *note*: alternative to regular hidiffusion method, but with different approach to scaling  
+- additional built-in 4 great custom trained **ControlNet SDXL** models from Xinsir: OpenPose, Canny, Scribble, AnimePainter  
+  thanks @lbeltrame
+- add torch **full deterministic mode**
+  enable in settings -> compute -> use deterministic mode  
+  typical differences are not large and its disabled by default as it does have some performance impact  
+
+### Improvements
+
+- further work on improving python 3.12 functionality and remove experimental flag  
+  note: recommended version remains python 3.11 for all users except if you're using directml and then its python 3.10  
+- improved **installer** for initial installs  
+  initial install will do single-pass install of all required packages with correct versions  
+  subsequent runs will check package versions as necessary  
+- add env variable `SD_PIP_DEBUG` to write `pip.log` for all pip operations  
+  also improved installer logging  
+- add python version check for `torch-directml`  
+- do not install `tensorflow` by default  
+- improve metadata/infotext parser  
+  add `cli/image-exif.py` that can be used to view/extract metadata from images  
+- lower overhead on generate calls  
+- auto-synchronize modernui and core branches  
+
+## Fixes
+
+- cumulative fixes since the last release
+- fix apply/unapply hidiffusion for sd15  
+- fix controlnet reference enabled check  
+- fix face-hires with control batch count  
+
 ## Update for 2024-06-02
 
 - fix textual inversion loading

TODO.md

@@ -2,37 +2,20 @@
 
 Main ToDo list can be found at [GitHub projects](https://github.com/users/vladmandic/projects)
 
-## Fix
-
-- ultralytics package install
-
 ## Future Candidates
 
 - stable diffusion 3.0: unreleased
-- boxdiff <https://github.com/huggingface/diffusers/pull/7947>
 - animatediff-sdxl <https://github.com/huggingface/diffusers/pull/6721>
 - async lowvram: <https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/14855>
 - fp8: <https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/14031>
 - profiling: <https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/716>
-- kohya-hires-fix: <https://github.com/huggingface/diffusers/pull/7633>
-- hunyuan-dit: <https://github.com/huggingface/diffusers/pull/8290>
 - init latents: variations, img2img
 - diffusers public callbacks  
 - include reference styles
 - lora: sc lora, dora, etc
-- controlnet: additional models
-- resadapter: <https://github.com/bytedance/res-adapter>
-- t-gate: <https://huggingface.co/docs/diffusers/main/en/optimization/tgate>
 
 ## Experimental
 
-- [MuLan](https://github.com/mulanai/MuLan) Multi-langunage prompts - wirte your prompts in ~110 auto-detected languages!
-  Compatible with SD15 and SDXL
-  Enable in scripts -> MuLan and set encoder to `InternVL-14B-224px` encoder
-  (that is currently only supported encoder, but others will be added)
-  Note: Model will be auto-downloaded on first use: note its huge size of 27GB
-  Even executing it in FP16 context will require ~16GB of VRAM for text encoder alone
-  *Note*: Uses fixed prompt parser, so no prompt attention will be used
 - [SDXL Flash Mini](https://huggingface.co/sd-community/sdxl-flash-mini)  
   SDXL type that weighs less, consumes less video memory, and the quality has not dropped much  
   to use, simply select from *networks -> models -> reference -> SDXL Flash Mini*  

cli/image-exif.py

@@ -4,71 +4,18 @@ import os
 import io
 import re
 import sys
-import json
+import importlib.util
 from PIL import Image, ExifTags, TiffImagePlugin, PngImagePlugin
 from rich import print # pylint: disable=redefined-builtin
 
 
-def unquote(text):
-    if len(text) == 0 or text[0] != '"' or text[-1] != '"':
-        return text
-    try:
-        return json.loads(text)
-    except Exception:
-        return text
+module_file = os.path.abspath(__file__)
+module_dir = os.path.dirname(module_file)
+module_spec = importlib.util.spec_from_file_location('infotext', os.path.join(module_dir, '..', 'modules', 'infotext.py'))
+infotext = importlib.util.module_from_spec(module_spec)
+module_spec.loader.exec_module(infotext)
 
 
-def parse_generation_parameters(infotext):
-    if not isinstance(infotext, str):
-        return {}
-    re_param = re.compile(r'\s*([\w ]+):\s*("(?:\\"[^,]|\\"|\\|[^\"])+"|[^,]*)(?:,|$)') # multi-word: value
-    re_size = re.compile(r"^(\d+)x(\d+)$") # int x int
-    basic_params = ['steps', 'seed', 'width', 'height', 'sampler', 'size', 'cfg scale', 'hires'] # first param is one of those
-
-    sanitized = infotext.replace('prompt:', 'Prompt:').replace('negative prompt:', 'Negative prompt:').replace('Negative Prompt', 'Negative prompt') # cleanup everything in brackets so re_params can work
-    sanitized = re.sub(r'<[^>]*>', lambda match: ' ' * len(match.group()), sanitized)
-    sanitized = re.sub(r'\([^)]*\)', lambda match: ' ' * len(match.group()), sanitized)
-    sanitized = re.sub(r'\{[^}]*\}', lambda match: ' ' * len(match.group()), sanitized)
-
-    params = dict(re_param.findall(sanitized))
-    params = { k.strip():params[k].strip() for k in params if k.lower() not in ['hashes', 'lora', 'embeddings', 'prompt', 'negative prompt']} # remove some keys
-    if len(list(params)) == 0:
-        first_param = None
-    else:
-        try:
-            first_param, first_param_idx = next((s, i) for i, s in enumerate(params) if any(x in s.lower() for x in basic_params))
-        except Exception:
-            first_param, first_param_idx = next(iter(params)), 0
-        if first_param_idx > 0:
-            for _i in range(first_param_idx):
-                params.pop(next(iter(params)))
-    params_idx = sanitized.find(f'{first_param}:') if first_param else -1
-    negative_idx = infotext.find("Negative prompt:")
-
-    prompt = infotext[:params_idx] if negative_idx == -1 else infotext[:negative_idx] # prompt can be with or without negative prompt
-    negative = infotext[negative_idx:params_idx] if negative_idx >= 0 else ''
-
-    for k, v in params.copy().items(): # avoid dict-has-changed
-        if len(v) > 0 and v[0] == '"' and v[-1] == '"':
-            v = unquote(v)
-        m = re_size.match(v)
-        if v.replace('.', '', 1).isdigit():
-            params[k] = float(v) if '.' in v else int(v)
-        elif v == "True":
-            params[k] = True
-        elif v == "False":
-            params[k] = False
-        elif m is not None:
-            params[f"{k}-1"] = int(m.group(1))
-            params[f"{k}-2"] = int(m.group(2))
-        elif k == 'VAE' and v == 'TAESD':
-            params["Full quality"] = False
-        else:
-            params[k] = v
-    params["Prompt"] = prompt.replace('Prompt:', '').strip()
-    params["Negative prompt"] = negative.replace('Negative prompt:', '').strip()
-    return params
-
 
 class Exif: # pylint: disable=single-string-used-for-slots
     __slots__ = ('__dict__') # pylint: disable=superfluous-parens
@@ -132,7 +79,7 @@ class Exif: # pylint: disable=single-string-used-for-slots
 
     def parse(self):
         x = self.exif.pop('parameters', None) or self.exif.pop('UserComment', None)
-        res = parse_generation_parameters(x)
+        res = infotext.parse(x)
         return res
 
     def get_bytes(self):

extensions-builtin/Lora/extra_networks_lora.py

@@ -104,7 +104,7 @@ class ExtraNetworkLora(extra_networks.ExtraNetwork):
             self.active = False
 
     def deactivate(self, p):
-        if shared.backend == shared.Backend.DIFFUSERS and hasattr(shared.sd_model, "unload_lora_weights") and hasattr(shared.sd_model, "text_encoder"):
+        if shared.native and hasattr(shared.sd_model, "unload_lora_weights") and hasattr(shared.sd_model, "text_encoder"):
             if 'CLIP' in shared.sd_model.text_encoder.__class__.__name__ and not (shared.compiled_model_state is not None and shared.compiled_model_state.is_compiled is True):
                 if shared.opts.lora_fuse_diffusers:
                     shared.sd_model.unfuse_lora()

extensions-builtin/Lora/lora_convert.py

@@ -106,7 +106,7 @@ def make_unet_conversion_map() -> Dict[str, str]:
 
 class KeyConvert:
     def __init__(self):
-        if shared.backend == shared.Backend.ORIGINAL:
+        if not shared.native:
             self.converter = self.original
             self.is_sd2 = 'model_transformer_resblocks' in shared.sd_model.network_layer_mapping
         else:

extensions-builtin/Lora/network.py

@@ -31,7 +31,8 @@ class NetworkOnDisk:
             self.metadata = m
         self.alias = self.metadata.get('ss_output_name', self.name)
         # self.set_hash(self.metadata.get('sshs_model_hash') or hashes.sha256_from_cache(self.filename, "lora/" + self.name, use_addnet_hash=self.is_safetensors) or '')
-        self.set_hash(hashes.sha256_from_cache(self.filename, "lora/" + self.name) or self.metadata.get('sshs_model_hash'))
+        sha256 = hashes.sha256_from_cache(self.filename, "lora/" + self.name) or hashes.sha256_from_cache(self.filename, "lora/" + self.name, use_addnet_hash=True) or self.metadata.get('sshs_model_hash')
+        self.set_hash(sha256)
         self.sd_version = self.detect_version()
 
     def detect_version(self):

extensions-builtin/Lora/network_overrides.py

@@ -1,7 +1,7 @@
 from modules import shared
 
 
-force_diffusers = [
+maybe_diffusers = [
     'aaebf6360f7d', # sd15-lcm
     '3d18b05e4f56', # sdxl-lcm
     'b71dcb732467', # sdxl-tcd
@@ -19,12 +19,16 @@ force_diffusers = [
     '8cca3706050b', # hyper-sdxl-1step
 ]
 
+force_diffusers = [
+    '816d0eed49fd', # flash-sdxl
+    'c2ec22757b46', # flash-sd15
+]
+
 def check_override(shorthash):
-    if not shared.opts.lora_maybe_diffusers:
-        return False
     if len(shorthash) < 4:
         return False
-    force = any(x.startswith(shorthash) for x in force_diffusers)
-    if force:
+    force = any(x.startswith(shorthash) for x in maybe_diffusers) if shared.opts.lora_maybe_diffusers else False
+    force = force or any(x.startswith(shorthash) for x in force_diffusers)
+    if force and shared.opts.lora_maybe_diffusers:
         shared.log.debug('LoRA override: force diffusers')
     return force

extensions-builtin/Lora/networks.py

@@ -47,7 +47,7 @@ convert_diffusers_name_to_compvis = lora_convert.convert_diffusers_name_to_compv
 
 def assign_network_names_to_compvis_modules(sd_model):
     network_layer_mapping = {}
-    if shared.backend == shared.Backend.DIFFUSERS:
+    if shared.native:
         if not hasattr(shared.sd_model, 'text_encoder') or not hasattr(shared.sd_model, 'unet'):
             return
         for name, module in shared.sd_model.text_encoder.named_modules():
@@ -85,7 +85,7 @@ def load_diffusers(name, network_on_disk, lora_scale=1.0) -> network.Network:
     shared.log.debug(f'LoRA load: name="{name}" file="{network_on_disk.filename}" type=diffusers {"cached" if cached else ""} fuse={shared.opts.lora_fuse_diffusers}')
     if cached is not None:
         return cached
-    if shared.backend != shared.Backend.DIFFUSERS:
+    if not shared.native:
         return None
     shared.sd_model.load_lora_weights(network_on_disk.filename)
     if shared.opts.lora_fuse_diffusers:
@@ -195,9 +195,9 @@ def load_networks(names, te_multipliers=None, unet_multipliers=None, dyn_dims=No
             try:
                 if recompile_model:
                     shared.compiled_model_state.lora_model.append(f"{name}:{te_multipliers[i] if te_multipliers else 1.0}")
-                if shared.backend == shared.Backend.DIFFUSERS and shared.opts.lora_force_diffusers: # OpenVINO only works with Diffusers LoRa loading
+                if shared.native and shared.opts.lora_force_diffusers: # OpenVINO only works with Diffusers LoRa loading
                     net = load_diffusers(name, network_on_disk, lora_scale=te_multipliers[i] if te_multipliers else 1.0)
-                elif shared.backend == shared.Backend.DIFFUSERS and network_overrides.check_override(shorthash):
+                elif shared.native and network_overrides.check_override(shorthash):
                     net = load_diffusers(name, network_on_disk, lora_scale=te_multipliers[i] if te_multipliers else 1.0)
                 else:
                     net = load_network(name, network_on_disk)

extensions-builtin/Lora/ui_extra_networks_lora.py

@@ -19,10 +19,10 @@ class ExtraNetworksPageLora(ui_extra_networks.ExtraNetworksPage):
         try:
             # path, _ext = os.path.splitext(l.filename)
             name = os.path.splitext(os.path.relpath(l.filename, shared.cmd_opts.lora_dir))[0]
-            if shared.backend == shared.Backend.ORIGINAL:
+            if not shared.native:
                 if l.sd_version == network.SdVersion.SDXL:
                     return None
-            elif shared.backend == shared.Backend.DIFFUSERS:
+            elif shared.native:
                 if shared.sd_model_type == 'none': # return all when model is not loaded
                     pass
                 elif shared.sd_model_type == 'sdxl':

html/reference.json

@@ -58,14 +58,14 @@
     "experimental": true
   },
 
-  "RunwayML SD 1.5": {
+  "RunwayML StableDiffusion 1.5": {
     "original": true,
     "path": "v1-5-pruned-fp16-emaonly.safetensors@https://huggingface.co/Aptronym/SDNext/resolve/main/Reference/v1-5-pruned-fp16-emaonly.safetensors?download=true",
     "preview": "v1-5-pruned-fp16-emaonly.jpg",
     "desc": "Stable Diffusion 1.5 is the base model all other 1.5 checkpoint were trained from. It's a latent text-to-image diffusion model capable of generating photo-realistic images given any text input. The Stable-Diffusion-v1-5 checkpoint was initialized with the weights of the Stable-Diffusion-v1-2 checkpoint and subsequently fine-tuned on 595k steps at resolution 512x512.",
     "extras": "width: 512, height: 512, sampler: DEIS, steps: 20, cfg_scale: 6.0"
   },
-  "StabilityAI SD 2.1": {
+  "StabilityAI StableDiffusion 2.1": {
     "path": "huggingface/stabilityai/stable-diffusion-2-1-base",
     "preview": "stabilityai--stable-diffusion-2-1-base.jpg",
     "skip": true,
@@ -73,7 +73,7 @@
     "desc": "This stable-diffusion-2-1-base model fine-tunes stable-diffusion-2-base (512-base-ema.ckpt) with 220k extra steps taken",
     "extras": "width: 512, height: 512, sampler: DEIS, steps: 20, cfg_scale: 6.0"
   },
-  "StabilityAI SD 2.1 V": {
+  "StabilityAI StableDiffusion 2.1 V": {
     "path": "huggingface/stabilityai/stable-diffusion-2-1",
     "preview": "stabilityai--stable-diffusion-2-1.jpg",
     "skip": true,
@@ -81,13 +81,12 @@
     "desc": "This stable-diffusion-2 model is resumed from stable-diffusion-2-base (512-base-ema.ckpt) and trained for 150k steps using a v-objective on the same dataset. Resumed for another 140k steps on 768x768 images",
     "extras": "width: 768, height: 768, sampler: DEIS, steps: 20, cfg_scale: 6.0"
   },
-  "StabilityAI SD-XL 1.0 Base": {
+  "StabilityAI StableDiffusion XL 1.0 Base": {
     "path": "sd_xl_base_1.0.safetensors@https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors?download=true",
     "preview": "sd_xl_base_1.0.jpg",
     "desc": "Stable Diffusion XL (SDXL) is the latest AI image generation model that is tailored towards more photorealistic outputs with more detailed imagery and composition compared to previous SD models, including SD 2.1. It can make realistic faces, legible text within the images, and better image composition, all while using shorter and simpler prompts at a greatly increased base resolution of 1024x1024. Just like its predecessors, SDXL has the ability to generate image variations using image-to-image prompting, inpainting (reimagining of the selected parts of an image), and outpainting (creating new parts that lie outside the image borders).",
     "extras": "width: 1024, height: 1024, sampler: DEIS, steps: 20, cfg_scale: 6.0"
   },
-
   "StabilityAI Stable Cascade": {
     "path": "huggingface/stabilityai/stable-cascade",
     "skip": true,
@@ -158,7 +157,14 @@
     "preview": "PixArt-alpha--pixart_sigma_sdxlvae_T5_diffusers.jpg",
     "extras": "width: 1024, height: 1024, sampler: Default, cfg_scale: 2.0"
   },
-
+  
+  "Tencent HunyuanDiT": {
+    "path": "Tencent-Hunyuan/HunyuanDiT-Diffusers",
+    "desc": "Hunyuan-DiT : A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding.",
+    "preview": "Tencent-Hunyuan-HunyuanDiT.jpg",
+    "extras": "width: 1024, height: 1024, sampler: Default, cfg_scale: 2.0"
+  },
+  
   "Kandinsky 2.1": {
     "path": "kandinsky-community/kandinsky-2-1",
     "desc": "Kandinsky 2.1 is a text-conditional diffusion model based on unCLIP and latent diffusion, composed of a transformer-based image prior model, a unet diffusion model, and a decoder. Kandinsky 2.1 inherits best practices from Dall-E 2 and Latent diffusion while introducing some new ideas. It uses the CLIP model as a text and image encoder, and diffusion image prior (mapping) between latent spaces of CLIP modalities. This approach increases the visual performance of the model and unveils new horizons in blending images and text-guided image manipulation.",

install-mm.py

@@ -1,44 +0,0 @@
-import os
-from installer import setup_logging
-setup_logging()
-
-
-checked_ok = False
-
-
-def check_dependencies():
-    from installer import installed, pip, log
-    global checked_ok # pylint: disable=global-statement
-    debug = log.trace if os.environ.get('SD_DWPOSE_DEBUG', None) is not None else lambda *args, **kwargs: None
-    packages = [
-        'openmim==0.3.9',
-        'mmengine==0.10.4',
-        'mmcv==2.1.0',
-        'mmpose==1.3.1',
-        'mmdet==3.3.0',
-    ]
-    status = [installed(p, reload=False, quiet=False) for p in packages]
-    status.append(False)
-    debug(f'DWPose required={packages} status={status}')
-    if not all(status):
-        log.info(f'Installing DWPose dependencies: {[packages]}')
-        cmd = 'install --upgrade --no-deps --force-reinstall '
-        pkgs = ' '.join(packages)
-        res = pip(cmd + pkgs, ignore=False, quiet=False)
-        debug(f'DWPose pip install: {res}')
-    try:
-        import pkg_resources
-        import imp # pylint: disable=deprecated-module
-        imp.reload(pkg_resources)
-        import mmcv # pylint: disable=unused-import
-        import mmengine # pylint: disable=unused-import
-        import mmpose # pylint: disable=unused-import
-        import mmdet # pylint: disable=unused-import
-        debug('DWPose import ok')
-        checked_ok = True
-    except Exception as e:
-        log.error(f'DWPose: {e}')
-    return checked_ok
-
-
-check_dependencies()

installer.py

@@ -23,8 +23,10 @@ class Dot(dict): # dot notation access to dictionary attributes
 
 
 version = None
+current_branch = None
 log = logging.getLogger("sd")
 debug = log.debug if os.environ.get('SD_INSTALL_DEBUG', None) is not None else lambda *args, **kwargs: None
+pip_log = '--log pip.log ' if os.environ.get('SD_PIP_DEBUG', None) is not None else ''
 log_file = os.path.join(os.path.dirname(__file__), 'sdnext.log')
 log_rolled = False
 first_call = True
@@ -83,7 +85,10 @@ def setup_logging():
         def get(self):
             return self.buffer
 
-    install('rich', 'rich')
+    install('rich', 'rich', quiet=True)
+    install('setuptools==69.5.1', 'setuptools', quiet=True)
+    install('psutil', 'psutil', quiet=True)
+    install('requests', 'requests', quiet=True)
     from functools import partial, partialmethod
     from logging.handlers import RotatingFileHandler
     from rich.theme import Theme
@@ -232,11 +237,11 @@ def uninstall(package, quiet = False):
 @lru_cache()
 def pip(arg: str, ignore: bool = False, quiet: bool = False):
     arg = arg.replace('>=', '==')
-    if not quiet:
-        log.info(f'Installing package: {arg.replace("install", "").replace("--upgrade", "").replace("--no-deps", "").replace("--force", "").replace("  ", " ").strip()}')
+    if not quiet and '-r ' not in arg:
+        log.info(f'Install: package="{arg.replace("install", "").replace("--upgrade", "").replace("--no-deps", "").replace("--force", "").replace("  ", " ").strip()}"')
     env_args = os.environ.get("PIP_EXTRA_ARGS", "")
-    log.debug(f"Running pip: {arg} {env_args}")
-    result = subprocess.run(f'"{sys.executable}" -m pip {arg} {env_args}', shell=True, check=False, env=os.environ, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    log.debug(f'Running: pip="{pip_log}{arg} {env_args}"')
+    result = subprocess.run(f'"{sys.executable}" -m pip {pip_log}{arg} {env_args}', shell=True, check=False, env=os.environ, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
     txt = result.stdout.decode(encoding="utf8", errors="ignore")
     if len(result.stderr) > 0:
         txt += ('\n' if len(txt) > 0 else '') + result.stderr.decode(encoding="utf8", errors="ignore")
@@ -252,14 +257,14 @@ def pip(arg: str, ignore: bool = False, quiet: bool = False):
 
 # install package using pip if not already installed
 @lru_cache()
-def install(package, friendly: str = None, ignore: bool = False, reinstall: bool = False, no_deps: bool = False):
+def install(package, friendly: str = None, ignore: bool = False, reinstall: bool = False, no_deps: bool = False, quiet: bool = False):
     res = ''
     if args.reinstall or args.upgrade:
         global quick_allowed # pylint: disable=global-statement
         quick_allowed = False
-    if args.reinstall or reinstall or not installed(package, friendly, quiet=False):
-        deps = '' if not no_deps else '--no-deps'
-        res = pip(f"install --upgrade {deps} {package}", ignore=ignore)
+    if args.reinstall or reinstall or not installed(package, friendly, quiet=quiet):
+        deps = '' if not no_deps else '--no-deps '
+        res = pip(f"install --upgrade {deps}{package}", ignore=ignore)
         try:
             import imp # pylint: disable=deprecated-module
             imp.reload(pkg_resources)
@@ -292,6 +297,7 @@ def git(arg: str, folder: str = None, ignore: bool = False):
         log.debug(f'Git output: {txt}')
     return txt
 
+
 # reattach as needed as head can get detached
 def branch(folder=None):
     # if args.experimental:
@@ -322,13 +328,13 @@ def branch(folder=None):
 
 
 # update git repository
-def update(folder, current_branch = False, rebase = True):
+def update(folder, keep_branch = False, rebase = True):
     try:
         git('config rebase.Autostash true')
     except Exception:
         pass
     arg = '--rebase --force' if rebase else ''
-    if current_branch:
+    if keep_branch:
         res = git(f'pull {arg}', folder)
         debug(f'Install update: folder={folder} args={arg} {res}')
         return res
@@ -386,15 +392,18 @@ def get_platform():
 
 
 # check python version
-def check_python():
-    supported_minors = [9, 10, 11]
+def check_python(supported_minors=[9, 10, 11, 12], reason=None):
     if args.quick:
         return
-    log.info(f'Python {platform.python_version()} on {platform.system()}')
+    log.info(f'Python version={platform.python_version()} platform={platform.system()} bin="{sys.executable}" venv="{sys.prefix}"')
     if not (int(sys.version_info.major) == 3 and int(sys.version_info.minor) in supported_minors):
         log.error(f"Incompatible Python version: {sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro} required 3.{supported_minors}")
+        if reason is not None:
+            log.error(reason)
         if not args.ignore:
             sys.exit(1)
+    if int(sys.version_info.minor) == 12:
+        os.environ.setdefault('SETUPTOOLS_USE_DISTUTILS', 'local') # hack for python 3.11 setuptools
     if not args.skip_git:
         git_cmd = os.environ.get('GIT', "git")
         if shutil.which(git_cmd) is None:
@@ -424,6 +433,215 @@ def check_onnx():
         install('onnxruntime', 'onnxruntime', ignore=True)
 
 
+def install_rocm_zluda(torch_command):
+    check_python(supported_minors=[10,11], reason='RocM or Zluda backends require Python 3.10 or 3.11')
+    is_windows = platform.system() == 'Windows'
+    log.info('AMD ROCm toolkit detected')
+    os.environ.setdefault('PYTORCH_HIP_ALLOC_CONF', 'garbage_collection_threshold:0.8,max_split_size_mb:512')
+    # if not is_windows:
+    #    os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow-rocm')
+    try:
+        if is_windows:
+            command = subprocess.run('hipinfo', shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            amd_gpus = command.stdout.decode(encoding="utf8", errors="ignore").split('\n')
+            amd_gpus = [x.split(' ')[-1].strip() for x in amd_gpus if x.startswith('gcnArchName:')]
+        else:
+            command = subprocess.run('rocm_agent_enumerator', shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            amd_gpus = command.stdout.decode(encoding="utf8", errors="ignore").split('\n')
+            amd_gpus = [x for x in amd_gpus if x and x != 'gfx000']
+        log.debug(f'ROCm agents detected: {amd_gpus}')
+    except Exception as e:
+        log.debug(f'ROCm agent enumerator failed: {e}')
+        amd_gpus = []
+
+    hip_visible_devices = [] # use the first available amd gpu by default
+    for idx, gpu in enumerate(amd_gpus):
+        if gpu in ['gfx1100', 'gfx1101', 'gfx1102']:
+            hip_visible_devices.append((idx, gpu, 'navi3x'))
+            break
+        if gpu in ['gfx1030', 'gfx1031', 'gfx1032', 'gfx1034']: # experimental navi 2x support
+            hip_visible_devices.append((idx, gpu, 'navi2x'))
+            break
+    if len(hip_visible_devices) > 0:
+        idx, gpu, arch = hip_visible_devices[0]
+        log.debug(f'ROCm agent used by default: idx={idx} gpu={gpu} arch={arch}')
+        os.environ.setdefault('HIP_VISIBLE_DEVICES', str(idx))
+        if arch == 'navi3x':
+            os.environ.setdefault('HSA_OVERRIDE_GFX_VERSION', '11.0.0')
+            # if os.environ.get('TENSORFLOW_PACKAGE') == 'tensorflow-rocm': # do not use tensorflow-rocm for navi 3x
+            #    os.environ['TENSORFLOW_PACKAGE'] = 'tensorflow==2.13.0'
+        elif arch == 'navi2x':
+            os.environ.setdefault('HSA_OVERRIDE_GFX_VERSION', '10.3.0')
+        else:
+            log.debug(f'HSA_OVERRIDE_GFX_VERSION auto config is skipped for {gpu}')
+    try:
+        command = subprocess.run('hipconfig --version', shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        arr = command.stdout.decode(encoding="utf8", errors="ignore").split('.')
+        rocm_ver = f'{arr[0]}.{arr[1]}' if len(arr) >= 2 else None
+        log.debug(f'ROCm version detected: {rocm_ver}')
+    except Exception as e:
+        log.debug(f'ROCm hipconfig failed: {e}')
+        rocm_ver = None
+    if args.use_zluda:
+        log.warning("ZLUDA support: experimental")
+        error = None
+        from modules import zluda_installer
+        try:
+            if args.reinstall_zluda:
+                zluda_installer.uninstall()
+            if args.experimental:
+                zluda_installer.enable_runtime_api()
+            zluda_path = zluda_installer.get_path()
+            zluda_installer.install(zluda_path)
+            zluda_installer.make_copy(zluda_path)
+        except Exception as e:
+            error = e
+            log.warning(f'Failed to install ZLUDA: {e}')
+        if error is None:
+            try:
+                zluda_installer.load(zluda_path)
+                torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.3.0 torchvision --index-url https://download.pytorch.org/whl/cu118')
+                log.info(f'Using ZLUDA in {zluda_path}')
+            except Exception as e:
+                error = e
+                log.warning(f'Failed to load ZLUDA: {e}')
+        if error is not None:
+            log.info('Using CPU-only torch')
+            torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision')
+    elif is_windows: # TODO TBD after ROCm for Windows is released
+        log.warning("HIP SDK is detected, but no Torch release for Windows available")
+        log.info("For ZLUDA support specify '--use-zluda'")
+        log.info('Using CPU-only torch')
+        torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision')
+
+        # conceal ROCm installed
+        os.environ.pop("ROCM_HOME", None)
+        os.environ.pop("ROCM_PATH", None)
+        paths = os.environ["PATH"].split(";")
+        paths_no_rocm = []
+        for path in paths:
+            if "ROCm" not in path:
+                paths_no_rocm.append(path)
+        os.environ["PATH"] = ";".join(paths_no_rocm)
+    else:
+        if rocm_ver is None: # assume the latest if version check fails
+            torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision --index-url https://download.pytorch.org/whl/rocm6.0')
+        elif rocm_ver == "6.1": # need nightlies
+            torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision --pre --index-url https://download.pytorch.org/whl/nightly/rocm6.1')
+        elif float(rocm_ver) < 5.5: # oldest supported version is 5.5
+            log.warning(f"Unsupported ROCm version detected: {rocm_ver}")
+            log.warning("Minimum supported ROCm version is 5.5")
+            torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision --index-url https://download.pytorch.org/whl/rocm5.5')
+        else:
+            torch_command = os.environ.get('TORCH_COMMAND', f'torch torchvision --index-url https://download.pytorch.org/whl/rocm{rocm_ver}')
+        if rocm_ver is not None:
+            ort_version = os.environ.get('ONNXRUNTIME_VERSION', None)
+            ort_package = os.environ.get('ONNXRUNTIME_PACKAGE', f"--pre onnxruntime-training{'' if ort_version is None else ('==' + ort_version)} --index-url https://pypi.lsh.sh/{rocm_ver[0]}{rocm_ver[2]} --extra-index-url https://pypi.org/simple")
+            install(ort_package, 'onnxruntime-training')
+    return torch_command
+
+
+def install_ipex(torch_command):
+    check_python(supported_minors=[10,11], reason='IPEX backend requires Python 3.10 or 3.11')
+    args.use_ipex = True # pylint: disable=attribute-defined-outside-init
+    log.info('Intel OneAPI Toolkit detected')
+    if os.environ.get("NEOReadDebugKeys", None) is None:
+        os.environ.setdefault('NEOReadDebugKeys', '1')
+    if os.environ.get("ClDeviceGlobalMemSizeAvailablePercent", None) is None:
+        os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
+    if "linux" in sys.platform:
+        torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.1.0.post0 torchvision==0.16.0.post0 intel-extension-for-pytorch==2.1.20+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/')
+        # os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow==2.15.0 intel-extension-for-tensorflow[xpu]==2.15.0.0')
+        if os.environ.get('DISABLE_VENV_LIBS', None) is None:
+            install(os.environ.get('MKL_PACKAGE', 'mkl==2024.1.0'), 'mkl')
+            install(os.environ.get('DPCPP_PACKAGE', 'mkl-dpcpp==2024.1.0'), 'mkl-dpcpp')
+            install(os.environ.get('ONECCL_PACKAGE', 'oneccl-devel==2021.12.0'), 'oneccl-devel')
+            install(os.environ.get('MPI_PACKAGE', 'impi-devel==2021.12.0'), 'impi-devel')
+    else:
+        if sys.version_info.minor == 11:
+            pytorch_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torch-2.1.0a0+cxx11.abi-cp311-cp311-win_amd64.whl'
+            torchvision_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torchvision-0.16.0a0+cxx11.abi-cp311-cp311-win_amd64.whl'
+            ipex_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/intel_extension_for_pytorch-2.1.10+xpu-cp311-cp311-win_amd64.whl'
+            torch_command = os.environ.get('TORCH_COMMAND', f'{pytorch_pip} {torchvision_pip} {ipex_pip}')
+        elif sys.version_info.minor == 10:
+            pytorch_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torch-2.1.0a0+cxx11.abi-cp310-cp310-win_amd64.whl'
+            torchvision_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torchvision-0.16.0a0+cxx11.abi-cp310-cp310-win_amd64.whl'
+            ipex_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/intel_extension_for_pytorch-2.1.10+xpu-cp310-cp310-win_amd64.whl'
+            torch_command = os.environ.get('TORCH_COMMAND', f'{pytorch_pip} {torchvision_pip} {ipex_pip}')
+        else:
+            torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.1.0.post0 torchvision==0.16.0.post0 intel-extension-for-pytorch==2.1.20+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/')
+            if os.environ.get('DISABLE_VENV_LIBS', None) is None:
+                install(os.environ.get('MKL_PACKAGE', 'mkl==2024.1.0'), 'mkl')
+                install(os.environ.get('DPCPP_PACKAGE', 'mkl-dpcpp==2024.1.0'), 'mkl-dpcpp')
+                install(os.environ.get('ONECCL_PACKAGE', 'oneccl-devel==2021.12.0'), 'oneccl-devel')
+                install(os.environ.get('MPI_PACKAGE', 'impi-devel==2021.12.0'), 'impi-devel')
+        torch_command = os.environ.get('TORCH_COMMAND', f'{pytorch_pip} {torchvision_pip} {ipex_pip}')
+    install(os.environ.get('OPENVINO_PACKAGE', 'openvino==2023.3.0'), 'openvino', ignore=True)
+    install('nncf==2.7.0', 'nncf', ignore=True)
+    install(os.environ.get('ONNXRUNTIME_PACKAGE', 'onnxruntime-openvino'), 'onnxruntime-openvino', ignore=True)
+    return torch_command
+
+
+def install_openvino(torch_command):
+    check_python(supported_minors=[10,11], reason='IPEX backend requires Python 3.10 or 3.11')
+    log.info('Using OpenVINO')
+    torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.2.0 torchvision==0.17.0 --index-url https://download.pytorch.org/whl/cpu')
+    install(os.environ.get('OPENVINO_PACKAGE', 'openvino==2023.3.0'), 'openvino')
+    install(os.environ.get('ONNXRUNTIME_PACKAGE', 'onnxruntime-openvino'), 'onnxruntime-openvino', ignore=True)
+    install('nncf==2.8.1', 'nncf')
+    os.environ.setdefault('PYTORCH_TRACING_MODE', 'TORCHFX')
+    if os.environ.get("NEOReadDebugKeys", None) is None:
+        os.environ.setdefault('NEOReadDebugKeys', '1')
+    if os.environ.get("ClDeviceGlobalMemSizeAvailablePercent", None) is None:
+        os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
+    return torch_command
+
+
+def is_rocm_available(allow_rocm):
+    if not allow_rocm:
+        return False
+    if installed('torch-directml', quiet=True):
+        log.debug('DirectML installation is detected. Skipping HIP SDK check.')
+        return False
+    if platform.system() == 'Windows':
+        from modules.zluda_installer import find_hip_sdk
+        return find_hip_sdk() is not None
+    else:
+        return shutil.which('rocminfo') is not None or os.path.exists('/opt/rocm/bin/rocminfo') or os.path.exists('/dev/kfd')
+
+
+def install_torch_addons():
+    xformers_package = os.environ.get('XFORMERS_PACKAGE', '--pre xformers') if opts.get('cross_attention_optimization', '') == 'xFormers' or args.use_xformers else 'none'
+    triton_command = os.environ.get('TRITON_COMMAND', 'triton') if sys.platform == 'linux' else None
+    if 'xformers' in xformers_package:
+        try:
+            install(f'--no-deps {xformers_package}', ignore=True)
+            import torch # pylint: disable=unused-import
+            import xformers # pylint: disable=unused-import
+        except Exception as e:
+            log.debug(f'Cannot install xformers package: {e}')
+    elif not args.experimental and not args.use_xformers and opts.get('cross_attention_optimization', '') != 'xFormers':
+        uninstall('xformers')
+    if opts.get('cuda_compile_backend', '') == 'hidet':
+        install('hidet', 'hidet')
+    if opts.get('cuda_compile_backend', '') == 'deep-cache':
+        install('DeepCache')
+    if opts.get('cuda_compile_backend', '') == 'olive-ai':
+        install('olive-ai')
+    if opts.get('nncf_compress_weights', False) and not args.use_openvino:
+        install('nncf==2.7.0', 'nncf')
+    if triton_command is not None:
+        install(triton_command, 'triton', quiet=True)
+
+
+def is_cuda_available(allow_cuda):
+    return allow_cuda and (shutil.which('nvidia-smi') is not None or args.use_xformers or os.path.exists(os.path.join(os.environ.get('SystemRoot') or r'C:\Windows', 'System32', 'nvidia-smi.exe')))
+
+
+def is_ipex_available(allow_ipex):
+    return allow_ipex and (args.use_ipex or shutil.which('sycl-ls') is not None or shutil.which('sycl-ls.exe') is not None or os.environ.get('ONEAPI_ROOT') is not None or os.path.exists('/opt/intel/oneapi') or os.path.exists("C:/Program Files (x86)/Intel/oneAPI") or os.path.exists("C:/oneAPI"))
+
+
 # check torch version
 def check_torch():
     if args.skip_torch:
@@ -440,175 +658,26 @@ def check_torch():
     log.debug(f'Torch overrides: cuda={args.use_cuda} rocm={args.use_rocm} ipex={args.use_ipex} diml={args.use_directml} openvino={args.use_openvino}')
     log.debug(f'Torch allowed: cuda={allow_cuda} rocm={allow_rocm} ipex={allow_ipex} diml={allow_directml} openvino={allow_openvino}')
     torch_command = os.environ.get('TORCH_COMMAND', '')
-    xformers_package = os.environ.get('XFORMERS_PACKAGE', '--pre xformers') if opts.get('cross_attention_optimization', '') == 'xFormers' or args.use_xformers else 'none'
-    triton_command = os.environ.get('TRITON_COMMAND', 'triton') if sys.platform == 'linux' else None
-
-    def is_rocm_available():
-        if not allow_rocm:
-            return False
-        if installed('torch-directml', quiet=True):
-            log.debug('DirectML installation is detected. Skipping HIP SDK check.')
-            return False
-        if platform.system() == 'Windows':
-            from modules.zluda_installer import find_hip_sdk
-            return find_hip_sdk() is not None
-        else:
-            return shutil.which('rocminfo') is not None or os.path.exists('/opt/rocm/bin/rocminfo') or os.path.exists('/dev/kfd')
 
     if torch_command != '':
         pass
-    elif allow_cuda and (shutil.which('nvidia-smi') is not None or args.use_xformers or os.path.exists(os.path.join(os.environ.get('SystemRoot') or r'C:\Windows', 'System32', 'nvidia-smi.exe'))):
+    elif is_cuda_available(allow_cuda):
         log.info('nVidia CUDA toolkit detected: nvidia-smi present')
         torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision --index-url https://download.pytorch.org/whl/cu121')
-        install('onnxruntime-gpu', 'onnxruntime-gpu', ignore=True)
-    elif is_rocm_available():
-        is_windows = platform.system() == 'Windows'
-        log.info('AMD ROCm toolkit detected')
-        os.environ.setdefault('PYTORCH_HIP_ALLOC_CONF', 'garbage_collection_threshold:0.8,max_split_size_mb:512')
-        if not is_windows:
-            os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow-rocm')
-        try:
-            if is_windows:
-                command = subprocess.run('hipinfo', shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-                amd_gpus = command.stdout.decode(encoding="utf8", errors="ignore").split('\n')
-                amd_gpus = [x.split(' ')[-1].strip() for x in amd_gpus if x.startswith('gcnArchName:')]
-            else:
-                command = subprocess.run('rocm_agent_enumerator', shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-                amd_gpus = command.stdout.decode(encoding="utf8", errors="ignore").split('\n')
-                amd_gpus = [x for x in amd_gpus if x and x != 'gfx000']
-            log.debug(f'ROCm agents detected: {amd_gpus}')
-        except Exception as e:
-            log.debug(f'ROCm agent enumerator failed: {e}')
-            amd_gpus = []
-
-        hip_visible_devices = [] # use the first available amd gpu by default
-        for idx, gpu in enumerate(amd_gpus):
-            if gpu in ['gfx1100', 'gfx1101', 'gfx1102']:
-                hip_visible_devices.append((idx, gpu, 'navi3x'))
-                break
-            if gpu in ['gfx1030', 'gfx1031', 'gfx1032', 'gfx1034']: # experimental navi 2x support
-                hip_visible_devices.append((idx, gpu, 'navi2x'))
-                break
-        if len(hip_visible_devices) > 0:
-            idx, gpu, arch = hip_visible_devices[0]
-            log.debug(f'ROCm agent used by default: idx={idx} gpu={gpu} arch={arch}')
-            os.environ.setdefault('HIP_VISIBLE_DEVICES', str(idx))
-            if arch == 'navi3x':
-                os.environ.setdefault('HSA_OVERRIDE_GFX_VERSION', '11.0.0')
-                if os.environ.get('TENSORFLOW_PACKAGE') == 'tensorflow-rocm': # do not use tensorflow-rocm for navi 3x
-                    os.environ['TENSORFLOW_PACKAGE'] = 'tensorflow==2.13.0'
-            elif arch == 'navi2x':
-                os.environ.setdefault('HSA_OVERRIDE_GFX_VERSION', '10.3.0')
-            else:
-                log.debug(f'HSA_OVERRIDE_GFX_VERSION auto config is skipped for {gpu}')
-        try:
-            command = subprocess.run('hipconfig --version', shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-            arr = command.stdout.decode(encoding="utf8", errors="ignore").split('.')
-            rocm_ver = f'{arr[0]}.{arr[1]}' if len(arr) >= 2 else None
-            log.debug(f'ROCm version detected: {rocm_ver}')
-        except Exception as e:
-            log.debug(f'ROCm hipconfig failed: {e}')
-            rocm_ver = None
-        if args.use_zluda:
-            log.warning("ZLUDA support: experimental")
-            error = None
-            from modules import zluda_installer
-            try:
-                if args.reinstall_zluda:
-                    zluda_installer.uninstall()
-                if args.experimental:
-                    zluda_installer.enable_runtime_api()
-                zluda_path = zluda_installer.get_path()
-                zluda_installer.install(zluda_path)
-                zluda_installer.make_copy(zluda_path)
-            except Exception as e:
-                error = e
-                log.warning(f'Failed to install ZLUDA: {e}')
-            if error is None:
-                try:
-                    zluda_installer.load(zluda_path)
-                    torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.3.0 torchvision --index-url https://download.pytorch.org/whl/cu118')
-                    log.info(f'Using ZLUDA in {zluda_path}')
-                except Exception as e:
-                    error = e
-                    log.warning(f'Failed to load ZLUDA: {e}')
-            if error is not None:
-                log.info('Using CPU-only torch')
-                torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision')
-        elif is_windows: # TODO TBD after ROCm for Windows is released
-            log.warning("HIP SDK is detected, but no Torch release for Windows available")
-            log.info("For ZLUDA support specify '--use-zluda'")
-            log.info('Using CPU-only torch')
-            torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision')
-        else:
-            if rocm_ver is None: # assume the latest if version check fails
-                torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision --index-url https://download.pytorch.org/whl/rocm6.0')
-            elif rocm_ver == "6.1": # need nightlies
-                torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision --pre --index-url https://download.pytorch.org/whl/nightly/rocm6.1')
-            elif float(rocm_ver) < 5.5: # oldest supported version is 5.5
-                log.warning(f"Unsupported ROCm version detected: {rocm_ver}")
-                log.warning("Minimum supported ROCm version is 5.5")
-                torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision --index-url https://download.pytorch.org/whl/rocm5.5')
-            else:
-                torch_command = os.environ.get('TORCH_COMMAND', f'torch torchvision --index-url https://download.pytorch.org/whl/rocm{rocm_ver}')
-            if rocm_ver is not None:
-                ort_version = os.environ.get('ONNXRUNTIME_VERSION', None)
-                ort_package = os.environ.get('ONNXRUNTIME_PACKAGE', f"--pre onnxruntime-training{'' if ort_version is None else ('==' + ort_version)} --index-url https://pypi.lsh.sh/{rocm_ver[0]}{rocm_ver[2]} --extra-index-url https://pypi.org/simple")
-                install(ort_package, 'onnxruntime-training')
-    elif allow_ipex and (args.use_ipex or shutil.which('sycl-ls') is not None or shutil.which('sycl-ls.exe') is not None or os.environ.get('ONEAPI_ROOT') is not None or os.path.exists('/opt/intel/oneapi') or os.path.exists("C:/Program Files (x86)/Intel/oneAPI") or os.path.exists("C:/oneAPI")):
-        args.use_ipex = True # pylint: disable=attribute-defined-outside-init
-        log.info('Intel OneAPI Toolkit detected')
-        if os.environ.get("NEOReadDebugKeys", None) is None:
-            os.environ.setdefault('NEOReadDebugKeys', '1')
-        if os.environ.get("ClDeviceGlobalMemSizeAvailablePercent", None) is None:
-            os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
-        if "linux" in sys.platform:
-            torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.1.0.post0 torchvision==0.16.0.post0 intel-extension-for-pytorch==2.1.20+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/')
-            os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow==2.15.0 intel-extension-for-tensorflow[xpu]==2.15.0.0')
-            if os.environ.get('DISABLE_VENV_LIBS', None) is None:
-                install(os.environ.get('MKL_PACKAGE', 'mkl==2024.1.0'), 'mkl')
-                install(os.environ.get('DPCPP_PACKAGE', 'mkl-dpcpp==2024.1.0'), 'mkl-dpcpp')
-                install(os.environ.get('ONECCL_PACKAGE', 'oneccl-devel==2021.12.0'), 'oneccl-devel')
-                install(os.environ.get('MPI_PACKAGE', 'impi-devel==2021.12.0'), 'impi-devel')
-        else:
-            if sys.version_info.minor == 11:
-                pytorch_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torch-2.1.0a0+cxx11.abi-cp311-cp311-win_amd64.whl'
-                torchvision_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torchvision-0.16.0a0+cxx11.abi-cp311-cp311-win_amd64.whl'
-                ipex_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/intel_extension_for_pytorch-2.1.10+xpu-cp311-cp311-win_amd64.whl'
-                torch_command = os.environ.get('TORCH_COMMAND', f'{pytorch_pip} {torchvision_pip} {ipex_pip}')
-            elif sys.version_info.minor == 10:
-                pytorch_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torch-2.1.0a0+cxx11.abi-cp310-cp310-win_amd64.whl'
-                torchvision_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/torchvision-0.16.0a0+cxx11.abi-cp310-cp310-win_amd64.whl'
-                ipex_pip = 'https://github.com/Nuullll/intel-extension-for-pytorch/releases/download/v2.1.10%2Bxpu/intel_extension_for_pytorch-2.1.10+xpu-cp310-cp310-win_amd64.whl'
-                torch_command = os.environ.get('TORCH_COMMAND', f'{pytorch_pip} {torchvision_pip} {ipex_pip}')
-            else:
-                torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.1.0.post0 torchvision==0.16.0.post0 intel-extension-for-pytorch==2.1.20+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/')
-                if os.environ.get('DISABLE_VENV_LIBS', None) is None:
-                    install(os.environ.get('MKL_PACKAGE', 'mkl==2024.1.0'), 'mkl')
-                    install(os.environ.get('DPCPP_PACKAGE', 'mkl-dpcpp==2024.1.0'), 'mkl-dpcpp')
-                    install(os.environ.get('ONECCL_PACKAGE', 'oneccl-devel==2021.12.0'), 'oneccl-devel')
-                    install(os.environ.get('MPI_PACKAGE', 'impi-devel==2021.12.0'), 'impi-devel')
-            torch_command = os.environ.get('TORCH_COMMAND', f'{pytorch_pip} {torchvision_pip} {ipex_pip}')
-        install(os.environ.get('OPENVINO_PACKAGE', 'openvino==2023.3.0'), 'openvino', ignore=True)
-        install('nncf==2.7.0', 'nncf', ignore=True)
-        install(os.environ.get('ONNXRUNTIME_PACKAGE', 'onnxruntime-openvino'), 'onnxruntime-openvino', ignore=True)
+        install('onnxruntime-gpu', 'onnxruntime-gpu', ignore=True, quiet=True)
+    elif is_rocm_available(allow_rocm):
+        torch_command = install_rocm_zluda(torch_command)
+    elif is_ipex_available(allow_ipex):
+        torch_command = install_ipex(torch_command)
     elif allow_openvino and args.use_openvino:
-        log.info('Using OpenVINO')
-        torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.2.0 torchvision==0.17.0 --index-url https://download.pytorch.org/whl/cpu')
-        install(os.environ.get('OPENVINO_PACKAGE', 'openvino==2023.3.0'), 'openvino')
-        install(os.environ.get('ONNXRUNTIME_PACKAGE', 'onnxruntime-openvino'), 'onnxruntime-openvino', ignore=True)
-        install('nncf==2.8.1', 'nncf')
-        os.environ.setdefault('PYTORCH_TRACING_MODE', 'TORCHFX')
-        if os.environ.get("NEOReadDebugKeys", None) is None:
-            os.environ.setdefault('NEOReadDebugKeys', '1')
-        if os.environ.get("ClDeviceGlobalMemSizeAvailablePercent", None) is None:
-            os.environ.setdefault('ClDeviceGlobalMemSizeAvailablePercent', '100')
+        torch_command = install_openvino(torch_command)
     else:
         machine = platform.machine()
         if sys.platform == 'darwin':
             torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision')
         elif allow_directml and args.use_directml and ('arm' not in machine and 'aarch' not in machine):
             log.info('Using DirectML Backend')
+            check_python(supported_minors=[10], reason='DirectML backend requires Python 3.10')
             torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.0.0 torchvision torch-directml')
             if 'torch' in torch_command and not args.version:
                 install(torch_command, 'torch torchvision')
@@ -619,11 +688,7 @@ def check_torch():
             log.info('Using CPU-only Torch')
             torch_command = os.environ.get('TORCH_COMMAND', 'torch torchvision')
     if 'torch' in torch_command and not args.version:
-        if not installed('torch', quiet=True):
-            log.debug(f'Installing torch: {torch_command}')
-        install(torch_command, 'torch torchvision')
-    if triton_command is not None:
-        install(triton_command, 'triton')
+        install(torch_command, 'torch torchvision', quiet=True)
     else:
         try:
             import torch
@@ -662,23 +727,7 @@ def check_torch():
     if args.version:
         return
     if not args.skip_all:
-        try:
-            if 'xformers' in xformers_package:
-                install(f'--no-deps {xformers_package}', ignore=True)
-                import torch
-                import xformers # pylint: disable=unused-import
-            elif not args.experimental and not args.use_xformers and opts.get('cross_attention_optimization', '') != 'xFormers':
-                uninstall('xformers')
-        except Exception as e:
-            log.debug(f'Cannot install xformers package: {e}')
-        if opts.get('cuda_compile_backend', '') == 'hidet':
-            install('hidet', 'hidet')
-        if opts.get('cuda_compile_backend', '') == 'deep-cache':
-            install('DeepCache')
-        if opts.get('cuda_compile_backend', '') == 'olive-ai':
-            install('olive-ai')
-        if opts.get('nncf_compress_weights', False) and not args.use_openvino:
-            install('nncf==2.7.0', 'nncf')
+        install_torch_addons()
     if args.profile:
         print_profile(pr, 'Torch')
 
@@ -710,14 +759,16 @@ def install_packages():
         pr.enable()
     log.info('Verifying packages')
     clip_package = os.environ.get('CLIP_PACKAGE', "git+https://github.com/openai/CLIP.git")
-    install(clip_package, 'clip')
-    tensorflow_package = os.environ.get('TENSORFLOW_PACKAGE', 'tensorflow==2.13.0')
-    install(tensorflow_package, 'tensorflow-rocm' if 'rocm' in tensorflow_package else 'tensorflow', ignore=True)
-    bitsandbytes_package = os.environ.get('BITSANDBYTES_PACKAGE', None)
-    if bitsandbytes_package is not None:
-        install(bitsandbytes_package, 'bitsandbytes', ignore=True)
-    elif not args.experimental:
-        uninstall('bitsandbytes')
+    install(clip_package, 'clip', quiet=True)
+    # tensorflow_package = os.environ.get('TENSORFLOW_PACKAGE', 'tensorflow==2.13.0')
+    # tensorflow_package = os.environ.get('TENSORFLOW_PACKAGE', None)
+    # if tensorflow_package is not None:
+    #    install(tensorflow_package, 'tensorflow-rocm' if 'rocm' in tensorflow_package else 'tensorflow', ignore=True, quiet=True)
+    # bitsandbytes_package = os.environ.get('BITSANDBYTES_PACKAGE', None)
+    # if bitsandbytes_package is not None:
+    #    install(bitsandbytes_package, 'bitsandbytes', ignore=True, quiet=True)
+    # elif not args.experimental:
+    #    uninstall('bitsandbytes')
     if args.profile:
         print_profile(pr, 'Packages')
 
@@ -860,11 +911,18 @@ def install_requirements():
         pr.enable()
     if args.skip_requirements and not args.requirements:
         return
+    if not installed('diffusers', quiet=True): # diffusers are not installed, so run initial installation
+        global quick_allowed # pylint: disable=global-statement
+        quick_allowed = False
+        log.info('Installing requirements: this make take a while...')
+        pip('install -r requirements.txt')
+    installed('torch', reload=True) # reload packages cache
     log.info('Verifying requirements')
     with open('requirements.txt', 'r', encoding='utf8') as f:
         lines = [line.strip() for line in f.readlines() if line.strip() != '' and not line.startswith('#') and line is not None]
         for line in lines:
-            _res = install(line)
+            if not installed(line, quiet=True):
+                _res = install(line)
     if args.profile:
         print_profile(pr, 'Requirements')
 
@@ -895,7 +953,7 @@ def set_environment():
     os.environ.setdefault('KINETO_LOG_LEVEL', '3')
     os.environ.setdefault('DO_NOT_TRACK', '1')
     os.environ.setdefault('HF_HUB_CACHE', opts.get('hfcache_dir', os.path.join(os.path.expanduser('~'), '.cache', 'huggingface', 'hub')))
-    log.debug(f'HF cache folder: {os.environ.get("HF_HUB_CACHE")}')
+    log.info(f'HF cache folder: {os.environ.get("HF_HUB_CACHE")}')
     allocator = f'garbage_collection_threshold:{opts.get("torch_gc_threshold", 80)/100:0.2f},max_split_size_mb:512'
     if opts.get("torch_malloc", "native") == 'cudaMallocAsync':
         allocator += ',backend:cudaMallocAsync'
@@ -934,9 +992,9 @@ def check_extensions():
     return round(newest_all)
 
 
-def get_version():
+def get_version(force=False):
     global version # pylint: disable=global-statement
-    if version is None:
+    if version is None or force:
         try:
             subprocess.run('git config log.showsignature false', stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell=True, check=True)
         except Exception:
@@ -958,9 +1016,41 @@ def get_version():
             }
         except Exception:
             version = { 'app': 'sd.next', 'version': 'unknown' }
+        try:
+            cwd = os.getcwd()
+            os.chdir('extensions-builtin/sdnext-modernui')
+            res = subprocess.run('git rev-parse --abbrev-ref HEAD', stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell=True, check=True)
+            os.chdir(cwd)
+            branch_ui = res.stdout.decode(encoding = 'utf8', errors='ignore') if len(res.stdout) > 0 else ''
+            branch_ui = 'dev' if 'dev' in branch_ui else 'main'
+            version['ui'] = branch_ui
+        except Exception:
+            os.chdir(cwd)
+            version['ui'] = 'unknown'
     return version
 
 
+def check_ui(ver):
+    if ver is None:
+        return
+    if ver['branch'] == ver['ui']:
+        return
+    log.debug(f'Branch mismatch: sdnext={ver["branch"]} ui={ver["ui"]}')
+    cwd = os.getcwd()
+    try:
+        os.chdir('extensions-builtin/sdnext-modernui')
+        git('checkout ' + ver['branch'], ignore=True)
+        os.chdir(cwd)
+        ver = get_version(force=True)
+        if ver['branch'] == ver['ui']:
+            log.debug(f'Branch synchronized: {ver["branch"]}')
+        else:
+            log.debug(f'Branch synch failed: sdnext={ver["branch"]} ui={ver["ui"]}')
+    except Exception as e:
+        log.debug(f'Branch switch: {e}')
+    os.chdir(cwd)
+
+
 # check version of the main repo and optionally upgrade it
 def check_version(offline=False, reset=True): # pylint: disable=unused-argument
     if args.skip_all:
@@ -968,9 +1058,11 @@ def check_version(offline=False, reset=True): # pylint: disable=unused-argument
     if not os.path.exists('.git'):
         log.warning('Not a git repository, all git operations are disabled')
         args.skip_git = True # pylint: disable=attribute-defined-outside-init
-    log.info(f'Version: {print_dict(get_version())}')
+    ver = get_version()
+    log.info(f'Version: {print_dict(ver)}')
     if args.version or args.skip_git:
         return
+    check_ui(ver)
     commit = git('rev-parse HEAD')
     global git_commit # pylint: disable=global-statement
     git_commit = commit[:7]
@@ -991,7 +1083,7 @@ def check_version(offline=False, reset=True): # pylint: disable=unused-argument
                 try:
                     git('add .')
                     git('stash')
-                    update('.', current_branch=True)
+                    update('.', keep_branch=True)
                     # git('git stash pop')
                     ver = git('log -1 --pretty=format:"%h %ad"')
                     log.info(f'Upgraded to version: {ver}')

launch.py

@@ -215,26 +215,25 @@ def main():
         installer.log.info('Startup: skip all')
         installer.quick_allowed = True
         init_paths()
-    elif installer.check_timestamp():
-        installer.log.info('Startup: quick launch')
-        installer.install_requirements()
-        installer.install_packages()
-        init_paths()
-        installer.check_extensions()
     else:
-        installer.log.info('Startup: standard')
         installer.install_requirements()
         installer.install_packages()
-        installer.install_submodules()
-        init_paths()
-        installer.install_extensions()
-        installer.install_requirements() # redo requirements since extensions may change them
-        installer.update_wiki()
-        if installer.errors == 0:
-            installer.log.debug(f'Setup complete without errors: {round(time.time())}')
+        if installer.check_timestamp():
+            installer.log.info('Startup: quick launch')
+            init_paths()
+            installer.check_extensions()
         else:
-            installer.log.warning(f'Setup complete with errors: {installer.errors}')
-            installer.log.warning(f'See log file for more details: {installer.log_file}')
+            installer.log.info('Startup: standard')
+            installer.install_submodules()
+            init_paths()
+            installer.install_extensions()
+            installer.install_requirements() # redo requirements since extensions may change them
+            installer.update_wiki()
+            if installer.errors == 0:
+                installer.log.debug(f'Setup complete without errors: {round(time.time())}')
+            else:
+                installer.log.warning(f'Setup complete with errors: {installer.errors}')
+                installer.log.warning(f'See log file for more details: {installer.log_file}')
     installer.extensions_preload(parser) # adds additional args from extensions
     args = installer.parse_args(parser)
 

modules/api/endpoints.py

@@ -146,7 +146,7 @@ def get_extensions_list():
     return ext_list
 
 def post_pnginfo(req: models.ReqImageInfo):
-    from modules import images, script_callbacks, generation_parameters_copypaste
+    from modules import images, script_callbacks, infotext
     if not req.image.strip():
         return models.ResImageInfo(info="")
     image = helpers.decode_base64_to_image(req.image.strip())
@@ -155,6 +155,6 @@ def post_pnginfo(req: models.ReqImageInfo):
     geninfo, items = images.read_info_from_image(image)
     if geninfo is None:
         geninfo = ""
-    params = generation_parameters_copypaste.parse_generation_parameters(geninfo)
+    params = infotext.parse(geninfo)
     script_callbacks.infotext_pasted_callback(geninfo, params)
     return models.ResImageInfo(info=geninfo, items=items, parameters=params)

modules/control/run.py

@@ -149,12 +149,34 @@ def control_run(units: List[unit.Unit] = [], inputs: List[Image.Image] = [], ini
         shared.log.debug('Control: override resize mode=mask')
         selected_scale_tab_mask = 1
 
-    # set initial resolution
+    # set control sizing
     if resize_mode_before != 0 or inputs is None or inputs == [None]:
         p.width, p.height = width_before, height_before # pylint: disable=attribute-defined-outside-init
+        p.width_before = width_before
+        p.height_before = height_before
+        if resize_name_before != 'None':
+            p.resize_mode_before = resize_mode_before
+            p.resize_name_before = resize_name_before
+        p.scale_by_before = scale_by_before
+        p.selected_scale_tab_before = selected_scale_tab_before
     else:
         del p.width
         del p.height
+    if resize_name_after != 'None':
+        p.resize_mode_after = resize_mode_after
+        p.resize_name_after = resize_name_after
+        p.width_after = width_after
+        p.height_after = height_after
+        p.scale_by_after = scale_by_after
+        p.selected_scale_tab_after = selected_scale_tab_after
+    if resize_name_mask != 'None':
+        p.resize_mode_mask = resize_mode_mask
+        p.resize_name_mask = resize_name_mask
+        p.width_mask = width_mask
+        p.height_mask = height_mask
+        p.scale_by_mask = scale_by_mask
+        p.selected_scale_tab_mask = selected_scale_tab_mask
+
     # hires/refine defined outside of main init
     p.enable_hr = enable_hr
     p.hr_sampler_name = processing.get_sampler_name(hr_sampler_index)
@@ -254,6 +276,8 @@ def control_run(units: List[unit.Unit] = [], inputs: List[Image.Image] = [], ini
             control_conditioning = active_strength[0] if len(active_strength) == 1 else list(active_strength) # strength or list[strength]
             control_guidance_start = active_start[0] if len(active_start) == 1 else list(active_start)
             control_guidance_end = active_end[0] if len(active_end) == 1 else list(active_end)
+    elif unit_type == 'reference':
+        has_models = any(u.enabled for u in units if u.type == 'reference')
     else:
         pass
 
@@ -299,7 +323,7 @@ def control_run(units: List[unit.Unit] = [], inputs: List[Image.Image] = [], ini
         pipe = instance.pipeline
         if inits is not None:
             shared.log.warning('Control: ControlLLLite does not support separate init image')
-    elif unit_type == 'reference':
+    elif unit_type == 'reference' and has_models:
         p.extra_generation_params["Control mode"] = 'Reference'
         p.extra_generation_params["Control attention"] = p.attention
         p.task_args['reference_attn'] = 'Attention' in p.attention
@@ -488,7 +512,7 @@ def control_run(units: List[unit.Unit] = [], inputs: List[Image.Image] = [], ini
                         debug('Control processed: using input direct')
                         processed_image = input_image
 
-                    if unit_type == 'reference':
+                    if unit_type == 'reference' and has_models:
                         p.ref_image = p.override or input_image
                         p.task_args.pop('image', None)
                         p.task_args['ref_image'] = p.ref_image
@@ -496,11 +520,11 @@ def control_run(units: List[unit.Unit] = [], inputs: List[Image.Image] = [], ini
                         if p.ref_image is None:
                             yield terminate('Control: attempting reference mode but image is none')
                             return [], '', '', 'Reference mode without image'
-                    elif unit_type == 'controlnet' and input_type == 1: # Init image same as control
+                    elif unit_type == 'controlnet' and input_type == 1 and has_models: # Init image same as control
                         p.task_args['control_image'] = p.init_images # switch image and control_image
                         p.task_args['strength'] = p.denoising_strength
                         p.init_images = [p.override or input_image] * len(active_model)
-                    elif unit_type == 'controlnet' and input_type == 2: # Separate init image
+                    elif unit_type == 'controlnet' and input_type == 2 and has_models: # Separate init image
                         if init_image is None:
                             shared.log.warning('Control: separate init image not provided')
                             init_image = input_image
@@ -517,7 +541,7 @@ def control_run(units: List[unit.Unit] = [], inputs: List[Image.Image] = [], ini
                     t2 += time.time() - t2
 
                     # determine txt2img, img2img, inpaint pipeline
-                    if unit_type == 'reference': # special case
+                    if unit_type == 'reference' and has_models: # special case
                         p.is_control = True
                         shared.sd_model = sd_models.set_diffuser_pipe(shared.sd_model, sd_models.DiffusersTaskType.TEXT_2_IMAGE)
                     elif not has_models: # run in txt2img/img2img/inpaint mode

modules/control/units/controlnet.py

@@ -50,6 +50,10 @@ predefined_sdxl = {
     'Depth Zoe XL': 'diffusers/controlnet-zoe-depth-sdxl-1.0',
     'Depth Mid XL': 'diffusers/controlnet-depth-sdxl-1.0-mid',
     'OpenPose XL': 'thibaud/controlnet-openpose-sdxl-1.0',
+    'Xinsir OpenPose XL': 'xinsir/controlnet-openpose-sdxl-1.0',
+    'Xinsir Canny XL': 'xinsir/controlnet-canny-sdxl-1.0',
+    'Xinsir Scribble XL': 'xinsir/controlnet-scribble-sdxl-1.0',
+    'Xinsir Anime Painter XL': 'xinsir/anime-painter',
     # 'StabilityAI Canny R128': 'stabilityai/control-lora/control-LoRAs-rank128/control-lora-canny-rank128.safetensors',
     # 'StabilityAI Depth R128': 'stabilityai/control-lora/control-LoRAs-rank128/control-lora-depth-rank128.safetensors',
     # 'StabilityAI Recolor R128': 'stabilityai/control-lora/control-LoRAs-rank128/control-lora-recolor-rank128.safetensors',

modules/control/units/t2iadapter.py

@@ -22,6 +22,12 @@ predefined_sd15 = {
     'Canny v2': 'TencentARC/t2iadapter_canny_sd15v2',
     'Sketch v1': 'TencentARC/t2iadapter_sketch_sd14v1',
     'Sketch v2': 'TencentARC/t2iadapter_sketch_sd15v2',
+    # 'Coadapter Canny': 'TencentARC/T2I-Adapter/models/coadapter-canny-sd15v1.pth',
+    # 'Coadapter Color': 'TencentARC/T2I-Adapter/models/coadapter-color-sd15v1.pth',
+    # 'Coadapter Depth': 'TencentARC/T2I-Adapter/models/coadapter-depth-sd15v1.pth',
+    # 'Coadapter Fuser': 'TencentARC/T2I-Adapter/models/coadapter-fuser-sd15v1.pth',
+    # 'Coadapter Sketch': 'TencentARC/T2I-Adapter/models/coadapter-sketch-sd15v1.pth',
+    # 'Coadapter Style': 'TencentARC/T2I-Adapter/models/coadapter-style-sd15v1.pth',
 }
 predefined_sdxl = {
     'Canny XL': 'TencentARC/t2i-adapter-canny-sdxl-1.0',
@@ -31,6 +37,7 @@ predefined_sdxl = {
     'OpenPose XL': 'TencentARC/t2i-adapter-openpose-sdxl-1.0',
     'Midas Depth XL': 'TencentARC/t2i-adapter-depth-midas-sdxl-1.0',
 }
+
 models = {}
 all_models = {}
 all_models.update(predefined_sd15)
@@ -94,7 +101,15 @@ class Adapter():
                 log.error(f'Control {what} model load failed: id="{model_id}" error=unknown model id')
                 return
             log.debug(f'Control {what} model loading: id="{model_id}" path="{model_path}"')
-            self.model = T2IAdapter.from_pretrained(model_path, **self.load_config)
+            if model_path.endswith('.pth') or model_path.endswith('.pt') or model_path.endswith('.safetensors'):
+                from huggingface_hub import hf_hub_download
+                parts = model_path.split('/')
+                repo_id = f'{parts[0]}/{parts[1]}'
+                filename = '/'.join(parts[2:])
+                model = hf_hub_download(repo_id, filename, **self.load_config)
+                self.model = T2IAdapter.from_pretrained(model, **self.load_config)
+            else:
+                self.model = T2IAdapter.from_pretrained(model_path, **self.load_config)
             if self.device is not None:
                 self.model.to(self.device)
             if self.dtype is not None:

modules/control/units/xs_model.py

@@ -36,7 +36,10 @@ try:
 except Exception:
     pass
 
-from diffusers.models.unet_2d_condition import UNet2DConditionModel
+try:
+    from diffusers.models.unet_2d_condition import UNet2DConditionModel
+except Exception:
+    from diffusers.models.unets.unet_2d_condition import UNet2DConditionModel
 from diffusers.utils import BaseOutput, logging, USE_PEFT_BACKEND
 
 

modules/devices.py

@@ -232,9 +232,13 @@ def set_cuda_params():
         if torch.backends.cudnn.is_available():
             try:
                 torch.backends.cudnn.deterministic = shared.opts.cudnn_deterministic
+                torch.use_deterministic_algorithms(shared.opts.cudnn_deterministic)
+                log.debug(f'Torch mode: deterministic={shared.opts.cudnn_deterministic}')
+                if shared.opts.cudnn_deterministic:
+                    os.environ.setdefault('CUBLAS_WORKSPACE_CONFIG', ':4096:8')
                 torch.backends.cudnn.benchmark = True
                 if shared.opts.cudnn_benchmark:
-                    log.debug('Torch enable cuDNN benchmark')
+                    log.debug('Torch cuDNN: enable benchmark')
                     torch.backends.cudnn.benchmark_limit = 0
                 torch.backends.cudnn.allow_tf32 = True
             except Exception:
@@ -363,10 +367,12 @@ def cond_cast_float(tensor):
     return tensor.float() if unet_needs_upcast else tensor
 
 
-def randn(seed, shape):
+def randn(seed, shape=None):
     torch.manual_seed(seed)
     if backend == 'ipex':
         torch.xpu.manual_seed_all(seed)
+    if shape is None:
+        return None
     if device.type == 'mps':
         return torch.randn(shape, device=cpu).to(device)
     elif shared.opts.diffusers_generator_device == "CPU":

modules/face/__init__.py

@@ -12,7 +12,7 @@ class Script(scripts.Script):
         return 'Face'
 
     def show(self, is_img2img):
-        return True if shared.backend == shared.Backend.DIFFUSERS else False
+        return True if shared.native else False
 
     def load_images(self, files):
         init_images = []
@@ -90,7 +90,7 @@ class Script(scripts.Script):
         return [mode, gallery, ip_model, ip_override, ip_cache, ip_strength, ip_structure, id_strength, id_conditioning, id_cache, pm_trigger, pm_strength, pm_start, fs_cache]
 
     def run(self, p: processing.StableDiffusionProcessing, mode, input_images, ip_model, ip_override, ip_cache, ip_strength, ip_structure, id_strength, id_conditioning, id_cache, pm_trigger, pm_strength, pm_start, fs_cache): # pylint: disable=arguments-differ, unused-argument
-        if shared.backend != shared.Backend.DIFFUSERS:
+        if not shared.native:
             return None
         if mode == 'None':
             return None

modules/face/faceid.py

@@ -70,7 +70,7 @@ def face_id(
 
         if shared.opts.cuda_compile_backend == 'none':
             sd_models.apply_token_merging(p.sd_model)
-            sd_hijack_freeu.apply_freeu(p, shared.backend == shared.Backend.ORIGINAL)
+            sd_hijack_freeu.apply_freeu(p, not shared.native)
 
         script_callbacks.before_process_callback(p)
 

modules/generation_parameters_copypaste.py

@@ -1,12 +1,11 @@
 import base64
 import io
 import os
-import re
-import json
 from PIL import Image
 import gradio as gr
 from modules.paths import data_path
 from modules import shared, gr_tempdir, script_callbacks, images
+from modules.infotext import parse, mapping, quote, unquote # pylint: disable=unused-import
 
 
 type_of_gr_update = type(gr.update())
@@ -14,7 +13,8 @@ paste_fields = {}
 registered_param_bindings = []
 debug = shared.log.trace if os.environ.get('SD_PASTE_DEBUG', None) is not None else lambda *args, **kwargs: None
 debug('Trace: PASTE')
-
+parse_generation_parameters = parse # compatibility
+infotext_to_setting_name_mapping = mapping # compatibility
 
 class ParamBinding:
     def __init__(self, paste_button, tabname, source_text_component=None, source_image_component=None, source_tabname=None, override_settings_component=None, paste_field_names=None):
@@ -32,21 +32,6 @@ def reset():
     paste_fields.clear()
 
 
-def quote(text):
-    if ',' not in str(text) and '\n' not in str(text) and ':' not in str(text):
-        return text
-    return json.dumps(text, ensure_ascii=False)
-
-
-def unquote(text):
-    if len(text) == 0 or text[0] != '"' or text[-1] != '"':
-        return text
-    try:
-        return json.loads(text)
-    except Exception:
-        return text
-
-
 def image_from_url_text(filedata):
     if filedata is None:
         return None
@@ -187,124 +172,13 @@ def send_image_and_dimensions(x):
     return img, w, h
 
 
-def parse_generation_parameters(infotext, no_prompt=False):
-    if not isinstance(infotext, str):
-        return {}
-    debug(f'Parse infotext: {infotext}')
-    re_param = re.compile(r'\s*([\w ]+):\s*("(?:\\"[^,]|\\"|\\|[^\"])+"|[^,]*)(?:,|$)') # multi-word: value
-    re_size = re.compile(r"^(\d+)x(\d+)$") # int x int
-    basic_params = ['steps:', 'seed:', 'width:', 'height:', 'sampler:', 'size:', 'cfg scale:'] # first param is one of those
-
-    infotext = infotext.replace('prompt:', 'Prompt:').replace('negative prompt:', 'Negative prompt:').replace('Negative Prompt', 'Negative prompt') # cleanup everything in brackets so re_params can work
-    infotext = infotext.replace(' Steps: ', ', Steps: ').replace('\nSteps: ', ', Steps: ') # fix cases where there is no delimiter between prompt and steps
-    sanitized = infotext
-    sanitized = re.sub(r'<[^>]*>', lambda match: ' ' * len(match.group()), sanitized)
-    sanitized = re.sub(r'\([^)]*\)', lambda match: ' ' * len(match.group()), sanitized)
-    sanitized = re.sub(r'\{[^}]*\}', lambda match: ' ' * len(match.group()), sanitized)
-
-    params = dict(re_param.findall(sanitized))
-    debug(f"Parse params: {params}")
-    params = { k.strip():params[k].strip() for k in params if k.lower() not in ['hashes', 'lora', 'embeddings', 'prompt', 'negative prompt']} # remove some keys
-    if len(list(params)) == 0:
-        first_param = None
-    else:
-        try:
-            first_param, first_param_idx = next((s, i) for i, s in enumerate(params) if any(x in s.lower() for x in basic_params))
-        except Exception:
-            first_param, first_param_idx = next(iter(params)), 0
-        if first_param_idx > 0:
-            for _i in range(first_param_idx):
-                params.pop(next(iter(params)))
-    params_idx = sanitized.find(f'{first_param}:') if first_param else -1
-    negative_idx = infotext.find("Negative prompt:")
-    if 'Steps:' in sanitized:
-        params_idx = max(params_idx, sanitized.find('Steps:'))
-
-    if negative_idx == -1: # prompt can be without negative prompt
-        prompt = infotext[:params_idx] if params_idx > 0 else infotext
-    else:
-        prompt = infotext[:negative_idx]
-    if prompt.startswith('Steps: '):
-        prompt = ''
-    if negative_idx >= 0:
-        negative = infotext[negative_idx:params_idx] if params_idx > 0 else infotext[negative_idx:]
-    else:
-        negative = ''
-
-    for k, v in params.copy().items(): # avoid dict-has-changed
-        if len(v) > 0 and v[0] == '"' and v[-1] == '"':
-            v = unquote(v)
-        m = re_size.match(v)
-        if v.replace('.', '', 1).isdigit():
-            params[k] = float(v) if '.' in v else int(v)
-        elif v == "True":
-            params[k] = True
-        elif v == "False":
-            params[k] = False
-        elif m is not None:
-            params[f"{k}-1"] = int(m.group(1))
-            params[f"{k}-2"] = int(m.group(2))
-        elif k == 'VAE' and v == 'TAESD':
-            params["Full quality"] = False
-        else:
-            params[k] = v
-    if not no_prompt:
-        params["Prompt"] = prompt.replace('Prompt:', '').strip(' ,\n')
-        params["Negative prompt"] = negative.replace('Negative prompt:', '').strip(' ,\n')
-    debug(f"Parse: {params}")
-    return params
-
-
-settings_map = {}
-
-
-infotext_to_setting_name_mapping = [
-    ('Backend', 'sd_backend'),
-    ('Model hash', 'sd_model_checkpoint'),
-    ('Refiner', 'sd_model_refiner'),
-    ('VAE', 'sd_vae'),
-    ('Parser', 'prompt_attention'),
-    ('Color correction', 'img2img_color_correction'),
-    # Samplers
-    ('Sampler Eta', 'scheduler_eta'),
-    ('Sampler ENSD', 'eta_noise_seed_delta'),
-    ('Sampler order', 'schedulers_solver_order'),
-    # Samplers diffusers
-    ('Sampler beta schedule', 'schedulers_beta_schedule'),
-    ('Sampler beta start', 'schedulers_beta_start'),
-    ('Sampler beta end', 'schedulers_beta_end'),
-    ('Sampler DPM solver', 'schedulers_dpm_solver'),
-    # Samplers original
-    ('Sampler brownian', 'schedulers_brownian_noise'),
-    ('Sampler discard', 'schedulers_discard_penultimate'),
-    ('Sampler dyn threshold', 'schedulers_use_thresholding'),
-    ('Sampler karras', 'schedulers_use_karras'),
-    ('Sampler low order', 'schedulers_use_loworder'),
-    ('Sampler quantization', 'enable_quantization'),
-    ('Sampler sigma', 'schedulers_sigma'),
-    ('Sampler sigma min', 's_min'),
-    ('Sampler sigma max', 's_max'),
-    ('Sampler sigma churn', 's_churn'),
-    ('Sampler sigma uncond', 's_min_uncond'),
-    ('Sampler sigma noise', 's_noise'),
-    ('Sampler sigma tmin', 's_tmin'),
-    ('Sampler ENSM', 'initial_noise_multiplier'), # img2img only
-    ('UniPC skip type', 'uni_pc_skip_type'),
-    ('UniPC variant', 'uni_pc_variant'),
-    # Token Merging
-    ('Mask weight', 'inpainting_mask_weight'),
-    ('ToMe', 'tome_ratio'),
-    ('ToDo', 'todo_ratio'),
-]
-
-
 def create_override_settings_dict(text_pairs):
     res = {}
     params = {}
     for pair in text_pairs:
         k, v = pair.split(":", maxsplit=1)
         params[k] = v.strip()
-    for param_name, setting_name in infotext_to_setting_name_mapping:
+    for param_name, setting_name in mapping:
         value = params.get(param_name, None)
         if value is None:
             continue
@@ -325,7 +199,7 @@ def connect_paste(button, local_paste_fields, input_comp, override_settings_comp
                 prompt = ''
         else:
             shared.log.debug(f'Paste prompt: type="current" prompt="{prompt}"')
-        params = parse_generation_parameters(prompt, no_prompt=False)
+        params = parse(prompt)
         script_callbacks.infotext_pasted_callback(prompt, params)
         res = []
         applied = {}

modules/hidiffusion/__init__.py

@@ -5,12 +5,14 @@ from modules import shared
 from modules.hidiffusion import hidiffusion
 
 
-def apply_hidiffusion(p, model_type):
+def apply(p, model_type):
+    if not shared.native:
+        return
     if model_type not in ['sd', 'sdxl'] and p.hidiffusion:
         shared.log.warning(f'HiDiffusion: class={shared.sd_model.__class__.__name__} not supported')
         return
-    remove_hidiffusion(p)
-    if p.hidiffusion:
+    unapply()
+    if getattr(p, 'hidiffusion', False) is True:
         t0 = time.time()
         hidiffusion.is_aggressive_raunet = shared.opts.hidiffusion_steps > 0
         hidiffusion.aggressive_step = shared.opts.hidiffusion_steps
@@ -36,6 +38,6 @@ def apply_hidiffusion(p, model_type):
         shared.log.debug(f'HiDiffusion apply: raunet={shared.opts.hidiffusion_raunet} attn={shared.opts.hidiffusion_attn} aggressive={shared.opts.hidiffusion_steps > 0}:{shared.opts.hidiffusion_steps} t1={shared.opts.hidiffusion_t1} t2={shared.opts.hidiffusion_t2} time={t1-t0:.2f} type={shared.sd_model_type} width={p.width} height={p.height}')
 
 
-def remove_hidiffusion(p):
+def unapply():
     if hasattr(shared.sd_model, "unet"):
         hidiffusion.remove_hidiffusion(shared.sd_model)

modules/hidiffusion/hidiffusion.py

@@ -3,7 +3,6 @@ import torch
 import torch.nn.functional as F
 from diffusers.utils.torch_utils import is_torch_version
 from diffusers.pipelines import auto_pipeline
-from modules.shared import log
 
 
 def sd15_hidiffusion_key():
@@ -229,7 +228,7 @@ def make_diffusers_transformer_block(block_class: Type[torch.nn.Module]) -> Type
                 norm_hidden_states = self.norm2(hidden_states)
                 norm_hidden_states = norm_hidden_states * (1 + scale_mlp) + shift_mlp
             if self._chunk_size is not None:
-                ff_output = _chunked_feed_forward(self.ff, norm_hidden_states, self._chunk_dim, self._chunk_size)
+                ff_output = _chunked_feed_forward(self.ff, norm_hidden_states, self._chunk_dim, self._chunk_size) # pylint: disable=undefined-variable # TODO hidiffusion undefined
             else:
                 ff_output = self.ff(norm_hidden_states)
             if self.use_ada_layer_norm_zero:
@@ -268,7 +267,7 @@ def make_diffusers_cross_attn_down_block(block_class: Type[torch.nn.Module]) ->
             encoder_attention_mask: Optional[torch.FloatTensor] = None,
             additional_residuals: Optional[torch.FloatTensor] = None,
         ) -> Tuple[torch.FloatTensor, Tuple[torch.FloatTensor, ...]]:
-            self.max_timestep = self.info['pipeline']._num_timesteps
+            self.max_timestep = self.info['pipeline']._num_timesteps # pylint: disable=protected-access
             # self.max_timestep = len(self.info['scheduler'].timesteps)
             ori_H, ori_W = self.info['size']
             if self.model == 'sd15':
@@ -303,7 +302,7 @@ def make_diffusers_cross_attn_down_block(block_class: Type[torch.nn.Module]) ->
                 self.T1 = int(self.max_timestep * self.T1_ratio)
 
             output_states = ()
-            cross_attention_kwargs.get("scale", 1.0) if cross_attention_kwargs is not None else 1.0
+            _scale = cross_attention_kwargs.get("scale", 1.0) if cross_attention_kwargs is not None else 1.0 # TODO hidiffusion unused
 
             blocks = list(zip(self.resnets, self.attentions))
 
@@ -407,7 +406,7 @@ def make_diffusers_cross_attn_up_block(block_class: Type[torch.nn.Module]) -> Ty
                     return F.interpolate(first, scale_factor=rescale, mode='bicubic')
                 return first
 
-            self.max_timestep = self.info['pipeline']._num_timesteps
+            self.max_timestep = self.info['pipeline']._num_timesteps # pylint: disable=protected-access
             ori_H, ori_W = self.info['size']
             if self.model == 'sd15':
                 if ori_H < 256 or ori_W < 256:
@@ -489,8 +488,8 @@ def make_diffusers_downsampler_block(block_class: Type[torch.nn.Module]) -> Type
         aggressive_raunet = False
         max_timestep = 50
 
-        def forward(self, hidden_states: torch.Tensor, scale = 1.0) -> torch.Tensor:
-            self.max_timestep = self.info['pipeline']._num_timesteps
+        def forward(self, hidden_states: torch.Tensor, scale = 1.0) -> torch.Tensor: # pylint: disable=unused-argument
+            self.max_timestep = self.info['pipeline']._num_timesteps # pylint: disable=protected-access
             # self.max_timestep = len(self.info['scheduler'].timesteps)
             ori_H, ori_W = self.info['size']
             if self.model == 'sd15':
@@ -522,20 +521,20 @@ def make_diffusers_downsampler_block(block_class: Type[torch.nn.Module]) -> Type
             else:
                 self.T1 = int(self.max_timestep * self.T1_ratio)
             if self.timestep < self.T1:
-                self.ori_stride = self.stride
-                self.ori_padding = self.padding
-                self.ori_dilation = self.dilation
-                self.stride = (4,4)
-                self.padding = (2,2)
-                self.dilation = (2,2)
+                self.ori_stride = self.stride # pylint: disable=access-member-before-definition, attribute-defined-outside-init
+                self.ori_padding = self.padding # pylint: disable=access-member-before-definition, attribute-defined-outside-init
+                self.ori_dilation = self.dilation # pylint: disable=access-member-before-definition, attribute-defined-outside-init
+                self.stride = (4,4) # pylint: disable=access-member-before-definition, attribute-defined-outside-init
+                self.padding = (2,2) # pylint: disable=access-member-before-definition, attribute-defined-outside-init
+                self.dilation = (2,2) # pylint: disable=access-member-before-definition, attribute-defined-outside-init
 
             hidden_states = F.conv2d(
                 hidden_states, self.weight, self.bias, self.stride, self.padding, self.dilation, self.groups
             )
             if self.timestep < self.T1:
-                self.stride = self.ori_stride
-                self.padding = self.ori_padding
-                self.dilation = self.ori_dilation
+                self.stride = self.ori_stride # pylint: disable=access-member-before-definition, attribute-defined-outside-init
+                self.padding = self.ori_padding # pylint: disable=access-member-before-definition, attribute-defined-outside-init
+                self.dilation = self.ori_dilation # pylint: disable=access-member-before-definition, attribute-defined-outside-init
             self.timestep += 1
             if self.timestep == self.max_timestep:
                 self.timestep = 0
@@ -557,8 +556,8 @@ def make_diffusers_upsampler_block(block_class: Type[torch.nn.Module]) -> Type[t
         aggressive_raunet = False
         max_timestep = 50
 
-        def forward(self, hidden_states: torch.Tensor, scale = 1.0) -> torch.Tensor:
-            self.max_timestep = self.info['pipeline']._num_timesteps
+        def forward(self, hidden_states: torch.Tensor, scale = 1.0) -> torch.Tensor: # pylint: disable=unused-argument
+            self.max_timestep = self.info['pipeline']._num_timesteps # pylint: disable=protected-access
             # self.max_timestep = len(self.info['scheduler'].timesteps)
             ori_H, ori_W = self.info['size']
             if self.model == 'sd15':
@@ -645,24 +644,21 @@ def apply_hidiffusion(
         modified_key = sd15_hidiffusion_key()
         for key, module in diffusion_model.named_modules():
             if apply_raunet and key in modified_key['down_module_key']:
-                make_block_fn = make_diffusers_downsampler_block
-                module.__class__ = make_block_fn(module.__class__)
+                module.__class__ = make_diffusers_downsampler_block(module.__class__)
                 module.switching_threshold_ratio = 'T1_ratio'
             if apply_raunet and key in modified_key['down_module_key_extra']:
-                make_block_fn = make_diffusers_cross_attn_down_block
-                module.__class__ = make_block_fn(module.__class__)
+                module.__class__ = make_diffusers_cross_attn_down_block(module.__class__)
                 module.switching_threshold_ratio = 'T2_ratio'
             if apply_raunet and key in modified_key['up_module_key']:
-                make_block_fn = make_diffusers_upsampler_block
-                module.__class__ = make_block_fn(module.__class__)
+                module.__class__ = make_diffusers_upsampler_block(module.__class__)
                 module.switching_threshold_ratio = 'T1_ratio'
             if apply_raunet and key in modified_key['up_module_key_extra']:
-                make_block_fn = make_diffusers_cross_attn_up_block
-                module.__class__ = make_block_fn(module.__class__)
+                module.__class__ = make_diffusers_cross_attn_up_block(module.__class__)
                 module.switching_threshold_ratio = 'T2_ratio'
             if apply_window_attn and key in modified_key['windown_attn_module_key']:
-                make_block_fn = make_diffusers_transformer_block
-                module.__class__ = make_block_fn(module.__class__)
+                module.__class__ = make_diffusers_transformer_block(module.__class__)
+            if hasattr(module, "_patched_forward"):
+                module.forward = module._patched_forward # pylint: disable=protected-access
             module.model = 'sd15'
             module.info = diffusion_model.info
 
@@ -685,7 +681,7 @@ def apply_hidiffusion(
             if apply_window_attn and key in modified_key['windown_attn_module_key']:
                 module.__class__ = make_diffusers_transformer_block(module.__class__)
             if hasattr(module, "_patched_forward"):
-                module.forward = module._patched_forward
+                module.forward = module._patched_forward # pylint: disable=protected-access
             module.model = 'sdxl'
             module.info = diffusion_model.info
     else:
@@ -702,7 +698,7 @@ def remove_hidiffusion(model: torch.nn.Module):
             module.info["hooks"].clear()
             del module.info
         if hasattr(module, "_forward"):
-            module.forward = module._forward
+            module.forward = module._forward # pylint: disable=protected-access
         if hasattr(module, "_parent"):
-            module.__class__ = module._parent
+            module.__class__ = module._parent # pylint: disable=protected-access
     return model

modules/infotext.py

@@ -0,0 +1,127 @@
+import os
+import re
+import json
+
+
+debug = lambda *args, **kwargs: None # pylint: disable=unnecessary-lambda-assignment
+re_size = re.compile(r"^(\d+)x(\d+)$") # int x int
+re_param = re.compile(r'\s*([\w ]+):\s*("(?:\\"[^,]|\\"|\\|[^\"])+"|[^,]*)(?:,|$)') # multi-word: value
+
+
+def quote(text):
+    if ',' not in str(text) and '\n' not in str(text) and ':' not in str(text):
+        return text
+    return json.dumps(text, ensure_ascii=False)
+
+
+def unquote(text):
+    if len(text) == 0 or text[0] != '"' or text[-1] != '"':
+        return text
+    try:
+        return json.loads(text)
+    except Exception:
+        return text
+
+
+def parse(infotext):
+    if not isinstance(infotext, str):
+        return {}
+    debug(f'Raw: {infotext}')
+    if 'negative prompt:' not in infotext.lower():
+        infotext = 'negative prompt: ' + infotext
+    if 'prompt:' not in infotext.lower():
+        infotext = 'prompt: ' + infotext
+
+    remaining = infotext.replace('\nSteps:', ' Steps:')
+    prompt = remaining[:infotext.lower().find('negative prompt:')]
+    remaining = remaining.replace(prompt, '')
+    if prompt.lower().startswith('prompt: '):
+        prompt = prompt[8:]
+    # debug(f'Prompt: {prompt}')
+
+    params = ['steps:', 'seed:', 'width:', 'height:', 'sampler:', 'size:', 'cfg scale:'] # first param is one of those
+    param_idx = [remaining.lower().find(p) for p in params if p in remaining.lower()]
+    param_idx = min(param_idx) if len(param_idx) > 0 else 0
+    negative = remaining[:param_idx] if param_idx > 0 else remaining
+    remaining = remaining.replace(negative, '')
+    if negative.lower().startswith('negative prompt: '):
+        negative = negative[16:]
+    # debug(f'Negative: {negative}')
+
+    params = dict(re_param.findall(remaining))
+    params['Prompt'] = prompt
+    params['Negative prompt'] = negative
+    for key, val in params.copy().items():
+        val = unquote(val).strip(" ,\n").replace('\\\n', '')
+        size = re_size.match(val)
+        if val.replace('.', '', 1).isdigit():
+            params[key] = float(val) if '.' in val else int(val)
+        elif val == "True":
+            params[key] = True
+        elif val == "False":
+            params[key] = False
+        elif key == 'VAE' and val == 'TAESD':
+            params["Full quality"] = False
+        elif size is not None:
+            params[f"{key}-1"] = int(size.group(1))
+            params[f"{key}-2"] = int(size.group(2))
+        elif isinstance(params[key], str):
+            params[key] = val
+        debug(f'Param parsed: type={type(params[key])} {key}={params[key]} raw="{val}"')
+
+    return params
+
+
+mapping = [
+    ('Backend', 'sd_backend'),
+    ('Model hash', 'sd_model_checkpoint'),
+    ('Refiner', 'sd_model_refiner'),
+    ('VAE', 'sd_vae'),
+    ('Parser', 'prompt_attention'),
+    ('Color correction', 'img2img_color_correction'),
+    # Samplers
+    ('Sampler Eta', 'scheduler_eta'),
+    ('Sampler ENSD', 'eta_noise_seed_delta'),
+    ('Sampler order', 'schedulers_solver_order'),
+    # Samplers diffusers
+    ('Sampler beta schedule', 'schedulers_beta_schedule'),
+    ('Sampler beta start', 'schedulers_beta_start'),
+    ('Sampler beta end', 'schedulers_beta_end'),
+    ('Sampler DPM solver', 'schedulers_dpm_solver'),
+    # Samplers original
+    ('Sampler brownian', 'schedulers_brownian_noise'),
+    ('Sampler discard', 'schedulers_discard_penultimate'),
+    ('Sampler dyn threshold', 'schedulers_use_thresholding'),
+    ('Sampler karras', 'schedulers_use_karras'),
+    ('Sampler low order', 'schedulers_use_loworder'),
+    ('Sampler quantization', 'enable_quantization'),
+    ('Sampler sigma', 'schedulers_sigma'),
+    ('Sampler sigma min', 's_min'),
+    ('Sampler sigma max', 's_max'),
+    ('Sampler sigma churn', 's_churn'),
+    ('Sampler sigma uncond', 's_min_uncond'),
+    ('Sampler sigma noise', 's_noise'),
+    ('Sampler sigma tmin', 's_tmin'),
+    ('Sampler ENSM', 'initial_noise_multiplier'), # img2img only
+    ('UniPC skip type', 'uni_pc_skip_type'),
+    ('UniPC variant', 'uni_pc_variant'),
+    # Token Merging
+    ('Mask weight', 'inpainting_mask_weight'),
+    ('ToMe', 'tome_ratio'),
+    ('ToDo', 'todo_ratio'),
+]
+
+
+if __name__ == '__main__':
+    import logging
+    log = logging.getLogger(__name__)
+    logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s | %(message)s')
+    debug = log.info
+
+    import sys
+    if len(sys.argv) > 1:
+        if os.path.exists(sys.argv[1]):
+            with open(sys.argv[1], 'r', encoding='utf8') as f:
+                parse(f.read())
+        else:
+            parse(sys.argv[1])

modules/interrogate.py

@@ -165,7 +165,7 @@ class InterrogateModels:
         res = ""
         shared.state.begin('Interrogate')
         try:
-            if shared.backend == shared.Backend.ORIGINAL and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
+            if not shared.native and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
                 lowvram.send_everything_to_cpu()
                 devices.torch_gc()
             self.load()
@@ -269,7 +269,7 @@ def interrogate(image, mode, caption=None):
 def interrogate_image(image, model, mode):
     shared.state.begin('Interrogate')
     try:
-        if shared.backend == shared.Backend.ORIGINAL and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
+        if not shared.native and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
             lowvram.send_everything_to_cpu()
             devices.torch_gc()
         load_interrogator(model)
@@ -297,7 +297,7 @@ def interrogate_batch(batch_files, batch_folder, batch_str, model, mode, write):
     shared.state.begin('Batch interrogate')
     prompts = []
     try:
-        if shared.backend == shared.Backend.ORIGINAL and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
+        if not shared.native and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
             lowvram.send_everything_to_cpu()
             devices.torch_gc()
         load_interrogator(model)

modules/ipadapter.py

@@ -76,7 +76,7 @@ def unapply(pipe): # pylint: disable=arguments-differ
     try:
         if hasattr(pipe, 'set_ip_adapter_scale'):
             pipe.set_ip_adapter_scale(0)
-        if hasattr(pipe, 'unet') and hasattr(pipe.unet, 'config')and pipe.unet.config.encoder_hid_dim_type == 'ip_image_proj':
+        if hasattr(pipe, 'unet') and hasattr(pipe.unet, 'config') and pipe.unet.config.encoder_hid_dim_type == 'ip_image_proj':
             pipe.unet.encoder_hid_proj = None
             pipe.config.encoder_hid_dim_type = None
             pipe.unet.set_default_attn_processor()
@@ -138,7 +138,7 @@ def apply(pipe, p: processing.StableDiffusionProcessing, adapter_names=[], adapt
     # init code
     if pipe is None:
         return False
-    if shared.backend != shared.Backend.DIFFUSERS:
+    if not shared.native:
         shared.log.warning('IP adapter: not in diffusers mode')
         return False
     if len(adapter_images) == 0:

modules/layerdiffuse/__init__.py

@@ -41,6 +41,8 @@ def apply_layerdiffuse_sdxl_conv(pipeline):
 
 
 def apply_layerdiffuse():
+    if not shared.native:
+        return
     try:
         if shared.sd_model_type == 'sd':
             shared.log.info(f'LayerDiffuse: class={shared.sd_model.__class__.__name__}')

modules/layerdiffuse/layerdiffuse_model.py

@@ -9,9 +9,12 @@ from typing import Optional, Tuple, Union
 from diffusers import AutoencoderKL
 from diffusers.configuration_utils import ConfigMixin, register_to_config
 from diffusers.models.modeling_utils import ModelMixin
-from diffusers.models.unet_2d_blocks import UNetMidBlock2D, get_down_block, get_up_block
 from diffusers.models.autoencoders.vae import DecoderOutput
 from diffusers.models.attention_processor import Attention, AttnProcessor
+try:
+    from diffusers.models.unet_2d_blocks import UNetMidBlock2D, get_down_block, get_up_block
+except Exception:
+    from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2D, get_down_block, get_up_block
 
 
 def zero_module(module):

modules/loader.py

@@ -10,14 +10,8 @@ from modules import timer, errors
 initialized = False
 errors.install()
 logging.getLogger("DeepSpeed").disabled = True
-# os.environ.setdefault('OMP_NUM_THREADS', 1)
-# os.environ.setdefault('MKL_NUM_THREADS', 1)
-
-# import tensorflow as tf # pylint: disable=C0411
 
 import torch # pylint: disable=C0411
-
-# torch.set_num_threads(1)
 try:
     import intel_extension_for_pytorch as ipex # pylint: disable=import-error, unused-import
     errors.log.debug(f'Load IPEX=={ipex.__version__}')

modules/masking.py

@@ -442,7 +442,7 @@ def run_mask(input_image: Image.Image, input_mask: Image.Image = None, return_ty
 
     return_type = return_type or opts.preview_type
 
-    shared.log.debug(f'Mask: size={input_image.width}x{input_image.height} masked={mask_size}px area={area_size/total_size:.2f} auto={opts.auto_mask} blur={opts.mask_blur} erode={opts.mask_erode} dilate={opts.mask_dilate} type={return_type} time={t1-t0:.2f}')
+    shared.log.debug(f'Mask: size={input_image.width}x{input_image.height} masked={mask_size}px area={area_size/total_size:.2f} auto={opts.auto_mask} blur={opts.mask_blur:.3f} erode={opts.mask_erode:.3f} dilate={opts.mask_dilate:.3f} type={return_type} time={t1-t0:.2f}')
     if return_type == 'None':
         return input_mask
     elif return_type == 'Opaque':

modules/modeldata.py

@@ -81,7 +81,7 @@ class Shared(sys.modules[__name__].__class__):
             if modules.sd_models.model_data.sd_model is None:
                 model_type = 'none'
                 return model_type
-            if shared.backend == shared.Backend.ORIGINAL:
+            if not shared.native:
                 model_type = 'ldm'
             elif "StableDiffusionXL" in self.sd_model.__class__.__name__:
                 model_type = 'sdxl'
@@ -110,7 +110,7 @@ class Shared(sys.modules[__name__].__class__):
             if modules.sd_models.model_data.sd_refiner is None:
                 model_type = 'none'
                 return model_type
-            if shared.backend == shared.Backend.ORIGINAL:
+            if not shared.native:
                 model_type = 'ldm'
             elif "StableDiffusionXL" in self.sd_refiner.__class__.__name__:
                 model_type = 'sdxl'

modules/pag/__init__.py

@@ -10,13 +10,15 @@ orig_pipeline = None
 
 def apply(p: processing.StableDiffusionProcessing): # pylint: disable=arguments-differ
     global orig_pipeline # pylint: disable=global-statement
+    if not shared.native:
+        return None
     c = shared.sd_model.__class__ if shared.sd_loaded else None
-    if p.pag_scale == 0:
+    if c == StableDiffusionPAGPipeline or c == StableDiffusionXLPAGPipeline:
         unapply()
         return None
-    if c == StableDiffusionPAGPipeline or c == StableDiffusionXLPAGPipeline:
-        pass
-    elif detect.is_sd15(c):
+    if p.pag_scale == 0:
+        return
+    if detect.is_sd15(c):
         orig_pipeline = shared.sd_model
         shared.sd_model = sd_models.switch_pipe(StableDiffusionPAGPipeline, shared.sd_model)
     elif detect.is_sdxl(c):

modules/postprocess/gfpgan_model.py

@@ -72,8 +72,8 @@ def setup_model(dirname):
     except Exception:
         pass
     try:
-        install('basicsr')
-        install('gfpgan')
+        install('basicsr', quiet=True)
+        install('gfpgan', quiet=True)
         import gfpgan
         import facexlib
         import modules.face_restoration

modules/postprocess/sdupscaler_model.py

@@ -8,7 +8,7 @@ class UpscalerSD(Upscaler):
     def __init__(self, dirname): # pylint: disable=super-init-not-called
         self.name = "SDUpscale"
         self.user_path = dirname
-        if shared.backend != shared.Backend.DIFFUSERS:
+        if not shared.native:
             super().__init__()
             return
         self.scalers = [
@@ -28,7 +28,6 @@ class UpscalerSD(Upscaler):
             shared.log.debug(f"Upscaler cached: type={scaler.name} model={path}")
             return self.models[path]
         else:
-            devices.set_cuda_params()
             model = diffusers.DiffusionPipeline.from_pretrained(path, cache_dir=shared.opts.diffusers_dir, torch_dtype=devices.dtype)
             if hasattr(model, "set_progress_bar_config"):
                 model.set_progress_bar_config(bar_format='Progress {rate_fmt}{postfix} {bar} {percentage:3.0f}% {n_fmt}/{total_fmt} {elapsed} {remaining} ' + '\x1b[38;5;71m' + 'Upscale', ncols=80, colour='#327fba')

modules/postprocessing.py

@@ -4,7 +4,7 @@ from typing import List
 
 from PIL import Image
 
-from modules import shared, images, devices, scripts, scripts_postprocessing, generation_parameters_copypaste
+from modules import shared, images, devices, scripts, scripts_postprocessing, infotext
 from modules.shared import opts
 
 
@@ -17,7 +17,7 @@ def run_postprocessing(extras_mode, image, image_folder: List[tempfile.NamedTemp
     image_ext = []
     outputs = []
     params = {}
-    infotext = ''
+    info = ''
     if extras_mode == 1:
         for img in image_folder:
             if isinstance(img, Image.Image):
@@ -63,7 +63,7 @@ def run_postprocessing(extras_mode, image, image_folder: List[tempfile.NamedTemp
     processed_images = []
     for image, name, ext in zip(image_data, image_names, image_ext): # pylint: disable=redefined-argument-from-local
         shared.log.debug(f'Process: image={image} {args}')
-        infotext = ''
+        info = ''
         if shared.state.interrupted:
             shared.log.debug('Postprocess interrupted')
             break
@@ -73,27 +73,27 @@ def run_postprocessing(extras_mode, image, image_folder: List[tempfile.NamedTemp
         pp = scripts_postprocessing.PostprocessedImage(image.convert("RGB"))
         scripts.scripts_postproc.run(pp, args)
         geninfo, items = images.read_info_from_image(image)
-        params = generation_parameters_copypaste.parse_generation_parameters(geninfo)
+        params = infotext.parse(geninfo)
         for k, v in items.items():
             pp.image.info[k] = v
         if 'parameters' in items:
-            infotext = items['parameters'] + ', '
-        infotext = infotext + ", ".join([k if k == v else f'{k}: {generation_parameters_copypaste.quote(v)}' for k, v in pp.info.items() if v is not None])
-        pp.image.info["postprocessing"] = infotext
+            info = items['parameters'] + ', '
+        info = info + ", ".join([k if k == v else f'{k}: {infotext.quote(v)}' for k, v in pp.info.items() if v is not None])
+        pp.image.info["postprocessing"] = info
         processed_images.append(pp.image)
         if save_output:
             if opts.use_original_name_batch and name is not None:
                 forced_filename = os.path.splitext(os.path.basename(name))[0]
-                images.save_image(pp.image, path=outpath, extension=ext or opts.samples_format, info=infotext, short_filename=True, no_prompt=True, grid=False, pnginfo_section_name="extras", existing_info=pp.image.info, forced_filename=forced_filename)
+                images.save_image(pp.image, path=outpath, extension=ext or opts.samples_format, info=info, short_filename=True, no_prompt=True, grid=False, pnginfo_section_name="extras", existing_info=pp.image.info, forced_filename=forced_filename)
             else:
-                images.save_image(pp.image, path=outpath, extension=ext or opts.samples_format, info=infotext, short_filename=True, no_prompt=True, grid=False, pnginfo_section_name="extras", existing_info=pp.image.info)
+                images.save_image(pp.image, path=outpath, extension=ext or opts.samples_format, info=info, short_filename=True, no_prompt=True, grid=False, pnginfo_section_name="extras", existing_info=pp.image.info)
         if extras_mode != 2 or show_extras_results:
             outputs.append(pp.image)
         image.close()
     scripts.scripts_postproc.postprocess(processed_images, args)
 
     devices.torch_gc()
-    return outputs, infotext, params
+    return outputs, info, params
 
 
 def run_extras(extras_mode, resize_mode, image, image_folder, input_dir, output_dir, show_extras_results, gfpgan_visibility, codeformer_visibility, codeformer_weight, upscaling_resize, upscaling_resize_w, upscaling_resize_h, upscaling_crop, extras_upscaler_1, extras_upscaler_2, extras_upscaler_2_visibility, upscale_first: bool, save_output: bool = True): #pylint: disable=unused-argument

modules/processing.py

@@ -3,7 +3,7 @@ import json
 import time
 from contextlib import nullcontext
 import numpy as np
-from PIL import Image
+from PIL import Image, ImageOps
 from modules import shared, devices, errors, images, scripts, memstats, lowvram, script_callbacks, extra_networks, face_restoration, sd_hijack_freeu, sd_models, sd_vae, processing_helpers
 from modules.sd_hijack_hypertile import context_hypertile_vae, context_hypertile_unet
 from modules.processing_class import StableDiffusionProcessing, StableDiffusionProcessingTxt2Img, StableDiffusionProcessingImg2Img, StableDiffusionProcessingControl # pylint: disable=unused-import
@@ -161,7 +161,7 @@ def process_images(p: StableDiffusionProcessing) -> Processed:
         pag.apply(p)
         if shared.opts.cuda_compile_backend == 'none':
             sd_models.apply_token_merging(p.sd_model)
-            sd_hijack_freeu.apply_freeu(p, shared.backend == shared.Backend.ORIGINAL)
+            sd_hijack_freeu.apply_freeu(p, not shared.native)
 
         if p.width is not None:
             p.width = 8 * int(p.width / 8)
@@ -247,7 +247,7 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
     else:
         assert p.prompt is not None
 
-    if shared.backend == shared.Backend.ORIGINAL:
+    if not shared.native:
         import modules.sd_hijack # pylint: disable=redefined-outer-name
         modules.sd_hijack.model_hijack.apply_circular(p.tiling)
         modules.sd_hijack.model_hijack.clear_comments()
@@ -256,7 +256,7 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
     output_images = []
 
     process_init(p)
-    if os.path.exists(shared.opts.embeddings_dir) and not p.do_not_reload_embeddings and shared.backend == shared.Backend.ORIGINAL:
+    if os.path.exists(shared.opts.embeddings_dir) and not p.do_not_reload_embeddings and not shared.native:
         modules.sd_hijack.model_hijack.embedding_db.load_textual_inversion_embeddings(force_reload=False)
     if p.scripts is not None and isinstance(p.scripts, scripts.ScriptRunner):
         p.scripts.process(p)
@@ -264,7 +264,7 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
     def infotext(_inxex=0): # dummy function overriden if there are iterations
         return ''
 
-    ema_scope_context = p.sd_model.ema_scope if shared.backend == shared.Backend.ORIGINAL else nullcontext
+    ema_scope_context = p.sd_model.ema_scope if not shared.native else nullcontext
     shared.state.job_count = p.n_iter
     with devices.inference_context(), ema_scope_context():
         t0 = time.time()
@@ -283,7 +283,7 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
                 shared.log.debug(f'Process interrupted: {n+1}/{p.n_iter}')
                 break
 
-            if shared.backend == shared.Backend.DIFFUSERS:
+            if shared.native:
                 from modules import ipadapter
                 ipadapter.apply(shared.sd_model, p)
             p.prompts = p.all_prompts[n * p.batch_size:(n+1) * p.batch_size]
@@ -304,10 +304,10 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
             if p.scripts is not None and isinstance(p.scripts, scripts.ScriptRunner):
                 x_samples_ddim = p.scripts.process_images(p)
             if x_samples_ddim is None:
-                if shared.backend == shared.Backend.ORIGINAL:
+                if not shared.native:
                     from modules.processing_original import process_original
                     x_samples_ddim = process_original(p)
-                elif shared.backend == shared.Backend.DIFFUSERS:
+                elif shared.native:
                     from modules.processing_diffusers import process_diffusers
                     x_samples_ddim = process_diffusers(p)
                 else:
@@ -316,7 +316,7 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
             if not shared.opts.keep_incomplete and shared.state.interrupted:
                 x_samples_ddim = []
 
-            if shared.backend == shared.Backend.ORIGINAL and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
+            if not shared.native and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
                 lowvram.send_everything_to_cpu()
                 devices.torch_gc()
             if p.scripts is not None and isinstance(p.scripts, scripts.ScriptRunner):
@@ -407,7 +407,7 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
                 if shared.opts.grid_save:
                     images.save_image(grid, p.outpath_grids, "", p.all_seeds[0], p.all_prompts[0], shared.opts.grid_format, info=infotext(-1), p=p, grid=True, suffix="-grid") # main save grid
 
-    if shared.backend == shared.Backend.DIFFUSERS:
+    if shared.native:
         from modules import ipadapter
         ipadapter.unapply(shared.sd_model)
 
@@ -415,7 +415,11 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
         extra_networks.deactivate(p, extra_network_data)
 
     if shared.opts.include_mask:
-        if getattr(p, 'image_mask', None) is not None and isinstance(p.image_mask, Image.Image):
+        if shared.opts.mask_apply_overlay and p.overlay_images is not None and len(p.overlay_images):
+            p.image_mask = create_binary_mask(p.overlay_images[0])
+            p.image_mask = ImageOps.invert(p.image_mask)
+            output_images.append(p.image_mask)
+        elif getattr(p, 'image_mask', None) is not None and isinstance(p.image_mask, Image.Image):
             if getattr(p, 'mask_for_facehires', None) is not None:
                 output_images.append(p.mask_for_facehires)
             else:

modules/processing_class.py

@@ -215,7 +215,7 @@ class StableDiffusionProcessingTxt2Img(StableDiffusionProcessing):
         self.script_args = []
 
     def init(self, all_prompts=None, all_seeds=None, all_subseeds=None):
-        if shared.backend == shared.Backend.DIFFUSERS:
+        if shared.native:
             shared.sd_model = sd_models.set_diffuser_pipe(self.sd_model, sd_models.DiffusersTaskType.TEXT_2_IMAGE)
         self.width = self.width or 512
         self.height = self.height or 512
@@ -252,7 +252,7 @@ class StableDiffusionProcessingTxt2Img(StableDiffusionProcessing):
                     self.hr_upscale_to_y = self.hr_resize_y
                 self.truncate_x = (self.hr_upscale_to_x - target_w) // 8
                 self.truncate_y = (self.hr_upscale_to_y - target_h) // 8
-        if shared.backend == shared.Backend.ORIGINAL: # diffusers are handled in processing_diffusers
+        if not shared.native: # diffusers are handled in processing_diffusers
             if (self.hr_upscale_to_x == self.width and self.hr_upscale_to_y == self.height) or upscaler is None or upscaler == 'None': # special case: the user has chosen to do nothing
                 self.is_hr_pass = False
                 return
@@ -303,9 +303,9 @@ class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
         self.script_args = []
 
     def init(self, all_prompts=None, all_seeds=None, all_subseeds=None):
-        if shared.backend == shared.Backend.DIFFUSERS and getattr(self, 'image_mask', None) is not None:
+        if shared.native and getattr(self, 'image_mask', None) is not None:
             shared.sd_model = sd_models.set_diffuser_pipe(self.sd_model, sd_models.DiffusersTaskType.INPAINTING)
-        elif shared.backend == shared.Backend.DIFFUSERS and getattr(self, 'init_images', None) is not None:
+        elif shared.native and getattr(self, 'init_images', None) is not None:
             shared.sd_model = sd_models.set_diffuser_pipe(self.sd_model, sd_models.DiffusersTaskType.IMAGE_2_IMAGE)
 
         if all_prompts is not None:
@@ -317,7 +317,7 @@ class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
 
         if self.sampler_name == "PLMS":
             self.sampler_name = 'UniPC'
-        if shared.backend == shared.Backend.ORIGINAL:
+        if not shared.native:
             self.sampler = sd_samplers.create_sampler(self.sampler_name, self.sd_model)
             if hasattr(self.sampler, "initialize"):
                 self.sampler.initialize(self)
@@ -331,7 +331,7 @@ class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
         if self.image_mask is not None:
             if type(self.image_mask) == list:
                 self.image_mask = self.image_mask[0]
-            if shared.backend == shared.Backend.ORIGINAL: # original way of processing mask
+            if not shared.native: # original way of processing mask
                 self.image_mask = processing_helpers.create_binary_mask(self.image_mask)
                 if self.inpainting_mask_invert:
                     self.image_mask = ImageOps.invert(self.image_mask)
@@ -341,7 +341,7 @@ class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
                     np_mask = cv2.GaussianBlur(np_mask, (kernel_size, 1), self.mask_blur)
                     np_mask = cv2.GaussianBlur(np_mask, (1, kernel_size), self.mask_blur)
                     self.image_mask = Image.fromarray(np_mask)
-            elif shared.backend == shared.Backend.DIFFUSERS:
+            elif shared.native:
                 if 'control' in self.ops:
                     self.image_mask = masking.run_mask(input_image=self.init_images, input_mask=self.image_mask, return_type='Grayscale', invert=self.inpainting_mask_invert==1) # blur/padding are handled in masking module
                 else:
@@ -411,9 +411,9 @@ class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
             self.overlay_images = self.overlay_images * self.batch_size
         if self.color_corrections is not None and len(self.color_corrections) == 1:
             self.color_corrections = self.color_corrections * self.batch_size
-        if shared.backend == shared.Backend.DIFFUSERS:
+        if shared.native:
             return # we've already set self.init_images and self.mask and we dont need any more processing
-        elif shared.backend == shared.Backend.ORIGINAL:
+        elif not shared.native:
             self.init_images = [np.moveaxis((np.array(image).astype(np.float32) / 255.0), 2, 0) for image in self.init_images]
             if len(self.init_images) == 1:
                 batch_images = np.expand_dims(self.init_images[0], axis=0).repeat(self.batch_size, axis=0)
@@ -469,6 +469,24 @@ class StableDiffusionProcessingControl(StableDiffusionProcessingImg2Img):
         self.fidelity = 0.5
         self.mask_image = None
         self.override = None
+        self.resize_mode_before = None
+        self.resize_name_before = None
+        self.width_before = None
+        self.height_before = None
+        self.scale_by_before = None
+        self.selected_scale_tab_before = None
+        self.resize_mode_after = None
+        self.resize_name_after = None
+        self.width_after = None
+        self.height_after = None
+        self.scale_by_after = None
+        self.selected_scale_tab_after = None
+        self.resize_mode_mask = None
+        self.resize_name_mask = None
+        self.width_mask = None
+        self.height_mask = None
+        self.scale_by_mask = None
+        self.selected_scale_tab_mask = None
 
     def sample(self, conditioning, unconditional_conditioning, seeds, subseeds, subseed_strength, prompts): # abstract
         pass

modules/processing_diffusers.py

@@ -100,6 +100,7 @@ def process_diffusers(p: processing.StableDiffusionProcessing):
         denoising_start=0 if use_refiner_start else p.refiner_start if use_denoise_start else None,
         denoising_end=p.refiner_start if use_refiner_start else 1 if use_denoise_start else None,
         output_type='latent' if hasattr(shared.sd_model, 'vae') else 'np',
+        # output_type='pil',
         clip_skip=p.clip_skip,
         desc='Base',
     )
@@ -112,13 +113,17 @@ def process_diffusers(p: processing.StableDiffusionProcessing):
         t0 = time.time()
         sd_models_compile.check_deepcache(enable=True)
         sd_models.move_model(shared.sd_model, devices.device)
-        hidiffusion.apply_hidiffusion(p, shared.sd_model_type)
+        hidiffusion.apply(p, shared.sd_model_type)
         # if 'image' in base_args:
         #    base_args['image'] = set_latents(p)
-        output = shared.sd_model(**base_args) # pylint: disable=not-callable
+        if hasattr(shared.sd_model, 'tgate') and getattr(p, 'gate_step', -1) > 0:
+            base_args['gate_step'] = p.gate_step
+            output = shared.sd_model.tgate(**base_args) # pylint: disable=not-callable
+        else:
+            output = shared.sd_model(**base_args)
         if isinstance(output, dict):
             output = SimpleNamespace(**output)
-        hidiffusion.remove_hidiffusion(p)
+        hidiffusion.unapply()
         sd_models_compile.openvino_post_compile(op="base") # only executes on compiled vino models
         sd_models_compile.check_deepcache(enable=False)
         if shared.cmd_opts.profile:
@@ -305,7 +310,9 @@ def process_diffusers(p: processing.StableDiffusionProcessing):
             if not hasattr(output, 'images') and hasattr(output, 'frames'):
                 shared.log.debug(f'Generated: frames={len(output.frames[0])}')
                 output.images = output.frames[0]
-            if hasattr(shared.sd_model, "vae") and output.images is not None and len(output.images) > 0:
+            if torch.is_tensor(output.images) and len(output.images) > 0 and any(s >= 512 for s in output.images.shape):
+                results = output.images.cpu().numpy()
+            elif hasattr(shared.sd_model, "vae") and output.images is not None and len(output.images) > 0:
                 results = processing_vae.vae_decode(latents=output.images, model=shared.sd_model, full_quality=p.full_quality)
             elif hasattr(output, 'images'):
                 results = output.images

modules/processing_helpers.py

@@ -35,9 +35,9 @@ def apply_color_correction(correction, original_image):
 
 
 def apply_overlay(image: Image, paste_loc, index, overlays):
-    debug(f'Apply overlay: image={image} loc={paste_loc} index={index} overlays={overlays}')
     if overlays is None or index >= len(overlays):
         return image
+    debug(f'Apply overlay: image={image} loc={paste_loc} index={index} overlays={overlays}')
     overlay = overlays[index]
     if paste_loc is not None:
         x, y, w, h = paste_loc
@@ -321,7 +321,7 @@ def img2img_image_conditioning(p, source_image, latent_image, image_mask=None):
 
     # HACK: Using introspection as the Depth2Image model doesn't appear to uniquely
     # identify itself with a field common to all models. The conditioning_key is also hybrid.
-    if shared.backend == shared.Backend.DIFFUSERS:
+    if shared.native:
         return diffusers_image_conditioning(source_image, latent_image, image_mask)
     if isinstance(p.sd_model, LatentDepth2ImageDiffusion):
         return depth2img_image_conditioning(source_image)
@@ -346,7 +346,7 @@ def validate_sample(tensor):
         sample = tensor
     else:
         shared.log.warning(f'Unknown sample type: {type(tensor)}')
-    sample = 255.0 * np.moveaxis(sample, 0, 2) if shared.backend == shared.Backend.ORIGINAL else 255.0 * sample
+    sample = 255.0 * np.moveaxis(sample, 0, 2) if not shared.native else 255.0 * sample
     with warnings.catch_warnings(record=True) as w:
         cast = sample.astype(np.uint8)
     if len(w) > 0:
@@ -482,7 +482,10 @@ def get_generator(p):
     else:
         generator_device = devices.cpu if shared.opts.diffusers_generator_device == "CPU" else shared.device
         try:
+            devices.randn(p.seeds[0])
             generator = [torch.Generator(generator_device).manual_seed(s) for s in p.seeds]
+            seeds = [g.initial_seed() for g in generator]
+            shared.log.debug(f'Torch generator: device={generator_device} seeds={seeds}')
         except Exception as e:
             shared.log.error(f'Torch generator: seeds={p.seeds} device={generator_device} {e}')
             generator = None
@@ -503,12 +506,19 @@ def set_latents(p):
     return latents
 
 
+last_circular = False
 def apply_circular(enable, model):
+    global last_circular # pylint: disable=global-statement
+    if not hasattr(model, 'unet') or not hasattr(model, 'vae'):
+        return
+    if last_circular == enable:
+        return
     try:
         for layer in [layer for layer in model.unet.modules() if type(layer) is torch.nn.Conv2d]:
             layer.padding_mode = 'circular' if enable else 'zeros'
         for layer in [layer for layer in model.vae.modules() if type(layer) is torch.nn.Conv2d]:
             layer.padding_mode = 'circular' if enable else 'zeros'
+        last_circular = enable
     except Exception as e:
         debug(f"Diffusers tiling failed: {e}")
 

modules/processing_info.py

@@ -4,7 +4,7 @@ from modules import shared, sd_samplers_common, sd_vae, generation_parameters_co
 from modules.processing_class import StableDiffusionProcessing
 
 
-if shared.backend == shared.Backend.ORIGINAL:
+if not shared.native:
     from modules import sd_hijack
 else:
     sd_hijack = None
@@ -57,7 +57,7 @@ def create_infotext(p: StableDiffusionProcessing, all_prompts=None, all_seeds=No
         "Styles": "; ".join(p.styles) if p.styles is not None and len(p.styles) > 0 else None,
         "Tiling": p.tiling if p.tiling else None,
         # sdnext
-        "Backend": 'Diffusers' if shared.backend == shared.Backend.DIFFUSERS else 'Original',
+        "Backend": 'Diffusers' if shared.native else 'Original',
         "App": 'SD.Next',
         "Version": git_commit,
         "Comment": comment,
@@ -98,6 +98,21 @@ def create_infotext(p: StableDiffusionProcessing, all_prompts=None, all_seeds=No
         # lookup by index
         if getattr(p, 'resize_mode', None) is not None:
             args['Resize mode'] = shared.resize_modes[p.resize_mode] if shared.resize_modes[p.resize_mode] != 'None' else None
+    if getattr(p, 'resize_mode_before', None) is not None:
+        args['Size before'] = f"{p.width_before}x{p.height_before}" if hasattr(p, 'width_before') and hasattr(p, 'height_before') else None
+        args['Size mode before'] = p.resize_mode_before
+        args['Size scale before'] = p.scale_by_before
+        args['Size name before'] = p.resize_name_before
+    if getattr(p, 'resize_mode_after', None) is not None:
+        args['Size after'] = f"{p.width_after}x{p.height_after}" if hasattr(p, 'width_after') and hasattr(p, 'height_after') else None
+        args['Size mode after'] = p.resize_mode_after
+        args['Size scale after'] = p.scale_by_after
+        args['Size name after'] = p.resize_name_after
+    if getattr(p, 'resize_mode_mask', None) is not None:
+        args['Size mask'] = f"{p.width_mask}x{p.height_mask}" if hasattr(p, 'width_mask') and hasattr(p, 'height_mask') else None
+        args['Size mode mask'] = p.resize_mode_mask
+        args['Size scale mask'] = p.scale_by_mask
+        args['Size name mask'] = p.resize_name_mask
     if 'face' in p.ops:
         args["Face restoration"] = shared.opts.face_restoration_model
     if 'color' in p.ops:
@@ -109,12 +124,12 @@ def create_infotext(p: StableDiffusionProcessing, all_prompts=None, all_seeds=No
     args["Sampler ENSD"] = shared.opts.eta_noise_seed_delta if shared.opts.eta_noise_seed_delta != 0 and sd_samplers_common.is_sampler_using_eta_noise_seed_delta(p) else None
     args["Sampler ENSM"] = p.initial_noise_multiplier if getattr(p, 'initial_noise_multiplier', 1.0) != 1.0 else None
     args['Sampler order'] = shared.opts.schedulers_solver_order if shared.opts.schedulers_solver_order != shared.opts.data_labels.get('schedulers_solver_order').default else None
-    if shared.backend == shared.Backend.DIFFUSERS:
+    if shared.native:
         args['Sampler beta schedule'] = shared.opts.schedulers_beta_schedule if shared.opts.schedulers_beta_schedule != shared.opts.data_labels.get('schedulers_beta_schedule').default else None
         args['Sampler beta start'] = shared.opts.schedulers_beta_start if shared.opts.schedulers_beta_start != shared.opts.data_labels.get('schedulers_beta_start').default else None
         args['Sampler beta end'] = shared.opts.schedulers_beta_end if shared.opts.schedulers_beta_end != shared.opts.data_labels.get('schedulers_beta_end').default else None
         args['Sampler DPM solver'] = shared.opts.schedulers_dpm_solver if shared.opts.schedulers_dpm_solver != shared.opts.data_labels.get('schedulers_dpm_solver').default else None
-    if shared.backend == shared.Backend.ORIGINAL:
+    if not shared.native:
         args['Sampler brownian'] = shared.opts.schedulers_brownian_noise if shared.opts.schedulers_brownian_noise != shared.opts.data_labels.get('schedulers_brownian_noise').default else None
         args['Sampler discard'] = shared.opts.schedulers_discard_penultimate if shared.opts.schedulers_discard_penultimate != shared.opts.data_labels.get('schedulers_discard_penultimate').default else None
         args['Sampler dyn threshold'] = shared.opts.schedulers_use_thresholding if shared.opts.schedulers_use_thresholding != shared.opts.data_labels.get('schedulers_use_thresholding').default else None

modules/prompt_parser.py

@@ -14,7 +14,7 @@ from typing import List
 import lark
 import torch
 from compel import Compel
-from modules.shared import opts, log, backend, Backend
+from modules.shared import opts, log, native
 
 # a prompt like this: "fantasy landscape with a [mountain:lake:0.25] and [an oak:a christmas tree:0.75][ in foreground::0.6][ in background:0.25] [shoddy:masterful:0.5]"
 # will be represented with prompt_schedule like this (assuming steps=100):
@@ -326,7 +326,7 @@ def parse_prompt_attention(text):
         whitespace = ''
     else:
         re_attention = re_attention_v1
-        if backend == Backend.DIFFUSERS:
+        if native:
             text = text.replace('\n', ' BREAK ')
         else:
             text = text.replace('\n', ' ')

modules/prompt_parser_diffusers.py

@@ -104,7 +104,7 @@ def get_prompt_schedule(prompt, steps):
 
 def get_tokens(msg, prompt):
     global token_dict, token_type # pylint: disable=global-statement
-    if shared.backend != shared.Backend.DIFFUSERS:
+    if not shared.native:
         return
     if shared.sd_loaded and hasattr(shared.sd_model, 'tokenizer') and shared.sd_model.tokenizer is not None:
         if token_dict is None or token_type != shared.sd_model_type:
@@ -133,7 +133,7 @@ def encode_prompts(pipe, p, prompts: list, negative_prompts: list, steps: int, c
     if 'StableDiffusion' not in pipe.__class__.__name__ and 'DemoFusion' not in pipe.__class__.__name__ and 'StableCascade' not in pipe.__class__.__name__:
         shared.log.warning(f"Prompt parser not supported: {pipe.__class__.__name__}")
         return
-    elif prompts == cache.get('prompts', None) and negative_prompts == cache.get('negative_prompts', None) and clip_skip == cache.get('clip_skip', None) and cache.get('model_type', None) == shared.sd_model_type:
+    elif prompts == cache.get('prompts', None) and negative_prompts == cache.get('negative_prompts', None) and clip_skip == cache.get('clip_skip', None) and cache.get('model_type', None) == shared.sd_model_type and steps == cache.get('steps', None):
         p.prompt_embeds = cache.get('prompt_embeds', None)
         p.positive_pooleds = cache.get('positive_pooleds', None)
         p.negative_embeds = cache.get('negative_embeds', None)
@@ -154,36 +154,28 @@ def encode_prompts(pipe, p, prompts: list, negative_prompts: list, steps: int, c
         for i in range(max(len(positive_schedule), len(negative_schedule))):
             positive_prompt = positive_schedule[i % len(positive_schedule)]
             negative_prompt = negative_schedule[i % len(negative_schedule)]
-            if cache.get('model_type', None) != shared.sd_model_type:
-                cache[positive_prompt + negative_prompt] = None
-                results = None
-            elif clip_skip == cache.get('clip_skip', None):
-                results = cache.get(positive_prompt + negative_prompt, None)
-            else:
-                results = None
-
-            if results is None:
-                results = get_weighted_text_embeddings(pipe, positive_prompt, negative_prompt, clip_skip)
-                cache[positive_prompt + negative_prompt] = results
-
-            prompt_embed, positive_pooled, negative_embed, negative_pooled = results
+            prompt_embed, positive_pooled, negative_embed, negative_pooled = get_weighted_text_embeddings(pipe, positive_prompt, negative_prompt, clip_skip)
             if prompt_embed is not None:
                 p.prompt_embeds.append(torch.cat([prompt_embed] * len(prompts), dim=0))
-                cache['prompt_embeds'] = p.prompt_embeds
             if negative_embed is not None:
                 p.negative_embeds.append(torch.cat([negative_embed] * len(negative_prompts), dim=0))
-                cache['negative_embeds'] = p.negative_embeds
             if positive_pooled is not None:
                 p.positive_pooleds.append(torch.cat([positive_pooled] * len(prompts), dim=0))
-                cache['positive_pooleds'] = p.positive_pooleds
             if negative_pooled is not None:
                 p.negative_pooleds.append(torch.cat([negative_pooled] * len(negative_prompts), dim=0))
-                cache['negative_pooleds'] = p.negative_pooleds
 
-        cache['prompts'] = prompts
-        cache['negative_prompts'] = negative_prompts
-        cache['clip_skip'] = clip_skip
-        cache['model_type'] = shared.sd_model_type
+        cache.update({
+            'prompt_embeds': p.prompt_embeds,
+            'negative_embeds': p.negative_embeds,
+            'positive_pooleds': p.positive_pooleds,
+            'negative_pooleds': p.negative_pooleds,
+            'scheduled_prompt': p.scheduled_prompt,
+            'prompts': prompts,
+            'negative_prompts': negative_prompts,
+            'clip_skip': clip_skip,
+            'steps': steps,
+            'model_type': shared.sd_model_type
+        })
         if debug_enabled:
             get_tokens('positive', prompts[0])
             get_tokens('negative', negative_prompts[0])

modules/sd_hijack.py

@@ -175,7 +175,7 @@ class StableDiffusionModelHijack:
         if m.cond_stage_key == "edit":
             sd_hijack_unet.hijack_ddpm_edit()
 
-        if "Model" in shared.opts.ipex_optimize and shared.backend == shared.Backend.ORIGINAL:
+        if "Model" in shared.opts.ipex_optimize and not shared.native:
             try:
                 import intel_extension_for_pytorch as ipex # pylint: disable=import-error, unused-import
                 m.model.eval()
@@ -185,7 +185,7 @@ class StableDiffusionModelHijack:
             except Exception as err:
                 shared.log.warning(f"IPEX Optimize not supported: {err}")
 
-        if "Model" in shared.opts.cuda_compile and shared.opts.cuda_compile_backend != 'none' and shared.backend == shared.Backend.ORIGINAL:
+        if "Model" in shared.opts.cuda_compile and shared.opts.cuda_compile_backend != 'none' and not shared.native:
             try:
                 import logging
                 shared.log.info(f"Compiling pipeline={m.model.__class__.__name__} mode={shared.opts.cuda_compile_backend}")

modules/sd_hijack_hypertile.py

@@ -186,7 +186,7 @@ def context_hypertile_vae(p):
     error_reported = False
     height, width = p.height, p.width
     max_h, max_w = 0, 0
-    vae = getattr(p.sd_model, "vae", None) if shared.backend == shared.Backend.DIFFUSERS else getattr(p.sd_model, "first_stage_model", None)
+    vae = getattr(p.sd_model, "vae", None) if shared.native else getattr(p.sd_model, "first_stage_model", None)
     if height % 8 != 0 or width % 8 != 0:
         log.warning(f'Hypertile VAE disabled: width={width} height={height} are not divisible by 8')
         return nullcontext()
@@ -211,7 +211,7 @@ def context_hypertile_unet(p):
     error_reported = False
     height, width = p.height, p.width
     max_h, max_w = 0, 0
-    unet = getattr(p.sd_model, "unet", None) if shared.backend == shared.Backend.DIFFUSERS else getattr(p.sd_model.model, "diffusion_model", None)
+    unet = getattr(p.sd_model, "unet", None) if shared.native else getattr(p.sd_model.model, "diffusion_model", None)
     if height % 8 != 0 or width % 8 != 0:
         log.warning(f'Hypertile UNet disabled: width={width} height={height} are not divisible by 8')
         return nullcontext()

modules/sd_models.py

@@ -127,7 +127,7 @@ def setup_model():
     list_models()
     sd_hijack_accelerate.hijack_hfhub()
     # sd_hijack_accelerate.hijack_torch_conv()
-    if shared.backend == shared.Backend.ORIGINAL:
+    if not shared.native:
         enable_midas_autodownload()
 
 
@@ -144,19 +144,19 @@ def list_models():
     global checkpoints_list # pylint: disable=global-statement
     checkpoints_list.clear()
     checkpoint_aliases.clear()
-    ext_filter = [".safetensors"] if shared.opts.sd_disable_ckpt or shared.backend == shared.Backend.DIFFUSERS else [".ckpt", ".safetensors"]
+    ext_filter = [".safetensors"] if shared.opts.sd_disable_ckpt or shared.native else [".ckpt", ".safetensors"]
     model_list = list(modelloader.load_models(model_path=model_path, model_url=None, command_path=shared.opts.ckpt_dir, ext_filter=ext_filter, download_name=None, ext_blacklist=[".vae.ckpt", ".vae.safetensors"]))
     for filename in sorted(model_list, key=str.lower):
         checkpoint_info = CheckpointInfo(filename)
         if checkpoint_info.name is not None:
             checkpoint_info.register()
-    if shared.backend == shared.Backend.DIFFUSERS:
+    if shared.native:
         for repo in modelloader.load_diffusers_models(clear=True):
             checkpoint_info = CheckpointInfo(repo['name'], sha=repo['hash'])
             if checkpoint_info.name is not None:
                 checkpoint_info.register()
     if shared.cmd_opts.ckpt is not None:
-        if not os.path.exists(shared.cmd_opts.ckpt) and shared.backend == shared.Backend.ORIGINAL:
+        if not os.path.exists(shared.cmd_opts.ckpt) and not shared.native:
             if shared.cmd_opts.ckpt.lower() != "none":
                 shared.log.warning(f"Requested checkpoint not found: {shared.cmd_opts.ckpt}")
         else:
@@ -414,7 +414,7 @@ def get_checkpoint_state_dict(checkpoint_info: CheckpointInfo, timer):
         checkpoints_loaded.move_to_end(checkpoint_info, last=True)  # FIFO -> LRU cache
         return checkpoints_loaded[checkpoint_info]
     res = read_state_dict(checkpoint_info.filename)
-    if shared.opts.sd_checkpoint_cache > 0 and shared.backend == shared.Backend.ORIGINAL:
+    if shared.opts.sd_checkpoint_cache > 0 and not shared.native:
         # cache newly loaded model
         checkpoints_loaded[checkpoint_info] = res
         # clean up cache if limit is reached
@@ -536,6 +536,7 @@ def change_backend():
     shared.log.warning('Full server restart required to apply all changes')
     unload_model_weights()
     shared.backend = shared.Backend.ORIGINAL if shared.opts.sd_backend == 'original' else shared.Backend.DIFFUSERS
+    shared.native = shared.backend == shared.Backend.DIFFUSERS
     checkpoints_loaded.clear()
     from modules.sd_samplers import list_samplers
     list_samplers(shared.backend)
@@ -564,29 +565,29 @@ def detect_pipeline(f: str, op: str = 'model', warning=True):
                 # elif size < 0: # unknown
                 #    guess = 'Stable Diffusion 2B'
                 elif size >= 5791 and size <= 5799: # 5795
-                    if shared.backend == shared.Backend.ORIGINAL:
+                    if not shared.native:
                         warn(f'Model detected as SD-XL refiner model, but attempting to load using backend=original: {op}={f} size={size} MB')
                     if op == 'model':
                         warn(f'Model detected as SD-XL refiner model, but attempting to load a base model: {op}={f} size={size} MB')
                     guess = 'Stable Diffusion XL Refiner'
                 elif (size >= 6611 and size <= 7220): # 6617, HassakuXL is 6776, monkrenRealisticINT_v10 is 7217
-                    if shared.backend == shared.Backend.ORIGINAL:
+                    if not shared.native:
                         warn(f'Model detected as SD-XL base model, but attempting to load using backend=original: {op}={f} size={size} MB')
                     guess = 'Stable Diffusion XL'
                 elif size >= 3361 and size <= 3369: # 3368
-                    if shared.backend == shared.Backend.ORIGINAL:
+                    if not shared.native:
                         warn(f'Model detected as SD upscale model, but attempting to load using backend=original: {op}={f} size={size} MB')
                     guess = 'Stable Diffusion Upscale'
                 elif size >= 4891 and size <= 4899: # 4897
-                    if shared.backend == shared.Backend.ORIGINAL:
+                    if not shared.native:
                         warn(f'Model detected as SD XL inpaint model, but attempting to load using backend=original: {op}={f} size={size} MB')
                     guess = 'Stable Diffusion XL Inpaint'
                 elif size >= 9791 and size <= 9799: # 9794
-                    if shared.backend == shared.Backend.ORIGINAL:
+                    if not shared.native:
                         warn(f'Model detected as SD XL instruct pix2pix model, but attempting to load using backend=original: {op}={f} size={size} MB')
                     guess = 'Stable Diffusion XL Instruct'
                 elif size > 3138 and size < 3142: #3140
-                    if shared.backend == shared.Backend.ORIGINAL:
+                    if not shared.native:
                         warn(f'Model detected as Segmind Vega model, but attempting to load using backend=original: {op}={f} size={size} MB')
                     guess = 'Stable Diffusion XL'
             # guess by name
@@ -597,25 +598,29 @@ def detect_pipeline(f: str, op: str = 'model', warning=True):
                 guess = 'Latent Consistency Model'
             """
             if 'instaflow' in f.lower():
-                if shared.backend == shared.Backend.ORIGINAL:
+                if not shared.native:
                     warn(f'Model detected as InstaFlow model, but attempting to load using backend=original: {op}={f} size={size} MB')
                 guess = 'InstaFlow'
             if 'segmoe' in f.lower():
-                if shared.backend == shared.Backend.ORIGINAL:
+                if not shared.native:
                     warn(f'Model detected as SegMoE model, but attempting to load using backend=original: {op}={f} size={size} MB')
                 guess = 'SegMoE'
+            if 'hunyuandit' in f.lower():
+                if not shared.native:
+                    warn(f'Model detected as Tenecent HunyuanDiT model, but attempting to load using backend=original: {op}={f} size={size} MB')
+                guess = 'HunyuanDiT'
             if 'pixart-xl' in f.lower():
-                if shared.backend == shared.Backend.ORIGINAL:
+                if not shared.native:
                     warn(f'Model detected as PixArt Alpha model, but attempting to load using backend=original: {op}={f} size={size} MB')
                 guess = 'PixArt-Alpha'
             if 'stable-cascade' in f.lower() or 'stablecascade' in f.lower() or 'wuerstchen3' in f.lower():
-                if shared.backend == shared.Backend.ORIGINAL:
+                if not shared.native:
                     warn(f'Model detected as Stable Cascade model, but attempting to load using backend=original: {op}={f} size={size} MB')
                 if devices.dtype == torch.float16:
                     warn('Stable Cascade does not support Float16')
                 guess = 'Stable Cascade'
             if 'pixart_sigma' in f.lower():
-                if shared.backend == shared.Backend.ORIGINAL:
+                if not shared.native:
                     warn(f'Model detected as PixArt-Sigma model, but attempting to load using backend=original: {op}={f} size={size} MB')
                 guess = 'PixArt-Sigma'
             # switch for specific variant
@@ -885,7 +890,6 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
         timer = Timer()
     logging.getLogger("diffusers").setLevel(logging.ERROR)
     timer.record("diffusers")
-    devices.set_cuda_params()
     diffusers_load_config = {
         "low_cpu_mem_usage": True,
         "torch_dtype": devices.dtype,
@@ -1197,7 +1201,7 @@ def switch_pipe(cls: diffusers.DiffusionPipeline, pipeline: diffusers.DiffusionP
         new_pipe = None
         signature = inspect.signature(cls.__init__, follow_wrapped=True, eval_str=True)
         possible = signature.parameters.keys()
-        if isinstance(pipeline, cls):
+        if isinstance(pipeline, cls) and args == {}:
             return pipeline
         pipe_dict = {}
         components_used = []
@@ -1411,11 +1415,10 @@ def load_model(checkpoint_info=None, already_loaded_state_dict=None, timer=None,
             current_checkpoint_info = model_data.sd_refiner.sd_checkpoint_info
             unload_model_weights(op=op)
 
-    if shared.backend == shared.Backend.ORIGINAL:
+    if not shared.native:
         from modules import sd_hijack_inpainting
         sd_hijack_inpainting.do_inpainting_hijack()
 
-    devices.set_cuda_params()
     if already_loaded_state_dict is not None:
         state_dict = already_loaded_state_dict
     else:
@@ -1468,7 +1471,7 @@ def load_model(checkpoint_info=None, already_loaded_state_dict=None, timer=None,
     else:
         shared.log.debug(f'Model weights loaded: {memory_stats()}')
     timer.record("load")
-    if shared.backend == shared.Backend.ORIGINAL and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
+    if not shared.native and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
         lowvram.setup_for_low_vram(sd_model, shared.cmd_opts.medvram)
     else:
         move_model(sd_model, devices.device)
@@ -1515,7 +1518,7 @@ def reload_model_weights(sd_model=None, info=None, reuse_dict=False, op='model',
         current_checkpoint_info = getattr(sd_model, 'sd_checkpoint_info', None)
         if current_checkpoint_info is not None and checkpoint_info is not None and current_checkpoint_info.filename == checkpoint_info.filename and not force:
             return None
-        if shared.backend == shared.Backend.ORIGINAL and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
+        if not shared.native and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
             lowvram.send_everything_to_cpu()
         else:
             move_model(sd_model, devices.cpu)
@@ -1527,12 +1530,12 @@ def reload_model_weights(sd_model=None, info=None, reuse_dict=False, op='model',
             sd_model = None
     timer = Timer()
     # TODO implement caching after diffusers implement state_dict loading
-    state_dict = get_checkpoint_state_dict(checkpoint_info, timer) if shared.backend == shared.Backend.ORIGINAL else None
+    state_dict = get_checkpoint_state_dict(checkpoint_info, timer) if not shared.native else None
     checkpoint_config = sd_models_config.find_checkpoint_config(state_dict, checkpoint_info)
     timer.record("config")
     if sd_model is None or checkpoint_config != getattr(sd_model, 'used_config', None):
         sd_model = None
-        if shared.backend == shared.Backend.ORIGINAL:
+        if not shared.native:
             load_model(checkpoint_info, already_loaded_state_dict=state_dict, timer=timer, op=op)
             model_data.sd_dict = shared.opts.sd_model_dict
         else:
@@ -1599,7 +1602,7 @@ def unload_model_weights(op='model'):
         shared.compiled_model_state.partitioned_modules.clear()
     if op == 'model' or op == 'dict':
         if model_data.sd_model:
-            if shared.backend == shared.Backend.ORIGINAL:
+            if not shared.native:
                 from modules import sd_hijack
                 move_model(model_data.sd_model, devices.cpu)
                 sd_hijack.model_hijack.undo_hijack(model_data.sd_model)
@@ -1611,7 +1614,7 @@ def unload_model_weights(op='model'):
             shared.log.debug(f'Unload weights {op}: {memory_stats()}')
     elif op == 'refiner':
         if model_data.sd_refiner:
-            if shared.backend == shared.Backend.ORIGINAL:
+            if not shared.native:
                 from modules import sd_hijack
                 move_model(model_data.sd_refiner, devices.cpu)
                 sd_hijack.model_hijack.undo_hijack(model_data.sd_refiner)

modules/sd_samplers.py

@@ -20,7 +20,7 @@ def list_samplers(backend_name = shared.backend):
     global samplers # pylint: disable=global-statement
     global samplers_for_img2img # pylint: disable=global-statement
     global samplers_map # pylint: disable=global-statement
-    if backend_name == shared.Backend.ORIGINAL:
+    if not shared.native:
         from modules import sd_samplers_compvis, sd_samplers_kdiffusion
         all_samplers = [*sd_samplers_compvis.samplers_data_compvis, *sd_samplers_kdiffusion.samplers_data_k_diffusion]
     else:
@@ -57,14 +57,14 @@ def create_sampler(name, model):
     if config is None or config.constructor is None:
         # shared.log.warning(f'Sampler: sampler="{name}" not found')
         return None
-    if shared.backend == shared.Backend.ORIGINAL:
+    if not shared.native:
         sampler = config.constructor(model)
         sampler.config = config
         sampler.name = name
         sampler.initialize(p=None)
         shared.log.debug(f'Sampler: sampler="{name}" config={config.options}')
         return sampler
-    elif shared.backend == shared.Backend.DIFFUSERS:
+    elif shared.native:
         sampler = config.constructor(model)
         if not hasattr(model, 'scheduler_config'):
             model.scheduler_config = sampler.sampler.config.copy()

modules/sd_samplers_common.py

@@ -49,7 +49,7 @@ def single_sample_to_image(sample, approximation=None):
         if len(sample.shape) == 4 and sample.shape[0]: # likely animatediff latent
             sample = sample.permute(1, 0, 2, 3)[0]
 
-        if shared.backend == shared.Backend.DIFFUSERS: # [-x,x] to [-5,5]
+        if shared.native: # [-x,x] to [-5,5]
             sample_max = torch.max(sample)
             if sample_max > 5:
                 sample = sample * (5 / sample_max)

modules/sd_vae.py

@@ -54,7 +54,7 @@ def refresh_vae_list():
     vae_path = shared.opts.vae_dir
     vae_dict.clear()
     vae_paths = []
-    if shared.backend == shared.Backend.ORIGINAL:
+    if not shared.native:
         if sd_models.model_path is not None and os.path.isdir(sd_models.model_path):
             vae_paths += [
                 os.path.join(sd_models.model_path, 'VAE', '**/*.vae.ckpt'),
@@ -73,7 +73,7 @@ def refresh_vae_list():
                 os.path.join(shared.opts.vae_dir, '**/*.pt'),
                 os.path.join(shared.opts.vae_dir, '**/*.safetensors'),
             ]
-    elif shared.backend == shared.Backend.DIFFUSERS:
+    elif shared.native:
         if sd_models.model_path is not None and os.path.isdir(sd_models.model_path):
             vae_paths += [os.path.join(sd_models.model_path, 'VAE', '**/*.vae.safetensors')]
         if shared.opts.ckpt_dir is not None and os.path.isdir(shared.opts.ckpt_dir):
@@ -92,7 +92,7 @@ def refresh_vae_list():
         name = get_filename(filepath)
         if name == 'VAE':
             continue
-        if shared.backend == shared.Backend.ORIGINAL:
+        if not shared.native:
             vae_dict[name] = filepath
         else:
             if filepath.endswith(".json"):
@@ -243,12 +243,12 @@ def reload_vae_weights(sd_model=None, vae_file=unspecified):
         vae_source = "function-argument"
     if loaded_vae_file == vae_file:
         return None
-    if shared.backend == shared.Backend.ORIGINAL and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
+    if not shared.native and (shared.cmd_opts.lowvram or shared.cmd_opts.medvram):
         lowvram.send_everything_to_cpu()
     # else:
     #    sd_models.move_model(sd_model, devices.cpu)
 
-    if shared.backend == shared.Backend.ORIGINAL:
+    if not shared.native:
         sd_hijack.model_hijack.undo_hijack(sd_model)
         if shared.cmd_opts.rollback_vae and devices.dtype_vae == torch.bfloat16:
             devices.dtype_vae = torch.float16

modules/shared.py

@@ -206,7 +206,7 @@ if cmd_opts.backend is not None: # override with args
 if cmd_opts.use_openvino: # override for openvino
     backend = Backend.DIFFUSERS
     from modules.intel.openvino import get_device_list as get_openvino_device_list # pylint: disable=ungrouped-imports
-
+native = backend == Backend.DIFFUSERS
 
 class OptionInfo:
     def __init__(self, default=None, label="", component=None, component_args=None, onchange=None, section=None, refresh=None, folder=None, submit=None, comment_before='', comment_after=''):
@@ -340,11 +340,11 @@ def temp_disable_extensions():
         for ext in disable_safe:
             if ext.lower() not in opts.disabled_extensions:
                 disabled.append(ext)
-    if backend == Backend.DIFFUSERS:
+    if native:
         for ext in disable_diffusers:
             if ext.lower() not in opts.disabled_extensions:
                 disabled.append(ext)
-    if backend == Backend.ORIGINAL:
+    if not native:
         for ext in disable_original:
             if ext.lower() not in opts.disabled_extensions:
                 disabled.append(ext)
@@ -366,13 +366,13 @@ if not (cmd_opts.lowvram or cmd_opts.medvram):
 
 
 if devices.backend == "directml": # Force BMM for DirectML instead of SDP
-    cross_attention_optimization_default = "Dynamic Attention BMM" if backend == Backend.DIFFUSERS else "Sub-quadratic"
-elif backend == Backend.DIFFUSERS and (cmd_opts.lowvram or cmd_opts.medvram):
+    cross_attention_optimization_default = "Dynamic Attention BMM" if native else "Sub-quadratic"
+elif native and (cmd_opts.lowvram or cmd_opts.medvram):
     cross_attention_optimization_default = "Dynamic Attention SDP"
 elif devices.backend == "cpu":
-    cross_attention_optimization_default = "Scaled-Dot-Product" if backend == Backend.DIFFUSERS else "Doggettx's"
+    cross_attention_optimization_default = "Scaled-Dot-Product" if native else "Doggettx's"
 elif devices.backend == "mps":
-    cross_attention_optimization_default = "Scaled-Dot-Product" if backend == Backend.DIFFUSERS else "Doggettx's"
+    cross_attention_optimization_default = "Scaled-Dot-Product" if native else "Doggettx's"
 else: # cuda, rocm, ipex
     cross_attention_optimization_default ="Scaled-Dot-Product"
 
@@ -392,12 +392,12 @@ options_templates.update(options_section(('sd', "Execution & Models"), {
     "sd_unet": OptionInfo("None", "UNET model", gr.Dropdown, lambda: {"choices": shared_items.sd_unet_items()}, refresh=shared_items.refresh_unet_list),
     "sd_checkpoint_autoload": OptionInfo(True, "Model autoload on start"),
     "sd_model_dict": OptionInfo('None', "Use separate base dict", gr.Dropdown, lambda: {"choices": ['None'] + list_checkpoint_tiles()}, refresh=refresh_checkpoints),
-    "stream_load": OptionInfo(False, "Load models using stream loading method", gr.Checkbox, {"visible": backend == Backend.ORIGINAL }),
+    "stream_load": OptionInfo(False, "Load models using stream loading method", gr.Checkbox, {"visible": not native }),
     "model_reuse_dict": OptionInfo(False, "Reuse loaded model dictionary", gr.Checkbox, {"visible": False}),
     "prompt_attention": OptionInfo("Full parser", "Prompt attention parser", gr.Radio, {"choices": ["Full parser", "Compel parser", "A1111 parser", "Fixed attention"] }),
-    "prompt_mean_norm": OptionInfo(True, "Prompt attention normalization", gr.Checkbox, {"visible": backend == Backend.ORIGINAL }),
-    "comma_padding_backtrack": OptionInfo(20, "Prompt padding", gr.Slider, {"minimum": 0, "maximum": 74, "step": 1, "visible": backend == Backend.ORIGINAL }),
-    "sd_checkpoint_cache": OptionInfo(0, "Cached models", gr.Slider, {"minimum": 0, "maximum": 10, "step": 1, "visible": backend == Backend.ORIGINAL }),
+    "prompt_mean_norm": OptionInfo(True, "Prompt attention normalization", gr.Checkbox, {"visible": not native }),
+    "comma_padding_backtrack": OptionInfo(20, "Prompt padding", gr.Slider, {"minimum": 0, "maximum": 74, "step": 1, "visible": not native }),
+    "sd_checkpoint_cache": OptionInfo(0, "Cached models", gr.Slider, {"minimum": 0, "maximum": 10, "step": 1, "visible": not native }),
     "sd_vae_checkpoint_cache": OptionInfo(0, "Cached VAEs", gr.Slider, {"minimum": 0, "maximum": 10, "step": 1, "visible": False}),
     "sd_disable_ckpt": OptionInfo(False, "Disallow models in ckpt format", gr.Checkbox, {"visible": False}),
 }))
@@ -406,6 +406,9 @@ options_templates.update(options_section(('cuda', "Compute Settings"), {
     "math_sep": OptionInfo("<h2>Execution precision</h2>", "", gr.HTML),
     "precision": OptionInfo("Autocast", "Precision type", gr.Radio, {"choices": ["Autocast", "Full"]}),
     "cuda_dtype": OptionInfo("FP32" if sys.platform == "darwin" or cmd_opts.use_openvino else "BF16" if devices.backend == "ipex" else "FP16", "Device precision type", gr.Radio, {"choices": ["FP32", "FP16", "BF16"]}),
+    "cudnn_deterministic": OptionInfo(False, "Use deterministic mode"),
+
+    "model_sep": OptionInfo("<h2>Model options</h2>", "", gr.HTML),
     "no_half": OptionInfo(False if not cmd_opts.use_openvino else True, "Full precision for model (--no-half)", None, None, None),
     "no_half_vae": OptionInfo(False if not cmd_opts.use_openvino else True, "Full precision for VAE (--no-half-vae)"),
     "upcast_sampling": OptionInfo(False if sys.platform != "darwin" else True, "Upcast sampling"),
@@ -415,20 +418,19 @@ options_templates.update(options_section(('cuda', "Compute Settings"), {
     "nan_skip": OptionInfo(False, "Skip Generation if NaN found in latents", gr.Checkbox, {"visible": True}),
     "rollback_vae": OptionInfo(False, "Attempt VAE roll back for NaN values"),
 
-    "cross_attention_sep": OptionInfo("<h2>Attention</h2>", "", gr.HTML),
-    "cross_attention_optimization": OptionInfo(cross_attention_optimization_default, "Attention optimization method", gr.Radio, lambda: {"choices": shared_items.list_crossattention(diffusers=backend == Backend.DIFFUSERS) }),
+    "cross_attention_sep": OptionInfo("<h2>Cross Attention</h2>", "", gr.HTML),
+    "cross_attention_optimization": OptionInfo(cross_attention_optimization_default, "Attention optimization method", gr.Radio, lambda: {"choices": shared_items.list_crossattention(native) }),
     "sdp_options": OptionInfo(sdp_options_default, "SDP options", gr.CheckboxGroup, {"choices": ['Flash attention', 'Memory attention', 'Math attention'] }),
     "xformers_options": OptionInfo(['Flash attention'], "xFormers options", gr.CheckboxGroup, {"choices": ['Flash attention'] }),
-    "dynamic_attention_slice_rate": OptionInfo(4, "Dynamic Attention slicing rate in GB", gr.Slider, {"minimum": 0.1, "maximum": 16, "step": 0.1, "visible": backend == Backend.DIFFUSERS}),
-    "sub_quad_sep": OptionInfo("<h3>Sub-quadratic options</h3>", "", gr.HTML, {"visible": backend == Backend.ORIGINAL}),
-    "sub_quad_q_chunk_size": OptionInfo(512, "Attention query chunk size", gr.Slider, {"minimum": 16, "maximum": 8192, "step": 8, "visible": backend == Backend.ORIGINAL}),
-    "sub_quad_kv_chunk_size": OptionInfo(512, "Attention kv chunk size", gr.Slider, {"minimum": 0, "maximum": 8192, "step": 8, "visible": backend == Backend.ORIGINAL}),
-    "sub_quad_chunk_threshold": OptionInfo(80, "Attention chunking threshold", gr.Slider, {"minimum": 0, "maximum": 100, "step": 1, "visible": backend == Backend.ORIGINAL}),
+    "dynamic_attention_slice_rate": OptionInfo(4, "Dynamic Attention slicing rate in GB", gr.Slider, {"minimum": 0.1, "maximum": 16, "step": 0.1, "visible": native}),
+    "sub_quad_sep": OptionInfo("<h3>Sub-quadratic options</h3>", "", gr.HTML, {"visible": not native}),
+    "sub_quad_q_chunk_size": OptionInfo(512, "Attention query chunk size", gr.Slider, {"minimum": 16, "maximum": 8192, "step": 8, "visible": not native}),
+    "sub_quad_kv_chunk_size": OptionInfo(512, "Attention kv chunk size", gr.Slider, {"minimum": 0, "maximum": 8192, "step": 8, "visible": not native}),
+    "sub_quad_chunk_threshold": OptionInfo(80, "Attention chunking threshold", gr.Slider, {"minimum": 0, "maximum": 100, "step": 1, "visible": not native}),
 
-    "other_sep": OptionInfo("<h2>Execution precision</h2>", "", gr.HTML),
+    "other_sep": OptionInfo("<h2>Execution options</h2>", "", gr.HTML),
     "opt_channelslast": OptionInfo(False, "Use channels last "),
     "cudnn_benchmark": OptionInfo(False, "Full-depth cuDNN benchmark feature"),
-    "cudnn_deterministic": OptionInfo(False, "Use deterministic options for cuDNN"),
     "diffusers_fuse_projections": OptionInfo(False, "Fused projections"),
     "torch_gc_threshold": OptionInfo(80, "Torch memory threshold for GC", gr.Slider, {"minimum": 0, "maximum": 100, "step": 1}),
     "torch_malloc": OptionInfo("native", "Torch memory allocator", gr.Radio, {"choices": ['native', 'cudaMallocAsync'] }),
@@ -445,7 +447,7 @@ options_templates.update(options_section(('cuda', "Compute Settings"), {
     "deep_cache_interval": OptionInfo(3, "DeepCache cache interval", gr.Slider, {"minimum": 1, "maximum": 10, "step": 1}),
 
     "nncf_sep": OptionInfo("<h2>Model Compress</h2>", "", gr.HTML),
-    "nncf_compress_weights": OptionInfo([], "Compress Model weights with NNCF", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder"], "visible": backend == Backend.DIFFUSERS}),
+    "nncf_compress_weights": OptionInfo([], "Compress Model weights with NNCF", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder"], "visible": native}),
 
     "ipex_sep": OptionInfo("<h2>IPEX</h2>", "", gr.HTML, {"visible": devices.backend == "ipex"}),
     "ipex_optimize": OptionInfo([], "IPEX Optimize for Intel GPUs", gr.CheckboxGroup, {"choices": ["Model", "VAE", "Text Encoder", "Upscaler"], "visible": devices.backend == "ipex"}),
@@ -668,7 +670,7 @@ options_templates.update(options_section(('ui', "User Interface Options"), {
     "keyedit_precision_attention": OptionInfo(0.1, "Ctrl+up/down precision when editing (attention:1.1)", gr.Slider, {"minimum": 0.01, "maximum": 0.2, "step": 0.001, "visible": False}),
     "keyedit_precision_extra": OptionInfo(0.05, "Ctrl+up/down precision when editing <extra networks:0.9>", gr.Slider, {"minimum": 0.01, "maximum": 0.2, "step": 0.001, "visible": False}),
     "keyedit_delimiters": OptionInfo(r".,\/!?%^*;:{}=`~()", "Ctrl+up/down word delimiters", gr.Textbox, { "visible": False }),
-    "quicksettings_list": OptionInfo(["sd_model_checkpoint"] if backend == Backend.ORIGINAL else ["sd_model_checkpoint", "sd_model_refiner"], "Quicksettings list", gr.Dropdown, lambda: {"multiselect":True, "choices": list(opts.data_labels.keys())}),
+    "quicksettings_list": OptionInfo(["sd_model_checkpoint"], "Quicksettings list", gr.Dropdown, lambda: {"multiselect":True, "choices": list(opts.data_labels.keys())}),
     "ui_scripts_reorder": OptionInfo("", "UI scripts order", gr.Textbox, { "visible": False }),
 }))
 
@@ -752,10 +754,10 @@ options_templates.update(options_section(('postprocessing', "Postprocessing"), {
     "facehires_iou": OptionInfo(0.5, "Max face overlap", gr.Slider, {"minimum": 0, "maximum": 1.0, "step": 0.05}),
     "facehires_min_size": OptionInfo(0, "Min face size", gr.Slider, {"minimum": 0, "maximum": 1024, "step": 1}),
     "facehires_max_size": OptionInfo(0, "Max face size", gr.Slider, {"minimum": 0, "maximum": 1024, "step": 1}),
-    "facehires_padding": OptionInfo(10, "Face padding", gr.Slider, {"minimum": 0, "maximum": 100, "step": 1}),
-    "face_restoration_unload": OptionInfo(False, "Move model to CPU when complete"),
+    "facehires_padding": OptionInfo(20, "Face padding", gr.Slider, {"minimum": 0, "maximum": 100, "step": 1}),
     "facehires_strength": OptionInfo(0.0, "Face restore strength", gr.Slider, {"minimum": 0, "maximum": 1, "step": 0.01}),
     "code_former_weight": OptionInfo(0.2, "CodeFormer weight parameter", gr.Slider, {"minimum": 0, "maximum": 1, "step": 0.01}),
+    "face_restoration_unload": OptionInfo(False, "Move model to CPU when complete"),
 
     "postprocessing_sep_upscalers": OptionInfo("<h2>Upscaling</h2>", "", gr.HTML),
     "upscaler_unload": OptionInfo(False, "Unload upscaler after processing"),
@@ -815,7 +817,7 @@ options_templates.update(options_section(('extra_networks', "Extra Networks"), {
     "extra_network_reference": OptionInfo(False, "Use reference values when available", gr.Checkbox),
     "extra_network_skip_indexing": OptionInfo(False, "Build info on first access", gr.Checkbox),
     "extra_networks_default_multiplier": OptionInfo(1.0, "Default multiplier for extra networks", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.01}),
-    "diffusers_convert_embed": OptionInfo(False, "Auto-convert SD 1.5 embeddings to SDXL ", gr.Checkbox, {"visible": backend==Backend.DIFFUSERS}),
+    "diffusers_convert_embed": OptionInfo(False, "Auto-convert SD 1.5 embeddings to SDXL ", gr.Checkbox, {"visible": native}),
     "extra_networks_sep3": OptionInfo("<h2>Extra networks settings</h2>", "", gr.HTML),
     "extra_networks_styles": OptionInfo(True, "Show built-in styles"),
     "lora_preferred_name": OptionInfo("filename", "LoRA preferred name", gr.Radio, {"choices": ["filename", "alias"]}),

modules/shared_items.py

@@ -90,6 +90,8 @@ def get_pipelines():
         pipelines['Stable Cascade'] = getattr(diffusers, 'StableCascadeCombinedPipeline', None)
     if hasattr(diffusers, 'PixArtSigmaPipeline'):
         pipelines['PixArt-Sigma'] = getattr(diffusers, 'PixArtSigmaPipeline', None)
+    if hasattr(diffusers, 'HunyuanDiTPipeline'):
+        pipelines['HunyuanDiT'] = getattr(diffusers, 'HunyuanDiTPipeline', None)
 
     for k, v in pipelines.items():
         if k != 'Autodetect' and v is None:

modules/styles.py

@@ -6,7 +6,7 @@ import csv
 import json
 import time
 import random
-from modules import files_cache, shared
+from modules import files_cache, shared, infotext
 
 
 class Style():
@@ -132,12 +132,11 @@ def apply_styles_to_extra(p, style: Style):
     name_exclude = [
         'size',
     ]
-    from modules.generation_parameters_copypaste import parse_generation_parameters
     reference_style = get_reference_style()
-    extra = parse_generation_parameters(reference_style) if shared.opts.extra_network_reference else {}
+    extra = infotext.parse(reference_style) if shared.opts.extra_network_reference else {}
 
     style_extra = apply_wildcards_to_prompt(style.extra, [style.wildcards], silent=True)
-    extra.update(parse_generation_parameters(style_extra))
+    extra.update(infotext.parse(style_extra))
     extra.pop('Prompt', None)
     extra.pop('Negative prompt', None)
     fields = []

modules/textual_inversion/textual_inversion.py

@@ -27,7 +27,7 @@ def list_textual_inversion_templates():
 
 
 def list_embeddings(*dirs):
-    is_ext = extension_filter(['.SAFETENSORS', '.PT' ] + ( ['.PNG', '.WEBP', '.JXL', '.AVIF', '.BIN' ] if shared.backend != shared.Backend.DIFFUSERS else [] ))
+    is_ext = extension_filter(['.SAFETENSORS', '.PT' ] + ( ['.PNG', '.WEBP', '.JXL', '.AVIF', '.BIN' ] if not shared.native else [] ))
     is_not_preview = lambda fp: not next(iter(os.path.splitext(fp))).upper().endswith('.PREVIEW') # pylint: disable=unnecessary-lambda-assignment
     return list(filter(lambda fp: is_ext(fp) and is_not_preview(fp) and os.stat(fp).st_size > 0, directory_files(*dirs)))
 
@@ -138,7 +138,7 @@ class EmbeddingDatabase:
         return embedding
 
     def get_expected_shape(self):
-        if shared.backend == shared.Backend.DIFFUSERS:
+        if shared.native:
             return 0
         if not shared.sd_loaded:
             shared.log.error('Model not loaded')
@@ -302,7 +302,7 @@ class EmbeddingDatabase:
         else:
             raise RuntimeError(f"Couldn't identify {filename} as textual inversion embedding")
 
-        if shared.backend == shared.Backend.DIFFUSERS:
+        if shared.native:
             return emb
 
         vec = emb.detach().to(devices.device, dtype=torch.float32)
@@ -326,7 +326,7 @@ class EmbeddingDatabase:
         if not os.path.isdir(embdir.path):
             return
         file_paths = list_embeddings(embdir.path)
-        if shared.backend == shared.Backend.DIFFUSERS:
+        if shared.native:
             self.load_diffusers_embedding(file_paths)
         else:
             for file_path in file_paths:

modules/ui.py

@@ -139,7 +139,7 @@ def create_ui(startup_timer = None):
     modules.scripts.scripts_current = None
 
     with gr.Blocks(analytics_enabled=False) as control_interface:
-        if shared.backend == shared.Backend.DIFFUSERS:
+        if shared.native:
             from modules import ui_control
             ui_control.create_ui()
             timer.startup.record("ui-control")

modules/ui_common.py

@@ -6,7 +6,7 @@ import platform
 import subprocess
 from functools import reduce
 import gradio as gr
-from modules import call_queue, shared, prompt_parser, ui_sections, ui_symbols, ui_components, generation_parameters_copypaste, images, scripts, script_callbacks
+from modules import call_queue, shared, prompt_parser, ui_sections, ui_symbols, ui_components, generation_parameters_copypaste, images, scripts, script_callbacks, infotext
 
 
 folder_symbol = ui_symbols.folder
@@ -23,8 +23,8 @@ def update_generation_info(generation_info, html_info, img_index):
         generation_info = json.loads(generation_info)
         if img_index < 0 or img_index >= len(generation_info["infotexts"]):
             return html_info, generation_info
-        infotext = generation_info["infotexts"][img_index]
-        html_info_formatted = infotext_to_html(infotext)
+        info = generation_info["infotexts"][img_index]
+        html_info_formatted = infotext_to_html(info)
         return html_info, html_info_formatted
     except Exception:
         pass
@@ -37,7 +37,7 @@ def plaintext_to_html(text):
 
 
 def infotext_to_html(text):
-    res = generation_parameters_copypaste.parse_generation_parameters(text)
+    res = infotext.parse(text)
     prompt = res.get('Prompt', '')
     negative = res.get('Negative prompt', '')
     res.pop('Prompt', None)
@@ -169,7 +169,7 @@ def save_files(js_data, files, html_info, index):
             if (js_data is None or len(js_data) == 0) and image is not None and image.info is not None:
                 info = image.info.pop('parameters', None) or image.info.pop('UserComment', None)
                 geninfo, _ = images.read_info_from_image(image)
-                items = generation_parameters_copypaste.parse_generation_parameters(geninfo)
+                items = infotext.parse(geninfo)
                 p = PObject(items)
             fullfn, txt_fullfn = images.save_image(image, shared.opts.outdir_save, "", seed=p.all_seeds[i], prompt=p.all_prompts[i], info=info, extension=shared.opts.samples_format, grid=is_grid, p=p)
             if fullfn is None:
@@ -262,7 +262,7 @@ def create_output_panel(tabname, preview=True, prompt=None, height=None):
                     clip_files.click(fn=None, _js='clip_gallery_urls', inputs=[result_gallery], outputs=[])
                 save = gr.Button('Save', elem_id=f'save_{tabname}')
                 delete = gr.Button('Delete', elem_id=f'delete_{tabname}')
-                if shared.backend == shared.Backend.ORIGINAL:
+                if not shared.native:
                     buttons = generation_parameters_copypaste.create_buttons(["img2img", "inpaint", "extras"])
                 else:
                     buttons = generation_parameters_copypaste.create_buttons(["txt2img", "img2img", "control", "extras"])
@@ -389,7 +389,7 @@ def update_token_counter(text, steps):
         return f"<span class='gr-box gr-text-input'>{token_count}/{max_length}</span>"
     from modules import extra_networks
     prompt, _ = extra_networks.parse_prompt(text)
-    if shared.backend == shared.Backend.ORIGINAL:
+    if not shared.native:
         from modules import sd_hijack
         try:
             _, prompt_flat_list, _ = prompt_parser.get_multicond_prompt_list([text])
@@ -399,7 +399,7 @@ def update_token_counter(text, steps):
         flat_prompts = reduce(lambda list1, list2: list1+list2, prompt_schedules)
         prompts = [prompt_text for _step, prompt_text in flat_prompts]
         token_count, max_length = max([sd_hijack.model_hijack.get_prompt_lengths(prompt) for prompt in prompts], key=lambda args: args[0])
-    elif shared.backend == shared.Backend.DIFFUSERS:
+    elif shared.native:
         if shared.sd_loaded and hasattr(shared.sd_model, 'tokenizer') and shared.sd_model.tokenizer is not None:
             has_bos_token = shared.sd_model.tokenizer.bos_token_id is not None
             has_eos_token = shared.sd_model.tokenizer.eos_token_id is not None

modules/ui_control.py

@@ -67,7 +67,7 @@ def generate_click(job_id: str, active_tab: str, *args):
 def create_ui(_blocks: gr.Blocks=None):
     helpers.initialize()
 
-    if shared.backend == shared.Backend.ORIGINAL:
+    if not shared.native:
         with gr.Blocks(analytics_enabled = False) as control_ui:
             pass
         return [(control_ui, 'Control', 'control')]
@@ -533,11 +533,27 @@ def create_ui(_blocks: gr.Blocks=None):
                 (negative, "Negative prompt"),
                 # input
                 (denoising_strength, "Denoising strength"),
-                # resize
+                # size basic
                 (width_before, "Size-1"),
                 (height_before, "Size-2"),
                 (resize_mode_before, "Resize mode"),
                 (scale_by_before, "Resize scale"),
+                # size control
+                (width_before, "Size before-1"),
+                (height_before, "Size before-2"),
+                (resize_mode_before, "Size mode before"),
+                (scale_by_before, "Size scale before"),
+                (resize_name_before, "Size name before"),
+                (width_after, "Size after-1"),
+                (height_after, "Size after-2"),
+                (resize_mode_after, "Size mode after"),
+                (scale_by_after, "Size scale after"),
+                (resize_name_after, "Size name after"),
+                (width_mask, "Size mask-1"),
+                (height_mask, "Size mask-2"),
+                (resize_mode_mask, "Size mode mask"),
+                (scale_by_mask, "Size scale mask"),
+                (resize_name_mask, "Size name mask"),
                 # sampler
                 (sampler_index, "Sampler"),
                 (steps, "Steps"),

modules/ui_extra_networks.py

@@ -16,7 +16,7 @@ from collections import OrderedDict
 import gradio as gr
 from PIL import Image
 from starlette.responses import FileResponse, JSONResponse
-from modules import paths, shared, scripts, files_cache, errors
+from modules import paths, shared, scripts, files_cache, errors, infotext
 from modules.ui_components import ToolButton
 import modules.ui_symbols as symbols
 
@@ -224,7 +224,7 @@ class ExtraNetworksPage:
                 tgt = tgt.path
                 if os.path.join(paths.models_path, 'Reference') in tgt:
                     subdirs['Reference'] = 1
-                if shared.backend == shared.Backend.DIFFUSERS and shared.opts.diffusers_dir in tgt:
+                if shared.native and shared.opts.diffusers_dir in tgt:
                     subdirs[os.path.basename(shared.opts.diffusers_dir)] = 1
                 if 'models--' in tgt:
                     continue
@@ -877,28 +877,26 @@ def create_ui(container, button_parent, tabname, skip_indexing = False):
         return ui_refresh_click(title)
 
     def ui_save_click():
-        from modules import generation_parameters_copypaste
         filename = os.path.join(paths.data_path, "params.txt")
         if os.path.exists(filename):
             with open(filename, "r", encoding="utf8") as file:
                 prompt = file.read()
         else:
             prompt = ''
-        params = generation_parameters_copypaste.parse_generation_parameters(prompt)
+        params = infotext.parse(prompt)
         res = show_details(text=None, img=None, desc=None, info=None, meta=None, parameters=None, description=None, prompt=None, negative=None, wildcards=None, params=params)
         return res
 
     def ui_quicksave_click(name):
         if name is None:
             return
-        from modules import generation_parameters_copypaste
         fn = os.path.join(paths.data_path, "params.txt")
         if os.path.exists(fn):
             with open(fn, "r", encoding="utf8") as file:
                 prompt = file.read()
         else:
             prompt = ''
-        params = generation_parameters_copypaste.parse_generation_parameters(prompt)
+        params = infotext.parse(prompt)
         fn = os.path.join(shared.opts.styles_dir, os.path.splitext(name)[0] + '.json')
         prompt = params.get('Prompt', '')
         item = {

modules/ui_extra_networks_checkpoints.py

@@ -16,7 +16,7 @@ class ExtraNetworksPageCheckpoints(ui_extra_networks.ExtraNetworksPage):
 
     def list_reference(self): # pylint: disable=inconsistent-return-statements
         for k, v in shared.reference_models.items():
-            if shared.backend != shared.Backend.DIFFUSERS:
+            if not shared.native:
                 if not v.get('original', False):
                     continue
                 url = v.get('alt', None) or v['path']
@@ -82,7 +82,7 @@ class ExtraNetworksPageCheckpoints(ui_extra_networks.ExtraNetworksPage):
         return items
 
     def allowed_directories_for_previews(self):
-        if shared.backend == shared.Backend.DIFFUSERS:
+        if shared.native:
             return [v for v in [shared.opts.ckpt_dir, shared.opts.diffusers_dir, reference_dir] if v is not None]
         else:
             return [v for v in [shared.opts.ckpt_dir, reference_dir, sd_models.model_path] if v is not None]

modules/ui_extra_networks_textual_inversion.py

@@ -13,7 +13,7 @@ class ExtraNetworksPageTextualInversion(ui_extra_networks.ExtraNetworksPage):
     def refresh(self):
         if sd_models.model_data.sd_model is None:
             return
-        if shared.backend == shared.Backend.ORIGINAL:
+        if not shared.native:
             sd_hijack.model_hijack.embedding_db.load_textual_inversion_embeddings(force_reload=True)
         elif hasattr(sd_models.model_data.sd_model, 'embedding_db'):
             sd_models.model_data.sd_model.embedding_db.load_textual_inversion_embeddings(force_reload=True)
@@ -48,7 +48,7 @@ class ExtraNetworksPageTextualInversion(ui_extra_networks.ExtraNetworksPage):
                 for embedding_path
                 in candidates
             ]
-        elif shared.backend == shared.Backend.ORIGINAL:
+        elif not shared.native:
             self.embeddings = list(sd_hijack.model_hijack.embedding_db.word_embeddings.values())
         elif hasattr(sd_models.model_data.sd_model, 'embedding_db'):
             self.embeddings = list(sd_models.model_data.sd_model.embedding_db.word_embeddings.values())

modules/ui_img2img.py

@@ -141,7 +141,7 @@ def create_ui():
                         with gr.Row():
                             inpainting_mask_invert = gr.Radio(label='Mode', choices=['masked', 'invert'], value='masked', type="index", elem_id="img2img_mask_mode")
                             inpaint_full_res = gr.Radio(label="Inpaint area", choices=["full", "masked"], type="index", value="full", elem_id="img2img_inpaint_full_res")
-                            inpainting_fill = gr.Radio(label='Masked content', choices=['fill', 'original', 'noise', 'nothing'], value='original', type="index", elem_id="img2img_inpainting_fill", visible=shared.backend == shared.Backend.ORIGINAL)
+                            inpainting_fill = gr.Radio(label='Masked content', choices=['fill', 'original', 'noise', 'nothing'], value='original', type="index", elem_id="img2img_inpainting_fill", visible=not shared.native)
 
                         def select_img2img_tab(tab):
                             return gr.update(visible=tab in [2, 3, 4]), gr.update(visible=tab == 3)

modules/ui_sections.py

@@ -166,18 +166,18 @@ def create_advanced_inputs(tab, base=True):
                 cfg_scale, cfg_end = None, None
             with gr.Row():
                 image_cfg_scale = gr.Slider(minimum=0.0, maximum=30.0, step=0.1, label='Secondary guidance', value=6.0, elem_id=f"{tab}_image_cfg_scale")
-                diffusers_guidance_rescale = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label='Rescale guidance', value=0.7, elem_id=f"{tab}_image_cfg_rescale", visible=shared.backend == shared.Backend.DIFFUSERS)
+                diffusers_guidance_rescale = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label='Rescale guidance', value=0.7, elem_id=f"{tab}_image_cfg_rescale", visible=shared.native)
             with gr.Row():
-                diffusers_pag_scale = gr.Slider(minimum=0.0, maximum=30.0, step=0.05, label='Attention guidance', value=0.0, elem_id=f"{tab}_pag_scale", visible=shared.backend == shared.Backend.DIFFUSERS)
-                diffusers_pag_adaptive = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label='Adaptive scaling', value=0.5, elem_id=f"{tab}_pag_adaptive", visible=shared.backend == shared.Backend.DIFFUSERS)
+                diffusers_pag_scale = gr.Slider(minimum=0.0, maximum=30.0, step=0.05, label='Attention guidance', value=0.0, elem_id=f"{tab}_pag_scale", visible=shared.native)
+                diffusers_pag_adaptive = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label='Adaptive scaling', value=0.5, elem_id=f"{tab}_pag_adaptive", visible=shared.native)
             with gr.Row():
                 clip_skip = gr.Slider(label='CLIP skip', value=1, minimum=0, maximum=12, step=0.1, elem_id=f"{tab}_clip_skip", interactive=True)
     return cfg_scale, clip_skip, image_cfg_scale, diffusers_guidance_rescale, diffusers_pag_scale, diffusers_pag_adaptive, cfg_end
 
 
 def create_correction_inputs(tab):
-    with gr.Accordion(open=False, label="Corrections", elem_id=f"{tab}_corrections", elem_classes=["small-accordion"], visible=shared.backend == shared.Backend.DIFFUSERS):
-        with gr.Group(visible=shared.backend == shared.Backend.DIFFUSERS):
+    with gr.Accordion(open=False, label="Corrections", elem_id=f"{tab}_corrections", elem_classes=["small-accordion"], visible=shared.native):
+        with gr.Group(visible=shared.native):
             with gr.Row(elem_id=f"{tab}_hdr_mode_row"):
                 hdr_mode = gr.Dropdown(label="Mode", choices=["Relative values", "Absolute values"], type="index", value="Relative values", elem_id=f"{tab}_hdr_mode", show_label=False)
                 gr.HTML('<br>')
@@ -243,7 +243,7 @@ def create_sampler_options(tabname):
             return '999,845,730,587,443,310,193,116,53,13'
         return ''
 
-    if shared.backend == shared.Backend.ORIGINAL:
+    if not shared.native:
         with gr.Row(elem_classes=['flex-break']):
             options = ['brownian noise', 'discard penultimate sigma']
             values = []
@@ -292,7 +292,7 @@ def create_hires_inputs(tab):
             with gr.Row(elem_id=f"{tab}_hires_row2"):
                 hr_second_pass_steps = gr.Slider(minimum=0, maximum=99, step=1, label='HiRes steps', elem_id=f"{tab}_steps_alt", value=20)
                 denoising_strength = gr.Slider(minimum=0.0, maximum=0.99, step=0.01, label='Strength', value=0.3, elem_id=f"{tab}_denoising_strength")
-        with gr.Group(visible=shared.backend == shared.Backend.DIFFUSERS):
+        with gr.Group(visible=shared.native):
             with gr.Row(elem_id=f"{tab}_refiner_row1", variant="compact"):
                 refiner_start = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label='Refiner start', value=0.0, elem_id=f"{tab}_refiner_start")
                 refiner_steps = gr.Slider(minimum=0, maximum=99, step=1, label="Refiner steps", elem_id=f"{tab}_refiner_steps", value=10)

modules/update.py

@@ -56,7 +56,7 @@ def apply_update(update_rebase, update_submodules, update_extensions):
         if update_rebase:
             i.git('add .')
             i.git('stash')
-        res = i.update('.', current_branch=True, rebase=update_rebase)
+        res = i.update('.', keep_branch=True, rebase=update_rebase)
         html.append(res.replace('\n', '<br>'))
     except Exception as e:
         html.append(f'Error during repository upgrade: {e}')

modules/xadapter/unet_adapter.py

@@ -28,20 +28,18 @@ from diffusers.models.embeddings import (
     ImageHintTimeEmbedding,
     ImageProjection,
     ImageTimeEmbedding,
-    PositionNet,
     TextImageProjection,
     TextImageTimeEmbedding,
     TextTimeEmbedding,
     TimestepEmbedding,
     Timesteps,
 )
+from modules.xadapter.xadapter_hijacks import PositionNet
 from diffusers.models.modeling_utils import ModelMixin
-from diffusers.models.unet_2d_blocks import (
-    UNetMidBlock2DCrossAttn,
-    UNetMidBlock2DSimpleCrossAttn,
-    get_down_block,
-    get_up_block,
-)
+try:
+    from diffusers.models.unet_2d_blocks import UNetMidBlock2DCrossAttn, UNetMidBlock2DSimpleCrossAttn, get_down_block, get_up_block
+except Exception:
+    from diffusers.models.unets.unet_2d_blocks import UNetMidBlock2DCrossAttn, UNetMidBlock2DSimpleCrossAttn, get_down_block, get_up_block
 
 
 logger = logging.get_logger(__name__)  # pylint: disable=invalid-name

modules/zluda.py

@@ -18,8 +18,11 @@ def _join_rocm_home(*paths) -> str:
 
 
 def is_zluda(device: DeviceLikeType):
-    device = torch.device(device)
-    return torch.cuda.get_device_name(device).endswith("[ZLUDA]")
+    try:
+        device = torch.device(device)
+        return torch.cuda.get_device_name(device).endswith("[ZLUDA]")
+    except Exception:
+        return False
 
 
 def test(device: DeviceLikeType) -> Union[Exception, None]:

requirements.txt

@@ -3,7 +3,6 @@ patch-ng
 anyio
 addict
 astunparse
-blendmodes
 clean-fid
 filetype
 future
@@ -15,32 +14,27 @@ kornia
 lark
 lpips
 omegaconf
-open-clip-torch
 optimum
 piexif
 psutil
 pyyaml
 resize-right
 rich
-scipy
 toml
-torchdiffeq
 voluptuous
 yapf
-scikit-image
 fasteners
-dctorch
-pymatting
 orjson
 invisible-watermark
 pi-heif
-diffusers==0.28.0
+diffusers==0.28.1
 safetensors==0.4.3
 tensordict==0.1.2
 peft==0.11.1
 httpx==0.24.1
 compel==2.0.2
 torchsde==0.2.6
+open-clip-torch
 clip-interrogator==0.6.0
 antlr4-python3-runtime==4.9.3
 requests==2.31.0
@@ -53,6 +47,8 @@ huggingface_hub==0.23.2
 numexpr==2.8.8
 numpy==1.26.4
 numba==0.59.1
+blendmodes
+scipy
 pandas
 protobuf==4.25.3
 pytorch_lightning==1.9.4
@@ -63,3 +59,6 @@ Pillow==10.3.0
 timm==0.9.16
 pydantic==1.10.15
 typing-extensions==4.11.0
+torchdiffeq
+dctorch
+scikit-image

scripts/animatediff.py

@@ -50,7 +50,7 @@ orig_pipe = None # original sd_model pipeline
 def set_adapter(adapter_name: str = 'None'):
     if not shared.sd_loaded:
         return
-    if shared.backend != shared.Backend.DIFFUSERS:
+    if not shared.native:
         shared.log.warning('AnimateDiff: not in diffusers mode')
         return
     global motion_adapter, loaded_adapter, orig_pipe # pylint: disable=global-statement
@@ -135,7 +135,7 @@ class Script(scripts.Script):
         return 'AnimateDiff'
 
     def show(self, _is_img2img):
-        return scripts.AlwaysVisible if shared.backend == shared.Backend.DIFFUSERS else False
+        return scripts.AlwaysVisible if shared.native else False
 
 
     def ui(self, _is_img2img):

scripts/blipdiffusion.py

@@ -10,7 +10,7 @@ class Script(scripts.Script):
         return title
 
     def show(self, is_img2img):
-        return is_img2img if shared.backend == shared.Backend.DIFFUSERS else False
+        return is_img2img if shared.native else False
 
     def ui(self, _is_img2img):
         with gr.Row():

scripts/demofusion.py

@@ -1225,7 +1225,7 @@ class Script(scripts.Script):
         return 'DemoFusion'
 
     def show(self, is_img2img):
-        return not is_img2img if shared.backend == shared.Backend.DIFFUSERS else False
+        return not is_img2img if shared.native else False
 
     # return signature is array of gradio components
     def ui(self, _is_img2img):

scripts/differential_diffusion.py

@@ -1875,7 +1875,7 @@ class Script(scripts.Script):
         return 'Differential diffusion'
 
     def show(self, is_img2img):
-        return is_img2img if shared.backend == shared.Backend.DIFFUSERS else False
+        return is_img2img if shared.native else False
 
     def ui(self, _is_img2img):
         with gr.Row():

scripts/example.py

@@ -67,7 +67,7 @@ class Script(scripts.Script):
         return title
 
     def show(self, is_img2img):
-        if shared.backend == shared.Backend.DIFFUSERS:
+        if shared.native:
             return img2img if is_img2img else txt2img
         return False
 

scripts/face-details.py

@@ -32,7 +32,7 @@ class FaceRestorerYolo(FaceRestoration):
 
     def dependencies(self):
         import installer
-        installer.install('ultralytics', ignore=True)
+        installer.install('ultralytics', ignore=True, quiet=True)
 
     def predict(
             self,
@@ -137,8 +137,10 @@ class FaceRestorerYolo(FaceRestoration):
             'width': resolution,
             'height': resolution,
         }
+        control_pipeline = None
         if getattr(p, 'is_control', False):
             from modules.control import run
+            control_pipeline = shared.sd_model
             run.restore_pipeline()
 
         p = processing_class.switch_class(p, processing.StableDiffusionProcessingImg2Img, args)
@@ -160,6 +162,7 @@ class FaceRestorerYolo(FaceRestoration):
                 continue
             p.init_images = [image]
             p.image_mask = [face.mask]
+            # mask_all.append(face.mask)
             p.recursion = True
             pp = processing.process_images_inner(p)
             del p.recursion
@@ -170,18 +173,21 @@ class FaceRestorerYolo(FaceRestoration):
                     mask_all.append(pp.images[1])
 
         # restore pipeline
+        if control_pipeline is not None:
+            shared.sd_model = control_pipeline
         p = processing_class.switch_class(p, orig_cls, orig_p)
         p.init_images = getattr(orig_p, 'init_images', None)
         p.image_mask = getattr(orig_p, 'image_mask', None)
         shared.opts.data['mask_apply_overlay'] = orig_apply_overlay
         np_image = np.array(image)
 
-        """
         if len(mask_all) > 0 and shared.opts.include_mask:
             from modules.control.util import blend
-            mask_all = blend([np.array(m) for m in mask_all])
-            mask_pil = Image.fromarray(mask_all)
-        """
+            p.image_mask = blend([np.array(m) for m in mask_all])
+            # combined = blend([np_image, p.image_mask])
+            # combined = Image.fromarray(combined)
+            # combined.save('/tmp/face.png')
+            p.image_mask = Image.fromarray(p.image_mask)
         return np_image
 
 

scripts/image2video.py

@@ -16,7 +16,7 @@ class Script(scripts.Script):
         return 'Image-to-Video'
 
     def show(self, is_img2img):
-        return is_img2img if shared.backend == shared.Backend.DIFFUSERS else False
+        return is_img2img if shared.native else False
         # return False
 
     # return signature is array of gradio components

scripts/init_latents.py

@@ -8,7 +8,7 @@ class Script(scripts.Script):
         return 'Init Latents'
 
     def show(self, is_img2img):
-        return scripts.AlwaysVisible if shared.backend == shared.Backend.DIFFUSERS else False
+        return scripts.AlwaysVisible if shared.native else False
 
     @staticmethod
     def get_latents(p):
@@ -31,7 +31,7 @@ class Script(scripts.Script):
 
     def process_batch(self, p: processing.StableDiffusionProcessing, *args, **kwargs): # pylint: disable=arguments-differ
         from modules.processing_helpers import create_random_tensors
-        if shared.backend != shared.Backend.DIFFUSERS:
+        if not shared.native:
             return
         args = list(args)
         if p.subseed_strength != 0 and getattr(shared.sd_model, '_execution_device', None) is not None:

scripts/ipadapter.py

@@ -14,7 +14,7 @@ class Script(scripts.Script):
         return 'IP Adapters'
 
     def show(self, is_img2img):
-        return scripts.AlwaysVisible if shared.backend == shared.Backend.DIFFUSERS else False
+        return scripts.AlwaysVisible if shared.native else False
 
     def load_images(self, files):
         init_images = []
@@ -83,7 +83,7 @@ class Script(scripts.Script):
         return [num_adapters] + adapters + scales + files + starts + ends + masks + [layers_active] + [layers]
 
     def process(self, p: processing.StableDiffusionProcessing, *args): # pylint: disable=arguments-differ
-        if shared.backend != shared.Backend.DIFFUSERS:
+        if not shared.native:
             return
         args = list(args) if args is not None else []
         if len(args) == 0:

scripts/kohya_hires_fix.py

@@ -0,0 +1,40 @@
+import gradio as gr
+import diffusers
+from modules import scripts, processing, shared, sd_models, devices
+
+
+class Script(scripts.Script):
+    def title(self):
+        return 'Kohya HiRes Fix'
+
+    def show(self, is_img2img):
+        return not is_img2img if shared.native else False
+
+    # return signature is array of gradio components
+    def ui(self, _is_img2img):
+        with gr.Row():
+            gr.HTML('<a href="https://github.com/huggingface/diffusers/pull/7633">&nbsp Kohya HiRes Fix</a><br>')
+        with gr.Row():
+            enabled = gr.Checkbox(label="Enabled", value=True)
+        with gr.Row():
+            scale_factor = gr.Slider(value=0.5, minimum=0, maximum=1, step=0.05, label="Scale factor")
+            timestep = gr.Number(value=600, minimum=0, maximum=1000, label="Timestep")
+            block_num = gr.Number(value=1, minimum=0, maximum=10, label="Block")
+        return [enabled, scale_factor, timestep, block_num]
+
+    def run(self, p: processing.StableDiffusionProcessing, enabled, scale_factor, timestep, block_num): # pylint: disable=arguments-differ
+        if not enabled:
+            return None
+        if shared.sd_model_type != 'sd':
+            shared.log.warning(f'Kohya Hires Fix: pipeline={shared.sd_model_type} required=sd')
+            return None
+        old_pipe = shared.sd_model
+        high_res_fix = [{'timestep': timestep, 'scale_factor': scale_factor, 'block_num': block_num}]
+        shared.sd_model = diffusers.StableDiffusionPipeline.from_pipe(shared.sd_model, **{ 'custom_pipeline': 'kohya_hires_fix', 'high_res_fix': high_res_fix })
+        sd_models.copy_diffuser_options(shared.sd_model, old_pipe)
+        sd_models.move_model(shared.sd_model, devices.device) # move pipeline to device
+        sd_models.set_diffuser_options(shared.sd_model, vae=None, op='model')
+        shared.log.debug(f'Kohya Hires Fix: pipeline={shared.sd_model.__class__.__name__} args={high_res_fix}')
+        processed = processing.process_images(p)
+        shared.sd_model = old_pipe
+        return processed

scripts/layerdiffuse.py

@@ -8,7 +8,7 @@ class Script(scripts.Script):
         return 'LayerDiffuse'
 
     def show(self, is_img2img):
-        return True if shared.backend == shared.Backend.DIFFUSERS else False
+        return True if shared.native else False
 
     def apply(self):
         from modules import layerdiffuse

scripts/ledits.py

@@ -8,7 +8,7 @@ class Script(scripts.Script):
         return 'LEdits++'
 
     def show(self, is_img2img):
-        return is_img2img if shared.backend == shared.Backend.DIFFUSERS else False
+        return is_img2img if shared.native else False
 
     # return signature is array of gradio components
     def ui(self, _is_img2img):

scripts/mixture_tiling.py

@@ -29,7 +29,7 @@ class Script(scripts.Script):
         return 'Mixture tiling'
 
     def show(self, is_img2img):
-        return not is_img2img if shared.backend == shared.Backend.DIFFUSERS else False
+        return not is_img2img if shared.native else False
 
     def ui(self, _is_img2img):
         with gr.Row():

scripts/mulan.py

@@ -50,7 +50,7 @@ class Script(scripts.Script):
 
     def show(self, is_img2img):
         if shared.cmd_opts.experimental:
-            return True if shared.backend == shared.Backend.DIFFUSERS else False
+            return True if shared.native else False
         else:
             return False
 

scripts/regional_prompting.py

@@ -24,7 +24,7 @@ class Script(scripts.Script):
         return 'Regional prompting'
 
     def show(self, is_img2img):
-        return not is_img2img if shared.backend == shared.Backend.DIFFUSERS else False
+        return not is_img2img if shared.native else False
 
     def change(self, mode):
         return [gr.update(visible='Col' in mode or 'Row' in mode), gr.update(visible='Prompt' in mode)]

scripts/resadapter.py

@@ -0,0 +1,57 @@
+from safetensors.torch import load_file
+from huggingface_hub import hf_hub_download
+import gradio as gr
+from modules import scripts, processing, shared, sd_models, devices
+
+
+repo = 'jiaxiangc/res-adapter'
+models = {
+    'None': '',
+    'SD15 v2 general': 'resadapter_v2_sd1.5',
+    'SDXL v2 general': 'resadapter_v2_sdxl',
+    'SD15 v1 general': 'resadapter_v1_sd1.5',
+    'SD15 v1 extrapolation': 'resadapter_v1_sd1.5_extrapolation',
+    'SD15 v1 interpolation': 'resadapter_v1_sd1.5_interpolation',
+    'SDXL v1 general': 'resadapter_v1_sdxl',
+    'SDXL v1 extrapolation': 'resadapter_v1_sdxl_extrapolation',
+    'SDXL v1 interpolation': 'resadapter_v1_sdxl_interpolation',
+}
+
+class Script(scripts.Script):
+    def title(self):
+        return 'ResAdapter'
+
+    def show(self, is_img2img):
+        return not is_img2img if shared.native else False
+
+    # return signature is array of gradio components
+    def ui(self, _is_img2img):
+        with gr.Row():
+            gr.HTML('<a href="https://github.com/bytedance/res-adapter">&nbsp ResAdapter</a><br>')
+        with gr.Row():
+            model = gr.Dropdown(label="Model", choices=list(models), value="None")
+            weight = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label="Weight", value=1.0)
+        return [model, weight]
+
+    def run(self, p: processing.StableDiffusionProcessing, model, weight): # pylint: disable=arguments-differ
+        if not shared.native or model == 'None':
+            return None
+        if shared.sd_model_type == 'sd':
+            if not model.startswith('SD15'):
+                shared.log.warning(f'ResAdapter: pipeline={shared.sd_model_type} selected={model}')
+                return None
+        if shared.sd_model_type == 'sdxl':
+            if not model.startswith('SDXL'):
+                shared.log.warning(f'ResAdapter: pipeline={shared.sd_model_type} selected={model}')
+                return None
+
+        old_pipe = shared.sd_model
+        shared.sd_model.load_lora_weights(hf_hub_download(repo_id=repo, subfolder=models[model], filename="pytorch_lora_weights.safetensors"), adapter_name="res_adapter")
+        shared.sd_model.set_adapters(["res_adapter"], adapter_weights=[weight])
+        shared.sd_model.unet.load_state_dict(load_file(hf_hub_download(repo_id=repo, subfolder=models[model], filename="diffusion_pytorch_model.safetensors")), strict=False)
+        sd_models.move_model(shared.sd_model, devices.device) # move pipeline to device
+        sd_models.set_diffuser_options(shared.sd_model, vae=None, op='model')
+        shared.log.debug(f'ResAdapter: pipeline={shared.sd_model.__class__.__name__} model="{model}" weight={weight} fn={models[model]}')
+        processed = processing.process_images(p)
+        shared.sd_model = old_pipe
+        return processed

scripts/stablevideodiffusion.py

@@ -19,7 +19,7 @@ class Script(scripts.Script):
         return 'Stable Video Diffusion'
 
     def show(self, is_img2img):
-        return is_img2img if shared.backend == shared.Backend.DIFFUSERS else False
+        return is_img2img if shared.native else False
 
     # return signature is array of gradio components
     def ui(self, _is_img2img):

scripts/t_gate.py

@@ -0,0 +1,45 @@
+import gradio as gr
+from modules import scripts, processing, shared, sd_models, devices
+from installer import install
+
+
+class Script(scripts.Script):
+    def title(self):
+        return 'T-Gate'
+
+    def show(self, is_img2img):
+        return not is_img2img if shared.native else False
+
+    # return signature is array of gradio components
+    def ui(self, _is_img2img):
+        with gr.Row():
+            gr.HTML('<a href="https://github.com/HaozheLiu-ST/T-GATE">&nbsp T-Gate</a><br>')
+        with gr.Row():
+            enabled = gr.Checkbox(label="Enabled", value=True)
+        with gr.Row():
+            gate_step = gr.Slider(minimum=1, maximum=50, step=1, label="Gate step", elem_id="t_gate_steps", value=10)
+        return [enabled, gate_step]
+
+    def run(self, p: processing.StableDiffusionProcessing, enabled, gate_step): # pylint: disable=arguments-differ
+        p.gate_step = min(gate_step, p.steps) if enabled else -1
+        if not enabled:
+            return None
+        install('tgate')
+        import tgate
+        if shared.sd_model_type == 'sd':
+            cls = tgate.TgateSDLoader
+        elif shared.sd_model_type == 'sdxl':
+            cls = tgate.TgateSDXLLoader
+        else:
+            shared.log.warning(f'T-Gate: pipeline={shared.sd_model_type} required=sd or sdxl')
+            return None
+        old_pipe = shared.sd_model
+        shared.sd_model = cls(shared.sd_model, gate_step=p.gate_step)
+        sd_models.copy_diffuser_options(shared.sd_model, old_pipe)
+        sd_models.move_model(shared.sd_model, devices.device) # move pipeline to device
+        sd_models.set_diffuser_options(shared.sd_model, vae=None, op='model')
+        shared.log.debug(f'T-Gate: pipeline={shared.sd_model.__class__.__name__} steps={p.gate_step}')
+        processed = processing.process_images(p)
+        shared.sd_model = old_pipe
+        del shared.sd_model.tgate
+        return processed

scripts/text2video.py

@@ -26,7 +26,7 @@ class Script(scripts.Script):
         return 'Text-to-Video'
 
     def show(self, is_img2img):
-        return not is_img2img if shared.backend == shared.Backend.DIFFUSERS else False
+        return not is_img2img if shared.native else False
 
     # return signature is array of gradio components
     def ui(self, _is_img2img):

scripts/x_adapter.py

@@ -16,7 +16,7 @@ class Script(scripts.Script):
 
     def show(self, is_img2img):
         return False
-        # return True if shared.backend == shared.Backend.DIFFUSERS else False
+        # return True if shared.native else False
 
     def ui(self, _is_img2img):
         with gr.Row():

webui.py

@@ -156,6 +156,7 @@ def initialize():
 
 
 def load_model():
+    modules.devices.set_cuda_params()
     if not opts.sd_checkpoint_autoload or (shared.cmd_opts.ckpt is not None and shared.cmd_opts.ckpt.lower() != 'none'):
         log.debug('Model auto load disabled')
     else:

webui.sh

@@ -73,8 +73,8 @@ fi
 
 if [[ -f "${venv_dir}"/bin/activate ]]
 then
-    echo "Activate python venv"
     source "${venv_dir}"/bin/activate
+    echo "Activate python venv: $VIRTUAL_ENV"
 else
     echo "Error: Cannot activate python venv"
     exit 1
@@ -103,6 +103,7 @@ then
     echo "Launch: ipexrun"
     exec ipexrun --multi-task-manager 'taskset' --memory-allocator 'jemalloc' launch.py "$@"
 else
-    echo "Launch"
+    PYTHON=`which python`
+    echo "Launch: ${PYTHON}"
     exec "${PYTHON}" launch.py "$@"
 fi