Merge pull request #2048 from vladmandic/dev

merge dev to master

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -61,6 +61,30 @@ body:
       value: |
         If unsure if this is a right place to ask your question, perhaps post on [Discussions](https://github.com/vladmandic/automatic/discussions)
         Or reach-out to us on [Discord](https://discord.gg/WqMzTUDC)
+  - type: dropdown
+    id: backend
+    attributes:
+      label: Backend
+      description: What is the backend you're using?
+      options:
+        - Original
+        - Diffusers
+      default: 0
+    validations:
+      required: true
+  - type: dropdown
+    id: model
+    attributes:
+      label: Model
+      description: What is the model type you're using?
+      options:
+        - SD 1.5
+        - SD-XL
+        - Kandinsky
+        - Other
+      default: 0
+    validations:
+      required: true
   - type: checkboxes
     attributes:
       label: Acknowledgements
@@ -68,5 +92,5 @@ body:
       options:
         - label: I have read the above and searched for existing issues
           required: true
-        - label: I confirm that this is classified correctly and its not an extension or diffusers-specific issue
+        - label: I confirm that this is classified correctly and its not an extension issue
           required: true

.github/ISSUE_TEMPLATE/diffusers_report.yml

@@ -1,76 +0,0 @@
-name: Diffusers Report
-description: Something is broken when using Diffusers backend
-title: "[Diffusers]: "
-labels: []
-
-body:
-  - type: textarea
-    id: description
-    attributes:
-      label: Issue Description
-      description: Tell us what happened in a very clear and simple way
-      value: Please fill this form with as much information as possible
-  - type: textarea
-    id: pipeline
-    attributes:
-      label: Diffusers pipeline used
-      description: Enter Diffusers pipeline and model used
-      value: 
-  - type: textarea
-    id: platform
-    attributes:
-      label: Version Platform Description
-      description: Describe your platform (program version, OS, browser)
-      value: 
-  - type: markdown
-    attributes:
-      value: |
-        Any issues without version information will be closed  
-        Provide any relevant platorm information:  
-        - Application version, OS details, GPU information, browser used  
-        
-        Easiest is to include top part of console log, for example:  
-        ```log
-        Starting SD.Next
-        Python 3.10.6 on Linux
-        Version: abd7d160 Sat Jun 10 07:37:42 2023 -0400
-        nVidia CUDA toolkit detected
-        Torch 2.1.0.dev20230519+cu121
-        Torch backend: nVidia CUDA 12.1 cuDNN 8801
-        Torch detected GPU: NVIDIA GeForce RTX 3060 VRAM 12288 Arch (8, 6) Cores 28
-        Enabled extensions-builtin: [...]
-        Enabled extensions: [...]
-        ```
-  - type: markdown
-    attributes:
-      value: |
-        If issue is setup, installation or startup related, please check `sdnext.log` before reporting  
-  - type: markdown
-    attributes:
-      value: |
-        If you have additional extensions installed, try to reproduce the issue with user extensions disabled  
-        And if the issue is with compatibility with specific extension, mark it as such when creating the issue  
-        Try running with `--safe` command line flag with disables loading of user-installed extensions  
-  - type: markdown
-    attributes:
-      value: |
-        If possible update to latest version before reporting the issue as older versions cannot be properly supported  
-        And search existing **issues** and **discussions** before creating a new one  
-  - type: textarea
-    id: logs
-    attributes:
-      label: Relevant log output
-      description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks
-      render: shell
-  - type: markdown
-    attributes:
-      value: |
-        If unsure if this is a right place to ask your question, perhaps post on [Discussions](https://github.com/vladmandic/automatic/discussions)
-        Or reach-out to us on [Discord](https://discord.gg/WqMzTUDC)
-  - type: checkboxes
-    attributes:
-      label: Acknowledgements
-      description: 
-      options:
-        - label: I have read the above and searched for existing issues
-          required: true

CHANGELOG.md

@@ -1,7 +1,29 @@
 # Change Log for SD.Next
 
+## Update for 2023-08-19
+
+Another larger release thats been baking in dev branch for a while...
+
+- general:
+  - caching of extra network information to enable much faster create/refresh operations  
+    thanks @midcoastal
+- diffusers:
+  - add **hires** support (*experimental*)  
+    applies to all model types that support img2img, including **sd** and **sd-xl**  
+    also supports all hires upscaler types as well as standard params like steps and denoising strength  
+    when used with **sd-xl**, it can be used with or without refiner loaded  
+    how to enable - there are no explicit checkboxes other than second pass itself:
+    - hires: upscaler is set and target resolution is not at default  
+    - refiner: if refiner model is loaded  
+  - images save options: *before hires*, *before refiner*
+  - redo `move model to cpu` logic in settings -> diffusers to be more reliable  
+    note that system defaults have also changed, so you may need to tweak to your liking  
+  - update dependencies
+
 ## Update for 2023-08-17
 
+Smaller update, but with some breaking changes (to prepare for future larger functionality)...
+
 - general:
   - update all metadata saved with images  
     see <https://github.com/vladmandic/automatic/wiki/Metadata> for details  

extensions-builtin/Lora/lora.py

@@ -3,6 +3,7 @@ import re
 from typing import Union
 import torch
 from modules import shared, devices, sd_models, errors, scripts, sd_hijack, hashes
+from modules.modelloader import directory_files, extension_filter
 
 metadata_tags_order = {"ss_sd_model_name": 1, "ss_resolution": 2, "ss_clip_skip": 3, "ss_num_train_images": 10, "ss_tag_frequency": 20}
 
@@ -444,10 +445,7 @@ def list_available_loras():
 
     os.makedirs(shared.cmd_opts.lora_dir, exist_ok=True)
 
-    candidates = list(shared.walk_files(shared.cmd_opts.lora_dir, allowed_extensions=[".pt", ".ckpt", ".safetensors"]))
-    for filename in sorted(candidates, key=str.lower):
-        if os.path.isdir(filename):
-            continue
+    for filename in sorted([*filter(extension_filter(['.PT', '.CKPT', '.SAFETENSORS']), directory_files(shared.cmd_opts.lora_dir))], key=str.lower):
 
         name = os.path.splitext(os.path.basename(filename))[0]
         entry = LoraOnDisk(name, filename)

installer.py

@@ -288,12 +288,9 @@ def check_python():
         log.debug(f'Git {git_version.replace("git version", "").strip()}')
 
 
-# Intel hasn't released a corresponding torchvision wheel along with torch and
-# ipex wheels, so we have to install official pytorch torchvision as a W/A.
-# However, the latest torchvision explicitly requires torch version == 2.0.1,
-# which is incompatible with the Intel torch version 2.0.0a0. This will cause
-# intel torch to be uninstalled when pip scans the dependencies of torchvision.
-# This function will check the torch version and force installing Intel torch
+# Intel hasn't released a corresponding torchvision wheel along with torch and ipex wheels, so we have to install official pytorch torchvision as a W/A.
+# However, the latest torchvision explicitly requires torch version == 2.0.1, which is incompatible with the Intel torch version 2.0.0a0. This will cause
+# intel torch to be uninstalled when pip scans the dependencies of torchvision. This function will check the torch version and force installing Intel torch
 # 2.0.0a0 to avoid the underlying dll version error.
 # TODO(Disty or Nuullll) remove this W/A when Intel releases torchvision wheel for windows.
 def fix_ipex_win_torch():
@@ -306,8 +303,8 @@ def fix_ipex_win_torch():
             log.warning(f'Incompatible torch version {installed_torch_ver} for ipex windows, reinstalling to {ipex_torch_ver}')
             torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.0.0a0 intel_extension_for_pytorch==2.0.110+gitba7f6c1 -f https://developer.intel.com/ipex-whl-stable-xpu')
             install(torch_command)
-            import torch
-            import intel_extension_for_pytorch as ipex
+            import torch # pylint: disable=unused-import
+            import intel_extension_for_pytorch as ipex # pylint: disable=unused-import
     except Exception as e:
         log.warning(e)
 
@@ -340,7 +337,6 @@ def check_torch():
         log.info('AMD ROCm toolkit detected')
         os.environ.setdefault('PYTORCH_HIP_ALLOC_CONF', 'garbage_collection_threshold:0.8,max_split_size_mb:512')
         os.environ.setdefault('TENSORFLOW_PACKAGE', 'tensorflow-rocm')
-
         try:
             command = subprocess.run('rocm_agent_enumerator', shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
             amd_gpus = command.stdout.decode(encoding="utf8", errors="ignore").split('\n')
@@ -350,31 +346,26 @@ def check_torch():
             log.debug(f'Run rocm_agent_enumerator failed: {e}')
             amd_gpus = []
 
-        # use the first available amd gpu by default
-        hip_visible_devices = []
+        hip_visible_devices = [] # use the first available amd gpu by default
         for idx, gpu in enumerate(amd_gpus):
             if gpu in ['gfx1100', 'gfx1101', 'gfx1102']:
                 hip_visible_devices.append((idx, gpu, 'navi3x'))
                 break
-            # experimental navi 2x support
-            if gpu in ['gfx1030', 'gfx1031', 'gfx1032', 'gfx1034']:
+            if gpu in ['gfx1030', 'gfx1031', 'gfx1032', 'gfx1034']: # experimental navi 2x support
                 hip_visible_devices.append((idx, gpu, 'navi2x'))
                 break
         if len(hip_visible_devices) > 0:
             idx, gpu, arch = hip_visible_devices[0]
             log.debug(f'ROCm agent used by default: idx={idx} gpu={gpu} arch={arch}')
-
             os.environ.setdefault('HIP_VISIBLE_DEVICES', str(idx))
             if arch == 'navi3x':
                 os.environ.setdefault('HSA_OVERRIDE_GFX_VERSION', '11.0.0')
-                # do not use tensorflow-rocm for navi 3x
-                if os.environ.get('TENSORFLOW_PACKAGE') == 'tensorflow-rocm':
+                if os.environ.get('TENSORFLOW_PACKAGE') == 'tensorflow-rocm': # do not use tensorflow-rocm for navi 3x
                     os.environ['TENSORFLOW_PACKAGE'] = 'tensorflow==2.13.0'
             elif arch == 'navi2x':
                 os.environ.setdefault('HSA_OVERRIDE_GFX_VERSION', '10.3.0')
             else:
                 log.debug(f'HSA_OVERRIDE_GFX_VERSION auto config is skipped for {gpu}')
-
         try:
             command = subprocess.run('hipconfig --version', shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
             major_ver, minor_ver, *_ = command.stdout.decode(encoding="utf8", errors="ignore").split('.')
@@ -383,13 +374,11 @@ def check_torch():
         except Exception as e:
             log.debug(f'Run hipconfig failed: {e}')
             rocm_ver = None
-
         if rocm_ver in ['5.5', '5.6']:
             # install torch nightly via torchvision to avoid wasting bandwidth when torchvision depends on torch from yesterday
             torch_command = os.environ.get('TORCH_COMMAND', f'torchvision --pre --index-url https://download.pytorch.org/whl/nightly/rocm{rocm_ver}')
         else:
             torch_command = os.environ.get('TORCH_COMMAND', 'torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/rocm5.4.2')
-
         xformers_package = os.environ.get('XFORMERS_PACKAGE', 'none')
     elif allow_ipex and (args.use_ipex or shutil.which('sycl-ls') is not None or shutil.which('sycl-ls.exe') is not None or os.environ.get('ONEAPI_ROOT') is not None or os.path.exists('/opt/intel/oneapi') or os.path.exists("C:/Program Files (x86)/Intel/oneAPI") or os.path.exists("C:/oneAPI")):
         args.use_ipex = True # pylint: disable=attribute-defined-outside-init

modules/generation_parameters_copypaste.py

@@ -306,7 +306,7 @@ infotext_to_setting_name_mapping = [
     ('Noise multiplier', 'initial_noise_multiplier'),
     ('Eta', 'eta_ancestral'),
     ('Eta DDIM', 'eta_ddim'),
-    ('Lora method', 'diffusers_lora_loader'),
+    ('LoRA method', 'diffusers_lora_loader'),
     ('Discard penultimate sigma', 'always_discard_next_to_last_sigma'),
     ('UniPC variant', 'uni_pc_variant'),
     ('UniPC skip type', 'uni_pc_skip_type'),

modules/images.py

@@ -209,9 +209,10 @@ def resize_image(resize_mode, im, width, height, upscaler_name=None):
     Resizes an image with the specified resize_mode, width, and height.
     Args:
         resize_mode: The mode to use when resizing the image.
-            0: Resize the image to the specified width and height.
-            1: Resize the image to fill the specified width and height, maintaining the aspect ratio, and then center the image within the dimensions, cropping the excess.
-            2: Resize the image to fit within the specified width and height, maintaining the aspect ratio, and then center the image within the dimensions, filling empty with data from image.
+            0: No resie
+            1: Resize the image to the specified width and height.
+            2: Resize the image to fill the specified width and height, maintaining the aspect ratio, and then center the image within the dimensions, cropping the excess.
+            3: Resize the image to fit within the specified width and height, maintaining the aspect ratio, and then center the image within the dimensions, filling empty with data from image.
         im: The image to resize.
         width: The width to resize the image to.
         height: The height to resize the image to.

modules/modelloader.py

@@ -1,4 +1,5 @@
 import os
+import time
 import shutil
 import importlib
 from typing import Dict
@@ -9,6 +10,49 @@ from modules.paths import script_path, models_path
 
 diffuser_repos = []
 
+def walk(top, onerror:callable=None):
+    # A near-exact copy of `os.path.walk()`, trimmed slightly. Probably not nessesary for most people's collections, but makes a difference on really large datasets.
+    nondirs = []
+    walk_dirs = []
+    try:
+        scandir_it = os.scandir(top)
+    except OSError as error:
+        if onerror is not None:
+            onerror(error, top)
+        return
+    with scandir_it:
+        while True:
+            try:
+                try:
+                    entry = next(scandir_it)
+                except StopIteration:
+                    break
+            except OSError as error:
+                if onerror is not None:
+                    onerror(error, top)
+                return
+            try:
+                is_dir = entry.is_dir()
+            except OSError:
+                is_dir = False
+            if not is_dir:
+                nondirs.append(entry.name)
+            else:
+                try:
+                    if entry.is_symlink() and not os.path.exists(entry.path):
+                        raise NotADirectoryError('Broken Symlink')
+                    walk_dirs.append(entry.path)
+                except OSError as error:
+                    if onerror is not None:
+                        onerror(error, entry.path)
+    # Recurse into sub-directories
+    for new_path in walk_dirs:
+        if os.path.basename(new_path).startswith('models--'):
+            continue
+        yield from walk(new_path, onerror)
+    # Yield after recursion if going bottom up
+    yield top, nondirs
+
 
 def download_civit_model(model_url: str, model_name: str, model_path: str, preview):
     model_file = os.path.join(shared.opts.ckpt_dir, model_path, model_name)
@@ -55,7 +99,6 @@ def download_civit_model(model_url: str, model_name: str, model_path: str, previ
 def download_diffusers_model(hub_id: str, cache_dir: str = None, download_config: Dict[str, str] = None, token = None, variant = None, revision = None, mirror = None):
     from diffusers import DiffusionPipeline
     import huggingface_hub as hf
-
     shared.state.begin()
     shared.state.job = 'downloload model'
     if download_config is None:
@@ -120,7 +163,6 @@ def load_diffusers_models(model_path: str, command_path: str = None):
 
 def find_diffuser(name: str):
     import huggingface_hub as hf
-
     if name in diffuser_repos:
         return name
     if shared.cmd_opts.no_download:
@@ -138,10 +180,95 @@ def find_diffuser(name: str):
     return None
 
 
+modelloader_directories = {}
+cache_last = 0
+cache_time = 1
+
+
+def directory_has_changed(dir:str, *, recursive:bool=True) -> bool: # pylint: disable=redefined-builtin
+    try:
+        dir = os.path.abspath(dir)
+        if dir not in modelloader_directories:
+            return True
+        if cache_last > (time.time() - cache_time):
+            return False
+        if not (os.path.exists(dir) and os.path.isdir(dir) and os.path.getmtime(dir) == modelloader_directories[dir][0]):
+            return True
+        if recursive:
+            for _dir in modelloader_directories:
+                if _dir.startswith(dir) and _dir != dir and not (os.path.exists(_dir) and os.path.isdir(_dir) and os.path.getmtime(_dir) == modelloader_directories[_dir][0]):
+                    return True
+    except Exception as e:
+        shared.log.error(f"Filesystem Error: {e.__class__.__name__}({e})")
+        return True
+    return False
+
+
+def directory_directories(dir:str, *, recursive:bool=True) -> dict[str,tuple[float,list[str]]]: # pylint: disable=redefined-builtin
+    dir = os.path.abspath(dir)
+    if directory_has_changed(dir, recursive=recursive):
+        for _dir in modelloader_directories:
+            try:
+                if (os.path.exists(_dir) and os.path.isdir(_dir)):
+                    continue
+            except Exception:
+                pass
+            del modelloader_directories[_dir]
+        for _dir, _files in walk(dir, lambda e, path: shared.log.debug(f"FS walk error: {e} {path}")):
+            try:
+                mtime = os.path.getmtime(_dir)
+                if _dir not in modelloader_directories or mtime != modelloader_directories[_dir][0]:
+                    modelloader_directories[_dir] = (mtime, [os.path.join(_dir, fn) for fn in _files])
+            except Exception as e:
+                shared.log.error(f"Filesystem Error: {e.__class__.__name__}({e})")
+                del modelloader_directories[_dir]
+    res = {}
+    for _dir in modelloader_directories:
+        if _dir == dir or (recursive and _dir.startswith(dir)):
+            res[_dir] = modelloader_directories[_dir]
+            if not recursive:
+                break
+    return res
+
+
+def directory_mtime(dir:str, *, recursive:bool=True) -> float: # pylint: disable=redefined-builtin
+    return float(max(0, *[mtime for mtime, _ in directory_directories(dir, recursive=recursive).values()]))
+
+
+def directories_file_paths(directories:dict) -> list[str]:
+    return sum([dat[1] for dat in directories.values()],[])
+
+
+def unique_directories(directories:list[str], *, recursive:bool=True) -> list[str]:
+    '''Ensure no empty, or duplicates'''
+    directories = { os.path.abspath(dir): True for dir in directories if dir }.keys()
+    if recursive:
+        '''If we are going recursive, then directories that are children of other directories are redundant'''
+        directories = [dir for dir in directories if not any(_dir != dir and dir.startswith(os.path.join(_dir,'')) for _dir in directories)]
+    return directories
+
+
+def unique_paths(paths:list[str]) -> list[str]:
+    return { fp: True for fp in paths }.keys()
+
+
+def directory_files(*directories:list[str], recursive:bool=True) -> list[str]:
+    return unique_paths(sum([[*directories_file_paths(directory_directories(dir, recursive=recursive))] for dir in unique_directories(directories, recursive=recursive)],[]))
+
+
+def extension_filter(ext_filter=None, ext_blacklist=None):
+    if ext_filter:
+        ext_filter = [*map(str.upper, ext_filter)]
+    if ext_blacklist:
+        ext_blacklist = [*map(str.upper, ext_blacklist)]
+    def filter(fp:str): # pylint: disable=redefined-builtin
+        return (not ext_filter or any(fp.upper().endswith(ew) for ew in ext_filter)) and (not ext_blacklist or not any(fp.upper().endswith(ew) for ew in ext_blacklist))
+    return filter
+
+
 def load_models(model_path: str, model_url: str = None, command_path: str = None, ext_filter=None, download_name=None, ext_blacklist=None) -> list:
     """
     A one-and done loader to try finding the desired models in specified directories.
-
     @param download_name: Specify to download from model_url immediately.
     @param model_url: If no other models are found, this will be downloaded on upscale.
     @param model_path: The location to store/find models in.
@@ -149,21 +276,11 @@ def load_models(model_path: str, model_url: str = None, command_path: str = None
     @param ext_filter: An optional list of filename extensions to filter by
     @return: A list of paths containing the desired model(s)
     """
-    places = []
-    places.append(model_path)
-    if command_path is not None and command_path != model_path and os.path.isdir(command_path):
-        places.append(command_path)
+    places = unique_directories([model_path, command_path])
+    #shared.log.debug(f"{inspect.currentframe().f_code.co_name}: {', '.join(places)}")
     output = []
     try:
-        for place in places:
-            for full_path in shared.walk_files(place, allowed_extensions=ext_filter):
-                if os.path.islink(full_path) and not os.path.exists(full_path):
-                    shared.log.error(f"Skipping broken symlink: {full_path}")
-                    continue
-                if ext_blacklist is not None and any(full_path.endswith(x) for x in ext_blacklist):
-                    continue
-                if full_path not in output:
-                    output.append(full_path)
+        output:list = [*filter(extension_filter(ext_filter, ext_blacklist), directory_files(*places))]
         if model_url is not None and len(output) == 0:
             if download_name is not None:
                 from basicsr.utils.download_util import load_file_from_url
@@ -249,7 +366,6 @@ def load_upscalers():
                 importlib.import_module(full_model)
             except Exception:
                 pass
-
     datas = []
     commandline_options = vars(shared.cmd_opts)
     # some of upscaler classes will not go away after reloading their modules, and we'll end up with two copies of those classes. The newest copy will always be the last in the list, so we go from end to beginning and ignore duplicates
@@ -258,7 +374,6 @@ def load_upscalers():
         classname = str(cls)
         if classname not in used_classes:
             used_classes[classname] = cls
-
     for cls in reversed(used_classes.values()):
         name = cls.__name__
         cmd_name = f"{name.lower().replace('upscaler', '')}_models_path"
@@ -267,7 +382,6 @@ def load_upscalers():
         scaler.user_path = commandline_model_path
         scaler.model_download_path = commandline_model_path or scaler.model_path
         datas += scaler.scalers
-
     shared.sd_upscalers = sorted(
         datas,
         # Special case for UpscalerNone keeps it at the beginning of the list.

modules/processing.py

@@ -711,7 +711,7 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
             else:
                 raise ValueError(f"Unknown backend {shared.backend}")
 
-            if shared.cmd_opts.lowvram or shared.cmd_opts.medvram:
+            if shared.cmd_opts.lowvram or shared.cmd_opts.medvram and shared.backend == shared.Backend.ORIGINAL:
                 lowvram.send_everything_to_cpu()
                 devices.torch_gc()
             if p.scripts is not None:

modules/processing_diffusers.py

@@ -1,8 +1,6 @@
 import inspect
 import typing
 import torch
-# import numpy as np
-# from PIL import Image
 import modules.devices as devices
 import modules.shared as shared
 import modules.sd_samplers as sd_samplers
@@ -23,27 +21,38 @@ except Exception as ex:
 
 def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_prompts):
     results = []
+    if p.enable_hr and p.hr_upscaler != 'None' and p.denoising_strength > 0 and len(getattr(p, 'init_images', [])) == 0:
+        p.is_hr_pass = True
+    is_refiner_enabled = p.enable_hr and shared.sd_refiner is not None
 
-    def diffusers_callback(step: int, _timestep: int, latents: torch.FloatTensor):
-        shared.state.sampling_step = step
+    def hires_resize(latents): # input=latents output=pil
+        latent_upscaler = shared.latent_upscale_modes.get(p.hr_upscaler, None)
+        shared.log.info(f'Diffusers Hires: upscaler={p.hr_upscaler} width={p.hr_upscale_to_x} height={p.hr_upscale_to_y} images={latents.shape[0]}')
+        if latent_upscaler is not None:
+            latents = torch.nn.functional.interpolate(latents, size=(p.hr_upscale_to_y // 8, p.hr_upscale_to_x // 8), mode=latent_upscaler["mode"], antialias=latent_upscaler["antialias"])
+        first_pass_images = vae_decode(latents=latents, model=shared.sd_model, full_quality=True, output_type='pil')
+        p.init_images = []
+        for first_pass_image in first_pass_images:
+            init_image = images.resize_image(1, first_pass_image, p.hr_upscale_to_x, p.hr_upscale_to_y, upscaler_name=p.hr_upscaler) if latent_upscaler is None else first_pass_image
+            p.init_images.append(init_image)
+        p.width = p.hr_upscale_to_x
+        p.height = p.hr_upscale_to_y
+
+    def save_intermediate(latents, suffix):
+        for i in range(len(latents)):
+            from modules.processing import create_infotext
+            info=create_infotext(p, p.all_prompts, p.all_seeds, p.all_subseeds, [], iteration=p.iteration, position_in_batch=i)
+            decoded = vae_decode(latents=latents, model=shared.sd_model, output_type='pil', full_quality=p.full_quality)
+            for i in range(len(decoded)):
+                images.save_image(decoded[i], path=p.outpath_samples, basename="", seed=seeds[i], prompt=prompts[i], extension=shared.opts.samples_format, info=info, p=p, suffix=suffix)
+
+    def diffusers_callback(_step: int, _timestep: int, latents: torch.FloatTensor):
+        shared.state.sampling_step += 1
         shared.state.sampling_steps = p.steps
+        if p.is_hr_pass:
+            shared.state.sampling_steps += p.hr_second_pass_steps
         shared.state.current_latent = latents
 
-    def hires_resize(latents):
-        return latents # TODO finish hires
-        if p.hr_upscaler == 'None':
-            return latents
-        scale = shared.latent_upscale_modes.get(p.hr_upscaler, None)
-        if scale is not None:
-            p.init_hr()
-            p.ops.append('hires')
-            shared.log.info(f'Diffusers Hires: upscaler={p.hr_upscaler} mode={scale["mode"]} antialias={scale["antialias"]} width={p.hr_upscale_to_x} height={p.hr_upscale_to_y} images={latents.shape[0]}')
-            hires_image = torch.nn.functional.interpolate(latents, size=(p.hr_upscale_to_y // 8, p.hr_upscale_to_x // 8), mode=scale["mode"], antialias=scale["antialias"])
-        else:
-            shared.log.warning(f'Diffusers hires unsupported: upscaler={p.hr_upscaler} supported=latent modes')
-            hires_image = latents
-        return hires_image
-
     def full_vae_decode(latents, model):
         shared.log.debug(f'Diffusers VAE decode: name={sd_vae.loaded_vae_file if sd_vae.loaded_vae_file is not None else "baked"} dtype={model.vae.dtype} upcast={model.vae.config.get("force_upcast", None)} images={latents.shape[0]}')
         if shared.opts.diffusers_move_unet and not model.has_accelerate:
@@ -51,6 +60,8 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
             unet_device = model.unet.device
             model.unet.to(devices.cpu)
             devices.torch_gc()
+        if not shared.cmd_opts.lowvram and not shared.opts.diffusers_seq_cpu_offload:
+            model.vae.to(devices.device)
         latents.to(model.vae.device)
         decoded = model.vae.decode(latents / model.vae.config.scaling_factor, return_dict=False)[0]
         if shared.opts.diffusers_move_unet and not model.has_accelerate:
@@ -65,19 +76,18 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         return decoded
 
     def vae_decode(latents, model, output_type='np', full_quality=True):
+        if not torch.is_tensor(latents): # already decoded
+            return latents
+        if latents.shape[0] == 0:
+            shared.log.error(f'VAE nothing to decode: {latents.shape}')
+            return []
         if shared.state.interrupted or shared.state.skipped:
             return []
         if not hasattr(model, 'vae'):
             shared.log.error('VAE not found in model')
             return []
-        if not torch.is_tensor(latents):
-            shared.log.error(f'VAE input is not latents: {type(latents)}')
-            return []
-        if latents.shape[0] == 0:
-            shared.log.error(f'VAE nothing to decode: {latents.shape}')
-            return []
-        if p.enable_hr:
-            latents = hires_resize(latents=latents)
+        if len(latents.shape) == 3: # lost a batch dim in hires
+            latents = latents.unsqueeze(0)
         if full_quality:
             decoded = full_vae_decode(latents=latents, model=shared.sd_model)
         else:
@@ -104,7 +114,9 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
                 negative_prompts_2.append(negative_prompts_2[-1])
         return prompts, negative_prompts, prompts_2, negative_prompts_2
 
-    def set_pipeline_args(model, prompts: list, negative_prompts: list, prompts_2: typing.Optional[list]=None, negative_prompts_2: typing.Optional[list]=None, is_refiner: bool=False, **kwargs):
+    def set_pipeline_args(model, prompts: list, negative_prompts: list, prompts_2: typing.Optional[list]=None, negative_prompts_2: typing.Optional[list]=None, is_refiner: bool=False, desc:str='', **kwargs):
+        if hasattr(model, "set_progress_bar_config"):
+            model.set_progress_bar_config(bar_format='Progress {rate_fmt}{postfix} {bar} {percentage:3.0f}% {n_fmt}/{total_fmt} {elapsed} {remaining} '+desc, ncols=80, colour='#327fba')
         args = {}
         pipeline = model
         signature = inspect.signature(type(pipeline).__call__)
@@ -117,13 +129,9 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         negative_pooled = None
         prompts, negative_prompts, prompts_2, negative_prompts_2 = fix_prompts(prompts, negative_prompts, prompts_2, negative_prompts_2)
         if shared.opts.prompt_attention in {'Compel parser', 'Full parser'}:
-            prompt_embed, pooled, negative_embed, negative_pooled = prompt_parser_diffusers.compel_encode_prompts(model,
-                                                                                                                  prompts,
-                                                                                                                  negative_prompts,
-                                                                                                                  prompts_2,
-                                                                                                                  negative_prompts_2,
-                                                                                                                  is_refiner,
-                                                                                                                  kwargs.pop("clip_skip", None))
+            prompt_embed, pooled, negative_embed, negative_pooled = prompt_parser_diffusers.compel_encode_prompts(model, prompts, negative_prompts,
+                                                                                                                  prompts_2, negative_prompts_2,
+                                                                                                                  is_refiner, kwargs.pop("clip_skip", None))
         if 'prompt' in possible:
             if hasattr(model, 'text_encoder') and 'prompt_embeds' in possible and prompt_embed is not None:
                 args['prompt_embeds'] = prompt_embed
@@ -141,7 +149,7 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
             else:
                 args['negative_prompt'] = negative_prompts
         if 'num_inference_steps' in possible:
-            args['num_inference_steps'] = p.steps
+            args['num_inference_steps'] = p.steps if not p.is_hr_pass else p.hr_second_pass_steps
         if 'guidance_scale' in possible:
             args['guidance_scale'] = p.cfg_scale
         if 'generator' in possible:
@@ -185,8 +193,9 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         return args
 
     is_karras_compatible = shared.sd_model.__class__.__init__.__annotations__.get("scheduler", None) == diffusers.schedulers.scheduling_utils.KarrasDiffusionSchedulers
-    if (not hasattr(shared.sd_model.scheduler, 'name')) or (shared.sd_model.scheduler.name != p.sampler_name) and (p.sampler_name != 'Default') and is_karras_compatible:
-        sampler = sd_samplers.all_samplers_map.get(p.sampler_name, None)
+    use_sampler = p.sampler_name if not p.is_hr_pass else p.latent_sampler
+    if (not hasattr(shared.sd_model.scheduler, 'name')) or (shared.sd_model.scheduler.name != use_sampler) and (use_sampler != 'Default') and is_karras_compatible:
+        sampler = sd_samplers.all_samplers_map.get(use_sampler, None)
         if sampler is None:
             sampler = sd_samplers.all_samplers_map.get("UniPC")
         sd_samplers.create_sampler(sampler.name, shared.sd_model) # TODO(Patrick): For wrapped pipelines this is currently a no-op
@@ -223,8 +232,7 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
     if shared.opts.diffusers_move_base and not shared.sd_model.has_accelerate:
         shared.sd_model.to(devices.device)
 
-    refiner_enabled = shared.sd_refiner is not None and p.enable_hr
-    pipe_args = set_pipeline_args(
+    base_args = set_pipeline_args(
         model=shared.sd_model,
         prompts=prompts,
         negative_prompts=negative_prompts,
@@ -232,36 +240,57 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         negative_prompts_2=[p.refiner_negative] if len(p.refiner_negative) > 0 else negative_prompts,
         eta=shared.opts.eta_ddim,
         guidance_rescale=p.diffusers_guidance_rescale,
-        denoising_start=0 if refiner_enabled and p.refiner_start > 0 and p.refiner_start < 1 else None,
-        denoising_end=p.refiner_start if refiner_enabled and p.refiner_start > 0 and p.refiner_start < 1 else None,
+        denoising_start=0 if is_refiner_enabled and p.refiner_start > 0 and p.refiner_start < 1 else None,
+        denoising_end=p.refiner_start if is_refiner_enabled and p.refiner_start > 0 and p.refiner_start < 1 else None,
         output_type='latent' if hasattr(shared.sd_model, 'vae') else 'np',
         is_refiner=False,
         clip_skip=p.clip_skip,
+        desc='Base',
         **task_specific_kwargs
     )
     p.extra_generation_params['CFG rescale'] = p.diffusers_guidance_rescale
     p.extra_generation_params["Eta DDIM"] = shared.opts.eta_ddim if shared.opts.eta_ddim is not None and shared.opts.eta_ddim > 0 else None
-    output = shared.sd_model(**pipe_args) # pylint: disable=not-callable
-    if shared.state.interrupted or shared.state.skipped:
-        unload_diffusers_lora()
-        return results
+    output = shared.sd_model(**base_args) # pylint: disable=not-callable
 
     if lora_state['active']:
-        p.extra_generation_params['Lora method'] = shared.opts.diffusers_lora_loader
+        p.extra_generation_params['LoRA method'] = shared.opts.diffusers_lora_loader
         unload_diffusers_lora()
 
-    if not refiner_enabled:
-        results = vae_decode(latents=output.images, model=shared.sd_model, full_quality=p.full_quality)
-    else:
-        for i in range(len(output.images)): # save images before refiner
-            if shared.opts.save and not p.do_not_save_samples and shared.opts.save_images_before_refiner and hasattr(shared.sd_model, 'vae'):
-                from modules.processing import create_infotext
-                info=create_infotext(p, p.all_prompts, p.all_seeds, p.all_subseeds, [], iteration=p.iteration, position_in_batch=i)
-                decoded = vae_decode(latents=output.images, model=shared.sd_model, output_type='pil', full_quality=p.full_quality)
-                for i in range(len(decoded)):
-                    images.save_image(decoded[i], path=p.outpath_samples, basename="", seed=seeds[i], prompt=prompts[i], extension=shared.opts.samples_format, info=info, p=p, suffix="-before-refiner")
+    if shared.state.interrupted or shared.state.skipped:
+        return results
 
-        if (shared.opts.diffusers_move_base or shared.cmd_opts.medvram or shared.opts.diffusers_model_cpu_offload) and not (shared.cmd_opts.lowvram or shared.opts.diffusers_seq_cpu_offload):
+    # optional hires pass
+    if p.is_hr_pass:
+        p.init_hr()
+        if p.width != p.hr_upscale_to_x or p.height != p.hr_upscale_to_y:
+            if shared.opts.save and not p.do_not_save_samples and shared.opts.save_images_before_highres_fix and hasattr(shared.sd_model, 'vae'):
+                save_intermediate(latents=output.images, suffix="-before-hires")
+            hires_resize(latents=output.images)
+            print('HERE', p.init_images)
+            sd_models.set_diffuser_pipe(shared.sd_model, sd_models.DiffusersTaskType.IMAGE_2_IMAGE)
+            p.ops.append('hires')
+            hires_args = set_pipeline_args(
+                model=shared.sd_model,
+                prompts=prompts,
+                negative_prompts=negative_prompts,
+                prompts_2=[p.refiner_prompt] if len(p.refiner_prompt) > 0 else prompts,
+                negative_prompts_2=[p.refiner_negative] if len(p.refiner_negative) > 0 else negative_prompts,
+                eta=shared.opts.eta_ddim,
+                guidance_rescale=p.diffusers_guidance_rescale,
+                output_type='latent' if hasattr(shared.sd_model, 'vae') else 'np',
+                is_refiner=False,
+                clip_skip=p.clip_skip,
+                image=p.init_images,
+                strength=p.denoising_strength,
+                desc='Hires',
+            )
+            output = shared.sd_model(**hires_args) # pylint: disable=not-callable
+
+    # optional refiner pass or decode
+    if is_refiner_enabled:
+        if shared.opts.save and not p.do_not_save_samples and shared.opts.save_images_before_refiner and hasattr(shared.sd_model, 'vae'):
+            save_intermediate(latents=output.images, suffix="-before-refiner")
+        if shared.opts.diffusers_move_base and not shared.sd_model.has_accelerate:
             shared.log.debug('Diffusers: Moving base model to CPU')
             shared.sd_model.to(devices.cpu)
             devices.torch_gc()
@@ -279,7 +308,7 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
             shared.sd_refiner.to(devices.device)
         p.ops.append('refine')
         for i in range(len(output.images)):
-            pipe_args = set_pipeline_args(
+            refiner_args = set_pipeline_args(
                 model=shared.sd_refiner,
                 prompts=[p.refiner_prompt] if len(p.refiner_prompt) > 0 else prompts[i],
                 negative_prompts=[p.refiner_negative] if len(p.refiner_negative) > 0 else negative_prompts[i],
@@ -294,19 +323,25 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
                 output_type='latent' if hasattr(shared.sd_refiner, 'vae') else 'np',
                 is_refiner=True,
                 clip_skip=p.clip_skip,
+                desc='Refiner',
             )
-            refiner_output = shared.sd_refiner(**pipe_args) # pylint: disable=not-callable
+            refiner_output = shared.sd_refiner(**refiner_args) # pylint: disable=not-callable
             p.extra_generation_params['Image CFG scale'] = p.image_cfg_scale if p.image_cfg_scale is not None else None
             p.extra_generation_params['Refiner start'] = p.refiner_start
             p.extra_generation_params["Hires steps"] = p.hr_second_pass_steps
 
             if not shared.state.interrupted and not shared.state.skipped:
                 refiner_images = vae_decode(latents=refiner_output.images, model=shared.sd_refiner, full_quality=True)
-                results.append(refiner_images[0])
+                for refiner_image in refiner_images:
+                    results.append(refiner_image)
 
         if shared.opts.diffusers_move_refiner and not shared.sd_refiner.has_accelerate:
             shared.log.debug('Diffusers: Moving refiner model to CPU')
             shared.sd_refiner.to(devices.cpu)
             devices.torch_gc()
 
+    # final decode since there is no refiner
+    if not is_refiner_enabled:
+        results = vae_decode(latents=output.images, model=shared.sd_model, full_quality=p.full_quality)
+
     return results

modules/sd_models.py

@@ -695,7 +695,7 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
 
         if (shared.opts.diffusers_model_cpu_offload or shared.cmd_opts.medvram) and (shared.opts.diffusers_seq_cpu_offload or shared.cmd_opts.lowvram):
             shared.log.warning(f'Diffusers {op}: Model CPU offload (--medvram) and Sequential CPU offload (--lowvram) are not compatible')
-            shared.log.debug(f'Diffusers {op}: disable model CPU offload and --medvram')
+            shared.log.debug(f'Diffusers {op}: disabling model CPU offload and --medvram')
             shared.opts.diffusers_model_cpu_offload=False
             shared.cmd_opts.medvram=False
 
@@ -705,11 +705,21 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
         if hasattr(sd_model, "enable_model_cpu_offload"):
             if (shared.cmd_opts.medvram and devices.backend != "directml") or shared.opts.diffusers_model_cpu_offload:
                 shared.log.debug(f'Diffusers {op}: enable model CPU offload')
+                if shared.opts.diffusers_move_base or shared.opts.diffusers_move_unet or shared.opts.diffusers_move_refiner:
+                    shared.opts.diffusers_move_base = False
+                    shared.opts.diffusers_move_unet = False
+                    shared.opts.diffusers_move_refiner = False
+                    shared.log.warning(f'Disabling {op} "Move model to CPU" since "Model CPU offload" is enabled')
                 sd_model.enable_model_cpu_offload()
                 sd_model.has_accelerate = True
         if hasattr(sd_model, "enable_sequential_cpu_offload"):
             if shared.cmd_opts.lowvram or shared.opts.diffusers_seq_cpu_offload:
                 shared.log.debug(f'Diffusers {op}: enable sequential CPU offload')
+                if shared.opts.diffusers_move_base or shared.opts.diffusers_move_unet or shared.opts.diffusers_move_refiner:
+                    shared.opts.diffusers_move_base = False
+                    shared.opts.diffusers_move_unet = False
+                    shared.opts.diffusers_move_refiner = False
+                    shared.log.warning(f'Disabling {op} "Move model to CPU" since "Sequential CPU offload" is enabled')
                 sd_model.enable_sequential_cpu_offload(device=devices.device)
                 sd_model.has_accelerate = True
         if hasattr(sd_model, "enable_vae_slicing"):
@@ -759,10 +769,11 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
                 else:
                     if not refiner_enough_vram and not (shared.opts.diffusers_move_base and shared.opts.diffusers_move_refiner):
                         shared.log.warning(f"Insufficient GPU memory, using system memory as fallback: free={free_vram} GB")
-                        shared.log.debug('Enabled moving base model to CPU')
-                        shared.log.debug('Enabled moving refiner model to CPU')
-                        shared.opts.diffusers_move_base=True
-                        shared.opts.diffusers_move_refiner=True
+                        if not shared.opts.shared.opts.diffusers_seq_cpu_offload and not shared.opts.diffusers_model_cpu_offload:
+                            shared.log.debug('Enabled moving base model to CPU')
+                            shared.log.debug('Enabled moving refiner model to CPU')
+                            shared.opts.diffusers_move_base=True
+                            shared.opts.diffusers_move_refiner=True
                     shared.log.debug('Moving base model to CPU')
                     model_data.sd_model.to(devices.cpu)
                     devices.torch_gc(force=True)

modules/sd_vae.py

@@ -124,6 +124,9 @@ def resolve_vae(checkpoint_file):
             return vae_dict[basename], 'in VAE dir'
     else:
         vae_from_options = vae_dict.get(shared.opts.sd_vae, None) # 5th
+        if vae_from_options is not None:
+            return vae_from_options, 'specified in settings'
+        vae_from_options = vae_dict.get(shared.opts.sd_vae + '.safetensors', None) # 6th
         if vae_from_options is not None:
             return vae_from_options, 'specified in settings'
         shared.log.warning(f"VAE not found: {shared.opts.sd_vae}")
@@ -188,10 +191,8 @@ def load_vae_diffusers(model_file, vae_file=None, vae_source="from unknown sourc
         pass
     else:
         diffusers_load_config['variant'] = shared.opts.diffusers_vae_load_variant
-
     if shared.opts.diffusers_vae_upcast != 'default':
         diffusers_load_config['force_upcast'] = True if shared.opts.diffusers_vae_upcast == 'true' else False
-
     shared.log.debug(f'Diffusers VAE load config: {diffusers_load_config}')
     try:
         import diffusers
@@ -251,8 +252,11 @@ def reload_vae_weights(sd_model=None, vae_file=unspecified):
         load_vae(sd_model, vae_file, vae_source)
         sd_hijack.model_hijack.hijack(sd_model)
         script_callbacks.model_loaded_callback(sd_model)
+        if vae_file is not None:
+            shared.log.info(f"VAE weights loaded: {vae_file}")
+    # else:
+    #    load_vae_diffusers(model_file, vae_file, vae_source)
 
     if not shared.cmd_opts.lowvram and not shared.cmd_opts.medvram and not sd_model.has_accelerate:
         sd_model.to(devices.device)
-    shared.log.info(f"VAE weights loaded: {vae_file}")
     return sd_model

modules/shared.py

@@ -268,7 +268,7 @@ def list_themes():
 
 
 def disable_extensions():
-    if opts.lora_disable:
+    if opts.lyco_patch_lora and backend != Backend.DIFFUSERS:
         if 'Lora' not in opts.disabled_extensions:
             opts.data['disabled_extensions'].append('Lora')
     else:
@@ -398,16 +398,16 @@ options_templates.update(options_section(('cuda', "Compute Settings"), {
 
 options_templates.update(options_section(('diffusers', "Diffusers Settings"), {
     "diffusers_pipeline": OptionInfo(pipelines[0], 'Diffusers pipeline', gr.Dropdown, lambda: {"choices": pipelines}),
-    "diffusers_move_base": OptionInfo(False, "Move base model to CPU when using refiner"),
-    "diffusers_move_unet": OptionInfo(False, "Move base model to CPU when using VAE"),
+    "diffusers_move_base": OptionInfo(True, "Move base model to CPU when using refiner"),
+    "diffusers_move_unet": OptionInfo(True, "Move base model to CPU when using VAE"),
     "diffusers_move_refiner": OptionInfo(True, "Move refiner model to CPU when not in use"),
     "diffusers_extract_ema": OptionInfo(True, "Use model EMA weights when possible"),
     "diffusers_generator_device": OptionInfo("default", "Generator device", gr.Radio, lambda: {"choices": ["default", "cpu"]}),
-    "diffusers_seq_cpu_offload": OptionInfo(False, "Enable sequential CPU offload"),
-    "diffusers_model_cpu_offload": OptionInfo(False, "Enable model CPU offload"),
+    "diffusers_model_cpu_offload": OptionInfo(False, "Enable model CPU offload (--medvram)"),
+    "diffusers_seq_cpu_offload": OptionInfo(False, "Enable sequential CPU offload (--lowvram)"),
     "diffusers_vae_upcast": OptionInfo("default", "VAE upcasting", gr.Radio, lambda: {"choices": ['default', 'true', 'false']}),
     "diffusers_vae_slicing": OptionInfo(True, "Enable VAE slicing"),
-    "diffusers_vae_tiling": OptionInfo(False, "Enable VAE tiling"),
+    "diffusers_vae_tiling": OptionInfo(True, "Enable VAE tiling"),
     "diffusers_attention_slicing": OptionInfo(False, "Enable attention slicing"),
     "diffusers_model_load_variant": OptionInfo("default", "Diffusers model loading variant", gr.Radio, lambda: {"choices": ['default', 'fp32', 'fp16']}),
     "diffusers_vae_load_variant": OptionInfo("default", "Diffusers VAE loading variant", gr.Radio, lambda: {"choices": ['default', 'fp32', 'fp16']}),
@@ -422,7 +422,7 @@ options_templates.update(options_section(('system-paths', "System Paths"), {
     "ckpt_dir": OptionInfo(os.path.join(paths.models_path, 'Stable-diffusion'), "Path to directory with stable diffusion checkpoints"),
     "diffusers_dir": OptionInfo(os.path.join(paths.models_path, 'Diffusers'), "Path to directory with stable diffusion diffusers"),
     "vae_dir": OptionInfo(os.path.join(paths.models_path, 'VAE'), "Path to directory with VAE files"),
-    "lora_dir": OptionInfo(os.path.join(paths.models_path, 'Lora'), "Path to directory with Lora network(s)"),
+    "lora_dir": OptionInfo(os.path.join(paths.models_path, 'Lora'), "Path to directory with LoRA network(s)"),
     "lyco_dir": OptionInfo(os.path.join(paths.models_path, 'LyCORIS'), "Path to directory with LyCORIS network(s)"),
     "styles_dir": OptionInfo(os.path.join(paths.data_path, 'styles.csv'), "Path to user-defined styles file"),
     "embeddings_dir": OptionInfo(os.path.join(paths.models_path, 'embeddings'), "Embeddings directory for textual inversion"),
@@ -626,9 +626,9 @@ options_templates.update(options_section(('extra_networks', "Extra Networks"), {
     "extra_networks_card_square": OptionInfo(True, "UI disable variable aspect ratio"),
     "extra_networks_card_fit": OptionInfo("cover", "UI image contain method", gr.Radio, lambda: {"choices": ["contain", "cover", "fill"]}),
     "extra_network_skip_indexing": OptionInfo(False, "Do not automatically build extra network pages", gr.Checkbox),
-    "lyco_patch_lora": OptionInfo(False, "Use LyCoris handler for all Lora types", gr.Checkbox),
-    "lora_disable": OptionInfo(False, "Disable built-in Lora handler", gr.Checkbox, { "visible": True }, onchange=disable_extensions),
-    "lora_functional": OptionInfo(False, "Use Kohya method for handling multiple Loras", gr.Checkbox),
+    "lyco_patch_lora": OptionInfo(False, "Use LyCoris handler for all LoRA types", gr.Checkbox),
+    # "lora_disable": OptionInfo(False, "Disable built-in Lora handler", gr.Checkbox, { "visible": True }, onchange=disable_extensions),
+    "lora_functional": OptionInfo(False, "Use Kohya method for handling multiple LoRA", gr.Checkbox),
     "extra_networks_add_text_separator": OptionInfo(" ", "Extra text to add before <...> when adding extra network to prompt", gr.Text, { "visible": False }),
     "extra_networks_default_multiplier": OptionInfo(1.0, "Multiplier for extra networks", gr.Slider, {"minimum": 0.0, "maximum": 1.0, "step": 0.01}),
     "sd_hypernetwork": OptionInfo("None", "Add hypernetwork to prompt", gr.Dropdown, lambda: {"choices": ["None"] + list(hypernetworks.keys())}, refresh=reload_hypernetworks),
@@ -708,10 +708,11 @@ class Options:
             diff = {}
             for k, v in self.data.items():
                 if k in self.data_labels:
+                    if type(v) is list:
+                        diff[k] = v
                     if self.data_labels[k].default != v:
                         diff[k] = v
-            output = json.dumps(diff, indent=2)
-            writefile(output, filename)
+            writefile(diff, filename)
         except Exception as e:
             log.error(f'Saving settings failed: {filename} {e}')
 

modules/textual_inversion/textual_inversion.py

@@ -13,6 +13,7 @@ import modules.textual_inversion.dataset
 from modules.textual_inversion.learn_schedule import LearnRateScheduler
 from modules.textual_inversion.image_embedding import embedding_to_b64, embedding_from_b64, insert_image_data_embed, extract_image_data_embed, caption_image_overlay
 from modules.textual_inversion.logging import save_settings_to_file
+from modules.modelloader import directory_files, extension_filter, directory_mtime
 
 TextualInversionTemplate = namedtuple("TextualInversionTemplate", ["name", "path"])
 textual_inversion_templates = {}
@@ -85,15 +86,13 @@ class DirWithTextualInversionEmbeddings:
         if not os.path.isdir(self.path):
             return False
 
-        mt = os.path.getmtime(self.path)
-        if self.mtime is None or mt > self.mtime:
-            return True
+        return directory_mtime(self.path) != self.mtime
 
     def update(self):
         if not os.path.isdir(self.path):
             return
 
-        self.mtime = os.path.getmtime(self.path)
+        self.mtime = directory_mtime(self.path)
 
 
 class EmbeddingDatabase:
@@ -216,16 +215,19 @@ class EmbeddingDatabase:
     def load_from_dir(self, embdir):
         if not os.path.isdir(embdir.path):
             return
-        for root, _dirs, fns in os.walk(embdir.path, followlinks=True):
-            for fn in fns:
-                try:
-                    fullfn = os.path.join(root, fn)
-                    if os.stat(fullfn).st_size == 0:
-                        continue
-                    self.load_from_file(fullfn, fn)
-                except Exception as e:
-                    errors.display(e, f'embedding load {fn}')
+
+        is_ext = extension_filter(['.PNG', '.WEBP', '.JXL', '.AVIF', '.BIN', '.PT', '.SAFETENSORS'])
+        is_not_preview = lambda fp: not next(iter(os.path.splitext(fp))).upper().endswith('.PREVIEW')
+
+        for file_path in [*filter(lambda fp: is_ext(fp) and is_not_preview(fp), directory_files(embdir.path))]:
+            try:
+                if os.stat(file_path).st_size == 0:
                     continue
+                fn = os.path.basename(file_path)
+                self.load_from_file(file_path, fn)
+            except Exception as e:
+                errors.display(e, f'embedding load {fn}')
+                continue
 
     def load_textual_inversion_embeddings(self, force_reload=False):
         if not force_reload:

modules/ui.py

@@ -92,8 +92,8 @@ def calc_resolution_hires(enable, width, height, hr_scale, hr_resize_x, hr_resiz
     from modules import processing, devices
     if not enable:
         return ""
-    if modules.shared.backend == modules.shared.Backend.DIFFUSERS:
-       return "Hires resize: disabled"
+    # if modules.shared.backend == modules.shared.Backend.DIFFUSERS:
+    #   return "Hires resize: disabled"
     p = processing.StableDiffusionProcessingTxt2Img(width=width, height=height, enable_hr=True, hr_scale=hr_scale, hr_resize_x=hr_resize_x, hr_resize_y=hr_resize_y)
     p.init_hr()
     with devices.autocast():
@@ -106,8 +106,8 @@ def resize_from_to_html(width, height, scale_by):
     target_height = int(height * scale_by)
     if not target_width or not target_height:
         return "no image selected"
-    if modules.shared.backend == modules.shared.Backend.DIFFUSERS:
-       return "Hires resize: disabled"
+    # if modules.shared.backend == modules.shared.Backend.DIFFUSERS:
+    #   return "Hires resize: disabled"
     return f"Hires resize: from <span class='resolution'>{width}x{height}</span> to <span class='resolution'>{target_width}x{target_height}</span>"
 
 

modules/ui_extra_networks.py

@@ -8,7 +8,7 @@ from pathlib import Path
 from collections import OrderedDict
 import gradio as gr
 from PIL import Image
-from modules import shared, scripts
+from modules import shared, scripts, modelloader
 from modules.generation_parameters_copypaste import image_from_url_text
 from modules.ui_components import ToolButton
 
@@ -31,7 +31,7 @@ def fetch_file(filename: str = ""):
         return FileResponse(filename, headers={"Accept-Ranges": "bytes"})
     if not any(Path(x).absolute() in Path(filename).absolute().parents for x in allowed_dirs):
         return JSONResponse({"error": f"File cannot be fetched: {filename}. Must be in one of directories registered by extra pages."})
-    if os.path.splitext(filename)[1].lower() not in (".png", ".jpg", ".webp"):
+    if os.path.splitext(filename)[1].lower() not in (".png", ".jpg", ".jpeg", ".webp"):
         return JSONResponse({"error": f"File cannot be fetched: {filename}. Only png and jpg and webp."})
     return FileResponse(filename, headers={"Accept-Ranges": "bytes"})
 
@@ -158,19 +158,17 @@ class ExtraNetworksPage:
             return f"<div id='{tabname}_{self_name_id}_subdirs' class='extra-network-subdirs'></div><div id='{tabname}_{self_name_id}_cards' class='extra-network-cards'>Extra network page not ready<br>Click refresh to try again</div>"
         subdirs = {}
         allowed_folders = [os.path.abspath(x) for x in self.allowed_directories_for_previews()]
-        for parentdir in [*set(allowed_folders)]:
-            for root, dirs, _files in os.walk(parentdir, followlinks=True):
-                for dirname in dirs:
-                    x = os.path.join(root, dirname)
-                    if shared.opts.diffusers_dir in x:
-                        subdirs[os.path.basename(shared.opts.diffusers_dir)] = 1
-                    if (not os.path.isdir(x)) or ('models--' in x):
-                        continue
-                    subdir = os.path.abspath(x)[len(parentdir):].replace("\\", "/")
-                    while subdir.startswith("/"):
-                        subdir = subdir[1:]
-                    if not self.is_empty(x):
-                        subdirs[subdir] = 1
+        for parentdir, dirs in {dir: modelloader.directory_directories(dir) for dir in allowed_folders}.items(): # pylint: disable=redefined-builtin
+            for dir in dirs.keys(): # pylint: disable=redefined-builtin
+                if shared.opts.diffusers_dir in dir:
+                    subdirs[os.path.basename(shared.opts.diffusers_dir)] = 1
+                if 'models--' in dir:
+                    continue
+                subdir = dir[len(parentdir):].replace("\\", "/")
+                while subdir.startswith("/"):
+                    subdir = subdir[1:]
+                if not self.is_empty(dir):
+                    subdirs[subdir] = 1
         if subdirs:
             subdirs = OrderedDict(sorted(subdirs.items()))
             subdirs = {"": 1, **subdirs}
@@ -189,10 +187,13 @@ class ExtraNetworksPage:
                 self.items = []
                 shared.log.error(f'Extra networks error listing items: {self.__class__}')
             self.create_xyz_grid()
-            for item in self.items:
+            htmls = []
+            items = self.items
+            for item in items:
                 self.metadata[item["name"]] = item.get("metadata", {})
                 self.info[item["name"]] = self.find_info(item['filename'])
-                self.html += self.create_html_for_item(item, tabname)
+                htmls.append(self.create_html_for_item(item, tabname))
+            self.html += ''.join(htmls)
             if len(subdirs_html) > 0 or len(self.html) > 0:
                 res = f"<div id='{tabname}_{self_name_id}_subdirs' class='extra-network-subdirs'>{subdirs_html}</div><div id='{tabname}_{self_name_id}_cards' class='extra-network-cards'>{self.html}</div>"
             else:
@@ -201,7 +202,7 @@ class ExtraNetworksPage:
             threading.Thread(target=self.create_thumb).start()
             return res
         except Exception as e:
-            shared.log.error(f'Extra networks page error: {e}')
+            shared.log.error(f'Extra networks {self.title} {tabname} page error: {e.__class__.__name__} -> {e}')
             return f"<div id='{tabname}_{self_name_id}_subdirs' class='extra-network-subdirs'></div><div id='{tabname}_{self_name_id}_cards' class='extra-network-cards'>Extra network error<br>{e}</div>"
 
     def list_items(self):
@@ -238,7 +239,7 @@ class ExtraNetworksPage:
                 args['title'] += f'\nAlias: {item["alias"]}'
             if item.get("tags", None) is not None:
                 args['title'] += f'\nTags: {", ".join(tags)}'
-            self.card.format(**args)
+            #self.card.format(**args)
             return self.card.format(**args)
         except Exception as e:
             shared.log.error(f'Extra networks item error: page={tabname} item={item["name"]} {e}')
@@ -246,36 +247,44 @@ class ExtraNetworksPage:
 
     def find_preview(self, path):
         preview_extensions = ["jpg", "jpeg", "png", "webp", "tiff", "jp2"]
-        for file in sum([[f'{path}.thumb.{ext}'] for ext in preview_extensions], []): # use thumbnail if exists
-            if os.path.isfile(file):
+        dir = os.path.dirname(path) # pylint: disable=redefined-builtin
+        paths = modelloader.directory_directories(dir, recursive=False)
+        for file in [f'{path}.thumb.{ext}' for ext in preview_extensions]: # use thumbnail if exists
+            if file in paths[dir][1] and os.path.exists(file):
                 return self.link_preview(file)
-        for file in sum([[f'{path}.preview.{ext}', f'{path}.{ext}'] for ext in preview_extensions], []):
-            if os.path.isfile(file):
+        for file in [f'{path}{mid}{ext}' for ext in preview_extensions for mid in ['.preview.', '.']]:
+            if file in paths[dir][1] and os.path.exists(file):
                 self.missing_thumbs.append(file)
                 return self.link_preview(file)
         return self.link_preview('html/card-no-preview.png')
 
     def find_description(self, path):
+        dir = os.path.dirname(path) # pylint: disable=redefined-builtin
+        paths = modelloader.directory_directories(dir, recursive=False)
         for file in [f"{path}.txt", f"{path}.description.txt"]:
-            try:
-                with open(file, "r", encoding="utf-8", errors="replace") as f:
-                    txt = f.read()
-                    txt = re.sub('[<>]', '', txt)
-                    return txt
-            except OSError:
-                pass
+            if file in paths[dir][1]:
+                try:
+                    with open(file, "r", encoding="utf-8", errors="replace") as f:
+                        txt = f.read()
+                        txt = re.sub('[<>]', '', txt)
+                        return txt
+                except OSError:
+                    pass
         return None
 
     def find_info(self, path):
+        dir = os.path.dirname(path) # pylint: disable=redefined-builtin
+        paths = modelloader.directory_directories(dir, recursive=False)
         basename, _ext = os.path.splitext(path)
         for file in [f"{path}.info", f"{path}.civitai.info", f"{basename}.info", f"{basename}.civitai.info"]:
-            try:
-                with open(file, "r", encoding="utf-8", errors="replace") as f:
-                    txt = f.read()
-                    txt = re.sub('[<>]', '', txt)
-                    return txt
-            except OSError:
-                pass
+            if file in paths[dir][1]:
+                try:
+                    with open(file, "r", encoding="utf-8", errors="replace") as f:
+                        txt = f.read()
+                        txt = re.sub('[<>]', '', txt)
+                        return txt
+                except OSError:
+                    pass
         return None
 
 
@@ -339,6 +348,7 @@ def create_ui(container, button, tabname, skip_indexing = False):
         ui.description_target_filename = gr.Textbox('Description save filename', elem_id=tabname+"_description_filename", visible=False)
 
         for page in ui.stored_extra_pages:
+            shared.log.debug(f"Create UI Extra Network Page: {page.title}")
             page_html = page.create_html(ui.tabname, skip_indexing)
             with gr.Tab(page.title, id=page.title.lower().replace(" ", "_"), elem_classes="extra-networks-tab"):
                 page_elem = gr.HTML(page_html, elem_id=tabname+page.name+"_extra_page", elem_classes="extra-networks-page")
@@ -354,6 +364,7 @@ def create_ui(container, button, tabname, skip_indexing = False):
     button_close.click(fn=toggle_visibility, inputs=[state_visible], outputs=[state_visible, container])
 
     def refresh():
+        shared.log.debug("Refreshing UI Extra Networks Pages")
         res = []
         for pg in ui.stored_extra_pages:
             pg.html = ''

modules/ui_extra_networks_checkpoints.py

@@ -16,6 +16,8 @@ class ExtraNetworksPageCheckpoints(ui_extra_networks.ExtraNetworksPage):
         checkpoint: sd_models.CheckpointInfo
         for name, checkpoint in sd_models.checkpoints_list.items():
             path, _ext = os.path.splitext(checkpoint.filename)
+            if not os.path.exists(path) and sd_models.model_path not in path:
+                path = os.path.abspath(os.path.join(checkpoint.path, os.pardir, os.pardir))
             yield {
                 "name": checkpoint.name_for_extra,
                 "filename": path,

requirements.txt

@@ -46,7 +46,7 @@ requests==2.31.0
 tqdm==4.65.0
 accelerate==0.20.3
 opencv-python-headless==4.7.0.72
-diffusers==0.19.3
+diffusers==0.20.0
 einops==0.4.1
 gradio==3.32.0
 huggingface_hub==0.16.4