redo diffusers scheduler

DIFFUSERS.md

@@ -60,6 +60,7 @@ whats implemented so far?
 
 - sdxl model  
 - new schedulers
+- settings -> schedulers
 
 ## Limitations
 
@@ -75,9 +76,7 @@ will need to handle in the code before we get out of alpha
 
 ## Issues
 
-- default model download ckpt vs hfhub?
 - new dependency hell (not diffuser related)?
-- extra networks ui auto-hide and transitions
 
 ## Notes for HF
 
@@ -90,7 +89,7 @@ will need to handle in the code before we get out of alpha
 - redone **lora** support, core is now in `modules/lora_diffusers.py`  
 - added support for diffuser models in **safetensors/ckpt** format  
 - when i use `diffusers.StableDiffusionPipeline.from_ckpt`  
-  first time it downloads something - what is that?
+  first time it downloads something - what is that? (could it be a default safety checker?)
   > Downloading (…)lve/main/config.json: 4.55k  
   > Downloading pytorch_model.bin: 1.22G  
 - loading safetensors model is very slow  
@@ -111,10 +110,19 @@ will need to handle in the code before we get out of alpha
   > global_step key not found in model  
   > Checkpoint /home/vlado/dev/automatic/models/Stable-diffusion/best/absolutereality_v1.safetensors has both EMA and non-EMA weights.  
   > In this conversion only the EMA weights are extracted. If you want to instead extract the non-EMA weights (useful to continue fine-tuning), please make sure to remove the `--extract_ema` flag.  
+- do you have plans to implement [Restart](https://github.com/vladmandic/automatic/issues/1537) sampler in diffusers?
+- scheduler config is really difficult to work with as its not possible to see which params each scheduler defines ahead of time and if passing params it doesn't have, it will result in runtime error
 
 ## Update
 
-- sortable models table in downloader ui
-- recommended scheduler: `deis`
-- `channels_last` and `cudnn_benchmark` now apply to diffusers
-- new settings section for diffusers fine-tuning
+- sortable models table in downloader ui  
+- system info tab -> benchmark is now working  
+- recommended scheduler: `deis`  
+- `channels_last` and `cudnn_benchmark` now apply to diffusers  
+- new settings section for diffusers fine-tuning  
+- fixed missed call to `devices.set_cuda_params`
+- had to reduce number of supported schedulers by a lot until i add param checking for the rest  
+  issue is that diffusers have completely different params for schedulers than a111, but passing unknown param causes runtime error
+  previously params were not passed at all, so you couldn't even use anything other than default scheduler (although ui showed you were)
+- fixed "it looks like the config file at 'xxx.safetensors' is not a valid JSON file"  
+  this is also related to schedulers as diffusers are trying to read default scheduler config from model itself, but that doesn't exist for safetensors  

html/locale_en.json

@@ -146,7 +146,7 @@
   {"id":"","label":"System Paths","localized":"","hint":""},
   {"id":"","label":"Image Options","localized":"","hint":""},
   {"id":"","label":"Image Processing","localized":"","hint":""},
-  {"id":"","label":"Output Paths","localized":"","hint":""},
+  {"id":"","label":"Image Paths","localized":"","hint":""},
   {"id":"","label":"User Interface","localized":"","hint":""},
   {"id":"","label":"Live Previews","localized":"","hint":""},
   {"id":"","label":"Sampler Settings","localized":"","hint":""},

javascript/style.css

@@ -100,7 +100,8 @@ button.custom-button{
 }
 
 #txt2img_gallery img, #img2img_gallery img, #extras_gallery img { object-fit: scale-down; width: -webkit-fill-available !important; }
-#txt2img_footer, #img2img_footer, #extras_footer{ height: fit-content; display: none; }
+#txt2img_footer, #img2img_footer, #extras_footer { height: fit-content; }
+#txt2img_footer, #img2img_footer { height: fit-content; display: none; }
 #txt2img_generate_box, #img2img_generate_box { gap: 0.5em; flex-wrap: wrap-reverse; }
 #txt2img_actions_column, #img2img_actions_column { gap: 0.5em; }
 #txt2img_generate_box > button, #img2img_generate_box > button { height: 2.2em; line-height: 0; }
@@ -538,11 +539,11 @@ table.settings-value-table td{
 .extra-networks .description { margin-top: 8px; }
 .extra-networks .tab-nav > button { margin-right: 0; height: auto; padding: 2px 4px 2px 4px; }
 .extra-networks-tab { padding: 0 !important; }
-.extra-network-subdirs { background: var(--input-background-fill); }
+.extra-network-subdirs { background: var(--input-background-fill); overflow-x: hidden; overflow-y: auto; max-height: 50vh; }
 .extra-networks-page { display: flex }
 .extra-networks .custom-button { min-width: 80px; max-width: 240px; width: 100%; background: none; justify-content: left; text-align: left; padding: 2px 8px 2px 8px; box-shadow: none; }
 .extra-networks .custom-button:hover { background: var(--button-primary-background-fill) }
-.extra-network-cards { display: flex; flex-wrap: wrap; height: 50vh; overflow-y: scroll; overflow-x: hidden; width: -webkit-fill-available; }
+.extra-network-cards { display: flex; flex-wrap: wrap; height: 50vh; max-height: 50vh; overflow-y: scroll; overflow-x: hidden; width: -webkit-fill-available; }
 .extra-network-cards .card { height: fit-content; margin: 0.5em; position: relative; scroll-snap-align: start; scroll-margin-top: 0; }
 .extra-network-cards .card .overlay { position: absolute; bottom: 0; padding: 0.2em; z-index: 10; width: 100%; background: none; }
 .extra-network-cards .card:hover .overlay { background: rgba(0, 0, 0, 0.40); }

modules/processing.py

@@ -692,11 +692,11 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
                 generator_device = 'cpu' if shared.opts.diffusers_generator_device == "cpu" else shared.device
                 generator = [torch.Generator(generator_device).manual_seed(s) for s in seeds]
                 if shared.sd_model.scheduler.name != p.sampler_name:
+                    #  sd_model.scheduler = diffusers.UniPCMultistepScheduler.from_config(sd_model.scheduler.config)
                     sampler = sd_samplers.all_samplers_map.get(p.sampler_name, None)
                     if sampler is None:
                         sampler = sd_samplers.all_samplers_map.get("UniPC")
-                    scheduler = sampler.constructor(shared.sd_model.sd_checkpoint_info.filename)
-                    shared.sd_model.scheduler = scheduler.sampler # TODO(Patrick): For wrapped pipelines this is currently a no-op
+                    shared.sd_model.scheduler = sd_samplers.create_sampler(sampler.name, shared.sd_model) # TODO(Patrick): For wrapped pipelines this is currently a no-op
 
                 cross_attention_kwargs={}
                 if lora_state['active']:
@@ -723,7 +723,6 @@ def process_images_inner(p: StableDiffusionProcessing) -> Processed:
                 if lora_state['active']:
                     unload_diffusers_lora()
 
-
             else:
                 raise ValueError(f"Unknown backend {backend}")
 

modules/sd_models.py

@@ -541,6 +541,8 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
         "low_cpu_mem_usage": True,
         "torch_dtype": devices.dtype,
         "safety_checker": None,
+        "requires_safety_checker": False,
+        "load_safety_checker": False,
         # "use_safetensors": True,  # TODO(PVP) - we can't enable this for all checkpoints just yet
     }
 
@@ -548,6 +550,7 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
         shared.opts.data['sd_model_checkpoint'] = "runwayml/stable-diffusion-v1-5"
     sd_model = None
     try:
+        devices.set_cuda_params() # todo
         if shared.cmd_opts.ckpt is not None and model_data.initial: # initial load
             model_name = modelloader.find_diffuser(shared.cmd_opts.ckpt)
             if model_name is not None:
@@ -564,12 +567,12 @@ def load_diffuser(checkpoint_info=None, already_loaded_state_dict=None, timer=No
                 sd_model = diffusers.DiffusionPipeline.from_pretrained(checkpoint_info.path, **diffusers_load_config)
             else:
                 diffusers_load_config["local_files_only "] = True
-                diffusers_load_config["extract_ema"] = True
+                diffusers_load_config["extract_ema"] = shared.opts.diffusers_extract_ema
                 sd_model = diffusers.StableDiffusionPipeline.from_ckpt(checkpoint_info.path, **diffusers_load_config)
 
         if "StableDiffusion" in sd_model.__class__.__name__:
-            sd_model.scheduler = diffusers.UniPCMultistepScheduler.from_config(sd_model.scheduler.config)
-            sd_model.scheduler.name = 'UniPC'
+            from modules.sd_samplers import create_sampler
+            create_sampler('UniPC', sd_model)
         elif "Kandinsky" in sd_model.__class__.__name__:
             sd_model.scheduler.name = 'DDIM'
 

modules/sd_samplers.py

@@ -3,14 +3,9 @@ from modules.sd_samplers_common import samples_to_image_grid, sample_to_image #
 from modules.shared import backend, Backend
 
 if backend == Backend.ORIGINAL:
-    all_samplers = [
-        *sd_samplers_kdiffusion.samplers_data_k_diffusion,
-        *sd_samplers_compvis.samplers_data_compvis,
-    ]
+    all_samplers = [*sd_samplers_kdiffusion.samplers_data_k_diffusion, *sd_samplers_compvis.samplers_data_compvis]
 else:
-    all_samplers = [
-        *sd_samplers_diffusers.samplers_data_diffusers,
-    ]
+    all_samplers = [*sd_samplers_diffusers.samplers_data_diffusers]
 all_samplers_map = {x.name: x for x in all_samplers}
 samplers = all_samplers
 samplers_for_img2img = all_samplers
@@ -34,10 +29,12 @@ def create_sampler(name, model):
         sampler = config.constructor(model)
         sampler.config = config
         return sampler
-    else:
-        sampler = config.constructor(model.sd_checkpoint_info.filename)
+    elif backend == Backend.DIFFUSERS:
+        sampler = config.constructor(model)
         model.scheduler = sampler.sampler
         return sampler.sampler
+    else:
+        return None
 
 
 def set_samplers():

modules/sd_samplers_compvis.py

@@ -37,7 +37,6 @@ class VanillaStableDiffusionSampler:
         self.eta = None
         self.config = None
         self.last_latent = None
-
         self.conditioning_key = sd_model.model.conditioning_key
 
     def number_of_needed_noises(self, p): # pylint: disable=unused-argument

modules/sd_samplers_diffusers.py

@@ -13,25 +13,43 @@ from diffusers import (
 )
 from modules import sd_samplers_common
 
+config = {
+    'All': { 'num_train_timesteps': 1000, 'beta_start': 0.0001, 'beta_end': 0.02, 'beta_schedule': 'linear', 'prediction_type': 'epsilon' },
+    'UniPC': { 'solver_order': 2, 'thresholding': False, 'dynamic_thresholding_ratio': 0.995, 'sample_max_value': 1.0, 'predict_x0': 'bh2', 'lower_order_final': True },
+    'DDIM': { 'clip_sample': True, 'set_alpha_to_one': True, 'steps_offset': 0, 'thresholding': False, 'dynamic_thresholding_ratio': 0.995, 'clip_sample_range': 1.0, 'sample_max_value': 1.0, 'timestep_spacing': 'leading', 'rescale_betas_zero_snr': False },
+    'DEIS': { 'solver_order': 2, 'thresholding': False, 'dynamic_thresholding_ratio': 0.995, 'sample_max_value': 1.0, 'algorithm_type': "deis", 'solver_type': "logrho", 'lower_order_final': True },
+    'Euler a': {},
+}
+
 samplers_data_diffusers = [
     sd_samplers_common.SamplerData('UniPC', lambda model: DiffusionSampler('UniPC', UniPCMultistepScheduler, model), [], {}),
     sd_samplers_common.SamplerData('DDIM', lambda model: DiffusionSampler('DDIM', DDIMScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('DDPM', lambda model: DiffusionSampler('DDPM', DDPMScheduler, model), [], {}),
+    # sd_samplers_common.SamplerData('DDPM', lambda model: DiffusionSampler('DDPM', DDPMScheduler, model), [], {}),
     sd_samplers_common.SamplerData('DEIS', lambda model: DiffusionSampler('DEIS', DEISMultistepScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('DPM++ 2M', lambda model: DiffusionSampler('DPM++ 2M', DPMSolverMultistepScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('DPM++ 1S', lambda model: DiffusionSampler('DPM++ 1S', DPMSolverSinglestepScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('DPM++ 2M SDE', lambda model: DiffusionSampler('DPM++ 2M SDE',  DPMSolverMultistepScheduler, model, algorithm_type="sde-dpmsolver++"), [], {}),
-    sd_samplers_common.SamplerData('DPM++ 2M Karras', lambda model: DiffusionSampler('DPM++ 2M Karras', DPMSolverMultistepScheduler, model, use_karras_sigmas=True), [], {}),
-    sd_samplers_common.SamplerData('DPM++ 1S Karras', lambda model: DiffusionSampler('DPM++ 1S Karras', DPMSolverSinglestepScheduler, model, use_karras_sigmas=True), [], {}),
-    sd_samplers_common.SamplerData('DPM++ 2M SDE Karras', lambda model: DiffusionSampler('DPM++ 2M SDE Karras', DPMSolverMultistepScheduler, model, use_karras_sigmas=True, algorithm_type="sde-dpmsolver++"), [], {}),
-    sd_samplers_common.SamplerData('Euler', lambda model: DiffusionSampler('Euler', EulerDiscreteScheduler, model), [], {}),
+    # sd_samplers_common.SamplerData('DPM++ 2M', lambda model: DiffusionSampler('DPM++ 2M', DPMSolverMultistepScheduler, model), [], {}),
+    # sd_samplers_common.SamplerData('DPM++ 1S', lambda model: DiffusionSampler('DPM++ 1S', DPMSolverSinglestepScheduler, model), [], {}),
+    # sd_samplers_common.SamplerData('DPM++ 2M SDE', lambda model: DiffusionSampler('DPM++ 2M SDE', DPMSolverMultistepScheduler, model, algorithm_type="sde-dpmsolver++"), [], {}),
+    # sd_samplers_common.SamplerData('DPM++ 2M Karras', lambda model: DiffusionSampler('DPM++ 2M Karras', DPMSolverMultistepScheduler, model, use_karras_sigmas=True), [], {}),
+    # sd_samplers_common.SamplerData('DPM++ 1S Karras', lambda model: DiffusionSampler('DPM++ 1S Karras', DPMSolverSinglestepScheduler, model, use_karras_sigmas=True), [], {}),
+    # sd_samplers_common.SamplerData('DPM++ 2M SDE Karras', lambda model: DiffusionSampler('DPM++ 2M SDE Karras', DPMSolverMultistepScheduler, model, use_karras_sigmas=True, algorithm_type="sde-dpmsolver++"), [], {}),
+    # sd_samplers_common.SamplerData('Euler', lambda model: DiffusionSampler('Euler', EulerDiscreteScheduler, model), [], {}),
     sd_samplers_common.SamplerData('Euler a', lambda model: DiffusionSampler('Euler a', EulerAncestralDiscreteScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('Heun', lambda model: DiffusionSampler('Heun', HeunDiscreteScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('DPM2++ 2M', lambda model: DiffusionSampler('KDPM2', KDPM2DiscreteScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('PNDM', lambda model: DiffusionSampler('PNDM', PNDMScheduler, model), [], {}),
+    # sd_samplers_common.SamplerData('Heun', lambda model: DiffusionSampler('Heun', HeunDiscreteScheduler, model), [], {}),
+    # sd_samplers_common.SamplerData('DPM2++ 2M', lambda model: DiffusionSampler('KDPM2', KDPM2DiscreteScheduler, model), [], {}),
+    # sd_samplers_common.SamplerData('PNDM', lambda model: DiffusionSampler('PNDM', PNDMScheduler, model), [], {}),
 ]
 
 class DiffusionSampler:
-    def __init__(self, name, constructor, sd_model, **kwargs):
-        self.sampler = constructor.from_pretrained(sd_model, subfolder="scheduler", **kwargs)
+    def __init__(self, name, constructor, model, **kwargs):
+        self.config = config['All'].copy()
+        for key, value in config.get(name, {}).items(): # diffusers defaults
+            if key in self.config:
+                self.config[key] = value
+        for key, value in model.scheduler.config.items(): # model defaults
+            if key in self.config:
+                self.config[key] = value
+        for key, value in kwargs.items(): # user args
+            if key in self.config:
+                self.config[key] = value
+        self.sampler = constructor(**self.config)
         self.sampler.name = name

modules/sd_samplers_diffusors.py

@@ -1,45 +0,0 @@
-from diffusers import (
-    DDIMScheduler,
-    DDPMScheduler,
-    DEISMultistepScheduler,
-    DPMSolverMultistepScheduler,
-    EulerAncestralDiscreteScheduler,
-    EulerDiscreteScheduler,
-    HeunDiscreteScheduler,
-    IPNDMScheduler,
-    KDPM2AncestralDiscreteScheduler,
-    PNDMScheduler,
-    UniPCMultistepScheduler,
-    # KarrasVeScheduler,
-    # RePaintScheduler,
-    # ScoreSdeVeScheduler,
-    # UnCLIPScheduler,
-    # VQDiffusionScheduler,
-)
-from modules import sd_samplers_common
-    # scheduler = diffusers.UniPCMultistepScheduler.from_pretrained(shared.cmd_opts.ckpt, subfolder="scheduler")
-
-samplers_data_diffusors = [
-    sd_samplers_common.SamplerData('UniPC', lambda model: DiffusionSampler('UniPC', UniPCMultistepScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('DDIM', lambda model: DiffusionSampler('DDIM', DDIMScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('DDPMS', lambda model: DiffusionSampler('DDPMS', DDPMScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('DEIS', lambda model: DiffusionSampler('DEIS', DEISMultistepScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('DPMSolver', lambda model: DiffusionSampler('DPMSolver', DPMSolverMultistepScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('Euler', lambda model: DiffusionSampler('Euler', EulerDiscreteScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('EulerAncestral', lambda model: DiffusionSampler('EulerAncestral', EulerAncestralDiscreteScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('Heun', lambda model: DiffusionSampler('Heun', HeunDiscreteScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('IPNDM', lambda model: DiffusionSampler('IPNDM', IPNDMScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('KDPM2Ancestral', lambda model: DiffusionSampler('KDPM2Ancestral', KDPM2AncestralDiscreteScheduler, model), [], {}),
-    sd_samplers_common.SamplerData('PNDMS', lambda model: DiffusionSampler('PNDMS', PNDMScheduler, model), [], {}),
-    # sd_samplers_common.SamplerData('KarrasVe', lambda model: DiffusionSampler('KarrasVe', KarrasVeScheduler, model), [], {}),
-    # sd_samplers_common.SamplerData('RePaint', lambda model: DiffusionSampler('RePaint', RePaintScheduler, model), [], {}),
-    # sd_samplers_common.SamplerData('ScoreSdeVe', lambda model: DiffusionSampler('ScoreSdeVe', ScoreSdeVeScheduler, model), [], {}),
-    # sd_samplers_common.SamplerData('UnCLIP', lambda model: DiffusionSampler('UnCLIP', UnCLIPScheduler, model), [], {}),
-    # sd_samplers_common.SamplerData('VQDiffusion', lambda model: DiffusionSampler('VQDiffusion', VQDiffusionScheduler, model), [], {}),
-]
-
-
-class DiffusionSampler:
-    def __init__(self, name, constructor, sd_model):
-        self.sampler = constructor.from_pretrained(sd_model, subfolder="scheduler")
-        self.sampler.name = name

modules/shared.py

@@ -343,6 +343,7 @@ options_templates.update(options_section(('cuda', "Compute Settings"), {
 }))
 
 options_templates.update(options_section(('diffusers', "Diffusers Settings"), {
+    "diffusers_extract_ema": OptionInfo(True, "Use model EMA weights when possible"),
     "diffusers_generator_device": OptionInfo("default", "Generator device", gr.Radio, lambda: {"choices": ["default", "cpu"]}),
     "diffusers_seq_cpu_offload": OptionInfo(False, "Enable sequential CPU offload"),
     "diffusers_model_cpu_offload": OptionInfo(False, "Enable model CPU offload"),
@@ -417,7 +418,7 @@ options_templates.update(options_section(('image-processing', "Image Processing"
 }))
 
 
-options_templates.update(options_section(('saving-paths', "Output Paths"), {
+options_templates.update(options_section(('saving-paths', "Image Paths"), {
     "outdir_samples": OptionInfo("", "Output directory for images", component_args=hide_dirs),
     "outdir_txt2img_samples": OptionInfo("outputs/text", 'Output directory for txt2img images', component_args=hide_dirs),
     "outdir_img2img_samples": OptionInfo("outputs/image", 'Output directory for img2img images', component_args=hide_dirs),

modules/ui_postprocessing.py

@@ -9,18 +9,16 @@ from modules.ui_common import infotext_to_html
 
 def wrap_pnginfo(image):
     _, geninfo, info = run_pnginfo(image)
-    return '', infotext_to_html(geninfo), info, geninfo
+    return infotext_to_html(geninfo), info, geninfo
 
 
 def submit_click(tab_index, extras_image, image_batch, extras_batch_input_dir, extras_batch_output_dir, show_extras_results, *script_inputs):
-
     result_images, geninfo, js_info = postprocessing.run_postprocessing(tab_index, extras_image, image_batch, extras_batch_input_dir, extras_batch_output_dir, show_extras_results, *script_inputs)
     return result_images, geninfo, json.dumps(js_info), ''
 
 
 def create_ui():
     tab_index = gr.State(value=0) # pylint: disable=abstract-class-instantiated
-
     with gr.Row().style(equal_height=False, variant='compact'):
         with gr.Column(variant='compact'):
             with gr.Tabs(elem_id="mode_extras"):
@@ -53,11 +51,10 @@ def create_ui():
     tab_single.select(fn=lambda: 0, inputs=[], outputs=[tab_index])
     tab_batch.select(fn=lambda: 1, inputs=[], outputs=[tab_index])
     tab_batch_dir.select(fn=lambda: 2, inputs=[], outputs=[tab_index])
-    _dummy = gr.HTML(visible=False)
     extras_image.change(
         fn=wrap_gradio_call(wrap_pnginfo),
         inputs=[extras_image],
-        outputs=[_dummy, html_info_formatted, exif_info, gen_info],
+        outputs=[html_info_formatted, exif_info, gen_info],
     )
     submit.click(
         _js="submit_postprocessing",