add resize modes to control plus simple outpaint

CHANGELOG.md

@@ -2,13 +2,12 @@
 
 ## TODO
 
-- resize type: strech, fill/color, edge, etc.
 - reference styles
 - quick apply style
 
-## Update for 2024-03-16
+## Update for 2024-03-17
 
-### Highlights 2024-03-16
+### Highlights 2024-03-17
 
 New models:
 - [Stable Cascade](https://github.com/Stability-AI/StableCascade) *Full* and *Lite*
@@ -16,17 +15,24 @@ New models:
 - [KOALA 700M](https://github.com/youngwanLEE/sdxl-koala)
 - [Stable Video Diffusion XT 1.1](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt-1-1)
 - [VGen](https://huggingface.co/ali-vilab/i2vgen-xl)  
+
 New pipelines and features:
 - Trajectory Consistency Distillation [TCD](https://mhh0318.github.io/tcd) for processing in even less steps
 - Img2img using [LEdit++](https://leditsplusplus-project.static.hf.space/index.html), context aware method with image analysis and positive/negative prompt handling
 - Visual Query & Answer using [moondream2](https://github.com/vikhyat/moondream) as an addition to standard interrogate methods
-- Face-HiRes: simple detailer for face refinements
+- Face-HiRes: simple built-in detailer for face refinements
+- Even simpler outpaint: when resizing image, simply pick outpaint method and if image has different aspect ratio, blank areas will be outpainted!
 - UI aspect-ratio controls and other UI improvements
 - User controllable invisibile and visible watermarking
 - Native composable LoRA
-**Styles**: Not just for prompts! Styles can apply *generate parameters* as templates and can be used to *apply wildcards* to prompts  
-**Reference models**: *Networks -> Models -> Reference*: All reference models now come with recommended settings that can be auto-applied if desired  
-Additional Improvements such as: Smooth tiling, Refine/HiRes workflow improvements, Control workflow improvements, Additional API endpoints  
+
+What else?
+
+- **Reference models**: *Networks -> Models -> Reference*: All reference models now come with recommended settings that can be auto-applied if desired  
+- **Styles**: Not just for prompts! Styles can apply *generate parameters* as templates and can be used to *apply wildcards* to prompts  
+improvements, Additional API endpoints  
+- Given the high interest in [ZLUDA](https://github.com/vosen/ZLUDA) engine introduced in last release we've updated much more flexible/automatic install procedure (see [wiki](https://github.com/vladmandic/automatic/wiki/ZLUDA) for details)  
+- Plus Additional Improvements such as: Smooth tiling, Refine/HiRes workflow improvements, Control workflow 
 
 Further details:  
 - For basic instructions, see [README](https://github.com/vladmandic/automatic/blob/master/README.md)  
@@ -34,7 +40,7 @@ Further details:
 - For documentation, see [WiKi](https://github.com/vladmandic/automatic/wiki)
 - [Discord](https://discord.com/invite/sd-next-federal-batch-inspectors-1101998836328697867) server  
 
-### Full Changelog 2024-03-16
+### Full Changelog 2024-03-17
 
 - [Stable Cascade](https://github.com/Stability-AI/StableCascade) *Full* and *Lite*
   - large multi-stage high-quality model from warp-ai/wuerstchen team and released by stabilityai  
@@ -84,6 +90,7 @@ Further details:
   - *note*: this is a very experimental feature and may not work as expected
 - **Control**
   - added *refiner/hires* workflows
+  - added resize methods to before/after/mask: fixed, crop, fill
 - **Samplers**
   - [TCD](https://mhh0318.github.io/tcd/): Trajectory Consistency Distillation  
     new sampler that produces consistent results in a very low number of steps (comparable to LCM but without reliance on LoRA)  

modules/images.py

@@ -123,13 +123,9 @@ class GridAnnotation:
 
 def get_font(fontsize):
     try:
-        return ImageFont.truetype(
-            shared.opts.font or "javascript/notosans-nerdfont-regular.ttf", fontsize
-        )
+        return ImageFont.truetype(shared.opts.font or "javascript/notosans-nerdfont-regular.ttf", fontsize)
     except Exception:
-        return ImageFont.truetype(
-            "javascript/notosans-nerdfont-regular.ttf", fontsize
-        )
+        return ImageFont.truetype("javascript/notosans-nerdfont-regular.ttf", fontsize)
 
 
 def draw_grid_annotations(im, width, height, hor_texts, ver_texts, margin=0, title=None):
@@ -216,27 +212,15 @@ def draw_prompt_matrix(im, width, height, all_prompts, margin=0):
 
 
 def resize_image(resize_mode, im, width, height, upscaler_name=None, output_type='image'):
-    shared.log.debug(f'Image resize: input={im} mode={resize_mode} target={width}x{height} upscaler={upscaler_name} fn={sys._getframe(1).f_code.co_name}') # pylint: disable=protected-access
-    """
-    Resizes an image with the specified resize_mode, width, and height.
-    Args:
-        resize_mode: The mode to use when resizing the image.
-            0: No resize
-            1: Resize the image to the specified width and height.
-            2: Resize the image to fill the specified width and height, maintaining the aspect ratio, and then center the image within the dimensions, cropping the excess.
-            3: Resize the image to fit within the specified width and height, maintaining the aspect ratio, and then center the image within the dimensions, filling empty with data from image.
-        im: The image to resize.
-        width: The width to resize the image to.
-        height: The height to resize the image to.
-        upscaler_name: The name of the upscaler to use. If not provided, defaults to opts.upscaler_for_img2img.
-    """
+    if im.width == width and im.height == height:
+        shared.log.debug(f'Image resize: input={im} target={width}x{height} mode={shared.resize_modes[resize_mode]} upscaler="{upscaler_name}" fn={sys._getframe(1).f_code.co_name}') # pylint: disable=protected-access
     upscaler_name = upscaler_name or shared.opts.upscaler_for_img2img
 
     def latent(im, w, h, upscaler):
         from modules.processing_vae import vae_encode, vae_decode
         import torch
         latents = vae_encode(im, shared.sd_model, full_quality=False) # TODO enable full VAE mode
-        latents = torch.nn.functional.interpolate(latents, size=(h // 8, w // 8), mode=upscaler["mode"], antialias=upscaler["antialias"])
+        latents = torch.nn.functional.interpolate(latents, size=(int(h // 8), int(w // 8)), mode=upscaler["mode"], antialias=upscaler["antialias"])
         im = vae_decode(latents, shared.sd_model, output_type='pil', full_quality=False)[0]
         return im
 
@@ -260,11 +244,7 @@ def resize_image(resize_mode, im, width, height, upscaler_name=None, output_type
             im = im.resize((w, h), resample=Image.Resampling.LANCZOS)
         return im
 
-    if resize_mode == 0 or (im.width == width and im.height == height):
-        res = im.copy()
-    elif resize_mode == 1:
-        res = resize(im, width, height)
-    elif resize_mode == 2:
+    def crop(im):
         ratio = width / height
         src_ratio = im.width / im.height
         src_w = width if ratio > src_ratio else im.width * height // im.height
@@ -272,7 +252,11 @@ def resize_image(resize_mode, im, width, height, upscaler_name=None, output_type
         resized = resize(im, src_w, src_h)
         res = Image.new(im.mode, (width, height))
         res.paste(resized, box=(width // 2 - src_w // 2, height // 2 - src_h // 2))
-    else:
+        return res
+
+    def fill(im, color=None):
+        color = color or shared.opts.image_background
+        """
         ratio = round(width / height, 1)
         src_ratio = round(im.width / im.height, 1)
         src_w = width if ratio < src_ratio else im.width * height // im.height
@@ -290,6 +274,27 @@ def resize_image(resize_mode, im, width, height, upscaler_name=None, output_type
             if height > 0 and fill_width > 0:
                 res.paste(resized.resize((fill_width, height), box=(0, 0, 0, height)), box=(0, 0))
                 res.paste(resized.resize((fill_width, height), box=(resized.width, 0, resized.width, height)), box=(fill_width + src_w, 0))
+        return res
+        """
+        ratio = min(width / im.width, height / im.height)
+        im = resize(im, im.width * ratio, im.height * ratio)
+        res = Image.new(im.mode, (width, height), color=color)
+        res.paste(im, box=((width - im.width)//2, (height - im.height)//2))
+        return res
+
+    if resize_mode == 0 or (im.width == width and im.height == height): # none
+        res = im.copy()
+    elif resize_mode == 1: # fixed
+        res = resize(im, width, height)
+    elif resize_mode == 2: # crop
+        res = crop(im)
+    elif resize_mode == 3: # fill
+        res = fill(im)
+    elif resize_mode == 4: # edge
+        from modules import masking
+        res = fill(im, color=0)
+        res, _mask = masking.outpaint(res)
+        res.save('/tmp/edge.png')
     if output_type == 'np':
         return np.array(res)
     return res

modules/processing_class.py

@@ -405,9 +405,8 @@ class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
             image = images.flatten(img, shared.opts.img2img_background_color)
             if self.width is None or self.height is None:
                 self.width, self.height = image.width, image.height
-            if crop_region is None and self.resize_mode != 4 and self.resize_mode > 0:
-                if image.width != self.width or image.height != self.height:
-                    image = images.resize_image(self.resize_mode, image, self.width, self.height, self.resize_name)
+            if crop_region is None and self.resize_mode > 0:
+                image = images.resize_image(self.resize_mode, image, self.width, self.height, self.resize_name)
                 self.width = image.width
                 self.height = image.height
             if self.image_mask is not None and shared.opts.mask_apply_overlay:
@@ -445,8 +444,6 @@ class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
             image = 2. * image - 1.
             image = image.to(device=shared.device, dtype=devices.dtype_vae)
             self.init_latent = self.sd_model.get_first_stage_encoding(self.sd_model.encode_first_stage(image))
-            if self.resize_mode == 4:
-                self.init_latent = torch.nn.functional.interpolate(self.init_latent, size=(self.height // 8, self.width // 8), mode="bilinear")
             if self.image_mask is not None:
                 init_mask = latent_mask
                 latmask = init_mask.convert('RGB').resize((self.init_latent.shape[3], self.init_latent.shape[2]))

modules/shared.py

@@ -74,7 +74,7 @@ restricted_opts = {
     "outdir_save",
     "outdir_init_images"
 }
-resize_modes = ["None", "Fixed", "Crop", "Fill", "Latent"]
+resize_modes = ["None", "Fixed", "Crop", "Fill", "Outpaint"]
 compatibility_opts = ['clip_skip', 'uni_pc_lower_order_final', 'uni_pc_order']
 console = Console(log_time=True, log_time_format='%H:%M:%S-%f')
 dir_timestamps = {}
@@ -515,6 +515,7 @@ options_templates.update(options_section(('saving-images', "Image Options"), {
     "save_selected_only": OptionInfo(True, "Save only saves selected image"),
     "include_mask": OptionInfo(False, "Include mask in outputs"),
     "samples_save_zip": OptionInfo(True, "Create ZIP archive"),
+    "image_background": OptionInfo("#000000", "Resize background color", gr.ColorPicker, {}),
 
     "image_sep_metadata": OptionInfo("<h2>Metadata/Logging</h2>", "", gr.HTML),
     "image_metadata": OptionInfo(True, "Include metadata"),
@@ -524,7 +525,7 @@ options_templates.update(options_section(('saving-images', "Image Options"), {
     "grid_save": OptionInfo(True, "Save all generated image grids"),
     "grid_format": OptionInfo('jpg', 'File format', gr.Dropdown, {"choices": ["jpg", "png", "webp", "tiff", "jp2"]}),
     "n_rows": OptionInfo(-1, "Row count", gr.Slider, {"minimum": -1, "maximum": 16, "step": 1}),
-    "grid_background": OptionInfo("#000000", "Background color", gr.ColorPicker, {}),
+    "grid_background": OptionInfo("#000000", "Grid background color", gr.ColorPicker, {}),
     "font": OptionInfo("", "Font file"),
     "font_color": OptionInfo("#FFFFFF", "Font color", gr.ColorPicker, {}),
 

modules/ui_control.py

@@ -95,11 +95,11 @@ def create_ui(_blocks: gr.Blocks=None):
                 with gr.Accordion(open=False, label="Size", elem_id="control_size", elem_classes=["small-accordion"]):
                     with gr.Tabs():
                         with gr.Tab('Before'):
-                            resize_mode_before, resize_name_before, width_before, height_before, scale_by_before, selected_scale_tab_before = ui_sections.create_resize_inputs('control', [], scale_visible=False, mode='Fixed', accordion=False, latent=True)
+                            resize_mode_before, resize_name_before, width_before, height_before, scale_by_before, selected_scale_tab_before = ui_sections.create_resize_inputs('control', [], accordion=False, latent=True)
                         with gr.Tab('After'):
-                            resize_mode_after, resize_name_after, width_after, height_after, scale_by_after, selected_scale_tab_after = ui_sections.create_resize_inputs('control', [], scale_visible=False, mode='Fixed', accordion=False, latent=False)
+                            resize_mode_after, resize_name_after, width_after, height_after, scale_by_after, selected_scale_tab_after = ui_sections.create_resize_inputs('control', [], accordion=False, latent=False)
                         with gr.Tab('Mask'):
-                            resize_mode_mask, resize_name_mask, width_mask, height_mask, scale_by_mask, selected_scale_tab_mask = ui_sections.create_resize_inputs('control', [], scale_visible=False, mode='Fixed', accordion=False, latent=False)
+                            resize_mode_mask, resize_name_mask, width_mask, height_mask, scale_by_mask, selected_scale_tab_mask = ui_sections.create_resize_inputs('control', [], accordion=False, latent=False)
 
                 with gr.Accordion(open=False, label="Sampler", elem_id="control_sampler", elem_classes=["small-accordion"]):
                     sd_samplers.set_samplers()

modules/ui_img2img.py

@@ -121,7 +121,7 @@ def create_ui():
                 with gr.Group(elem_classes="settings-accordion"):
 
                     steps, sampler_index = ui_sections.create_sampler_inputs('img2img')
-                    resize_mode, resize_name, width, height, scale_by, selected_scale_tab = ui_sections.create_resize_inputs('img2img', [init_img, sketch])
+                    resize_mode, resize_name, width, height, scale_by, selected_scale_tab = ui_sections.create_resize_inputs('img2img', [init_img, sketch], latent=True)
                     batch_count, batch_size = ui_sections.create_batch_inputs('img2img')
                     seed, reuse_seed, subseed, reuse_subseed, subseed_strength, seed_resize_from_h, seed_resize_from_w = ui_sections.create_seed_inputs('img2img')
 

modules/ui_sections.py

@@ -229,7 +229,7 @@ def create_hires_inputs(tab):
                 denoising_strength = gr.Slider(minimum=0.0, maximum=0.99, step=0.01, label='Strength', value=0.3, elem_id=f"{tab}_denoising_strength")
         with gr.Group(visible=shared.backend == shared.Backend.DIFFUSERS):
             with gr.Row(elem_id=f"{tab}_refiner_row1", variant="compact"):
-                refiner_start = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label='Refiner start', value=0.8, elem_id=f"{tab}_refiner_start")
+                refiner_start = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label='Refiner start', value=0.0, elem_id=f"{tab}_refiner_start")
                 refiner_steps = gr.Slider(minimum=0, maximum=99, step=1, label="Refiner steps", elem_id=f"{tab}_refiner_steps", value=10)
             with gr.Row(elem_id=f"{tab}_refiner_row3", variant="compact"):
                 refiner_prompt = gr.Textbox(value='', label='Secondary prompt', elem_id=f"{tab}_refiner_prompt")
@@ -238,23 +238,14 @@ def create_hires_inputs(tab):
     return enable_hr, hr_sampler_index, denoising_strength, hr_upscaler, hr_force, hr_second_pass_steps, hr_scale, hr_resize_x, hr_resize_y, refiner_steps, refiner_start, refiner_prompt, refiner_negative
 
 
-def create_resize_inputs(tab, images, scale_visible=True, mode=None, accordion=True, latent=False):
-    def resize_from_to_html(width, height, scale_by):
-        target_width = int(width * scale_by)
-        target_height = int(height * scale_by)
-        if not target_width or not target_height:
-            return "Hires resize: no image selected"
-        return f"Hires resize: from <span class='resolution'>{width}x{height}</span> to <span class='resolution'>{target_width}x{target_height}</span>"
-
+def create_resize_inputs(tab, images, accordion=True, latent=False):
     dummy_component = gr.Number(visible=False, value=0)
     with gr.Accordion(open=False, label="Resize", elem_classes=["small-accordion"], elem_id=f"{tab}_resize_group") if accordion else gr.Group():
+        # with gr.Row():
+        #    resize_mode = gr.Radio(label="Mode", elem_id=f"{tab}_resize_mode", choices=shared.resize_modes, type="index", value='Fixed')
         with gr.Row():
-            if mode is not None:
-                resize_mode = gr.Radio(label="Resize mode", elem_id=f"{tab}_resize_mode", choices=shared.resize_modes, type="index", value=mode, visible=False)
-            else:
-                resize_mode = gr.Radio(label="Resize mode", elem_id=f"{tab}_resize_mode", choices=shared.resize_modes, type="index", value='None')
-        with gr.Row():
-            resize_name = gr.Dropdown(label="Resize method", elem_id=f"{tab}_resize_name", choices=([] if not latent else list(shared.latent_upscale_modes)) + [x.name for x in shared.sd_upscalers], value=shared.latent_upscale_default_mode)
+            resize_mode = gr.Dropdown(label="Mode", elem_id=f"{tab}_resize_mode", choices=shared.resize_modes, type="index", value='Fixed')
+            resize_name = gr.Dropdown(label="Method", elem_id=f"{tab}_resize_name", choices=([] if not latent else list(shared.latent_upscale_modes)) + [x.name for x in shared.sd_upscalers], value=shared.latent_upscale_default_mode)
             ui_common.create_refresh_button(resize_name, modelloader.load_upscalers, lambda: {"choices": modelloader.load_upscalers()}, 'refresh_upscalers')
 
         with gr.Row(visible=True) as _resize_group:
@@ -275,21 +266,10 @@ def create_resize_inputs(tab, images, scale_visible=True, mode=None, accordion=T
                                     res_switch_btn.click(lambda w, h: (h, w), inputs=[width, height], outputs=[width, height], show_progress=False)
                                     detect_image_size_btn = ToolButton(value=ui_symbols.detect, elem_id=f"{tab}_detect_image_size_btn")
                                     detect_image_size_btn.click(fn=lambda w, h, _: (w or gr.update(), h or gr.update()), _js=f'currentImageResolution{tab}', inputs=[dummy_component, dummy_component, dummy_component], outputs=[width, height], show_progress=False)
-
                     with gr.Tab(label="Scale") as tab_scale_by:
                         scale_by = gr.Slider(minimum=0.05, maximum=8.0, step=0.05, label="Scale", value=1.0, elem_id=f"{tab}_scale")
-                        if scale_visible:
-                            with gr.Row():
-                                scale_by_html = gr.HTML(resize_from_to_html(0, 0, 0.0), elem_id=f"{tab}_scale_resolution_preview")
-                                gr.Slider(label="Unused", elem_id=f"{tab}_unused_scale_by_slider")
-                                button_update_resize_to = gr.Button(visible=False, elem_id=f"{tab}_update_resize_to")
-                            on_change_args = dict(fn=resize_from_to_html, _js=f'currentImageResolution{tab}', inputs=[dummy_component, dummy_component, scale_by], outputs=scale_by_html, show_progress=False)
-                            scale_by.release(**on_change_args)
-                            button_update_resize_to.click(**on_change_args)
-
                     for component in images:
                         component.change(fn=lambda: None, _js="updateImg2imgResizeToTextAfterChangingImage", inputs=[], outputs=[], show_progress=False)
-
             tab_scale_to.select(fn=lambda: 0, inputs=[], outputs=[selected_scale_tab])
             tab_scale_by.select(fn=lambda: 1, inputs=[], outputs=[selected_scale_tab])
             # resize_mode.change(fn=lambda x: gr.update(visible=x != 0), inputs=[resize_mode], outputs=[_resize_group])