add stable-video-diffusion

CHANGELOG.md

@@ -3,10 +3,6 @@
 ## Update for 2023-11-29
 
 - **Diffusers**
-  - [SDXL Turbo](https://huggingface.co/stabilityai/sdxl-turbo) support  
-    - just set CFG scale (0.0-1.0) and steps (1-3) to a very low value  
-    - compatible with original StabilityAI SDXL-Turbo or any of the newer merges
-    - download safetensors or select from networks -> reference
   - **HDR latent control**, based on [article](https://huggingface.co/blog/TimothyAlexisVass/explaining-the-sdxl-latent-space#long-prompts-at-high-guidance-scales-becoming-possible)  
     - in *Advanced* params
     - allows control of *latent clamping*, *color centering* and *range maximimization*  
@@ -15,20 +11,33 @@
     - lightweight implementation of T2I adapters which can guide generation towards specific image style  
     - supports most T2I models, not limited to SD 1.5  
     - models are auto-downloaded on first use
-    - for IP adapter support in Original backend, use standard *ControlNet* extension  
+    - for IP adapter support in *Original* backend, use standard *ControlNet* extension  
   - **AnimateDiff**
     - lightweight implementation of AnimateDiff basic models *(1.4, 1.5, 1.5.2)*
     - supports SD 1.5 only  
     - models are auto-downloaded on first use  
-    - *note*: AnimateDiff can be combined with IP-Adapter for even better results!  
-    - for AnimateDiff support in Original backend, use standard *AnimateDiff* extension  
+    - can create animated GIF and MP4 video files  
+    - can be combined with IP-Adapter for even better results!  
+    - for AnimateDiff support in *Original* backend, use standard *AnimateDiff* extension  
+  - [SDXL Turbo](https://huggingface.co/stabilityai/sdxl-turbo) support  
+    - just set CFG scale (0.0-1.0) and steps (1-3) to a very low value  
+    - compatible with original StabilityAI SDXL-Turbo or any of the newer merges
+    - download safetensors or select from networks -> reference
+  - [Stable Video Diffusion](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid) and [Stable Video Diffusion XT](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt) support  
+    - download using built-in model downloader or simply select from *networks -> reference*  
+      support for manually downloaded safetensors models will be added later  
+    - go to *image* tab, enter input image and select *script* -> *stable video diffusion*
   - [Kandinsky 3](https://huggingface.co/kandinsky-community/kandinsky-3) support  
-    - download using built-in model downloader or simply select from networks -> reference  
+    - download using built-in model downloader or simply select from *networks -> reference*  
     - this model is absolutely massive at 27.5GB at fp16, so be patient  
     - model params count is at 11.9B (compared to SD-XL at 3.3B) and its trained on mixed resolutions from 256px to 1024px  
     - use either model offload or sequential cpu offload to be able to use it  
   - better autodetection of *inpaint* and *instruct* pipelines  
   - support long seconary prompt for refiner  
+- **Video support**
+  - applies to any model that supports video generation, e.g. AnimateDiff and StableVideoDiffusion  
+  - support for GIF and MP4  
+  - output folder for videos is in *settings -> image paths -> video*  
 - **Model merge**
   - add **SD-XL ReBasin** support, thanks @AI-Casanova  
 - **General**

html/reference.json

@@ -19,6 +19,16 @@
     "desc": "SDXL-Turbo is a distilled version of SDXL 1.0, trained for real-time synthesis. SDXL-Turbo is based on a novel training method called Adversarial Diffusion Distillation (ADD) (see the technical report), which allows sampling large-scale foundational image diffusion models in 1 to 4 steps at high image quality. This approach uses score distillation to leverage large-scale off-the-shelf image diffusion models as a teacher signal and combines this with an adversarial loss to ensure high image fidelity even in the low-step regime of one or two sampling steps.",
     "preview": "stabilityai--sdxl-turbo.jpg"
   },
+  "StabilityAI Stable Video Diffusion": {
+    "path": "stabilityai/stable-video-diffusion-img2vid",
+    "desc": "(SVD) Image-to-Video is a latent diffusion model trained to generate short video clips from an image conditioning. This model was trained to generate 14 frames at resolution 576x1024 given a context frame of the same size. We also finetune the widely used f8-decoder for temporal consistency.",
+    "preview": "stabilityai--stable-video-diffusion-img2vid.jpg"
+  },
+  "StabilityAI Stable Video Diffusion XT": {
+    "path": "stabilityai/stable-video-diffusion-img2vid-xt",
+    "desc": "(SVD) Image-to-Video is a latent diffusion model trained to generate short video clips from an image conditioning. This model was trained to generate 25 frames at resolution 576x1024 given a context frame of the same size, finetuned from SVD Image-to-Video [14 frames]. We also finetune the widely used f8-decoder for temporal consistency.",
+    "preview": "stabilityai--stable-video-diffusion-img2vid-xt.jpg"
+  },
   "Segmind SSD-1B": {
     "path": "segmind/SSD-1B",
     "desc": "The Segmind Stable Diffusion Model (SSD-1B) offers a compact, efficient, and distilled version of the SDXL model. At 50% smaller and 60% faster than Stable Diffusion XL (SDXL), it provides quick and seamless performance without sacrificing image quality.",

modules/images.py

@@ -621,6 +621,51 @@ def save_image(image, path, basename='', seed=None, prompt=None, extension=share
     return params.filename, filename_txt
 
 
+def save_video_atomic(images, video_type, filename, duration, loop):
+    try:
+        import cv2
+    except Exception as e:
+        shared.log.error(f'Save video: cv2: {e}')
+        return
+    os.makedirs(os.path.dirname(filename), exist_ok=True)
+    if video_type == 'mp4':
+        video_frames = [np.array(frame) for frame in images]
+        fourcc = "mp4v"
+        h, w, _c = video_frames[0].shape
+        video_writer = cv2.VideoWriter(filename, fourcc=cv2.VideoWriter_fourcc(*fourcc), fps=len(images)/duration, frameSize=(w, h))
+        for i in range(len(video_frames)):
+            img = cv2.cvtColor(video_frames[i], cv2.COLOR_RGB2BGR)
+            video_writer.write(img)
+        shared.log.info(f'Save video: file="{filename}" frames={len(images)} duration={duration} fourcc={fourcc}')
+    if video_type == 'gif':
+        append = images.copy()
+        image = append.pop(0)
+        if loop:
+            append += append[::-1]
+        frames=len(append) + 1
+        image.save(
+            filename,
+            save_all = True,
+            append_images = append,
+            optimize = False,
+            duration = 1000.0 * duration / frames,
+            loop = 0 if loop else 1,
+        )
+        shared.log.info(f'Save video: file="{filename}" frames={len(append) + 1} duration={duration} loop={loop}')
+
+
+def save_video(p, images, video_type, filename = None, duration = 2, loop = True):
+    if images is None or len(images) < 2:
+        return
+    image = images[0]
+    namegen = FilenameGenerator(p, seed=p.all_seeds[0], prompt=p.all_prompts[0], image=image)
+    if filename is None:
+        filename = namegen.apply(shared.opts.samples_filename_pattern if shared.opts.samples_filename_pattern and len(shared.opts.samples_filename_pattern) > 0 else "[seq]-[prompt_words]")
+        filename = namegen.sanitize(os.path.join(shared.opts.outdir_video, filename))
+        filename = namegen.sequence(filename, shared.opts.outdir_video, '')
+    threading.Thread(target=save_video_atomic, args=(images, video_type, f'{filename}.{video_type}', duration, loop)).start()
+
+
 def safe_decode_string(s: bytes):
     remove_prefix = lambda text, prefix: text[len(prefix):] if text.startswith(prefix) else text # pylint: disable=unnecessary-lambda-assignment
     for encoding in ['utf-8', 'utf-16', 'ascii', 'latin_1', 'cp1252', 'cp437']: # try different encodings

modules/modelloader.py

@@ -205,7 +205,7 @@ def download_diffusers_model(hub_id: str, cache_dir: str = None, download_config
         download_config["mirror"] = mirror
     if custom_pipeline is not None and len(custom_pipeline) > 0:
         download_config["custom_pipeline"] = custom_pipeline
-    shared.log.debug(f"Diffusers downloading: {hub_id} {download_config}")
+    shared.log.debug(f"Diffusers downloading: {hub_id} args={download_config}")
     if token is not None and len(token) > 2:
         shared.log.debug(f"Diffusers authentication: {token}")
         hf.login(token)

modules/processing.py

@@ -784,7 +784,7 @@ def validate_sample(tensor):
     if shared.backend == shared.Backend.ORIGINAL:
         sample = 255.0 * np.moveaxis(tensor.cpu().numpy(), 0, 2)
     else:
-        sample = 255. * tensor
+        sample = 255.0 * tensor
     with warnings.catch_warnings(record=True) as w:
         cast = sample.astype(np.uint8)
     if len(w) > 0:

modules/processing_diffusers.py

@@ -5,7 +5,6 @@ import inspect
 import typing
 import torch
 import torchvision.transforms.functional as TF
-import diffusers
 import modules.devices as devices
 import modules.shared as shared
 import modules.sd_samplers as sd_samplers
@@ -73,10 +72,7 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
                 time.sleep(0.1)
 
     def diffusers_callback(_pipe, step: int, timestep: int, kwargs: dict):
-        latents = correction_callback(p, timestep, kwargs)
-        latents = kwargs['latents']
         shared.state.sampling_step = step
-        shared.state.current_latent = latents
         if shared.state.interrupted or shared.state.skipped:
             raise AssertionError('Interrupted...')
         if shared.state.paused:
@@ -85,7 +81,12 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
                 if shared.state.interrupted or shared.state.skipped:
                     raise AssertionError('Interrupted...')
                 time.sleep(0.1)
-        return {'latents': latents}
+        if kwargs.get('latents', None) is None:
+            print('HERE NO')
+            return kwargs
+        kwargs = correction_callback(p, timestep, kwargs)
+        shared.state.current_latent = kwargs['latents']
+        return kwargs
 
     def full_vae_decode(latents, model):
         t0 = time.time()
@@ -129,7 +130,7 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         if len(latents) == 0:
             return []
         decoded = torch.zeros((len(latents), 3, latents.shape[2] * 8, latents.shape[3] * 8), dtype=devices.dtype_vae, device=devices.device)
-        for i in range(len(output.images)):
+        for i in range(latents.shape[0]):
             decoded[i] = sd_vae_taesd.decode(latents[i])
         return decoded
 
@@ -151,6 +152,8 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         if not hasattr(model, 'vae'):
             shared.log.error('VAE not found in model')
             return []
+        if latents.shape[0] == 4 and latents.shape[1] != 4: # likely animatediff latent
+            latents = latents.permute(1, 0, 2, 3)
         if len(latents.shape) == 3: # lost a batch dim in hires
             latents = latents.unsqueeze(0)
         if full_quality:
@@ -200,16 +203,27 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
 
     def task_specific_kwargs(model):
         task_args = {}
-        is_img2img_model = bool("Zero123" in shared.sd_model.__class__.__name__)
+        is_img2img_model = bool('Zero123' in shared.sd_model.__class__.__name__)
         if sd_models.get_diffusers_task(model) == sd_models.DiffusersTaskType.TEXT_2_IMAGE and not is_img2img_model:
             p.ops.append('txt2img')
-            task_args = {"height": 8 * math.ceil(p.height / 8), "width": 8 * math.ceil(p.width / 8)}
+            task_args = {
+                'height': 8 * math.ceil(p.height / 8),
+                'width': 8 * math.ceil(p.width / 8),
+            }
         elif (sd_models.get_diffusers_task(model) == sd_models.DiffusersTaskType.IMAGE_2_IMAGE or is_img2img_model) and len(getattr(p, 'init_images' ,[])) > 0:
             p.ops.append('img2img')
-            task_args = {"image": p.init_images, "strength": p.denoising_strength}
+            task_args = {
+                'image': p.init_images,
+                'strength': p.denoising_strength,
+            }
         elif sd_models.get_diffusers_task(model) == sd_models.DiffusersTaskType.INSTRUCT and len(getattr(p, 'init_images' ,[])) > 0:
             p.ops.append('instruct')
-            task_args = {"height": 8 * math.ceil(p.height / 8), "width": 8 * math.ceil(p.width / 8), "image": p.init_images, "strength": p.denoising_strength}
+            task_args = {
+                'height': 8 * math.ceil(p.height / 8),
+                'width': 8 * math.ceil(p.width / 8),
+                'image': p.init_images,
+                'strength': p.denoising_strength,
+            }
         elif (sd_models.get_diffusers_task(model) == sd_models.DiffusersTaskType.INPAINTING or is_img2img_model) and len(getattr(p, 'init_images' ,[])) > 0:
             p.ops.append('inpaint')
             if getattr(p, 'mask', None) is None:
@@ -217,7 +231,13 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
             width = 8 * math.ceil(p.init_images[0].width / 8)
             height = 8 * math.ceil(p.init_images[0].height / 8)
             # option-1: use images as inputs
-            task_args = {"image": p.init_images, "mask_image": p.mask, "strength": p.denoising_strength, "height": height, "width": width}
+            task_args = {
+                'image': p.init_images,
+                'mask_image': p.mask,
+                'strength': p.denoising_strength,
+                'height': height,
+                'width': width,
+            }
             """ # option-2: preprocess images into latents using diffusers
             vae_scale_factor = 2 ** (len(model.vae.config.block_out_channels) - 1)
             image_processor = diffusers.image_processor.VaeImageProcessor(vae_scale_factor=vae_scale_factor)
@@ -237,11 +257,16 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
             task_args = {"image": p.init_images, "mask_image": mask_image, "masked_image_latents": masked_image_latents, "strength": p.denoising_strength, "height": height, "width": width}
             """
         if model.__class__.__name__ == 'LatentConsistencyModelPipeline' and hasattr(p, 'init_images') and len(p.init_images) > 0:
+            p.ops.append('lcm')
             init_latents = [vae_encode(image, model=shared.sd_model, full_quality=p.full_quality).squeeze(dim=0) for image in p.init_images]
             init_latent = torch.stack(init_latents, dim=0).to(shared.device)
             init_noise = p.denoising_strength * create_random_tensors(init_latent.shape[1:], seeds=p.all_seeds, subseeds=p.all_subseeds, subseed_strength=p.subseed_strength, p=p)
             init_latent = (1 - p.denoising_strength) * init_latent + init_noise
-            task_args = {"latents": init_latent.to(model.dtype), "width": p.width, "height": p.height }
+            task_args = {
+                'latents': init_latent.to(model.dtype),
+                'width': p.width,
+                'height': p.height,
+            }
         return task_args
 
     def set_pipeline_args(model, prompts: list, negative_prompts: list, prompts_2: typing.Optional[list]=None, negative_prompts_2: typing.Optional[list]=None, desc:str='', **kwargs):
@@ -309,12 +334,13 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
                 pass
         task_kwargs = task_specific_kwargs(model)
         for arg in task_kwargs:
-            if arg in possible and arg not in args: # task specific args should not override args
+            # if arg in possible and arg not in args: # task specific args should not override args
+            if arg in possible:
                 args[arg] = task_kwargs[arg]
-            else:
-                pass
-        for k, v in getattr(p, 'task_args', {}).items():
+        task_args = getattr(p, 'task_args', {})
+        for k, v in task_args.items():
             args[k] = v
+
         hypertile_set(p, hr=len(getattr(p, 'init_images', [])))
         clean = args.copy()
         clean.pop('callback', None)
@@ -394,10 +420,6 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
             # TODO extra_generation_params add sampler options
             # p.extra_generation_params['Sampler options'] = ''
 
-    recompile_model()
-    update_sampler(shared.sd_model)
-    p.extra_generation_params['Pipeline'] = shared.sd_model.__class__.__name__
-
     if len(getattr(p, 'init_images', [])) > 0:
         while len(p.init_images) < len(prompts):
             p.init_images.append(p.init_images[-1])
@@ -475,7 +497,10 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         clip_skip=p.clip_skip,
         desc='Base',
     )
+    recompile_model()
+    update_sampler(shared.sd_model)
     shared.state.sampling_steps = base_args['num_inference_steps']
+    p.extra_generation_params['Pipeline'] = shared.sd_model.__class__.__name__
     p.extra_generation_params["Sampler Eta"] = shared.opts.scheduler_eta if shared.opts.scheduler_eta is not None and shared.opts.scheduler_eta > 0 and shared.opts.scheduler_eta < 1 else None
     try:
         output = shared.sd_model(**base_args) # pylint: disable=not-callable
@@ -486,9 +511,13 @@ def process_diffusers(p: StableDiffusionProcessing, seeds, prompts, negative_pro
         shared.log.info(e)
     except ValueError as e:
         shared.state.interrupted = True
-        shared.log.error(f'Processing: {e}')
+        shared.log.error(f'Processing: args={base_args} {e}')
         if shared.cmd_opts.debug:
             errors.display(e, 'Processing')
+    except RuntimeError as e:
+        shared.state.interrupted = True
+        shared.log.error(f'Processing: args={base_args} {e}')
+        errors.display(e, 'Processing')
 
     if hasattr(shared.sd_model, 'embedding_db') and len(shared.sd_model.embedding_db.embeddings_used) > 0:
         p.extra_generation_params['Embeddings'] = ', '.join(shared.sd_model.embedding_db.embeddings_used)

modules/shared.py

@@ -446,11 +446,12 @@ options_templates.update(options_section(('saving-paths', "Image Naming & Paths"
 
     "outdir_sep_dirs": OptionInfo("<h2>Directories</h2>", "", gr.HTML),
     "outdir_samples": OptionInfo("", "Output directory for images", component_args=hide_dirs, folder=True),
-    "outdir_txt2img_samples": OptionInfo("outputs/text", 'Output directory for txt2img images', component_args=hide_dirs, folder=True),
-    "outdir_img2img_samples": OptionInfo("outputs/image", 'Output directory for img2img images', component_args=hide_dirs, folder=True),
-    "outdir_extras_samples": OptionInfo("outputs/extras", 'Output directory for images from extras tab', component_args=hide_dirs, folder=True),
-    "outdir_save": OptionInfo("outputs/save", "Directory for saving images using the Save button", component_args=hide_dirs, folder=True),
-    "outdir_init_images": OptionInfo("outputs/init-images", "Directory for saving init images when using img2img", component_args=hide_dirs, folder=True),
+    "outdir_txt2img_samples": OptionInfo("outputs/text", 'Directory for text generate', component_args=hide_dirs, folder=True),
+    "outdir_img2img_samples": OptionInfo("outputs/image", 'Directory for image generate', component_args=hide_dirs, folder=True),
+    "outdir_extras_samples": OptionInfo("outputs/extras", 'Directory for processed images', component_args=hide_dirs, folder=True),
+    "outdir_save": OptionInfo("outputs/save", "Directory for manually saved images", component_args=hide_dirs, folder=True),
+    "outdir_video": OptionInfo("outputs/video", "Directory for videos", component_args=hide_dirs, folder=True),
+    "outdir_init_images": OptionInfo("outputs/init-images", "Directory for init images", component_args=hide_dirs, folder=True),
 
     "outdir_sep_grids": OptionInfo("<h2>Grids</h2>", "", gr.HTML),
     "grid_extended_filename": OptionInfo(True, "Add extended info (seed, prompt) to filename when saving grid", gr.Checkbox, {"visible": False}),

modules/ui.py

@@ -421,7 +421,7 @@ def create_ui(startup_timer = None):
                                 cfg_scale = gr.Slider(minimum=0.0, maximum=30.0, step=0.1, label='CFG scale', value=6.0, elem_id="txt2img_cfg_scale")
                                 clip_skip = gr.Slider(label='CLIP skip', value=1, minimum=1, maximum=14, step=1, elem_id='txt2img_clip_skip', interactive=True)
                             with FormRow():
-                                image_cfg_scale = gr.Slider(minimum=1.0, maximum=30.0, step=0.1, label='Secondary CFG scale', value=6.0, elem_id="txt2img_image_cfg_scale")
+                                image_cfg_scale = gr.Slider(minimum=0.0, maximum=30.0, step=0.1, label='Secondary CFG scale', value=6.0, elem_id="txt2img_image_cfg_scale")
                                 diffusers_guidance_rescale = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label='Guidance rescale', value=0.7, elem_id="txt2img_image_cfg_rescale")
                         with gr.Group():
                             with FormRow():
@@ -726,8 +726,8 @@ def create_ui(startup_timer = None):
 
                     with gr.Accordion(open=False, label="Advanced", elem_classes=["small-accordion"], elem_id="img2img_advanced_group"):
                         with FormRow():
-                            cfg_scale = gr.Slider(minimum=1.0, maximum=30.0, step=0.1, label='CFG scale', value=6.0, elem_id="img2img_cfg_scale")
-                            image_cfg_scale = gr.Slider(minimum=1.0, maximum=30.0, step=0.15, label='Image CFG scale', value=1.5, elem_id="img2img_image_cfg_scale")
+                            cfg_scale = gr.Slider(minimum=0.0, maximum=30.0, step=0.1, label='CFG scale', value=6.0, elem_id="img2img_cfg_scale")
+                            image_cfg_scale = gr.Slider(minimum=0.0, maximum=30.0, step=0.15, label='Image CFG scale', value=1.5, elem_id="img2img_image_cfg_scale")
                         with FormRow():
                             clip_skip = gr.Slider(label='CLIP skip', value=1, minimum=1, maximum=4, step=1, elem_id='img2img_clip_skip', interactive=True)
                             diffusers_guidance_rescale = gr.Slider(minimum=0.0, maximum=1.0, step=0.05, label='Guidance rescale', value=0.7, elem_id="txt2img_image_cfg_rescale")

scripts/animatediff.py

@@ -13,7 +13,6 @@ TODO:
 - AnimateFace: https://huggingface.co/nlper2022/animatediff_face_512/tree/main
 """
 
-import os
 import gradio as gr
 import diffusers
 from modules import scripts, processing, shared, devices, sd_models
@@ -66,7 +65,7 @@ def set_adapter(name: str = None):
         shared.log.warning(f'AnimateDiff: unsupported model type: {shared.sd_model.__class__.__name__}')
         return
     if motion_adapter is not None and loaded_adapter == adapter_name:
-        shared.log.info(f'AnimateDiff cache: adapter="{adapter_name}"')
+        shared.log.debug(f'AnimateDiff cache: adapter="{adapter_name}"')
         return
     try:
         shared.log.info(f'AnimateDiff load: adapter="{adapter_name}"')
@@ -117,50 +116,35 @@ class Script(scripts.Script):
                 lora_index = gr.Dropdown(label='Lora', choices=list(LORAS), value='None')
                 strength = gr.Slider(label='Strength', minimum=0.0, maximum=2.0, step=0.05, value=1.0)
             with gr.Row():
-                override = gr.Checkbox(label='Override sampler', value=False)
+                latent_mode = gr.Checkbox(label='Latent mode', value=False)
             with gr.Row():
-                create_gif = gr.Checkbox(label='Create GIF', value=False)
+                create_gif = gr.Checkbox(label='GIF', value=False)
+                create_mp4 = gr.Checkbox(label='MP4', value=False)
                 loop = gr.Checkbox(label='Loop', value=True)
                 duration = gr.Slider(label='Duration', minimum=0.25, maximum=10, step=0.25, value=2)
-        return [adapter_index, frames, lora_index, strength, override, create_gif, duration, loop]
+        return [adapter_index, frames, lora_index, strength, latent_mode, create_gif, create_mp4, duration, loop]
 
-    def process(self, p: processing.StableDiffusionProcessing, adapter_index, frames, lora_index, strength, override, create_gif, duration, loop): # pylint: disable=arguments-differ, unused-argument
+    def process(self, p: processing.StableDiffusionProcessing, adapter_index, frames, lora_index, strength, latent_mode, create_gif, create_mp4, duration, loop): # pylint: disable=arguments-differ, unused-argument
         adapter = ADAPTERS[adapter_index]
         lora = LORAS[lora_index]
         set_adapter(adapter)
         if motion_adapter is None:
             return
-        shared.log.debug(f'AnimateDiff: adapter="{adapter}" lora="{lora}" strength={strength} sampler={override} gif={create_gif}')
-        p.extra_generation_params['AnimateDiff'] = loaded_adapter
-        if override:
-            shared.sd_model.scheduler = diffusers.DDIMScheduler.from_pretrained('SG161222/Realistic_Vision_V5.1_noVAE', subfolder="scheduler", clip_sample=False, timestep_spacing="linspace", steps_offset=1)
+        shared.log.debug(f'AnimateDiff: adapter="{adapter}" lora="{lora}" strength={strength} gif={create_gif} mp4={create_mp4}')
         if lora is not None and lora != 'None':
             shared.sd_model.load_lora_weights(lora, adapter_name=lora)
             shared.sd_model.set_adapters([lora], adapter_weights=[strength])
             p.extra_generation_params['AnimateDiff Lora'] = f'{lora}:{strength}'
+        p.extra_generation_params['AnimateDiff'] = loaded_adapter
         p.do_not_save_grid = True
         p.task_args['num_frames'] = frames
-        p.task_args['output_type'] = 'np' # TODO: AnimateDiff use latents and update vae_decode
         p.task_args['num_inference_steps'] = p.steps
+        if not latent_mode:
+            p.task_args['output_type'] = 'np'
 
-    def postprocess(self, p: processing.StableDiffusionProcessing, processed: processing.Processed, adapter_index, frames, lora_index, strength, override, create_gif, duration, loop): # pylint: disable=arguments-differ, unused-argument
-        if not create_gif or len(processed.images) < 2:
-            return
-        from modules.images import FilenameGenerator
-        image = processed.images[0]
-        namegen = FilenameGenerator(p, seed=p.all_seeds[0], prompt=p.all_prompts[0], image=image)
-        fn = namegen.apply(shared.opts.samples_filename_pattern if shared.opts.samples_filename_pattern and len(shared.opts.samples_filename_pattern) > 0 else "[seq]-[prompt_words]")
-        fn = namegen.sanitize(os.path.join(shared.opts.outdir_save, fn))
-        fn = namegen.sequence(fn, shared.opts.outdir_save, '')
-        images = processed.images[1:]
-        if loop:
-            images += processed.images[::-1]
-        image.save(
-            f'{fn}.gif',
-            save_all = True,
-            append_images = images,
-            optimize = False,
-            duration = 1000.0 * duration / frames,
-            loop = 0 if loop else 1,
-        )
-        shared.log.info(f'AnimateDiff saved: file="{fn}" frames={len(images) + 1} duration={duration} loop={loop}')
+    def postprocess(self, p: processing.StableDiffusionProcessing, processed: processing.Processed, adapter_index, frames, lora_index, strength, latent_mode, create_gif, create_mp4, duration, loop): # pylint: disable=arguments-differ, unused-argument
+        from modules.images import save_video
+        if create_gif:
+            save_video(p, images=processed.images, video_type='gif', duration=duration, loop=loop)
+        if create_mp4:
+            save_video(p, images=processed.images, video_type='mp4', duration=duration, loop=loop)

scripts/stablevideodiffusion.py

@@ -0,0 +1,68 @@
+"""
+Additional params for StableVideoDiffusion
+"""
+import torch
+import gradio as gr
+from modules import scripts, processing, shared, sd_models, images
+
+
+class Script(scripts.Script):
+    def title(self):
+        return 'Stable Video Diffusion'
+
+    def show(self, is_img2img):
+        return is_img2img if shared.backend == shared.Backend.DIFFUSERS else False
+
+    # return signature is array of gradio components
+    def ui(self, _is_img2img):
+        with gr.Row():
+            num_frames = gr.Slider(label='Frames', minimum=1, maximum=50, step=1, value=14)
+            min_guidance_scale = gr.Slider(label='Min guidance', minimum=0.0, maximum=10.0, step=0.1, value=1.0)
+            max_guidance_scale = gr.Slider(label='Max guidance', minimum=0.0, maximum=10.0, step=0.1, value=3.0)
+        with gr.Row():
+            decode_chunk_size = gr.Slider(label='Decode chunks', minimum=1, maximum=25, step=1, value=6)
+            motion_bucket_id = gr.Slider(label='Motion level', minimum=0, maximum=1, step=0.05, value=0.5)
+            noise_aug_strength = gr.Slider(label='Noise strength', minimum=0.0, maximum=1.0, step=0.01, value=0.1)
+        with gr.Row():
+            override_resolution = gr.Checkbox(label='Override resolution', value=True)
+            create_gif = gr.Checkbox(label='GIF', value=False)
+            create_mp4 = gr.Checkbox(label='MP4', value=False)
+            loop = gr.Checkbox(label='Loop', value=True)
+            duration = gr.Slider(label='Duration', minimum=0.25, maximum=10, step=0.25, value=2)
+        return [num_frames, override_resolution, min_guidance_scale, max_guidance_scale, decode_chunk_size, motion_bucket_id, noise_aug_strength, create_gif, create_mp4, duration, loop]
+
+    def run(self, p: processing.StableDiffusionProcessing, num_frames, override_resolution, min_guidance_scale, max_guidance_scale, decode_chunk_size, motion_bucket_id, noise_aug_strength, create_gif, create_mp4, duration, loop): # pylint: disable=arguments-differ, unused-argument
+        if shared.sd_model is None or shared.sd_model.__class__.__name__ != 'StableVideoDiffusionPipeline':
+            return None
+        if hasattr(p, 'init_images') and len(p.init_images) > 0:
+            if override_resolution:
+                p.width = 1024
+                p.height = 576
+                p.task_args['image'] = images.resize_image(resize_mode=2, im=p.init_images[0], width=p.width, height=p.height, upscaler_name=None, output_type='pil')
+            else:
+                p.task_args['image'] = p.init_images[0]
+            p.ops.append('svd')
+            p.do_not_save_grid = True
+            p.sampler_name = 'Default' # svd does not support non-default sampler
+            p.task_args['generator'] = torch.manual_seed(p.seed) # svd does not support gpu based generator
+            p.task_args['width'] = p.width
+            p.task_args['height'] = p.height
+            p.task_args['num_frames'] = num_frames
+            p.task_args['decode_chunk_size'] = decode_chunk_size
+            p.task_args['motion_bucket_id'] = round(255 * motion_bucket_id)
+            p.task_args['noise_aug_strength'] = noise_aug_strength
+            p.task_args['num_inference_steps'] = p.steps
+            p.task_args['min_guidance_scale'] = min_guidance_scale
+            p.task_args['max_guidance_scale'] = max_guidance_scale
+            p.task_args['output_type'] = 'np'
+            shared.log.debug(f'StableVideo: args={p.task_args}')
+            shared.sd_model = sd_models.set_diffuser_pipe(shared.sd_model, sd_models.DiffusersTaskType.IMAGE_2_IMAGE)
+            processed = processing.process_images(p)
+            if create_gif:
+                images.save_video(p, images=processed.images, video_type='gif', duration=duration, loop=loop)
+            if create_mp4:
+                images.save_video(p, images=processed.images, video_type='mp4', duration=duration, loop=loop)
+            return processed
+        else:
+            shared.log.error('StableVideo: no init_images')
+            return None