animatediff updates

CHANGELOG.md

@@ -50,18 +50,22 @@ To use and of the new models, simply select model from *Networks -> Reference* a
   This is an SDXL style model that replaces standard CLiP-L and CLiP-G text encoders with a massive `chatglm3-6b` encoder supporting both English and Chinese prompting  
 - [HunyuanDiT 1.2](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers)  
   Hunyuan-DiT is a powerful multi-resolution diffusion transformer (DiT) with fine-grained Chinese understanding  
+- [AnimateDiff](https://github.com/guoyww/animatediff/)  
+  support for additional models: **SD 1.5 v3** (Sparse), **SD Lightning** (4-step), **SDXL Beta**  
 
 **New Features...**
 
 - support for **Balanced Offload**, thanks @Disty0!  
-  balanced offload will dynamically split and offload models from the GPU based on the max gpu and cpu memory size: model parts that dont fit in the GPU will be dynamically sliced and offloaded to the CPU  
+  balanced offload will dynamically split and offload models from the GPU based on the max configured GPU and CPU memory size  
+  model parts that dont fit in the GPU will be dynamically sliced and offloaded to the CPU  
   see *Settings -> Diffusers Settings -> Max GPU memory and Max CPU memory*  
-  *note*: balanced offload will force loading LoRA with Diffusers method and it is not compatible with Optimum Quanto  
+  *note*: balanced offload will force loading LoRA with Diffusers method  
+  *note*: balanced offload is not compatible with Optimum Quanto  
 - support for **Optimum Quanto** with 8 bit and 4 bit quantization options, thanks @Disty0 and @Trojaner!  
   to use, go to Settings -> Compute Settings and enable "Quantize Model weights with Optimum Quanto" option  
-  note: Optimum Quanto requires PyTorch 2.4  
+  *note*: Optimum Quanto requires PyTorch 2.4  
 - new prompt attention mode: **xhinker** which brings support for prompt attention to new models such as FLUX.1 and SD3  
-  to use, enable in Settings -> Execution -> Prompt attention
+  to use, enable in *Settings -> Execution -> Prompt attention*
 
 **Changes & Fixes...**
 

README.md

@@ -31,7 +31,7 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG
 - Multiple UIs!  
   ▹ **Standard | Modern**  
 - Multiple diffusion models!  
-  ▹ **Stable Diffusion 1.5/2.1/XL/3.0 | LCM | Lightning | Segmind | Kandinsky | Pixart-α | Pixart-Σ | Stable Cascade | Würstchen | aMUSEd | DeepFloyd IF | UniDiffusion | SD-Distilled | BLiP Diffusion | KOALA | SDXS | Hyper-SD | HunyuanDiT | etc.**
+  ▹ **Stable Diffusion 1.5/2.1/XL/3.0 | LCM | Lightning | Segmind | Kandinsky | Pixart-α | Pixart-Σ | Stable Cascade | FLUX.1 | AuraFlow | Würstchen | Lumina | Kolors | aMUSEd | DeepFloyd IF | UniDiffusion | SD-Distilled | BLiP Diffusion | KOALA | SDXS | Hyper-SD | HunyuanDiT | etc.**
 - Built-in Control for Text, Image, Batch and video processing!  
   ▹ **ControlNet | ControlNet XS | Control LLLite | T2I Adapters | IP Adapters**  
 - Multiplatform!  

TODO.md

@@ -5,21 +5,11 @@ Main ToDo list can be found at [GitHub projects](https://github.com/users/vladma
 ## Future Candidates
 
 - cogvideo-x: <https://huggingface.co/THUDM/CogVideoX-5b>
-- animatediff-sdxl: <https://github.com/huggingface/diffusers/pull/6721>
 - animatediff prompt-travel: <https://github.com/huggingface/diffusers/pull/9231>
 - async lowvram: <https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/14855>
 - fp8: <https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/14031>
 - init latents: variations, img2img
-- diffusers public callbacks  
 - include reference styles
-- lora: sc lora, etc
-
-## Experimental
-
-- [SDXL Flash Mini](https://huggingface.co/sd-community/sdxl-flash-mini)  
-  SDXL type that weighs less, consumes less video memory, and the quality has not dropped much  
-  to use, simply select from *networks -> models -> reference -> SDXL Flash Mini*  
-  recommended parameters: steps: 6-9, cfg scale: 2.5-3.5, sampler: DPM++ SDE  
 
 ### Missing
 

scripts/animatediff.py

@@ -13,6 +13,7 @@ TODO animatediff items:
 import os
 import gradio as gr
 import diffusers
+from safetensors.torch import load_file
 from modules import scripts, processing, shared, devices, sd_models
 
 
@@ -25,7 +26,9 @@ ADAPTERS = {
     'Motion 1.4': 'guoyww/animatediff-motion-adapter-v1-4',
     'TemporalDiff': 'vladmandic/temporaldiff',
     'AnimateFace': 'vladmandic/animateface',
-    'SDXL Beta': 'guoyww/animatediff-motion-adapter-sdxl-beta',
+    'Lightning': 'ByteDance/AnimateDiff-Lightning/animatediff_lightning_4step_diffusers.safetensors',
+    'SDXL Beta': 'a-r-r-o-w/animatediff-motion-adapter-sdxl-beta',
+    # 'SDXL Beta': 'guoyww/animatediff-motion-adapter-sdxl-beta',
     # 'LongAnimateDiff 32': 'vladmandic/longanimatediff-32',
     # 'LongAnimateDiff 64': 'vladmandic/longanimatediff-64',
 }
@@ -81,11 +84,14 @@ def set_adapter(adapter_name: str = 'None'):
     try:
         shared.log.info(f'AnimateDiff load: adapter="{adapter_name}"')
         motion_adapter = None
-        if shared.sd_model_type == 'sd':
+        if adapter_name.endswith('.safetensors'):
+            motion_adapter = diffusers.MotionAdapter().to(shared.device, devices.dtype)
+            motion_adapter.load_state_dict(load_file(adapter_name))
+        elif shared.sd_model_type == 'sd':
             motion_adapter = diffusers.MotionAdapter.from_pretrained(adapter_name, cache_dir=shared.opts.diffusers_dir, torch_dtype=devices.dtype, low_cpu_mem_usage=False, device_map=None)
         elif shared.sd_model_type == 'sdxl':
             motion_adapter = diffusers.MotionAdapter.from_pretrained(adapter_name, cache_dir=shared.opts.diffusers_dir, torch_dtype=devices.dtype, low_cpu_mem_usage=False, device_map=None, variant='fp16')
-        motion_adapter.to(shared.device)
+        sd_models.move_model(motion_adapter, devices.device) # move pipeline to device
         sd_models.set_diffuser_options(motion_adapter, vae=None, op='adapter')
         loaded_adapter = adapter_name
         new_pipe = None
@@ -123,6 +129,7 @@ def set_adapter(adapter_name: str = 'None'):
         sd_models.move_model(shared.sd_model, devices.device) # move pipeline to device
         sd_models.copy_diffuser_options(new_pipe, orig_pipe)
         sd_models.set_diffuser_options(shared.sd_model, vae=None, op='model')
+        sd_models.move_model(shared.sd_model.unet, devices.device) # move pipeline to device
         shared.log.debug(f'AnimateDiff create: pipeline="{shared.sd_model.__class__}" adapter="{loaded_adapter}"')
     except Exception as e:
         motion_adapter = None