diff --git a/CHANGELOG.md b/CHANGELOG.md index e2c11e212..78e9d05bf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,18 +50,22 @@ To use and of the new models, simply select model from *Networks -> Reference* a This is an SDXL style model that replaces standard CLiP-L and CLiP-G text encoders with a massive `chatglm3-6b` encoder supporting both English and Chinese prompting - [HunyuanDiT 1.2](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers) Hunyuan-DiT is a powerful multi-resolution diffusion transformer (DiT) with fine-grained Chinese understanding +- [AnimateDiff](https://github.com/guoyww/animatediff/) + support for additional models: **SD 1.5 v3** (Sparse), **SD Lightning** (4-step), **SDXL Beta** **New Features...** - support for **Balanced Offload**, thanks @Disty0! - balanced offload will dynamically split and offload models from the GPU based on the max gpu and cpu memory size: model parts that dont fit in the GPU will be dynamically sliced and offloaded to the CPU + balanced offload will dynamically split and offload models from the GPU based on the max configured GPU and CPU memory size + model parts that dont fit in the GPU will be dynamically sliced and offloaded to the CPU see *Settings -> Diffusers Settings -> Max GPU memory and Max CPU memory* - *note*: balanced offload will force loading LoRA with Diffusers method and it is not compatible with Optimum Quanto + *note*: balanced offload will force loading LoRA with Diffusers method + *note*: balanced offload is not compatible with Optimum Quanto - support for **Optimum Quanto** with 8 bit and 4 bit quantization options, thanks @Disty0 and @Trojaner! to use, go to Settings -> Compute Settings and enable "Quantize Model weights with Optimum Quanto" option - note: Optimum Quanto requires PyTorch 2.4 + *note*: Optimum Quanto requires PyTorch 2.4 - new prompt attention mode: **xhinker** which brings support for prompt attention to new models such as FLUX.1 and SD3 - to use, enable in Settings -> Execution -> Prompt attention + to use, enable in *Settings -> Execution -> Prompt attention* **Changes & Fixes...** diff --git a/README.md b/README.md index bfc145e04..0fd319378 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ All individual features are not listed here, instead check [ChangeLog](CHANGELOG - Multiple UIs! ▹ **Standard | Modern** - Multiple diffusion models! - ▹ **Stable Diffusion 1.5/2.1/XL/3.0 | LCM | Lightning | Segmind | Kandinsky | Pixart-α | Pixart-Σ | Stable Cascade | Würstchen | aMUSEd | DeepFloyd IF | UniDiffusion | SD-Distilled | BLiP Diffusion | KOALA | SDXS | Hyper-SD | HunyuanDiT | etc.** + ▹ **Stable Diffusion 1.5/2.1/XL/3.0 | LCM | Lightning | Segmind | Kandinsky | Pixart-α | Pixart-Σ | Stable Cascade | FLUX.1 | AuraFlow | Würstchen | Lumina | Kolors | aMUSEd | DeepFloyd IF | UniDiffusion | SD-Distilled | BLiP Diffusion | KOALA | SDXS | Hyper-SD | HunyuanDiT | etc.** - Built-in Control for Text, Image, Batch and video processing! ▹ **ControlNet | ControlNet XS | Control LLLite | T2I Adapters | IP Adapters** - Multiplatform! diff --git a/TODO.md b/TODO.md index 2729def18..3c90ee485 100644 --- a/TODO.md +++ b/TODO.md @@ -5,21 +5,11 @@ Main ToDo list can be found at [GitHub projects](https://github.com/users/vladma ## Future Candidates - cogvideo-x: -- animatediff-sdxl: - animatediff prompt-travel: - async lowvram: - fp8: - init latents: variations, img2img -- diffusers public callbacks - include reference styles -- lora: sc lora, etc - -## Experimental - -- [SDXL Flash Mini](https://huggingface.co/sd-community/sdxl-flash-mini) - SDXL type that weighs less, consumes less video memory, and the quality has not dropped much - to use, simply select from *networks -> models -> reference -> SDXL Flash Mini* - recommended parameters: steps: 6-9, cfg scale: 2.5-3.5, sampler: DPM++ SDE ### Missing diff --git a/scripts/animatediff.py b/scripts/animatediff.py index 8c82a2cfc..86177cf17 100644 --- a/scripts/animatediff.py +++ b/scripts/animatediff.py @@ -13,6 +13,7 @@ TODO animatediff items: import os import gradio as gr import diffusers +from safetensors.torch import load_file from modules import scripts, processing, shared, devices, sd_models @@ -25,7 +26,9 @@ ADAPTERS = { 'Motion 1.4': 'guoyww/animatediff-motion-adapter-v1-4', 'TemporalDiff': 'vladmandic/temporaldiff', 'AnimateFace': 'vladmandic/animateface', - 'SDXL Beta': 'guoyww/animatediff-motion-adapter-sdxl-beta', + 'Lightning': 'ByteDance/AnimateDiff-Lightning/animatediff_lightning_4step_diffusers.safetensors', + 'SDXL Beta': 'a-r-r-o-w/animatediff-motion-adapter-sdxl-beta', + # 'SDXL Beta': 'guoyww/animatediff-motion-adapter-sdxl-beta', # 'LongAnimateDiff 32': 'vladmandic/longanimatediff-32', # 'LongAnimateDiff 64': 'vladmandic/longanimatediff-64', } @@ -81,11 +84,14 @@ def set_adapter(adapter_name: str = 'None'): try: shared.log.info(f'AnimateDiff load: adapter="{adapter_name}"') motion_adapter = None - if shared.sd_model_type == 'sd': + if adapter_name.endswith('.safetensors'): + motion_adapter = diffusers.MotionAdapter().to(shared.device, devices.dtype) + motion_adapter.load_state_dict(load_file(adapter_name)) + elif shared.sd_model_type == 'sd': motion_adapter = diffusers.MotionAdapter.from_pretrained(adapter_name, cache_dir=shared.opts.diffusers_dir, torch_dtype=devices.dtype, low_cpu_mem_usage=False, device_map=None) elif shared.sd_model_type == 'sdxl': motion_adapter = diffusers.MotionAdapter.from_pretrained(adapter_name, cache_dir=shared.opts.diffusers_dir, torch_dtype=devices.dtype, low_cpu_mem_usage=False, device_map=None, variant='fp16') - motion_adapter.to(shared.device) + sd_models.move_model(motion_adapter, devices.device) # move pipeline to device sd_models.set_diffuser_options(motion_adapter, vae=None, op='adapter') loaded_adapter = adapter_name new_pipe = None @@ -123,6 +129,7 @@ def set_adapter(adapter_name: str = 'None'): sd_models.move_model(shared.sd_model, devices.device) # move pipeline to device sd_models.copy_diffuser_options(new_pipe, orig_pipe) sd_models.set_diffuser_options(shared.sd_model, vae=None, op='model') + sd_models.move_model(shared.sd_model.unet, devices.device) # move pipeline to device shared.log.debug(f'AnimateDiff create: pipeline="{shared.sd_model.__class__}" adapter="{loaded_adapter}"') except Exception as e: motion_adapter = None