Public/automatic
1
0
Fork 0
mirror of https://github.com/vladmandic/automatic synced 2026-09-18 16:54:33 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
dev
automatic/TODO.md
T
Vladimir Mandic 516b9f4a45 lint 
Signed-off-by: Vladimir Mandic <mandic00@live.com>
2026-09-14 12:11:52 +02:00

175 lines
9.6 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# TODO
## Short-term
- Control tab verify overrides handling, @vladmandic
- LTX: Implement LTX2DFRPipeline
- ROCm: v10
- Video: unify execution path for ui and api
- Torch: update ipex, rocm to torch==2.14
## Issues
- [Inpaint](https://discord.com/channels/1101998836328697867/1130536562422186044/1506850651035144322), @vladmandic
## Features
### Assigned
- Object clear remover for Kanvas: [Object clear](https://huggingface.co/jixin0101/ObjectClear), @vladmandic
- OpenAI API interface for image generation, @vladmandic
- Lightweight scheduler/queue manager, @vladmandic
- Integrate natural language image search: [ImageDB](https://github.com/vladmandic/imagedb), @vladmandic
- Support cloud providers, @CalamitousFelicitousness
- Benchmark tool productize: @CalamitousFelicitousness
### Roadmap
- Automated testing and integrate models repo
- Video upscaling: LTX-Upscaler
- Video upscaling: [MiniMax-Upscaler](https://huggingface.co/LBH-123-AI/Minimax_h3_latent_Upscaler)
- Video capabilities to processing tab, add RIFE, upscaling (once available)
- Distraction-free UI mode with prompt-only, chat-based interface
- Video models: support finetunes
- Incorporate [prompting guides](https://github.com/CalamitousFelicitousness/ai-prompting-guides)
- Video models: use Networks/Reference instead of custom
- UI Lite vs Expert mode
- Remove obsolete code: `olive-ai`
### OnHold
- [nVidia-VFX](https://pypi.org/project/nvidia-vfx/): not compatible with latest nVidia drivers
- [QuantFunc](https://huggingface.co/QuantFunc/Klein-9B-Series): once its released as sdk
- LoRA add OMI format support for SD35/FLUX.1
- Remote Text-Encoder support, sidelined for the moment
- Auto handle scheduler `prediction_type`
- Multi-user support
- Settings profile manager
- TensorRT acceleration
- Cache models in memory
- Unify *huggingface* and *diffusers* model folders
- JSON image metadata
## New models / Pipelines
TODO: Investigate which models are diffusers-compatible and prioritize!
### Image
- [nVidia Cosmos-Predict-2.5](https://huggingface.co/nvidia/Cosmos-Predict2.5-2B) (in diffusers)
- [nVidia Cosmos-Transfer-2.5](https://huggingface.co/nvidia/Cosmos-Transfer2.5-2B) (in diffusers)
- [Tencent HY-WU](https://huggingface.co/tencent/HY-WU) (transformers-compatible)
### Video
- [ByteDance Lance](https://github.com/bytedance/Lance)
- [HY-OmniWeaving](https://huggingface.co/tencent/HY-OmniWeaving)
- [OpenMOSS MOVA](https://huggingface.co/OpenMOSS-Team/MOVA-720p)
- [Wan2.2-Animate](https://huggingface.co/Wan-AI/Wan2.2-Animate-14B)
- [Wan2.1-T2V-14B-CausVid](https://huggingface.co/lightx2v/Wan2.1-T2V-14B-CausVid)
- [Wan2.1-StepDistill-CfgDistill](https://huggingface.co/lightx2v/Wan2.1-T2V-14B-StepDistill-CfgDistill)
- [Krea Realtime Video](https://huggingface.co/krea/krea-realtime-video)
- [MAGI-1](https://github.com/SandAI-org/MAGI-1)
- [MUG-V 10B](https://huggingface.co/MUG-V/MUG-V-inference)
- [Ovi](https://github.com/character-ai/Ovi)
- [LucyEdit](https://huggingface.co/decart-ai/Lucy-Edit-1.1-Dev)
- [HunyuanVideo-Avatar](https://huggingface.co/tencent/HunyuanVideo-Avatar)
- [Sana I2V](https://huggingface.co/Efficient-Large-Model/SANA-Video_2B_480p_diffusers)
- [Wan-2.2 S2V](https://huggingface.co/Wan-AI/Wan2.2-S2V-14B)
- [Meituan LongCat-Video](https://huggingface.co/meituan-longcat/LongCat-Video)
- [Phantom HuMo](https://github.com/Phantom-video/Phantom)
- [CausVid-Plus](https://github.com/goatWu/CausVid-Plus/)
- [LivePortrait](https://github.com/KwaiVGI/LivePortrait)
- [SandAI Magi](https://github.com/SandAI-org/MAGI-1)
- [inclusionAI Ming](https://github.com/inclusionAI/Ming)
- [HummingbirdXT](https://huggingface.co/amd/HummingbirdXT)
- [DiffusionForcing](https://github.com/kwsong0113/diffusion-forcing-transformer)
- [ByteDance Lynx](https://github.com/bytedance/lynx)
- [LanDiff](https://github.com/landiff/landiff)
### Other/Unsorted
- [TryOnDiffusion](https://github.com/fashn-AI/tryondiffusion)
- [GPEN Face Restoration](https://github.com/yangxy/GPEN)
- [ByteDance DreamO](https://github.com/bytedance/DreamO)
- Unified image customization framework combining face identity preservation, virtual try-on, style transfer, etc.
- Created: 2025-05 | Updated: 2025-08 | Stars: 1,700
- [ControlNeXt](https://github.com/dvlab-research/ControlNeXt/)
- Lightweight controllable generation framework for images and videos (SD1.5, SDXL, SVD) that uses up to 90% fewer trainable parameters than ControlNet
- Created: 2024-08 | Updated: 2024-08 | Stars: 1,600
- [ByteDance USO](https://github.com/bytedance/USO)
- Unified model for both style-transfer and subject-driven image generation from one or two reference images
- Created: 2025-08 | Updated: 2025-09 | Stars: 1,200
- [TwinFlow](https://github.com/inclusionAI/TwinFlow)
- Distillation technique that converts large image generation models into 1–2 step generators without requiring a separate teacher model
- Created: 2025-12 | Updated: 2026-02 | Stars: 506
- [FlashFace](https://github.com/ali-vilab/FlashFace)
- Zero-shot face personalization method that generates images of a specific person from one or a few reference photos
- Created: 2024-03 | Updated: 2024-05 | Stars: 436
- [DiffSynth-Engine](https://github.com/modelscope/DiffSynth-Engine)
- Alternative to diffusers library that unlocks some diffsynth specific capabilities
- Created: 2024-05 | Updated: 2026-03 | Stars: 393
- [MS-Diffusion](https://github.com/MS-Diffusion/MS-Diffusion)
- Multi-subject image personalization framework that uses layout guidance to place multiple reference subjects in a single generated image without identity confusion
- Created: 2024-04 | Updated: 2025-07 | Stars: 309
- [RamTorch](https://github.com/lodestone-rock/ramtorch)
- Alternative memory management and offloading library
- Created: 2025-09 | Updated: 2026-04 | Stars: 266
- [UniRef](https://github.com/FoundationVision/UniRef)
- Unified segmentation model that handles referring image segmentation and few-shot segmentation
- Created: 2023-04 | Updated: 2025-04 | Stars: 238
- [FreeFuse](https://github.com/yaoliliu/FreeFuse)
- Training-free method to combine multiple subject LoRAs in one image generation without conflicts, by automatically routing each LoRA's influence to its target spatial region.
- Created: 2026-01 | Updated: 2026-03 | Stars: 178
- [mmgp](https://github.com/deepbeepmeep/mmgp)
- Alternative memory management and offloading library
- Created: 2024-03 | Updated: 2026-02 | Stars: 175
- [ReNO](https://github.com/ExplainableML/ReNO)
- Inference-time technique that improves one-step text-to-image models by iteratively optimizing the initial noise using reward model signals, boosting prompt accuracy in 20–50 seconds
- Created: 2024-06 | Updated: 2025-09 | Stars: 166
- [RegionE](https://github.com/Peyton-Chen/RegionE)
- Speeds up instruction-based image editing by skipping redundant computation in image regions that are not being changed.
- Created: 2025-10 | Updated: 2026-02 | Stars: 98
- [Make-It-Count](https://github.com/Litalby1/make-it-count)
- Method that reliably generates the exact number of objects requested by tracking instance identities during denoising
- Created: 2024-04 | Updated: 2025-04 | Stars: 96
- [FaceClip](https://huggingface.co/ByteDance/FaceCLIP)
- Identity-preserving image generation model that jointly encodes a face and a text prompt into a shared embedding to produce portraits matching both the subject's appearance and the scene description
- Created: 2025-04 | Updated: 2025-04 | Likes: 88
- [T5Gemma Adapter](https://github.com/NeuroSenko/ComfyUI_LLM_SDXL_Adapter)
- Experiment that replaces the SDXL text encoder with a T5Gemma LLM via a trained adapter for richer prompt understanding
- Created: 2025-07 | Updated: 2025-10 | Stars: 67
- [Sonic Inpaint](https://github.com/ubc-vision/sonic)
- Image inpainting method that optimizes for better masked-region filling
- Created: 2025-11 | Updated: 2026-01 | Stars: 23
- [SEVA](https://github.com/Stability-AI/stable-virtual-camera)
- Model that generates novel-view images of a scene from a single input photo.
- Created: 2025-04 | Updated: 2025-06 | Stars: N/A (draft PR)
- [Bria FIBO RMBG](https://huggingface.co/briaai/RMBG-2.0)
- Background removal model trained on Bria FIBO dataset
- Created: 2025-08 | Updated: 2025-09 | Stars: N/A (private model)
### Rejected
- [Mugen](https://huggingface.co/CabalResearch/Mugen) (sdxl with flux vae experiment, not clean)
- [Liquid](https://github.com/FoundationVision/Liquid) (autoregressive, not clean)
## Code TODO
> pnpm run todo
```code
installer.py:652:15: W0511: TODO rocm: switch to pytorch source when it becomes available (fixme)
modules/sd_models_compile.py:90:5: W0511: TODO pruna: enable when it supports transformers==5.5 (fixme)
modules/transformer_cache.py:29:61: W0511: TODO fc: autodetect tensor format based on model (fixme)
modules/transformer_cache.py:30:50: W0511: TODO fc: autodetect distilled based on model (fixme)
modules/processing_class.py:406:32: W0511: TODO processing: remove duplicate mask params (fixme)
modules/sd_samplers_diffusers.py:370:31: W0511: TODO enso-required (fixme)
modules/sd_models.py:1424:5: W0511: TODO model load: implement model in-memory caching (fixme)
modules/ui_models_load.py:257:5: W0511: TODO loader: load receipe (fixme)
modules/ui_models_load.py:264:5: W0511: TODO loader: save receipe (fixme)
modules/sd_hijack_hypertile.py:123:17: W0511: TODO hypertile: vae breaks when using non-standard sizes (fixme)
modules/sd_unet.py:77:39: W0511: TODO model load: force-reloading entire model as loading transformers only leads to massive memory usage (fixme)
modules/modular_guiders.py:66:51: W0511: TODO: guiders (fixme)
```
Reference in New Issue View Git Blame Copy Permalink