update todo

TODO.md

@@ -42,11 +42,9 @@ Tech that can be integrated as part of the core workflow...
 - [Weighted merges](https://github.com/bbc-mc/sdweb-merge-block-weighted-gui/tree/master)
 - [Prune models](https://github.com/Akegarasu/sd-webui-model-converter)
 - [Use scripts from API](https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/6469)
-- [Aesthetic Gradients](https://github.com/AUTOMATIC1111/stable-diffusion-webui-aesthetic-gradients)
-- [Latent blending](https://github.com/lunarring/latentblending/)
+- [Aesthetic gradients](https://github.com/AUTOMATIC1111/stable-diffusion-webui-aesthetic-gradients)
 - [Face swap](https://github.com/kex0/batch-face-swap)
-- [Animator extension](https://github.com/Animator-Anon/animator_extension)
-- [Custom Diffusion](https://github.com/guaneec/custom-diffusion-webui)
+- [Custom diffusion](https://github.com/guaneec/custom-diffusion-webui)
 - [LORA](https://github.com/cloneofsimo/lora)
   - <https://github.com/kohya-ss/sd-webui-additional-networks>
   - <https://github.com/kohya-ss/sd-scripts>
@@ -54,14 +52,23 @@ Tech that can be integrated as part of the core workflow...
   - <https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/2670#discussioncomment-4372336>
   - <https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/2670#discussioncomment-4582025>
 
+
+## Video Generation
+
+- [Deforum](https://github.com/deforum-art/deforum-for-automatic1111-webui)
+- [Latent blending](https://github.com/lunarring/latentblending/)
+- [VIDM: video implicit diffusion models](https://github.com/MKFMIKU/vidm)
+- [Tune-a-Video](https://github.com/showlab/Tune-A-Video)
+- [Animator extension](https://github.com/Animator-Anon/animator_extension)
+- [KLMC2 animation](https://colab.research.google.com/github/dmarx/notebooks/blob/main/Stable_Diffusion_KLMC2_Animation.ipynb)
+- [Disco diffusion](https://colab.research.google.com/github/alembics/disco-diffusion/blob/main/Disco_Diffusion.ipynb)
+- [Video killed the radio star](https://colab.research.google.com/github/dmarx/video-killed-the-radio-star/blob/main/Video_Killed_The_Radio_Star_Defusion.ipynb)
+
 ## Experimental
 
 Cool stuff that is not integrated anywhere...
 
 - [TensorRT](https://www.photoroom.com/tech/stable-diffusion-25-percent-faster-and-save-seconds/)
-- [KLMC2 Animation](https://colab.research.google.com/github/dmarx/notebooks/blob/main/Stable_Diffusion_KLMC2_Animation.ipynb)
-- [Disco Diffusion](https://colab.research.google.com/github/alembics/disco-diffusion/blob/main/Disco_Diffusion.ipynb)
-- [Video Killed the Radio Star](https://colab.research.google.com/github/dmarx/video-killed-the-radio-star/blob/main/Video_Killed_The_Radio_Star_Defusion.ipynb)
 - Bunch of stuff:<https://pharmapsychotic.com/tools.html>
 - Prevalent colors to interrogate
 - Auto-Sort inputs by face recognition

cli/train.py

@@ -276,7 +276,10 @@ async def train(params):
     })
     log.info({ 'learn-rate': args.train_embedding.learn_rate })
     log.debug({ 'train args': args.train_embedding })
+    t0 = time.time()
     res = await post('/sdapi/v1/train/embedding', args.train_embedding)
+    t1 = time.time()
+    log.info({ 'train embedding finished': round(t1 - t0) })
     log.debug({ 'train end': res.info })
     return
 
@@ -316,6 +319,9 @@ async def monitor(params):
     while True:
         await asyncio.sleep(10)
         res = await progress()
+        if not 'state' in res:
+            log.info({ 'monitor disconnected': res })
+            break
         if (res.state.job_count == params.steps and res.state.job_no >= res.state.job_count) or (res.eta_relative < 0) or (res.interrupted) or (res.state.job_count == 0): # need exit case if interrupted or failed
             if res.interrupted:
                 log.info({ 'monitor interrupted': { 'embedding': params.name } })