implement complete face module

CHANGELOG.md

@@ -21,14 +21,15 @@ OPTIONAL:
 - masking api
 - preprocess api
 
-## Update for 2023-01-25
+## Update for 2023-01-27
 
 Another big release, highlights being:  
 - A lot more functionality in the **Control** module:
   - Inpaint and outpaint support, flexible resizing options, optional hires  
   - Built-in support for many new processors and models which are auto-downloaded on first use  
   - Full support for scripts and extensions  
-- Fully baked-in **FaceID**, **FaceSwap** and **PhotoMaker** modules
+- Complete **Face** module  
+  implements all variations of **FaceID**, **FaceSwap** and latest **PhotoMaker** and **InstantID**  
 - Much enhanced **IPAdapter** modules  
 - Brand new **intelligent masking**, manual or automatic  
   Using ML models (object removal, background removal, segmentation, etc.) and with live previews  
@@ -92,27 +93,34 @@ As of this release, default backend is set to **diffusers** as its more feature
   - fix batch/folder/video modes  
   - fix processor switching within same unit  
   - fix pipeline switching between different modes  
-- [FaceID/FaceSwap](https://huggingface.co/h94/IP-Adapter-FaceID)  
-  - full implementation for *SD15* and *SD-XL*, to use simply select from *Scripts*  
-    **Base** (93MB) uses *InsightFace* to generate face embeds and *OpenCLIP-ViT-H-14* (2.5GB) as image encoder  
-    **SXDL** (1022MB) uses *InsightFace* to generate face embeds and *OpenCLIP-ViT-bigG-14* (3.7GB) as image encoder  
-    **Plus** (150MB) uses *InsightFace* to generate face embeds and *CLIP-ViT-H-14-laion2B* (3.8GB) as image encoder  
-  - **FaceSwap**  
-    you can use just faceid or just faceswap or both at the same time  
-    faceid guides image generation given the input image while face swap performs face swapping at the end of generation  
-  - *note*: all models are downloaded on first use  
-  - enable use via api, thanks @trojaner
+- **Face** module  
+  implements all variations of **FaceID**, **FaceSwap** and latest **PhotoMaker** and **InstantID**  
+  simply select from scripts and choose your favorite method and model  
+  *note*: all models are auto-downloaded on first use  
+  - [FaceID](https://huggingface.co/h94/IP-Adapter-FaceID)  
+    - faceid guides image generation given the input image  
+    - full implementation for *SD15* and *SD-XL*, to use simply select from *Scripts*  
+      **Base** (93MB) uses *InsightFace* to generate face embeds and *OpenCLIP-ViT-H-14* (2.5GB) as image encoder  
+      **Plus** (150MB) uses *InsightFace* to generate face embeds and *CLIP-ViT-H-14-laion2B* (3.8GB) as image encoder  
+      **SXDL** (1022MB) uses *InsightFace* to generate face embeds and *OpenCLIP-ViT-bigG-14* (3.7GB) as image encoder  
+  - [FaceSwap](https://github.com/deepinsight/insightface/blob/master/examples/in_swapper/README.md)  
+    - face swap performs face swapping at the end of generation  
+    - based on InsightFace in-swapper  
+  - [PhotoMaker](https://github.com/TencentARC/PhotoMaker)  
+    - for *SD-XL* only  
+    - new model from TenencentARC using similar concept as IPAdapter, but with different implementation and  
+      allowing full concept swaps between input images and generated images using trigger words  
+    - note: trigger word must match exactly one term in prompt for model to work  
+  - [InstantID](https://github.com/InstantID/InstantID)  
+    - for *SD-XL* only  
+    - based on custom trained ip-adapter and controlnet combined concepts  
+    - note: controlnet appears to be heavily watermarked  
+  - enable use via api, thanks @trojaner  
 - [IPAdapter](https://huggingface.co/h94/IP-Adapter)  
   - additional models for *SD15* and *SD-XL*, to use simply select from *Scripts*:  
     **SD15**: Base, Base ViT-G, Light, Plus, Plus Face, Full Face  
     **SDXL**: Base SXDL, Base ViT-H SXDL, Plus ViT-H SXDL, Plus Face ViT-H SXDL  
   - enable use via api, thanks @trojaner  
-- [PhotoMaker](https://github.com/TencentARC/PhotoMaker)  
-  - for *SD-XL* only  
-  - simply select from *scripts*  
-  - new model from TenencentARC using similar concept as IPAdapter, but with different implementation and  
-    allowing full concept swaps between input images and generated images using trigger words  
-  - note: trigger word must match exactly one term in prompt for model to work  
 - [Self-attention guidance](https://github.com/SusungHong/Self-Attention-Guidance)  
   - simply select scale in advanced menu  
   - can drastically improve image coherence as well as reduce artifacts  
@@ -250,6 +258,7 @@ As of this release, default backend is set to **diffusers** as its more feature
   - cli: fix cmd args parsing  
   - global crlf->lf switch  
   - model type switch if there is loaded submodels  
+  - cleanup samplers use of compute devices, thanks @Disty0  
 - **other**
   - updated core requirements  
   - major internal ui module refactoring  

modules/control/proc/reference_sd15.py

@@ -212,7 +212,7 @@ class StableDiffusionReferencePipeline(StableDiffusionPipeline):
             num_images_per_prompt (`int`, *optional*, defaults to 1):
                 The number of images to generate per prompt.
             eta (`float`, *optional*, defaults to 0.0):
-                Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to
+                Corresponds to parameter eta in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to
                 [`schedulers.DDIMScheduler`], will be ignored for others.
             generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
                 One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html)
@@ -246,7 +246,7 @@ class StableDiffusionReferencePipeline(StableDiffusionPipeline):
                 [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py).
             guidance_rescale (`float`, *optional*, defaults to 0.0):
                 Guidance rescale factor proposed by [Common Diffusion Noise Schedules and Sample Steps are
-                Flawed](https://arxiv.org/pdf/2305.08891.pdf) `guidance_scale` is defined as `φ` in equation 16. of
+                Flawed](https://arxiv.org/pdf/2305.08891.pdf) `guidance_scale` is defined as . in equation 16. of
                 [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://arxiv.org/pdf/2305.08891.pdf).
                 Guidance rescale factor should fix overexposure when using zero terminal SNR.
             attention_auto_machine_weight (`float`):

modules/face/__init__.py

@@ -0,0 +1,140 @@
+import os
+import gradio as gr
+from PIL import Image
+from modules import scripts, processing, shared, images
+
+
+debug = shared.log.trace if os.environ.get('SD_FACE_DEBUG', None) is not None else lambda *args, **kwargs: None
+
+
+class Script(scripts.Script):
+    def title(self):
+        return 'Face'
+
+    def show(self, is_img2img):
+        return True if shared.backend == shared.Backend.DIFFUSERS else False
+
+    def load_images(self, files):
+        init_images = []
+        for file in files or []:
+            try:
+                if isinstance(file, str):
+                    from modules.api.api import decode_base64_to_image
+                    image = decode_base64_to_image(file)
+                elif isinstance(file, Image.Image):
+                    image = file
+                elif isinstance(file, dict) and 'name' in file:
+                    image = Image.open(file['name']) # _TemporaryFileWrapper from gr.Files
+                elif hasattr(file, 'name'):
+                    image = Image.open(file.name) # _TemporaryFileWrapper from gr.Files
+                else:
+                    raise ValueError(f'PhotoMaker unknown input: {file}')
+                init_images.append(image)
+            except Exception as e:
+                shared.log.warning(f'PhotoMaker failed to load image: {e}')
+        return init_images
+
+    def mode_change(self, mode):
+        return [
+            gr.update(visible=mode=='FaceID'),
+            gr.update(visible=mode=='FaceSwap'),
+            gr.update(visible=mode=='InstantID'),
+            gr.update(visible=mode=='PhotoMaker'),
+        ]
+
+    # return signature is array of gradio components
+    def ui(self, _is_img2img):
+        with gr.Row():
+            mode = gr.Dropdown(label='Mode', choices=['None', 'FaceID', 'FaceSwap', 'InstantID', 'PhotoMaker'], value='None')
+        with gr.Group(visible=False) as cfg_faceid:
+            with gr.Row():
+                gr.HTML('<a href="https://huggingface.co/h94/IP-Adapter-FaceID" target="_blank">&nbsp Tencent AI Lab IP-Adapter FaceID</a><br>')
+            with gr.Row():
+                from modules.face.faceid import FACEID_MODELS
+                ip_model = gr.Dropdown(choices=list(FACEID_MODELS), label='FaceID Model', value='FaceID Base')
+            with gr.Row(visible=True):
+                ip_override = gr.Checkbox(label='Override sampler', value=True)
+                ip_cache = gr.Checkbox(label='Cache model', value=True)
+            with gr.Row(visible=True):
+                ip_strength = gr.Slider(label='Strength', minimum=0.0, maximum=2.0, step=0.01, value=1.0)
+                ip_structure = gr.Slider(label='Structure', minimum=0.0, maximum=1.0, step=0.01, value=1.0)
+        with gr.Group(visible=False) as cfg_faceswap:
+            with gr.Row():
+                gr.HTML('<a href="https://github.com/deepinsight/insightface/blob/master/examples/in_swapper/README.md" target="_blank">&nbsp InsightFace InSwapper</a><br>')
+            with gr.Row(visible=True):
+                fs_cache = gr.Checkbox(label='Cache model', value=True)
+        with gr.Group(visible=False) as cfg_instantid:
+            with gr.Row():
+                gr.HTML('<a href="https://github.com/InstantID/InstantID" target="_blank">&nbsp InstantX InstantID</a><br>')
+            with gr.Row():
+                id_strength = gr.Slider(label='Strength', minimum=0.0, maximum=2.0, step=0.01, value=1.0)
+                id_conditioning = gr.Slider(label='Control', minimum=0.0, maximum=2.0, step=0.01, value=0.5)
+            with gr.Row(visible=True):
+                id_cache = gr.Checkbox(label='Cache model', value=True)
+        with gr.Group(visible=False) as cfg_photomaker:
+            with gr.Row():
+                gr.HTML('<a href="https://photo-maker.github.io/" target="_blank">&nbsp Tenecent ARC Lab PhotoMaker</a><br>')
+            with gr.Row():
+                pm_trigger = gr.Text(label='Trigger word', value="person")
+                pm_strength = gr.Slider(label='Strength', minimum=0.0, maximum=2.0, step=0.01, value=1.0)
+                pm_start = gr.Slider(label='Start', minimum=0.0, maximum=1.0, step=0.01, value=0.5)
+        with gr.Row():
+            files = gr.File(label='Input images', file_count='multiple', file_types=['image'], type='file', interactive=True, height=100)
+        with gr.Row():
+            gallery = gr.Gallery(show_label=False, value=[])
+        files.change(fn=self.load_images, inputs=[files], outputs=[gallery])
+        mode.change(fn=self.mode_change, inputs=[mode], outputs=[cfg_faceid, cfg_faceswap, cfg_instantid, cfg_photomaker])
+
+        return [mode, gallery, ip_model, ip_override, ip_cache, ip_strength, ip_structure, id_strength, id_conditioning, id_cache, pm_trigger, pm_strength, pm_start, fs_cache]
+
+    def run(self, p: processing.StableDiffusionProcessing, mode, input_images, ip_model, ip_override, ip_cache, ip_strength, ip_structure, id_strength, id_conditioning, id_cache, pm_trigger, pm_strength, pm_start, fs_cache): # pylint: disable=arguments-differ, unused-argument
+        if input_images is None or len(input_images) == 0:
+            shared.log.error('Face: no init images')
+            return None
+        if shared.sd_model_type != 'sd' and shared.sd_model_type != 'sdxl':
+            shared.log.error('Face: base model not supported')
+            return None
+
+        for i, image in enumerate(input_images):
+            if isinstance(image, str):
+                from modules.api.api import decode_base64_to_image
+                input_images[i] = decode_base64_to_image(image).convert("RGB")
+
+        processed = None
+        for i, image in enumerate(input_images):
+            input_images[i] = Image.open(image['name'])
+        source_image = input_images[0]
+
+        if mode == 'FaceID': # faceid runs as ipadapter in its own pipeline
+            from modules.face.faceid import face_id
+            from modules.face.insightface import get_app
+            processed_images = face_id(p, app=get_app('buffalo_l'), source_image=source_image, model=ip_model, override=ip_override, cache=ip_cache, scale=ip_strength, structure=ip_structure) # run faceid pipeline
+            processed = processing.Processed(p, images_list=processed_images, seed=p.seed, subseed=p.subseed, index_of_first_image=0) # manually created processed object
+        elif mode == 'PhotoMaker': # photomaker creates pipeline and triggers original process_images
+            from modules.face.photomaker import photo_maker
+            processed = photo_maker(p, input_images=input_images, trigger=pm_trigger, strength=pm_strength, start=pm_start)
+        elif mode == 'InstantID':
+            from modules.face.instantid import instant_id # instantid creates pipeline and triggers original process_images
+            from modules.face.insightface import get_app
+            processed = instant_id(p, app=get_app('antelopev2'), source_image=source_image, strength=id_strength, conditioning=id_conditioning, cache=id_cache)
+
+        if processed is None: # run normal pipeline
+            processed = processing.process_images(p)
+
+        if mode == 'FaceSwap': # faceswap runs as postprocessing
+            from modules.face.faceswap import face_swap
+            from modules.face.insightface import get_app
+            if shared.opts.save_images_before_face_restoration and not p.do_not_save_samples:
+                for i, image in enumerate(processed.images):
+                    info = processing.create_infotext(p, index=i)
+                    images.save_image(image, path=p.outpath_samples, seed=p.all_seeds[i], prompt=p.all_prompts[i], info=info, p=p, suffix="-before-faceswap")
+            processed.images = face_swap(p, app=get_app('buffalo_l'), input_images=processed.images, source_image=source_image, cache=fs_cache)
+
+        processed.info = processed.infotext(p, 0)
+        processed.infotexts = [processed.info]
+        if shared.opts.samples_save and not p.do_not_save_samples:
+            for i, image in enumerate(processed.images):
+                info = processing.create_infotext(p, index=i)
+                images.save_image(image, path=p.outpath_samples, seed=p.all_seeds[i], prompt=p.all_prompts[i], info=info, p=p)
+
+        return processed

modules/face/faceid.py

@@ -0,0 +1,125 @@
+import os
+import cv2
+import torch
+import numpy as np
+import diffusers
+import huggingface_hub as hf
+from PIL import Image
+from modules import processing, shared, devices
+
+
+FACEID_MODELS = {
+    'FaceID Base': 'h94/IP-Adapter-FaceID/ip-adapter-faceid_sd15.bin',
+    'FaceID Plus v1': 'h94/IP-Adapter-FaceID/ip-adapter-faceid-plus_sd15.bin',
+    'FaceID Plus v2': 'h94/IP-Adapter-FaceID/ip-adapter-faceid-plusv2_sd15.bin',
+    'FaceID XL': 'h94/IP-Adapter-FaceID/ip-adapter-faceid_sdxl.bin'
+}
+faceid_model = None
+faceid_model_name = None
+debug = shared.log.trace if os.environ.get('SD_FACE_DEBUG', None) is not None else lambda *args, **kwargs: None
+
+
+def face_id(p: processing.StableDiffusionProcessing, app, source_image: Image.Image, model: str, override: bool, cache: bool, scale: float, structure: float):
+    global faceid_model, faceid_model_name # pylint: disable=global-statement
+    from insightface.utils import face_align
+    from ip_adapter.ip_adapter_faceid import IPAdapterFaceID, IPAdapterFaceIDPlus, IPAdapterFaceIDXL
+
+    ip_ckpt = FACEID_MODELS[model]
+    folder, filename = os.path.split(ip_ckpt)
+    basename, _ext = os.path.splitext(filename)
+    model_path = hf.hf_hub_download(repo_id=folder, filename=filename, cache_dir=shared.opts.diffusers_dir)
+    if model_path is None:
+        shared.log.error(f'FaceID download failed: model={model} file={ip_ckpt}')
+        return None
+
+    processing.process_init(p)
+    if override:
+        shared.sd_model.scheduler = diffusers.DDIMScheduler(
+            num_train_timesteps=1000,
+            beta_start=0.00085,
+            beta_end=0.012,
+            beta_schedule="scaled_linear",
+            clip_sample=False,
+            set_alpha_to_one=False,
+            steps_offset=1,
+        )
+    shortcut = None
+    if faceid_model is None or faceid_model_name != model or not cache:
+        shared.log.debug(f'FaceID load: model={model} file={ip_ckpt}')
+        if 'Plus' in model:
+            image_encoder_path = "laion/CLIP-ViT-H-14-laion2B-s32B-b79K"
+            faceid_model = IPAdapterFaceIDPlus(
+                sd_pipe=shared.sd_model,
+                image_encoder_path=image_encoder_path,
+                ip_ckpt=model_path,
+                lora_rank=128, num_tokens=4, device=devices.device, torch_dtype=devices.dtype,
+            )
+            shortcut = 'v2' in model
+        elif 'XL' in model:
+            faceid_model = IPAdapterFaceIDXL(
+                sd_pipe=shared.sd_model,
+                ip_ckpt=model_path,
+                lora_rank=128, num_tokens=4, device=devices.device, torch_dtype=devices.dtype,
+            )
+        else:
+            faceid_model = IPAdapterFaceID(
+                sd_pipe=shared.sd_model,
+                ip_ckpt=model_path,
+                lora_rank=128, num_tokens=4, device=devices.device, torch_dtype=devices.dtype,
+            )
+        faceid_model_name = model
+    else:
+        shared.log.debug(f'FaceID cached: model={model} file={ip_ckpt}')
+
+    processed_images = []
+    np_image = cv2.cvtColor(np.array(source_image), cv2.COLOR_RGB2BGR)
+    faces = app.get(np_image)
+    if len(faces) == 0:
+        shared.log.error('FaceID: no faces found')
+        return None
+    face_embeds = torch.from_numpy(faces[0].normed_embedding).unsqueeze(0)
+    face_image = face_align.norm_crop(np_image, landmark=faces[0].kps, image_size=224) # you can also segment the face
+
+    for i, face in enumerate(faces):
+        shared.log.debug(f'FaceID face: i={i+1} score={face.det_score:.2f} gender={"female" if face.gender==0 else "male"} age={face.age} bbox={face.bbox}')
+        p.extra_generation_params[f"FaceID {i+1}"] = f'{face.det_score:.2f} {"female" if face.gender==0 else "male"} {face.age}y'
+    ip_model_dict = { # main generate dict
+        'num_samples': p.batch_size,
+        'width': p.width,
+        'height': p.height,
+        'num_inference_steps': p.steps,
+        'scale': scale,
+        'guidance_scale': p.cfg_scale,
+        'faceid_embeds': face_embeds.shape,
+    }
+    # optional generate dict
+    if shortcut is not None:
+        ip_model_dict['shortcut'] = shortcut
+    if 'Plus' in model:
+        ip_model_dict['s_scale'] = structure
+        ip_model_dict['face_image'] = face_image.shape
+    shared.log.debug(f'FaceID args: {ip_model_dict}')
+    if 'Plus' in model:
+        ip_model_dict['face_image'] = face_image
+    ip_model_dict['faceid_embeds'] = face_embeds
+    # run generate
+    faceid_model.set_scale(scale)
+    for i in range(p.n_iter):
+        ip_model_dict.update({
+                'prompt': p.all_prompts[i],
+                'negative_prompt': p.all_negative_prompts[i],
+                'seed': int(p.all_seeds[i]),
+            })
+        debug(f'FaceID: {ip_model_dict}')
+        res = faceid_model.generate(**ip_model_dict)
+        if isinstance(res, list):
+            processed_images += res
+    faceid_model.set_scale(0)
+
+    if not cache:
+        faceid_model = None
+        faceid_model_name = None
+    devices.torch_gc()
+
+    p.extra_generation_params["IP Adapter"] = f'{basename}:{scale}'
+    return processed_images

modules/face/faceswap.py

@@ -0,0 +1,41 @@
+from typing import List
+import os
+import cv2
+import numpy as np
+import huggingface_hub as hf
+from PIL import Image
+from modules import processing, shared, devices
+
+
+debug = shared.log.trace if os.environ.get('SD_FACE_DEBUG', None) is not None else lambda *args, **kwargs: None
+insightface_app = None
+swapper = None
+
+
+def face_swap(p: processing.StableDiffusionProcessing, app, input_images: List[Image.Image], source_image: Image.Image, cache: bool):
+    import insightface.model_zoo
+    global swapper # pylint: disable=global-statement
+    if swapper is None:
+        model_path = hf.hf_hub_download(repo_id='ezioruan/inswapper_128.onnx', filename='inswapper_128.onnx', cache_dir=shared.opts.diffusers_dir)
+        router: insightface.model_zoo.model_zoo.INSwapper = insightface.model_zoo.model_zoo.ModelRouter(model_path)
+        swapper = router.get_model()
+
+    np_image = cv2.cvtColor(np.array(source_image), cv2.COLOR_RGB2BGR)
+    faces = app.get(np_image)
+    source_face = faces[0]
+    processed_images = []
+    for image in input_images:
+        np_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR)
+        faces = app.get(np_image)
+        for i, face in enumerate(faces):
+            debug(f'FaceSwap: face={i} source={source_face.bbox} target={face.bbox}')
+            np_image = swapper.get(img=np_image, target_face=face, source_face=source_face, paste_back=True) # pylint: disable=unexpected-keyword-arg, no-value-for-parameter
+        p.extra_generation_params["FaceSwap"] = f'{len(faces)}'
+        np_image = cv2.cvtColor(np_image, cv2.COLOR_BGR2RGB)
+        processed_images.append(Image.fromarray(np_image))
+
+    if not cache:
+        swapper = None
+    devices.torch_gc()
+
+    return processed_images

modules/face/insightface.py

@@ -0,0 +1,49 @@
+import os
+from modules.shared import log, opts
+
+
+insightface_app = None
+instightface_mp = None
+
+
+def get_app(mp_name):
+    from installer import installed, install
+    packages = [
+        ('insightface', 'insightface'),
+        ('git+https://github.com/tencent-ailab/IP-Adapter.git', 'ip_adapter'),
+    ]
+    for pkg in packages:
+        if not installed(pkg[1], reload=False, quiet=True):
+            install(pkg[0], pkg[1], ignore=True)
+    global insightface_app, instightface_mp # pylint: disable=global-statement
+    if insightface_app is None or mp_name != instightface_mp:
+        import onnxruntime
+        from insightface.app import FaceAnalysis
+        import huggingface_hub as hf
+        import zipfile
+        log.debug(f"InsightFace: mp={mp_name} device={onnxruntime.get_device()} providers={onnxruntime.get_available_providers()}")
+        root_dir = os.path.join(opts.diffusers_dir, 'models--vladmandic--insightface-faceanalysis')
+        local_dir = os.path.join(root_dir, 'models')
+        extract_dir = os.path.join(local_dir, mp_name)
+        model_path = hf.hf_hub_download(
+            repo_id='vladmandic/insightface-faceanalysis',
+            filename=f'{mp_name}.zip',
+            local_dir_use_symlinks=False,
+            cache_dir=opts.diffusers_dir,
+            local_dir=local_dir
+        )
+        if not os.path.exists(extract_dir):
+            log.debug(f"InsightFace extract: folder={extract_dir}")
+            os.makedirs(extract_dir)
+            with zipfile.ZipFile(model_path) as zf:
+                zf.extractall(local_dir)
+        kwargs = {
+            'root': root_dir,
+            'download': False,
+            'download_zip': False,
+        }
+        insightface_app = FaceAnalysis(name=mp_name, providers=['CUDAExecutionProvider', 'CPUExecutionProvider'], **kwargs)
+        instightface_mp = mp_name
+        onnxruntime.set_default_logger_severity(3)
+        insightface_app.prepare(ctx_id=0, det_thresh=0.5, det_size=(640, 640))
+    return insightface_app

modules/face/instantid.py

@@ -0,0 +1,86 @@
+import os
+import cv2
+import numpy as np
+import huggingface_hub as hf
+from modules import shared, processing, sd_models, devices
+
+
+REPO_ID = "InstantX/InstantID"
+controlnet_model = None
+debug = shared.log.trace if os.environ.get('SD_FACE_DEBUG', None) is not None else lambda *args, **kwargs: None
+
+
+def instant_id(p: processing.StableDiffusionProcessing, app, source_image, strength=1.0, conditioning=0.5, cache=True): # pylint: disable=arguments-differ
+    from modules.face.instantid_model import StableDiffusionXLInstantIDPipeline, draw_kps
+    from diffusers.models import ControlNetModel
+    global controlnet_model # pylint: disable=global-statement
+
+    # prepare pipeline
+    if source_image is None:
+        shared.log.warning('InstantID: no input images')
+        return None
+
+    c = shared.sd_model.__class__.__name__ if shared.sd_model is not None else ''
+    if c != 'StableDiffusionXLPipeline':
+        shared.log.warning(f'InstantID invalid base model: current={c} required=StableDiffusionXLPipeline')
+        return None
+
+    # prepare face emb
+    faces = app.get(cv2.cvtColor(np.array(source_image), cv2.COLOR_RGB2BGR))
+    face = sorted(faces, key=lambda x:(x['bbox'][2]-x['bbox'][0])*x['bbox'][3]-x['bbox'][1])[-1]  # only use the maximum face
+    face_emb = face['embedding']
+    face_kps = draw_kps(source_image, face['kps'])
+
+    shared.log.debug(f'InstantID face: score={face.det_score:.2f} gender={"female" if face.gender==0 else "male"} age={face.age} bbox={face.bbox}')
+    shared.log.debug(f'InstantID loading: model={REPO_ID}')
+    face_adapter = hf.hf_hub_download(repo_id=REPO_ID, filename="ip-adapter.bin")
+    if controlnet_model is None:
+        controlnet_model = ControlNetModel.from_pretrained(REPO_ID, subfolder="ControlNetModel", torch_dtype=devices.dtype, cache_dir=shared.opts.diffusers_dir)
+
+    processing.process_init(p)
+
+    # create new pipeline
+    orig_pipeline = shared.sd_model # backup current pipeline definition
+    shared.sd_model = StableDiffusionXLInstantIDPipeline(
+        vae = shared.sd_model.vae,
+        text_encoder=shared.sd_model.text_encoder,
+        text_encoder_2=shared.sd_model.text_encoder_2,
+        tokenizer=shared.sd_model.tokenizer,
+        tokenizer_2=shared.sd_model.tokenizer_2,
+        unet=shared.sd_model.unet,
+        scheduler=shared.sd_model.scheduler,
+        controlnet=controlnet_model,
+        force_zeros_for_empty_prompt=shared.opts.diffusers_force_zeros,
+    )
+    sd_models.copy_diffuser_options(shared.sd_model, orig_pipeline) # copy options from original pipeline
+    sd_models.set_diffuser_options(shared.sd_model) # set all model options such as fp16, offload, etc.
+    shared.sd_model.load_ip_adapter_instantid(face_adapter, scale=strength)
+    shared.sd_model.set_ip_adapter_scale(strength)
+    if not ((shared.opts.diffusers_model_cpu_offload or shared.cmd_opts.medvram) or (shared.opts.diffusers_seq_cpu_offload or shared.cmd_opts.lowvram)):
+        shared.sd_model.to(shared.device, devices.dtype) # move pipeline if needed, but don't touch if its under automatic managment
+
+    # pipeline specific args
+    orig_prompt_attention = shared.opts.prompt_attention
+    shared.opts.data['prompt_attention'] = 'Fixed attention' # otherwise need to deal with class_tokens_mask
+    p.task_args['prompt'] = p.prompt # override all logic
+    p.task_args['image_embeds'] = face_emb
+    p.task_args['image'] = face_kps
+    p.task_args['controlnet_conditioning_scale'] = float(conditioning)
+    p.task_args['ip_adapter_scale'] = float(strength)
+    debug(f'InstantID: args={p.task_args}')
+
+    # run processing
+    shared.log.debug(f'InstantID: strength={strength} conditioning={conditioning} image={source_image}')
+    processed: processing.Processed = processing.process_images(p)
+    shared.sd_model.set_ip_adapter_scale(0)
+    p.extra_generation_params['InstantID'] = f'{strength}/{conditioning}'
+    p.extra_generation_params["Face"] = f'{face.det_score:.2f} {"female" if face.gender==0 else "male"} {face.age}y'
+
+    if not cache:
+        controlnet_model = None
+        devices.torch_gc()
+
+    # restore original pipeline
+    shared.opts.data['prompt_attention'] = orig_prompt_attention
+    shared.sd_model = orig_pipeline
+    return processed

modules/face/photomaker.py

@@ -0,0 +1,73 @@
+import os
+import huggingface_hub as hf
+from modules import shared, processing, sd_models
+
+
+def photo_maker(p: processing.StableDiffusionProcessing, input_images, trigger, strength, start): # pylint: disable=arguments-differ
+    from modules.face.photomaker_model import PhotoMakerStableDiffusionXLPipeline
+
+    # prepare pipeline
+    if len(input_images) == 0:
+        shared.log.warning('PhotoMaker: no input images')
+        return None
+
+    c = shared.sd_model.__class__.__name__ if shared.sd_model is not None else ''
+    if c != 'StableDiffusionXLPipeline':
+        shared.log.warning(f'PhotoMaker invalid base model: current={c} required=StableDiffusionXLPipeline')
+        return None
+
+    # validate prompt
+    trigger_ids = shared.sd_model.tokenizer.encode(trigger) + shared.sd_model.tokenizer_2.encode(trigger)
+    prompt_ids1 = shared.sd_model.tokenizer.encode(p.prompt)
+    prompt_ids2 = shared.sd_model.tokenizer_2.encode(p.prompt)
+    for t in trigger_ids:
+        if prompt_ids1.count(t) != 1:
+            shared.log.error(f'PhotoMaker: trigger word not matched in prompt: {trigger} ids={trigger_ids} prompt={p.prompt} ids={prompt_ids1}')
+            return None
+        if prompt_ids2.count(t) != 1:
+            shared.log.error(f'PhotoMaker: trigger word not matched in prompt: {trigger} ids={trigger_ids} prompt={p.prompt} ids={prompt_ids1}')
+            return None
+
+    # create new pipeline
+    orig_pipeline = shared.sd_model # backup current pipeline definition
+    shared.sd_model = PhotoMakerStableDiffusionXLPipeline(
+        vae = shared.sd_model.vae,
+        text_encoder=shared.sd_model.text_encoder,
+        text_encoder_2=shared.sd_model.text_encoder_2,
+        tokenizer=shared.sd_model.tokenizer,
+        tokenizer_2=shared.sd_model.tokenizer_2,
+        unet=shared.sd_model.unet,
+        scheduler=shared.sd_model.scheduler,
+        force_zeros_for_empty_prompt=shared.opts.diffusers_force_zeros,
+    )
+    sd_models.copy_diffuser_options(shared.sd_model, orig_pipeline) # copy options from original pipeline
+    sd_models.set_diffuser_options(shared.sd_model) # set all model options such as fp16, offload, etc.
+    if not ((shared.opts.diffusers_model_cpu_offload or shared.cmd_opts.medvram) or (shared.opts.diffusers_seq_cpu_offload or shared.cmd_opts.lowvram)):
+        shared.sd_model.to(shared.device) # move pipeline if needed, but don't touch if its under automatic managment
+
+    orig_prompt_attention = shared.opts.prompt_attention
+    shared.opts.data['prompt_attention'] = 'Fixed attention' # otherwise need to deal with class_tokens_mask
+    p.task_args['input_id_images'] = input_images
+    p.task_args['start_merge_step'] = int(start * p.steps)
+    p.task_args['prompt'] = p.prompt # override all logic
+
+    photomaker_path = hf.hf_hub_download(repo_id="TencentARC/PhotoMaker", filename="photomaker-v1.bin", repo_type="model", cache_dir=shared.opts.diffusers_dir)
+    shared.log.debug(f'PhotoMaker: model={photomaker_path} images={len(input_images)} trigger={trigger} args={p.task_args}')
+
+    # load photomaker adapter
+    shared.sd_model.load_photomaker_adapter(
+        os.path.dirname(photomaker_path),
+        subfolder="",
+        weight_name=os.path.basename(photomaker_path),
+        trigger_word=trigger
+    )
+    shared.sd_model.set_adapters(["photomaker"], adapter_weights=[strength])
+
+    # run processing
+    processed: processing.Processed = processing.process_images(p)
+    p.extra_generation_params['PhotoMaker'] = f'{strength}'
+
+    # restore original pipeline
+    shared.opts.data['prompt_attention'] = orig_prompt_attention
+    shared.sd_model = orig_pipeline
+    return processed

modules/face/photomaker_model.py

@@ -533,7 +533,7 @@ class PhotoMakerStableDiffusionXLPipeline(StableDiffusionXLPipeline):
             self.upcast_vae()
             latents = latents.to(next(iter(self.vae.post_quant_conv.parameters())).dtype)
 
-        if not output_type == "latent":
+        if output_type != "latent":
             image = self.vae.decode(latents / self.vae.config.scaling_factor, return_dict=False)[0]
         else:
             image = latents

modules/processing_diffusers.py

@@ -297,6 +297,8 @@ def process_diffusers(p: processing.StableDiffusionProcessing):
             clean['negative_prompt_embeds'] = clean['negative_prompt_embeds'].shape if torch.is_tensor(clean['negative_prompt_embeds']) else type(clean['negative_prompt_embeds'])
         if 'negative_pooled_prompt_embeds' in clean:
             clean['negative_pooled_prompt_embeds'] = clean['negative_pooled_prompt_embeds'].shape if torch.is_tensor(clean['negative_pooled_prompt_embeds']) else type(clean['negative_pooled_prompt_embeds'])
+        if 'image_embeds' in clean:
+            clean['image_embeds'] = clean['image_embeds'].shape if torch.is_tensor(clean['image_embeds']) else type(clean['image_embeds'])
         clean['generator'] = generator_device
         clean['parser'] = parser
         shared.log.debug(f'Diffuser pipeline: {model.__class__.__name__} task={sd_models.get_diffusers_task(model)} set={clean}')

modules/scripts.py

@@ -247,7 +247,7 @@ def load_scripts():
     scripts_data.clear()
     postprocessing_scripts_data.clear()
     script_callbacks.clear_callbacks()
-    scripts_list = list_scripts("scripts", ".py")
+    scripts_list = list_scripts('scripts', '.py') + list_scripts(os.path.join('modules', 'face'), '.py')
     syspath = sys.path
 
     def register_scripts_from_module(module, scriptfile):

modules/ui_control.py

@@ -10,7 +10,6 @@ from modules.control.units import xs # vislearn ControlNet-XS
 from modules.control.units import lite # vislearn ControlNet-XS
 from modules.control.units import t2iadapter # TencentARC T2I-Adapter
 from modules.control.units import reference # reference pipeline
-from scripts import ipadapter # pylint: disable=no-name-in-module
 from modules import errors, shared, progress, sd_samplers, ui_components, ui_symbols, ui_common, ui_sections, generation_parameters_copypaste, call_queue, scripts, masking # pylint: disable=ungrouped-imports
 
 
@@ -438,7 +437,8 @@ def create_ui(_blocks: gr.Blocks=None):
                     with gr.Row():
                         with gr.Column():
                             gr.HTML('<a href="https://github.com/tencent-ailab/IP-Adapter">IP-Adapter</a>')
-                            ip_adapter = gr.Dropdown(label='Adapter', choices=ipadapter.ADAPTERS, value='none')
+                            from scripts.ipadapter import ADAPTERS # pylint: disable=no-name-in-module
+                            ip_adapter = gr.Dropdown(label='Adapter', choices=ADAPTERS, value='none')
                             ip_scale = gr.Slider(label='Scale', minimum=0.0, maximum=1.0, step=0.01, value=0.5)
                         with gr.Column():
                             ip_image = gr.Image(label="Input", show_label=False, type="pil", source="upload", interactive=True, tool="editor", height=256, width=256)

modules/ui_extensions.py

@@ -6,7 +6,7 @@ import html
 from datetime import datetime, timedelta
 import git
 import gradio as gr
-from modules import extensions, shared, paths, errors
+from modules import extensions, shared, paths, errors, ui_symbols
 
 
 extensions_index = "https://vladmandic.github.io/sd-data/pages/extensions.json"
@@ -372,27 +372,27 @@ def create_html(search_text, sort_column):
         if ext.get('status', None) is None or type(ext['status']) == str: # old format
             ext['status'] = 0
         if ext['url'] is None or ext['url'] == '':
-            status = "<span style='cursor:pointer;color:#00C0FD' title='Local'>⬤</span>"
+            status = f"<span style='cursor:pointer;color:#00C0FD' title='Local'>{ui_symbols.bullet}</span>"
         elif ext['status'] > 0:
             if ext['status'] == 1:
-                status = "<span style='cursor:pointer;color:#00FD9C ' title='Verified'>⬤</span>"
+                status = f"<span style='cursor:pointer;color:#00FD9C ' title='Verified'>{ui_symbols.bullet}</span>"
             elif ext['status'] == 2:
-                status = "<span style='cursor:pointer;color:#FFC300' title='Supported only with backend:Original'>⬤</span>"
+                status = f"<span style='cursor:pointer;color:#FFC300' title='Supported only with backend:Original'>{ui_symbols.bullet}</span>"
             elif ext['status'] == 3:
-                status = "<span style='cursor:pointer;color:#FFC300' title='Supported only with backend:Diffusers'>⬤</span>"
+                status = f"<span style='cursor:pointer;color:#FFC300' title='Supported only with backend:Diffusers'>{ui_symbols.bullet}</span>"
             elif ext['status'] == 4:
-                status = f"<span style='cursor:pointer;color:#4E22FF' title=\"{ext.get('note', 'custom value')}\">⬤</span>"
+                status = f"<span style='cursor:pointer;color:#4E22FF' title=\"{ext.get('note', 'custom value')}\">{ui_symbols.bullet}</span>"
             elif ext['status'] == 5:
-                status = "<span style='cursor:pointer;color:#CE0000' title='Not supported'>⬤</span>"
+                status = f"<span style='cursor:pointer;color:#CE0000' title='Not supported'>{ui_symbols.bullet}</span>"
             elif ext['status'] == 6:
-                status = "<span style='cursor:pointer;color:#AEAEAE' title='Just discovered'>⬤</span>"
+                status = f"<span style='cursor:pointer;color:#AEAEAE' title='Just discovered'>{ui_symbols.bullet}</span>"
             else:
-                status = "<span style='cursor:pointer;color:#008EBC' title='Unknown status'>⬤</span>"
+                status = f"<span style='cursor:pointer;color:#008EBC' title='Unknown status'>{ui_symbols.bullet}</span>"
         else:
             if updated < datetime.timestamp(datetime.now() - timedelta(6*30)):
-                status = "<span style='cursor:pointer;color:#C000CF' title='Unmaintained'>⬤</span>"
+                status = f"<span style='cursor:pointer;color:#C000CF' title='Unmaintained'>{ui_symbols.bullet}</span>"
             else:
-                status = "<span style='cursor:pointer;color:#7C7C7C' title='No info'>⬤</span>"
+                status = f"<span style='cursor:pointer;color:#7C7C7C' title='No info'>{ui_symbols.bullet}</span>"
 
         code += f"""
             <tr style="display: {visible}">
@@ -434,7 +434,12 @@ def create_ui():
                         check = gr.Button(value="Update all installed", variant="primary")
                         apply = gr.Button(value="Apply changes", variant="primary")
                 list_extensions()
-                gr.HTML('<span style="color: var(--body-text-color)"><h2>Extension list</h2>⯀ Refesh extension list to download latest list with status<br>⯀ Check status of an extension by looking at status icon before installing it<br>⯀ After any operation such as install/uninstall or enable/disable, please restart the server<br></span>')
+                gr.HTML('''<span style="color: var(--body-text-color)">
+                            <h2>Extension list</h2>
+                            - Refesh extension list to download latest list with status<br>
+                            - Check status of an extension by looking at status icon before installing it<br>
+                            - After any operation such as install/uninstall or enable/disable, please restart the server<br>
+                        </span>''')
                 gr.HTML('')
                 info = gr.HTML('')
                 extensions_table = gr.HTML(create_html(search_text.value, sort_column.value))

modules/ui_symbols.py

@@ -27,6 +27,7 @@ mark_diag = '※'
 mark_flag = '⁜'
 int_clip = '✎'
 int_blip = '✐'
+bullet = '⃝'
 """
 refresh = '🔄'
 close = '🛗'

pyproject.toml

@@ -42,6 +42,7 @@ exclude = [
   "modules/control/proc/normalbae/nets/submodules/efficientnet_repo/geffnet",
   "modules/control/units/*_model.py",
   "modules/control/units/*_pipe.py",
+  "modules/pipelines/*.py",
 ]
 ignore = [
 	"A003", # Class attirbute shadowing builtin

scripts/faceid.py

@@ -1,273 +0,0 @@
-import os
-import cv2
-import torch
-import numpy as np
-import gradio as gr
-import diffusers
-import huggingface_hub as hf
-from PIL import Image
-from modules import scripts, processing, shared, devices, images
-
-
-debug = shared.log.trace if os.environ.get('SD_FACEID_DEBUG', None) is not None else lambda *args, **kwargs: None
-MODELS = {
-    'FaceID Base': 'h94/IP-Adapter-FaceID/ip-adapter-faceid_sd15.bin',
-    'FaceID Plus': 'h94/IP-Adapter-FaceID/ip-adapter-faceid-plus_sd15.bin',
-    'FaceID Plus v2': 'h94/IP-Adapter-FaceID/ip-adapter-faceid-plusv2_sd15.bin',
-    'FaceID XL': 'h94/IP-Adapter-FaceID/ip-adapter-faceid_sdxl.bin'
-}
-app = None
-ip_model = None
-ip_model_name = None
-ip_model_tokens = None
-ip_model_rank = None
-swapper = None
-
-
-def dependencies():
-    from installer import installed, install
-    packages = [
-        ('insightface', 'insightface'),
-        ('git+https://github.com/tencent-ailab/IP-Adapter.git', 'ip_adapter'),
-    ]
-    for pkg in packages:
-        if not installed(pkg[1], reload=False, quiet=True):
-            install(pkg[0], pkg[1], ignore=True)
-
-
-def face_id(p: processing.StableDiffusionProcessing, faces, image, model, override, tokens, rank, cache, scale, structure):
-    global ip_model, ip_model_name, ip_model_tokens, ip_model_rank # pylint: disable=global-statement
-    from insightface.utils import face_align
-    from ip_adapter.ip_adapter_faceid import IPAdapterFaceID, IPAdapterFaceIDPlus, IPAdapterFaceIDXL
-
-    face_embeds = torch.from_numpy(faces[0].normed_embedding).unsqueeze(0)
-    face_image = face_align.norm_crop(image, landmark=faces[0].kps, image_size=224) # you can also segment the face
-
-    ip_ckpt = MODELS[model]
-    folder, filename = os.path.split(ip_ckpt)
-    basename, _ext = os.path.splitext(filename)
-    model_path = hf.hf_hub_download(repo_id=folder, filename=filename, cache_dir=shared.opts.diffusers_dir)
-    if model_path is None:
-        shared.log.error(f'FaceID download failed: model={model} file={ip_ckpt}')
-        return None
-
-    processing.process_init(p)
-    if override:
-        shared.sd_model.scheduler = diffusers.DDIMScheduler(
-            num_train_timesteps=1000,
-            beta_start=0.00085,
-            beta_end=0.012,
-            beta_schedule="scaled_linear",
-            clip_sample=False,
-            set_alpha_to_one=False,
-            steps_offset=1,
-        )
-    shortcut = None
-    if ip_model is None or ip_model_name != model or ip_model_tokens != tokens or ip_model_rank != rank or not cache:
-        shared.log.debug(f'FaceID load: model={model} file={ip_ckpt} tokens={tokens} rank={rank}')
-        if 'Plus' in model:
-            image_encoder_path = "laion/CLIP-ViT-H-14-laion2B-s32B-b79K"
-            ip_model = IPAdapterFaceIDPlus(
-                sd_pipe=shared.sd_model,
-                image_encoder_path=image_encoder_path,
-                ip_ckpt=model_path,
-                lora_rank=rank,
-                num_tokens=tokens,
-                device=devices.device,
-                torch_dtype=devices.dtype,
-            )
-            shortcut = 'v2' in model
-        elif 'XL' in model:
-            ip_model = IPAdapterFaceIDXL(
-                sd_pipe=shared.sd_model,
-                ip_ckpt=model_path,
-                lora_rank=rank,
-                num_tokens=tokens,
-                device=devices.device,
-                torch_dtype=devices.dtype,
-            )
-        else:
-            ip_model = IPAdapterFaceID(
-                sd_pipe=shared.sd_model,
-                ip_ckpt=model_path,
-                lora_rank=rank,
-                num_tokens=tokens,
-                device=devices.device,
-                torch_dtype=devices.dtype,
-            )
-        ip_model_name = model
-        ip_model_tokens = tokens
-        ip_model_rank = rank
-    else:
-        shared.log.debug(f'FaceID cached: model={model} file={ip_ckpt} tokens={tokens} rank={rank}')
-
-    # main generate dict
-    ip_model_dict = {
-        'num_samples': p.batch_size,
-        'width': p.width,
-        'height': p.height,
-        'num_inference_steps': p.steps,
-        'scale': scale,
-        'guidance_scale': p.cfg_scale,
-        'faceid_embeds': face_embeds.shape,
-    }
-
-    # optional generate dict
-    if shortcut is not None:
-        ip_model_dict['shortcut'] = shortcut
-    if 'Plus' in model:
-        ip_model_dict['s_scale'] = structure
-        ip_model_dict['face_image'] = face_image.shape
-    shared.log.debug(f'FaceID args: {ip_model_dict}')
-    if 'Plus' in model:
-        ip_model_dict['face_image'] = face_image
-    ip_model_dict['faceid_embeds'] = face_embeds
-
-    # run generate
-    processed_images = []
-    ip_model.set_scale(scale)
-    for i in range(p.n_iter):
-        ip_model_dict.update(
-            {
-                'prompt': p.all_prompts[i],
-                'negative_prompt': p.all_negative_prompts[i],
-                'seed': int(p.all_seeds[i]),
-            }
-        )
-        debug(f'FaceID: {ip_model_dict}')
-        res = ip_model.generate(**ip_model_dict)
-        if isinstance(res, list):
-            processed_images += res
-    ip_model.set_scale(0)
-
-    if not cache:
-        ip_model = None
-        ip_model_name = None
-    devices.torch_gc()
-
-    p.extra_generation_params["IP Adapter"] = f'{basename}:{scale}'
-    return processed_images
-
-
-def face_swap(p: processing.StableDiffusionProcessing, image, source_face):
-    import insightface.model_zoo
-    global swapper # pylint: disable=global-statement
-    if swapper is None:
-        model_path = hf.hf_hub_download(repo_id='ezioruan/inswapper_128.onnx', filename='inswapper_128.onnx', cache_dir=shared.opts.diffusers_dir)
-        router = insightface.model_zoo.model_zoo.ModelRouter(model_path)
-        swapper = router.get_model()
-
-    np_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR)
-    faces = app.get(np_image)
-
-    res = np_image.copy()
-    for target_face in faces:
-        res = swapper.get(res, target_face, source_face, paste_back=True) # pylint: disable=too-many-function-args, unexpected-keyword-arg
-
-    p.extra_generation_params["FaceSwap"] = f'{len(faces)}'
-    np_image = cv2.cvtColor(res, cv2.COLOR_BGR2RGB)
-    return Image.fromarray(np_image)
-
-
-class Script(scripts.Script):
-    def title(self):
-        return 'FaceID'
-
-    def show(self, is_img2img):
-        return True if shared.backend == shared.Backend.DIFFUSERS else False
-
-    # return signature is array of gradio components
-    def ui(self, _is_img2img):
-        with gr.Row():
-            mode = gr.CheckboxGroup(label='Mode', choices=['FaceID', 'FaceSwap'], value=['FaceID'])
-            model = gr.Dropdown(choices=list(MODELS), label='FaceID Model', value='FaceID Base')
-        with gr.Row(visible=True):
-            override = gr.Checkbox(label='Override sampler', value=True)
-            cache = gr.Checkbox(label='Cache model', value=True)
-        with gr.Row(visible=True):
-            scale = gr.Slider(label='Strength', minimum=0.0, maximum=2.0, step=0.01, value=1.0)
-            structure = gr.Slider(label='Structure', minimum=0.0, maximum=1.0, step=0.01, value=1.0)
-        with gr.Row(visible=False):
-            rank = gr.Slider(label='Rank', minimum=4, maximum=256, step=4, value=128)
-            tokens = gr.Slider(label='Tokens', minimum=1, maximum=16, step=1, value=4)
-        with gr.Row():
-            image = gr.Image(image_mode='RGB', label='Image', source='upload', type='pil', width=512)
-        return [mode, model, scale, image, override, rank, tokens, structure, cache]
-
-    def run(self, p: processing.StableDiffusionProcessing, mode, model, scale, image, override, rank, tokens, structure, cache): # pylint: disable=arguments-differ, unused-argument
-        if len(mode) == 0:
-            return None
-        dependencies()
-        try:
-            import onnxruntime
-            from insightface.app import FaceAnalysis
-        except Exception as e:
-            shared.log.error(f'FaceID: {e}')
-            return None
-        if image is None:
-            shared.log.error('FaceID: no init_images')
-            return None
-        if shared.sd_model_type != 'sd' and shared.sd_model_type != 'sdxl':
-            shared.log.error('FaceID: base model not supported')
-            return None
-
-        global app # pylint: disable=global-statement
-        if app is None:
-            shared.log.debug(f"ONNX: device={onnxruntime.get_device()} providers={onnxruntime.get_available_providers()}")
-            app = FaceAnalysis(name="buffalo_l", providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
-            onnxruntime.set_default_logger_severity(3)
-            app.prepare(ctx_id=0, det_thresh=0.5, det_size=(640, 640))
-
-        if isinstance(image, str):
-            from modules.api.api import decode_base64_to_image
-            image = decode_base64_to_image(image).convert("RGB")
-
-        np_image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR)
-        faces = app.get(np_image)
-        if len(faces) == 0:
-            shared.log.error('FaceID: no faces found')
-            return None
-        for i, face in enumerate(faces):
-            shared.log.debug(f'FaceID face: i={i+1} score={face.det_score:.2f} gender={"female" if face.gender==0 else "male"} age={face.age} bbox={face.bbox}')
-            p.extra_generation_params[f"FaceID {i+1}"] = f'{face.det_score:.2f} {"female" if face.gender==0 else "male"} {face.age}y'
-
-        processed_images = []
-        if 'FaceID' in mode:
-            processed_images = face_id(p, faces, np_image, model, override, tokens, rank, cache, scale, structure) # run faceid pipeline
-            processed = processing.Processed(
-                p,
-                images_list=processed_images,
-                seed=p.seed,
-                subseed=p.subseed,
-                index_of_first_image=0,
-            )
-            if 'FaceSwap' not in mode:
-                if shared.opts.samples_save and not p.do_not_save_samples:
-                    for i, image in enumerate(processed.images):
-                        info = processing.create_infotext(p, index=i)
-                        images.save_image(image, path=p.outpath_samples, seed=p.all_seeds[i], prompt=p.all_prompts[i], info=info, p=p)
-            else:
-                if shared.opts.save_images_before_face_restoration and not p.do_not_save_samples:
-                    for i, image in enumerate(processed.images):
-                        info = processing.create_infotext(p, index=i)
-                        images.save_image(image, path=p.outpath_samples, seed=p.all_seeds[i], prompt=p.all_prompts[i], info=info, p=p, suffix="-before-face-swap")
-
-        else:
-            processed = processing.process_images(p) # run normal pipeline
-            processed_images = processed.images
-
-        if 'FaceSwap' in mode: # replace faces as postprocess
-            processed.images = []
-            for batch_image in processed_images:
-                swapped_image = face_swap(p, batch_image, source_face=faces[0])
-                processed.images.append(swapped_image)
-
-            if shared.opts.samples_save and not p.do_not_save_samples:
-                for i, image in enumerate(processed.images):
-                    info = processing.create_infotext(p, index=i)
-                    images.save_image(image, path=p.outpath_samples, seed=p.all_seeds[i], prompt=p.all_prompts[i], info=info, p=p)
-
-        processed.info = processed.infotext(p, 0)
-        processed.infotexts = [processed.info]
-
-        return processed

scripts/photomaker.py

@@ -1,118 +0,0 @@
-import os
-import gradio as gr
-import huggingface_hub as hf
-from PIL import Image
-from modules import shared, processing, sd_models, scripts
-
-
-class Script(scripts.Script):
-    def title(self):
-        return 'PhotoMaker'
-
-    def show(self, is_img2img):
-        return True if shared.backend == shared.Backend.DIFFUSERS else False
-
-    def load_images(self, files):
-        init_images = []
-        for file in files or []:
-            try:
-                if isinstance(file, str):
-                    from modules.api.api import decode_base64_to_image
-                    image = decode_base64_to_image(file)
-                elif isinstance(file, Image.Image):
-                    image = file
-                elif isinstance(file, dict) and 'name' in file:
-                    image = Image.open(file['name']) # _TemporaryFileWrapper from gr.Files
-                elif hasattr(file, 'name'):
-                    image = Image.open(file.name) # _TemporaryFileWrapper from gr.Files
-                else:
-                    raise ValueError(f'PhotoMaker unknown input: {file}')
-                init_images.append(image)
-            except Exception as e:
-                shared.log.warning(f'PhotoMaker failed to load image: {e}')
-        return init_images
-
-    def ui(self, _is_img2img):
-        with gr.Row():
-            trigger = gr.Text(label='Trigger word', value="person")
-            strength = gr.Slider(label='Strength', minimum=0.0, maximum=2.0, step=0.01, value=1.0)
-            start = gr.Slider(label='Start', minimum=0.0, maximum=1.0, step=0.01, value=0.5)
-        with gr.Row():
-            files = gr.File(label='Input images', file_count='multiple', file_types=['image'], type='file', interactive=True, height=100)
-        with gr.Row():
-            gallery = gr.Gallery(show_label=False, value=[])
-        with gr.Row():
-            gr.HTML('<a href="https://github.com/TencentARC/PhotoMaker>PhotoMaker</a>"')
-        files.change(fn=self.load_images, inputs=[files], outputs=[gallery])
-        return [trigger, strength, start, gallery]
-
-    # Run pipeline
-    def run(self, p: processing.StableDiffusionProcessing, trigger, strength, start, images): # pylint: disable=arguments-differ
-        from scripts.photomaker_model import PhotoMakerStableDiffusionXLPipeline # pylint: disable=no-name-in-module
-
-        # prepare pipeline
-        input_images = self.load_images(images)
-        if len(input_images) == 0:
-            shared.log.warning('PhotoMaker: no input images')
-            return None
-
-        c = shared.sd_model.__class__.__name__ if shared.sd_model is not None else ''
-        if c != 'StableDiffusionXLPipeline':
-            shared.log.warning(f'PhotoMaker invalid base model: current={c} required=StableDiffusionXLPipeline')
-            return None
-
-        # validate prompt
-        trigger_ids = shared.sd_model.tokenizer.encode(trigger) + shared.sd_model.tokenizer_2.encode(trigger)
-        prompt_ids1 = shared.sd_model.tokenizer.encode(p.prompt)
-        prompt_ids2 = shared.sd_model.tokenizer_2.encode(p.prompt)
-        for t in trigger_ids:
-            if prompt_ids1.count(t) != 1:
-                shared.log.error(f'PhotoMaker: trigger word not matched in prompt: {trigger} ids={trigger_ids} prompt={p.prompt} ids={prompt_ids1}')
-                return None
-            if prompt_ids2.count(t) != 1:
-                shared.log.error(f'PhotoMaker: trigger word not matched in prompt: {trigger} ids={trigger_ids} prompt={p.prompt} ids={prompt_ids1}')
-                return None
-
-        # create new pipeline
-        orig_pipeline = shared.sd_model # backup current pipeline definition
-        shared.sd_model = PhotoMakerStableDiffusionXLPipeline(
-            vae = shared.sd_model.vae,
-            text_encoder=shared.sd_model.text_encoder,
-            text_encoder_2=shared.sd_model.text_encoder_2,
-            tokenizer=shared.sd_model.tokenizer,
-            tokenizer_2=shared.sd_model.tokenizer_2,
-            unet=shared.sd_model.unet,
-            scheduler=shared.sd_model.scheduler,
-            force_zeros_for_empty_prompt=shared.opts.diffusers_force_zeros,
-        )
-        sd_models.copy_diffuser_options(shared.sd_model, orig_pipeline) # copy options from original pipeline
-        sd_models.set_diffuser_options(shared.sd_model) # set all model options such as fp16, offload, etc.
-        if not ((shared.opts.diffusers_model_cpu_offload or shared.cmd_opts.medvram) or (shared.opts.diffusers_seq_cpu_offload or shared.cmd_opts.lowvram)):
-            shared.sd_model.to(shared.device) # move pipeline if needed, but don't touch if its under automatic managment
-
-        orig_prompt_attention = shared.opts.prompt_attention
-        shared.opts.data['prompt_attention'] = 'Fixed attention' # otherwise need to deal with class_tokens_mask
-        p.task_args['input_id_images'] = input_images
-        p.task_args['start_merge_step'] = int(start * p.steps)
-        p.task_args['prompt'] = p.prompt # override all logic
-
-        photomaker_path = hf.hf_hub_download(repo_id="TencentARC/PhotoMaker", filename="photomaker-v1.bin", repo_type="model", cache_dir=shared.opts.diffusers_dir)
-        shared.log.debug(f'PhotoMaker: model={photomaker_path} images={len(input_images)} trigger={trigger} args={p.task_args}')
-
-        # load photomaker adapter
-        shared.sd_model.load_photomaker_adapter(
-            os.path.dirname(photomaker_path),
-            subfolder="",
-            weight_name=os.path.basename(photomaker_path),
-            trigger_word=trigger
-        )
-        shared.sd_model.set_adapters(["photomaker"], adapter_weights=[strength])
-
-        # run processing
-        processed: processing.Processed = processing.process_images(p)
-        p.extra_generation_params['PhotoMaker'] = f'{strength}'
-
-        # restore original pipeline
-        shared.opts.data['prompt_attention'] = orig_prompt_attention
-        shared.sd_model = orig_pipeline
-        return processed