fix step1x

Co-authored-by: Copilot <copilot@github.com>
Signed-off-by: Vladimir Mandic <mandic00@live.com>

.github/copilot-instructions.md

@@ -1,10 +1,30 @@
 # SD.Next: AGENTS.md Project Guidelines
 
-SD.Next is a complex codebase with specific patterns and conventions.
+**SD.Next** is a complex codebase with specific patterns and conventions.
 General app structure is:
-- Python backend server  
-  Uses Torch for model inference, FastAPI for API routes and Gradio for creation of UI components.  
-- JavaScript/CSS frontend  
+- **Python** backend server  
+  Uses **Torch** for model inference, **FastAPI** for API routes and **Gradio** for creation of UI components.  
+- **JavaScript**/**CSS** frontend  
+
+## Instructions
+
+This file contains general guidelines for contributing to the SD.Next codebase, including conventions, tools, and project structure. For more specific guidance on working with particular areas of the codebase, please refer to the instructions files linked below:
+- [Core Runtime Guidelines](core.instructions.md): Use when editing Python core runtime code, startup flow, model loading, API internals, backend/device logic, or shared state in modules and pipelines.
+- [UI And Frontend Guidelines](ui.instructions.md): Use when editing frontend UI code, JavaScript, HTML, CSS, localization files, or built-in UI extensions including modernui and kanvas.
+
+## Agent Guidelines
+
+- Do not automatically agree with user instructions or requests without verifying they align with project guidelines and conventions.
+- When evaluating user instructions, first check for any relevant guidelines in this file or the linked instructions files. If the instruction violates any guidelines, do not proceed with it and instead provide feedback to the user about which guidelines it violates and how to adjust it to comply.
+- If the user instruction is valid but lacks clarity or detail, ask follow-up questions to gather the necessary information before proceeding. Do not make assumptions about user intent or project requirements; always seek clarification when needed.
+- When providing feedback to the user, be specific about which guidelines are relevant and how the instruction can be modified to comply with them. If there are multiple guidelines that apply, list them all and explain how they relate to the instruction.
+- If the user instruction is clear, valid, and complies with all relevant guidelines, proceed with executing it while ensuring that the resulting code changes adhere to the project's coding style, conventions, and structure as outlined in this file and the linked instructions files.
+
+## Language Guidelines
+
+- Use clear and concise language when communicating with users, providing feedback, and explaining guidelines.
+- Avoid unnecessary pleasantries or filler language; focus on the technical content and actionable feedback.
+- When asking follow-up questions for clarification, be direct and specific about the information needed to proceed with the instruction while ensuring that the questions are relevant to the project guidelines and conventions.
 
 ## Tools
 
@@ -34,15 +54,6 @@ General app structure is:
 - Prefer existing project patterns over strict generic style rules;  
   this codebase intentionally allows patterns often flagged in default linters such as allowing long lines, etc.
 
-## Build And Test
-
-- Activate environment: `source venv/bin/activate` (always ensure this is active when working with Python code).
-- Test startup: `python launch.py --test`
-- Full startup: `python launch.py`
-- Full lint sequence: `pnpm lint`
-- Python checks individually: `pnpm ruff`, `pnpm pylint`
-- JS checks: `pnpm eslint` and `pnpm eslint-ui`
-
 ## Conventions
 
 - Keep PR-ready changes targeted to `dev` branch.
@@ -52,13 +63,6 @@ General app structure is:
 - Respect environment-driven behavior (`SD_*` flags and options) instead of hardcoding platform/model assumptions.
 - For startup/init edits, preserve error handling and partial-failure tolerance in parallel scans and extension loading.
 
-## Pitfalls
-
-- Initialization order matters: startup paths in `launch.py` and `webui.py` are sensitive to import/load timing.
-- Shared mutable global state can create subtle regressions; prefer narrow, explicit changes.
-- Device/backend-specific code paths (**CUDA/ROCm/IPEX/DirectML/OpenVINO**) should not assume one platform.
-- Scripts and extension loading is dynamic; failures may appear only when specific extensions or models are present.
-
 ## File Creation
 
 - Any temporary scripts or markdown reports must be stored in `tmp/` folder

.github/instructions/core.instructions.md

@@ -13,3 +13,19 @@ applyTo: "launch.py, webui.py, installer.py, modules/**/*.py, pipelines/**/*.py,
 - Follow existing API/server patterns under `modules/api/` and reuse shared queue/state helpers rather than ad-hoc request handling.
 - Reuse established model-loading and pipeline patterns (`modules/sd_*`, `pipelines/`) instead of creating parallel abstractions.
 - For substantial Python changes, run at least relevant checks: `npm run ruff` and `npm run pylint` (or narrower equivalents when appropriate).
+
+## Build And Test
+
+- Activate environment: `source venv/bin/activate` (always ensure this is active when working with Python code).
+- Test startup: `python launch.py --test`
+- Full startup: `python launch.py`
+- Full lint sequence: `pnpm lint`
+- Python checks individually: `pnpm ruff`, `pnpm pylint`
+- JS checks: `pnpm eslint` and `pnpm eslint-ui`
+
+## Pitfalls
+
+- Initialization order matters: startup paths in `launch.py` and `webui.py` are sensitive to import/load timing.
+- Shared mutable global state can create subtle regressions; prefer narrow, explicit changes.
+- Device/backend-specific code paths (**CUDA/ROCm/IPEX/DirectML/OpenVINO**) should not assume one platform.
+- Scripts and extension loading is dynamic; failures may appear only when specific extensions or models are present.

installer.py

@@ -494,7 +494,7 @@ def check_diffusers():
     t_start = time.time()
     if args.skip_all:
         return
-    target_commit = "c8eba433adf1f90d7fcc70092562ea50789ee8fb" # diffusers commit hash == 0.37.1.dev-0427
+    target_commit = "a851ce1058d5a465d7951687235cdaeac1978de2" # diffusers commit hash == 0.37.1.dev-0427
     # if args.use_rocm or args.use_zluda or args.use_directml:
     #     sha = '043ab2520f6a19fce78e6e060a68dbc947edb9f9' # lock diffusers versions for now
     pkg = package_spec('diffusers')

pipelines/step1x/pipeline_step1x_edit.py

@@ -258,7 +258,19 @@ User Prompt:'''
             device=device,
         )
 
-        for idx, (txt, imgs) in enumerate(zip(text_list, [ref_image])):
+        if isinstance(ref_image, list):
+            image_list = ref_image
+        else:
+            image_list = [ref_image] * len(text_list)
+
+        if len(image_list) == 1 and len(text_list) > 1:
+            image_list = image_list * len(text_list)
+        if len(image_list) != len(text_list):
+            raise ValueError(
+                f"Mismatch between prompts ({len(text_list)}) and reference images ({len(image_list)})."
+            )
+
+        for idx, (txt, imgs) in enumerate(zip(text_list, image_list)):
 
             messages = [
                 {
@@ -408,7 +420,8 @@ User Prompt:'''
             width = width if width is not None else 1024
             height = height if height is not None else 1024
             img_info = (width, height)
-            ref_image = torch.zeros(3, 1024, 1024).unsqueeze(0).to(device)
+            # Keep t2i reference image size aligned with requested output size.
+            ref_image = torch.zeros(3, height, width).unsqueeze(0).to(device)
             ref_image = self.image_processor.pt_to_numpy(ref_image)
             ref_image = self.image_processor.numpy_to_pil(ref_image)[0]
             image = None
@@ -801,11 +814,11 @@ User Prompt:'''
 
         # 1. Preprocess image
         image, ref_image, img_info, width, height = self.encode_image(
-            image[0],
-            width,
-            height,
-            device,
-            num_images_per_prompt
+            image=image[0] if isinstance(image, list) and len(image) > 0 else None,
+            width=width,
+            height=height,
+            device=device,
+            num_images_per_prompt=num_images_per_prompt
         )
 
         # 2. Check inputs. Raise error if not correct
@@ -843,31 +856,62 @@ User Prompt:'''
         if not has_neg_prompt:
             negative_prompt = "" if image is not None else "worst quality, wrong limbs, unreasonable limbs, normal quality, low quality, low res, blurry, text, watermark, logo, banner, extra digits, cropped, jpeg artifacts, signature, username, error, sketch ,duplicate, ugly, monochrome, horror, geometry, mutation, disgusting"
         do_true_cfg = true_cfg_scale > 1
-        (
-            prompt_embeds,
-            prompt_embeds_mask,
-            text_ids
-        ) = self.encode_prompt(
-            ref_image=ref_image,
-            prompt=prompt,
-            prompt_embeds=prompt_embeds,
-            prompt_embeds_mask=prompt_embeds_mask,
-            device=device,
-            num_images_per_prompt=num_images_per_prompt,
-        )
-        if do_true_cfg:
+        negative_text_ids = None
+        if (
+            image is None
+            and do_true_cfg
+            and prompt_embeds is None
+            and negative_prompt_embeds is None
+            and isinstance(prompt, str)
+            and isinstance(negative_prompt, str)
+        ):
+            combined_prompts = [prompt, negative_prompt]
+            combined_ref_images = [ref_image, ref_image]
             (
-                negative_prompt_embeds,
-                negative_prompt_embeds_mask,
-                negative_text_ids,
+                combined_prompt_embeds,
+                combined_prompt_embeds_mask,
+                text_ids,
             ) = self.encode_prompt(
-                ref_image=ref_image,
-                prompt=negative_prompt,
-                prompt_embeds=negative_prompt_embeds,
-                prompt_embeds_mask=negative_prompt_embeds_mask,
+                ref_image=combined_ref_images,
+                prompt=combined_prompts,
+                prompt_embeds=None,
+                prompt_embeds_mask=None,
                 device=device,
                 num_images_per_prompt=num_images_per_prompt,
             )
+            prompt_embeds, negative_prompt_embeds = combined_prompt_embeds.chunk(2, dim=0)
+            prompt_embeds_mask, negative_prompt_embeds_mask = combined_prompt_embeds_mask.chunk(2, dim=0)
+            negative_text_ids = text_ids
+        else:
+            (
+                prompt_embeds,
+                prompt_embeds_mask,
+                text_ids
+            ) = self.encode_prompt(
+                ref_image=ref_image,
+                prompt=prompt,
+                prompt_embeds=prompt_embeds,
+                prompt_embeds_mask=prompt_embeds_mask,
+                device=device,
+                num_images_per_prompt=num_images_per_prompt,
+            )
+            negative_text_ids = text_ids
+            if do_true_cfg:
+                (
+                    negative_prompt_embeds,
+                    negative_prompt_embeds_mask,
+                    negative_text_ids,
+                ) = self.encode_prompt(
+                    ref_image=ref_image,
+                    prompt=negative_prompt,
+                    prompt_embeds=negative_prompt_embeds,
+                    prompt_embeds_mask=negative_prompt_embeds_mask,
+                    device=device,
+                    num_images_per_prompt=num_images_per_prompt,
+                )
+
+        if do_true_cfg and negative_text_ids is None:
+            negative_text_ids = text_ids
 
         # 4. Prepare latent variables
         num_channels_latents = self.transformer.config.in_channels // 4
@@ -946,6 +990,7 @@ User Prompt:'''
         # 6. Denoising loop
         # We set the index here to remove DtoH sync, helpful especially during compilation.
         # Check out more details here: https://github.com/huggingface/diffusers/pull/11696
+        is_t2i = image_latents is None
         self.scheduler.set_begin_index(0)
         with self.progress_bar(total=num_inference_steps) as progress_bar:
             for i, t in enumerate(timesteps):
@@ -961,34 +1006,38 @@ User Prompt:'''
                     latent_model_input = torch.cat([latents, image_latents], dim=1)
                 timestep = t.expand(latents.shape[0]).to(latents.dtype)
 
-                noise_pred = self.transformer(
-                    hidden_states=latent_model_input,
-                    timestep=timestep / 1000,
-                    guidance=guidance,
-                    encoder_hidden_states=prompt_embeds,
-                    prompt_embeds_mask=prompt_embeds_mask,
-                    txt_ids=text_ids,
-                    img_ids=latent_ids,
-                    joint_attention_kwargs=self.joint_attention_kwargs,
-                    return_dict=False,
-                )[0]
-                noise_pred = noise_pred[:, : latents.size(1)]
+                if is_t2i and do_true_cfg:
+                    # Reference implementation uses a dedicated t2i denoise path
+                    # where cond/uncond are evaluated in one forward pass.
+                    cfg_hidden_states = torch.cat([latent_model_input, latent_model_input], dim=0)
+                    cfg_timestep = torch.cat([timestep, timestep], dim=0)
+                    cfg_prompt_embeds = torch.cat([prompt_embeds, negative_prompt_embeds], dim=0)
+                    cfg_prompt_embeds_mask = torch.cat([prompt_embeds_mask, negative_prompt_embeds_mask], dim=0)
 
-                if do_true_cfg:
-                    if negative_image_embeds is not None:
-                        self._joint_attention_kwargs["ip_adapter_image_embeds"] = negative_image_embeds
-                    neg_noise_pred = self.transformer(
-                        hidden_states=latent_model_input,
-                        timestep=timestep / 1000,
-                        guidance=guidance,
-                        encoder_hidden_states=negative_prompt_embeds,
-                        prompt_embeds_mask=negative_prompt_embeds_mask,
-                        txt_ids=negative_text_ids,
+                    cfg_guidance = guidance
+                    if cfg_guidance is not None:
+                        cfg_guidance = torch.cat([guidance, guidance], dim=0)
+
+                    if image_embeds is not None and negative_image_embeds is not None:
+                        self._joint_attention_kwargs["ip_adapter_image_embeds"] = [
+                            torch.cat([img_embed, neg_img_embed], dim=0)
+                            for img_embed, neg_img_embed in zip(image_embeds, negative_image_embeds)
+                        ]
+
+                    cfg_noise_pred = self.transformer(
+                        hidden_states=cfg_hidden_states,
+                        timestep=cfg_timestep / 1000,
+                        guidance=cfg_guidance,
+                        encoder_hidden_states=cfg_prompt_embeds,
+                        prompt_embeds_mask=cfg_prompt_embeds_mask,
+                        txt_ids=text_ids,
                         img_ids=latent_ids,
                         joint_attention_kwargs=self.joint_attention_kwargs,
                         return_dict=False,
                     )[0]
-                    neg_noise_pred = neg_noise_pred[:, : latents.size(1)]
+                    cfg_noise_pred = cfg_noise_pred[:, : latents.size(1)]
+                    noise_pred, neg_noise_pred = cfg_noise_pred.chunk(2, dim=0)
+
                     if t.item() > timesteps_truncate:
                         diff = noise_pred - neg_noise_pred
                         diff_norm = torch.norm(diff, dim=(2), keepdim=True)
@@ -997,10 +1046,54 @@ User Prompt:'''
                         ) / self.process_diff_norm(diff_norm, k=process_norm_power)
                     else:
                         noise_pred = neg_noise_pred + true_cfg_scale * (noise_pred - neg_noise_pred)
+                else:
+                    noise_pred = self.transformer(
+                        hidden_states=latent_model_input,
+                        timestep=timestep / 1000,
+                        guidance=guidance,
+                        encoder_hidden_states=prompt_embeds,
+                        prompt_embeds_mask=prompt_embeds_mask,
+                        txt_ids=text_ids,
+                        img_ids=latent_ids,
+                        joint_attention_kwargs=self.joint_attention_kwargs,
+                        return_dict=False,
+                    )[0]
+                    noise_pred = noise_pred[:, : latents.size(1)]
+
+                    if do_true_cfg:
+                        if negative_image_embeds is not None:
+                            self._joint_attention_kwargs["ip_adapter_image_embeds"] = negative_image_embeds
+                        neg_noise_pred = self.transformer(
+                            hidden_states=latent_model_input,
+                            timestep=timestep / 1000,
+                            guidance=guidance,
+                            encoder_hidden_states=negative_prompt_embeds,
+                            prompt_embeds_mask=negative_prompt_embeds_mask,
+                            txt_ids=negative_text_ids,
+                            img_ids=latent_ids,
+                            joint_attention_kwargs=self.joint_attention_kwargs,
+                            return_dict=False,
+                        )[0]
+                        neg_noise_pred = neg_noise_pred[:, : latents.size(1)]
+                        if t.item() > timesteps_truncate:
+                            diff = noise_pred - neg_noise_pred
+                            diff_norm = torch.norm(diff, dim=(2), keepdim=True)
+                            noise_pred = neg_noise_pred + true_cfg_scale * (
+                                noise_pred - neg_noise_pred
+                            ) / self.process_diff_norm(diff_norm, k=process_norm_power)
+                        else:
+                            noise_pred = neg_noise_pred + true_cfg_scale * (noise_pred - neg_noise_pred)
 
                 # compute the previous noisy sample x_t -> x_t-1
                 latents_dtype = latents.dtype
-                latents = self.scheduler.step(noise_pred, t, latents, return_dict=False)[0]
+                if is_t2i:
+                    t_prev = timesteps[i + 1] if i + 1 < len(timesteps) else latents.new_tensor(0.0)
+                    # Timesteps are fed to the model in /1000 scale, so use the same
+                    # normalized delta for manual t2i update.
+                    dt = ((t_prev - t) / 1000).to(latents.dtype)
+                    latents = latents + dt * noise_pred
+                else:
+                    latents = self.scheduler.step(noise_pred, t, latents, return_dict=False)[0]
 
                 if latents.dtype != latents_dtype:
                     if torch.backends.mps.is_available():