update ideogram4 requirements and global skills update

Signed-off-by: Vladimir Mandic <mandic00@live.com>
This commit is contained in:
Vladimir Mandic
2026-06-05 07:54:03 +02:00
parent 9c6d6907ea
commit 9ade767836
18 changed files with 98 additions and 10 deletions
+25 -2
View File
@@ -1,8 +1,22 @@
---
description: "Use when editing Python core runtime code, startup flow, model loading, API internals, backend/device logic, or shared state in modules and pipelines."
name: "Core Runtime Guidelines"
applyTo: "launch.py, webui.py, installer.py, modules/**/*.py, pipelines/**/*.py, scripts/**/*.py, extensions-builtin/**/*.py"
applyTo: "launch.py, webui.py, installer.py, modules/**/*.py, pipelines/**/*.py, scripts/**/*.py, extensions-builtin/**/*.py, cli/**/*.py"
---
## Agent Guidelines
1. Verify the user instruction against relevant guidelines in this file and linked instruction files before proceeding.
2. If the instruction conflicts with any guideline, do not proceed. Explain which guideline(s) it conflicts with and how to adjust the instruction to comply.
3. If the instruction is valid but unclear or incomplete, ask targeted follow-up questions before implementation. Do not assume user intent or requirements.
4. When giving feedback, name the applicable guideline(s) and explain how each one applies.
5. If the instruction is clear and compliant, proceed and keep resulting changes aligned with project coding style, conventions, and structure.
## Language Guidelines
- Use clear and concise language when communicating with users, providing feedback, and explaining guidelines.
- Avoid unnecessary pleasantries or filler language; focus on the technical content and actionable feedback.
- When asking follow-up questions for clarification, be direct and specific about the information needed to proceed with the instruction while ensuring that the questions are relevant to the project guidelines and conventions.
# Core Runtime Guidelines
1. Preserve startup ordering and import timing in `launch.py` and `webui.py`; avoid moving initialization steps unless required to fix a critical startup bug or implement a new startup feature.
@@ -14,6 +28,16 @@ applyTo: "launch.py, webui.py, installer.py, modules/**/*.py, pipelines/**/*.py,
7. Reuse established model-loading and pipeline patterns (`modules/sd_*`, `pipelines/`) instead of creating parallel abstractions.
8. For substantial Python changes, run at least relevant checks: `pnpm run ruff` and `pnpm run pylint` (or narrower equivalents when appropriate).
## Tools
- `venv` for Python environment management, activated with `source venv/bin/activate` (Linux) or `venv\Scripts\activate` (Windows).
venv MUST be activated before running any Python commands or scripts to ensure correct dependencies and environment variables.
- `python` 3.10+.
- `pyproject.toml` for Python configuration, including linting and type checking settings.
- `pnpm` for managing JavaScript dependencies and scripts, with key commands defined in `package.json`.
- `ruff` and `pylint` for Python linting, with configurations in `pyproject.toml` and executed via `pnpm ruff` and `pnpm pylint`.
- `pre-commit` hooks which also check line-endings and other formatting issues, configured in `.pre-commit-config.yaml`.
## Build And Test
- Activate environment: `source venv/bin/activate` (always ensure this is active when working with Python code).
@@ -21,7 +45,6 @@ applyTo: "launch.py, webui.py, installer.py, modules/**/*.py, pipelines/**/*.py,
- Full startup: `python launch.py`
- Full lint sequence: `pnpm lint`
- Python checks individually: `pnpm ruff`, `pnpm pylint`
- TypeScript checks: `pnpm eslint`, `pnpm tsc`
## Pitfalls
+14
View File
@@ -3,6 +3,20 @@ description: "Use when editing frontend UI code, TypeScript, JavaScript, HTML, C
name: "UI And Frontend Guidelines"
applyTo: "ui/**/*, extensions-builtin/sdnext-modernui/**/*, extensions-builtin/sdnext-kanvas/**/*"
---
## Agent Guidelines
1. Verify the user instruction against relevant guidelines in this file and linked instruction files before proceeding.
2. If the instruction conflicts with any guideline, do not proceed. Explain which guideline(s) it conflicts with and how to adjust the instruction to comply.
3. If the instruction is valid but unclear or incomplete, ask targeted follow-up questions before implementation. Do not assume user intent or requirements.
4. When giving feedback, name the applicable guideline(s) and explain how each one applies.
5. If the instruction is clear and compliant, proceed and keep resulting changes aligned with project coding style, conventions, and structure.
## Language Guidelines
- Use clear and concise language when communicating with users, providing feedback, and explaining guidelines.
- Avoid unnecessary pleasantries or filler language; focus on the technical content and actionable feedback.
- When asking follow-up questions for clarification, be direct and specific about the information needed to proceed with the instruction while ensuring that the questions are relevant to the project guidelines and conventions.
# UI And Frontend Guidelines
Apply these rules in priority order:
+4
View File
@@ -14,6 +14,10 @@ Given an external model URL, inspect how the model is implemented and estimate h
- User wants effort estimation before implementation work
- User wants to classify whether integration should reuse Diffusers, use custom Diffusers code, or require full custom implementation
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and model integration guidance before proceeding.
## Accepted Inputs
- Hugging Face model URL (preferred)
+4
View File
@@ -15,6 +15,10 @@ Read modules/api/api.py, enumerate all registered endpoints, and validate that e
- OpenAPI docs look wrong or clients report schema mismatches
- You need a pre-PR API contract sanity pass
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and API guidance before proceeding.
## Primary File
- `modules/api/api.py`
+4
View File
@@ -15,6 +15,10 @@ Run a consolidated model-integration audit that combines loader checks, detect/r
- A custom pipeline was ported and needs contract validation
- You want a pre-PR integration quality gate for model-related changes
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and model integration guidance before proceeding.
## Combined Scope
This skill combines four audit surfaces. Run them in this order unless user scope limits categories:
+4
View File
@@ -15,6 +15,10 @@ Perform a detailed step-by-step trace of generation workflows from UI definition
- A new parameter was added to UI or processing classes/functions and needs end-to-end validation
- You want a pre-PR contract audit for generation flow integrity
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
## Required Workflow Coverage
Run workflow coverage in this order to keep checks focused and complete:
+4
View File
@@ -29,6 +29,10 @@ Related files:
- `modules/schedulers/**/*.py` for custom scheduler implementations
- `modules/res4lyf/**/*.py` for Res4Lyf scheduler classes (if installed/enabled)
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
## What "Loaded And Compiled" Means
Treat this as a two-level check:
+4
View File
@@ -15,6 +15,10 @@ Audit all Python scripts in `scripts/*.py` and validate that script class overri
- A script UI was changed and runtime args no longer match
- You want a pre-PR quality gate for script API compatibility
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
## Scope
Primary audit scope:
+4
View File
@@ -15,6 +15,10 @@ Audit SD.Next UI integration points where Python uses Gradio `_js=...` bindings
- You need to validate UI contract consistency for Gradio-bound JS methods.
- User adds or updates extension with JavaScript code in `extensions/*/javascript`.
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
## Primary Files
- `ui/globals.d.ts`
+4
View File
@@ -17,6 +17,10 @@ Read the error, identify which integration layer is failing, isolate the smalles
- Sampling fails due to tensor shape, dtype, device, or scheduler issues
- The model loads but outputs corrupted images, wrong output type, or obviously incorrect results
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and model debugging guidance before proceeding.
## Debugging Order
Always debug from the outside in.
+4
View File
@@ -16,6 +16,10 @@ Use this skill to implement, edit, review, and prepare pull-request-ready change
- Adding tests and docs for diffusers changes
- Preparing a PR that targets the diffusers repository
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and diffusers integration guidance before proceeding.
## Primary Objectives
1. Keep behavior explicit, minimal, and inference-focused.
+4
View File
@@ -15,6 +15,10 @@ Run the project lint workflow in the required order, fix findings, and re-run af
- Multiple files changed and style/static checks may have drifted
- You need a repeatable full-lint remediation pass
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
## Required Environment Step
Always start from repository root and activate virtual environment first:
+4
View File
@@ -16,6 +16,10 @@ Read the task, identify the model architecture and artifact layout, choose the i
- A model already exists in Diffusers but is not yet wired into SD.Next
- A custom architecture needs a repo-local `pipelines/<model>` package and loader
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and model porting guidance before proceeding.
## Core Rule
Prefer the smallest correct integration path.
+4
View File
@@ -17,6 +17,10 @@ This skill targets SD.Next repo-local pipeline ports only.
- The task requires preserving generation behavior without introducing new dependencies
- The task requires removing hard-coded runtime assumptions (device or attention backend)
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and pipeline integration guidance before proceeding.
## Mandatory Clarification Gate
Before implementation, confirm these required inputs with the user:
+6 -2
View File
@@ -15,6 +15,10 @@ Use this skill to audit and update SD.Next model reference catalogs using a phas
- Verifying category placement across `base/cloud/quant/distilled/nunchaku/community`
- Syncing catalog entries with thumbnail files in `models/Reference`
## Guidance
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
## Catalog Files In Scope
- `data/reference.json` (base)
@@ -40,7 +44,7 @@ Priority 2 - schema and formatting consistency:
Priority 3 - assets and size backfill:
- Do not overwrite real thumbnails with placeholders.
- For `size` backfill, use `cli/hf-info.py` as the primary source of truth.
- For `size` backfill, use `cli/hf-info.py` -> section `info` -> field `size` as the primary source of truth.
## Validation Checklist
@@ -76,7 +80,7 @@ Priority 3 - assets and size backfill:
7. Size backfill checks (`size: 0`)
- Enumerate all entries with `"size": 0` across `data/reference*.json`.
- For each Hugging Face repo-style path (`owner/name`), run `cli/hf-info.py`.
- Parse `data.size` from tool output when present (format is MB string, e.g. `"23933.4MB"`).
- Parse `info.data.size` from tool output when present (format is MB string, e.g. `"23933.4MB"`).
- Convert MB to GB using deterministic rounding: `gb = round(mb / 1024, 2)`.
- Update only the `size` field for resolvable records; do not modify unrelated fields.
- If `cli/hf-info.py` returns `ok: false`, missing `data.size`, or non-repo paths, leave `size` unchanged and report as unresolved.
+1 -1
View File
@@ -533,7 +533,7 @@ def check_diffusers():
t_start = time.time()
if args.skip_all:
return
target_commit = "9b0818cf87413b4b9ca2501bf49406eed6d881af" # diffusers commit hash == 0.39.0.dev0 == 06-03-2026 (adds Ideogram 4)
target_commit = "79e408a705d6aff9762c58e4601b11e7ea00ea86" # diffusers commit hash == 0.39.0.dev0 == 06-03-2026 (adds Ideogram 4)
# if args.use_rocm or args.use_zluda or args.use_directml:
# sha = '043ab2520f6a19fce78e6e060a68dbc947edb9f9' # lock diffusers versions for now
pkg = package_spec('diffusers')
+1 -2
View File
@@ -1,6 +1,5 @@
import os
import transformers
from transformers.models.qwen3_vl import Qwen3VLModel
shared_te_map = {
@@ -98,7 +97,7 @@ shared_te_map = {
},
'Qwen3-VL 8B Base': {
'cls': Qwen3VLModel,
'cls': transformers.Qwen3VLModel,
'target_repo': 'Qwen/Qwen3-VL-8B-Instruct',
},
}
+3 -3
View File
@@ -42,7 +42,7 @@ class Ideogram4Pipeline(diffusers.Ideogram4Pipeline):
self.text_encoder.to(devices.cpu)
def pin_transformers_if_fit(transformer, unconditional_transformer) -> bool:
def pin_transformers(transformer, unconditional_transformer) -> bool:
"""Keep both transformers resident under balanced offload when they fit the budget.
Every denoise step runs both transformers, so balanced offload ping-pongs them across
@@ -60,7 +60,7 @@ def pin_transformers_if_fit(transformer, unconditional_transformer) -> bool:
if fits:
transformer.offload_never = True
unconditional_transformer.offload_never = True
log.info(f'Load model: type=Ideogram4 offload=balanced transformers={size_gb:.1f} budget={budget_gb:.1f} action={"pin-resident" if fits else "offload"}')
log.debug(f'Load model: type=Ideogram4 offload=balanced transformers={size_gb:.1f} budget={budget_gb:.1f} action={"pin" if fits else "default"}')
return fits
@@ -80,7 +80,7 @@ def load_ideogram4(checkpoint_info, diffusers_load_config=None):
cls = diffusers.Ideogram4Transformer2DModel
transformer = generic.load_transformer(repo_id, cls_name=cls, subfolder="transformer", load_config=diffusers_load_config)
unconditional_transformer = generic.load_transformer(repo_id, cls_name=cls, subfolder="unconditional_transformer", load_config=diffusers_load_config)
pin_transformers_if_fit(transformer, unconditional_transformer)
pin_transformers(transformer, unconditional_transformer)
# shared_te_map redirects to the shared Qwen3-VL repo (deduped with VQA + prompt-enhance);
# the bundled text_encoder is the fallback when sharing is off. The vae, tokenizer, and
# scheduler load from the repo via from_pretrained.