mirror of
https://github.com/vladmandic/automatic
synced 2026-08-26 15:16:01 +02:00
update ideogram4 requirements and global skills update
Signed-off-by: Vladimir Mandic <mandic00@live.com>
This commit is contained in:
@@ -1,8 +1,22 @@
|
||||
---
|
||||
description: "Use when editing Python core runtime code, startup flow, model loading, API internals, backend/device logic, or shared state in modules and pipelines."
|
||||
name: "Core Runtime Guidelines"
|
||||
applyTo: "launch.py, webui.py, installer.py, modules/**/*.py, pipelines/**/*.py, scripts/**/*.py, extensions-builtin/**/*.py"
|
||||
applyTo: "launch.py, webui.py, installer.py, modules/**/*.py, pipelines/**/*.py, scripts/**/*.py, extensions-builtin/**/*.py, cli/**/*.py"
|
||||
---
|
||||
## Agent Guidelines
|
||||
|
||||
1. Verify the user instruction against relevant guidelines in this file and linked instruction files before proceeding.
|
||||
2. If the instruction conflicts with any guideline, do not proceed. Explain which guideline(s) it conflicts with and how to adjust the instruction to comply.
|
||||
3. If the instruction is valid but unclear or incomplete, ask targeted follow-up questions before implementation. Do not assume user intent or requirements.
|
||||
4. When giving feedback, name the applicable guideline(s) and explain how each one applies.
|
||||
5. If the instruction is clear and compliant, proceed and keep resulting changes aligned with project coding style, conventions, and structure.
|
||||
|
||||
## Language Guidelines
|
||||
|
||||
- Use clear and concise language when communicating with users, providing feedback, and explaining guidelines.
|
||||
- Avoid unnecessary pleasantries or filler language; focus on the technical content and actionable feedback.
|
||||
- When asking follow-up questions for clarification, be direct and specific about the information needed to proceed with the instruction while ensuring that the questions are relevant to the project guidelines and conventions.
|
||||
|
||||
# Core Runtime Guidelines
|
||||
|
||||
1. Preserve startup ordering and import timing in `launch.py` and `webui.py`; avoid moving initialization steps unless required to fix a critical startup bug or implement a new startup feature.
|
||||
@@ -14,6 +28,16 @@ applyTo: "launch.py, webui.py, installer.py, modules/**/*.py, pipelines/**/*.py,
|
||||
7. Reuse established model-loading and pipeline patterns (`modules/sd_*`, `pipelines/`) instead of creating parallel abstractions.
|
||||
8. For substantial Python changes, run at least relevant checks: `pnpm run ruff` and `pnpm run pylint` (or narrower equivalents when appropriate).
|
||||
|
||||
## Tools
|
||||
|
||||
- `venv` for Python environment management, activated with `source venv/bin/activate` (Linux) or `venv\Scripts\activate` (Windows).
|
||||
venv MUST be activated before running any Python commands or scripts to ensure correct dependencies and environment variables.
|
||||
- `python` 3.10+.
|
||||
- `pyproject.toml` for Python configuration, including linting and type checking settings.
|
||||
- `pnpm` for managing JavaScript dependencies and scripts, with key commands defined in `package.json`.
|
||||
- `ruff` and `pylint` for Python linting, with configurations in `pyproject.toml` and executed via `pnpm ruff` and `pnpm pylint`.
|
||||
- `pre-commit` hooks which also check line-endings and other formatting issues, configured in `.pre-commit-config.yaml`.
|
||||
|
||||
## Build And Test
|
||||
|
||||
- Activate environment: `source venv/bin/activate` (always ensure this is active when working with Python code).
|
||||
@@ -21,7 +45,6 @@ applyTo: "launch.py, webui.py, installer.py, modules/**/*.py, pipelines/**/*.py,
|
||||
- Full startup: `python launch.py`
|
||||
- Full lint sequence: `pnpm lint`
|
||||
- Python checks individually: `pnpm ruff`, `pnpm pylint`
|
||||
- TypeScript checks: `pnpm eslint`, `pnpm tsc`
|
||||
|
||||
## Pitfalls
|
||||
|
||||
|
||||
@@ -3,6 +3,20 @@ description: "Use when editing frontend UI code, TypeScript, JavaScript, HTML, C
|
||||
name: "UI And Frontend Guidelines"
|
||||
applyTo: "ui/**/*, extensions-builtin/sdnext-modernui/**/*, extensions-builtin/sdnext-kanvas/**/*"
|
||||
---
|
||||
## Agent Guidelines
|
||||
|
||||
1. Verify the user instruction against relevant guidelines in this file and linked instruction files before proceeding.
|
||||
2. If the instruction conflicts with any guideline, do not proceed. Explain which guideline(s) it conflicts with and how to adjust the instruction to comply.
|
||||
3. If the instruction is valid but unclear or incomplete, ask targeted follow-up questions before implementation. Do not assume user intent or requirements.
|
||||
4. When giving feedback, name the applicable guideline(s) and explain how each one applies.
|
||||
5. If the instruction is clear and compliant, proceed and keep resulting changes aligned with project coding style, conventions, and structure.
|
||||
|
||||
## Language Guidelines
|
||||
|
||||
- Use clear and concise language when communicating with users, providing feedback, and explaining guidelines.
|
||||
- Avoid unnecessary pleasantries or filler language; focus on the technical content and actionable feedback.
|
||||
- When asking follow-up questions for clarification, be direct and specific about the information needed to proceed with the instruction while ensuring that the questions are relevant to the project guidelines and conventions.
|
||||
|
||||
# UI And Frontend Guidelines
|
||||
|
||||
Apply these rules in priority order:
|
||||
|
||||
@@ -14,6 +14,10 @@ Given an external model URL, inspect how the model is implemented and estimate h
|
||||
- User wants effort estimation before implementation work
|
||||
- User wants to classify whether integration should reuse Diffusers, use custom Diffusers code, or require full custom implementation
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and model integration guidance before proceeding.
|
||||
|
||||
## Accepted Inputs
|
||||
|
||||
- Hugging Face model URL (preferred)
|
||||
|
||||
@@ -15,6 +15,10 @@ Read modules/api/api.py, enumerate all registered endpoints, and validate that e
|
||||
- OpenAPI docs look wrong or clients report schema mismatches
|
||||
- You need a pre-PR API contract sanity pass
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and API guidance before proceeding.
|
||||
|
||||
## Primary File
|
||||
|
||||
- `modules/api/api.py`
|
||||
|
||||
@@ -15,6 +15,10 @@ Run a consolidated model-integration audit that combines loader checks, detect/r
|
||||
- A custom pipeline was ported and needs contract validation
|
||||
- You want a pre-PR integration quality gate for model-related changes
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and model integration guidance before proceeding.
|
||||
|
||||
## Combined Scope
|
||||
|
||||
This skill combines four audit surfaces. Run them in this order unless user scope limits categories:
|
||||
|
||||
@@ -15,6 +15,10 @@ Perform a detailed step-by-step trace of generation workflows from UI definition
|
||||
- A new parameter was added to UI or processing classes/functions and needs end-to-end validation
|
||||
- You want a pre-PR contract audit for generation flow integrity
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
|
||||
|
||||
## Required Workflow Coverage
|
||||
|
||||
Run workflow coverage in this order to keep checks focused and complete:
|
||||
|
||||
@@ -29,6 +29,10 @@ Related files:
|
||||
- `modules/schedulers/**/*.py` for custom scheduler implementations
|
||||
- `modules/res4lyf/**/*.py` for Res4Lyf scheduler classes (if installed/enabled)
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
|
||||
|
||||
## What "Loaded And Compiled" Means
|
||||
|
||||
Treat this as a two-level check:
|
||||
|
||||
@@ -15,6 +15,10 @@ Audit all Python scripts in `scripts/*.py` and validate that script class overri
|
||||
- A script UI was changed and runtime args no longer match
|
||||
- You want a pre-PR quality gate for script API compatibility
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
|
||||
|
||||
## Scope
|
||||
|
||||
Primary audit scope:
|
||||
|
||||
@@ -15,6 +15,10 @@ Audit SD.Next UI integration points where Python uses Gradio `_js=...` bindings
|
||||
- You need to validate UI contract consistency for Gradio-bound JS methods.
|
||||
- User adds or updates extension with JavaScript code in `extensions/*/javascript`.
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
|
||||
|
||||
## Primary Files
|
||||
|
||||
- `ui/globals.d.ts`
|
||||
|
||||
@@ -17,6 +17,10 @@ Read the error, identify which integration layer is failing, isolate the smalles
|
||||
- Sampling fails due to tensor shape, dtype, device, or scheduler issues
|
||||
- The model loads but outputs corrupted images, wrong output type, or obviously incorrect results
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and model debugging guidance before proceeding.
|
||||
|
||||
## Debugging Order
|
||||
|
||||
Always debug from the outside in.
|
||||
|
||||
@@ -16,6 +16,10 @@ Use this skill to implement, edit, review, and prepare pull-request-ready change
|
||||
- Adding tests and docs for diffusers changes
|
||||
- Preparing a PR that targets the diffusers repository
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and diffusers integration guidance before proceeding.
|
||||
|
||||
## Primary Objectives
|
||||
|
||||
1. Keep behavior explicit, minimal, and inference-focused.
|
||||
|
||||
@@ -15,6 +15,10 @@ Run the project lint workflow in the required order, fix findings, and re-run af
|
||||
- Multiple files changed and style/static checks may have drifted
|
||||
- You need a repeatable full-lint remediation pass
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
|
||||
|
||||
## Required Environment Step
|
||||
|
||||
Always start from repository root and activate virtual environment first:
|
||||
|
||||
@@ -16,6 +16,10 @@ Read the task, identify the model architecture and artifact layout, choose the i
|
||||
- A model already exists in Diffusers but is not yet wired into SD.Next
|
||||
- A custom architecture needs a repo-local `pipelines/<model>` package and loader
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and model porting guidance before proceeding.
|
||||
|
||||
## Core Rule
|
||||
|
||||
Prefer the smallest correct integration path.
|
||||
|
||||
@@ -17,6 +17,10 @@ This skill targets SD.Next repo-local pipeline ports only.
|
||||
- The task requires preserving generation behavior without introducing new dependencies
|
||||
- The task requires removing hard-coded runtime assumptions (device or attention backend)
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime and pipeline integration guidance before proceeding.
|
||||
|
||||
## Mandatory Clarification Gate
|
||||
|
||||
Before implementation, confirm these required inputs with the user:
|
||||
|
||||
@@ -15,6 +15,10 @@ Use this skill to audit and update SD.Next model reference catalogs using a phas
|
||||
- Verifying category placement across `base/cloud/quant/distilled/nunchaku/community`
|
||||
- Syncing catalog entries with thumbnail files in `models/Reference`
|
||||
|
||||
## Guidance
|
||||
|
||||
- Consult `.github/instructions/core.instructions.md` for relevant core runtime guidance before proceeding.
|
||||
|
||||
## Catalog Files In Scope
|
||||
|
||||
- `data/reference.json` (base)
|
||||
@@ -40,7 +44,7 @@ Priority 2 - schema and formatting consistency:
|
||||
Priority 3 - assets and size backfill:
|
||||
|
||||
- Do not overwrite real thumbnails with placeholders.
|
||||
- For `size` backfill, use `cli/hf-info.py` as the primary source of truth.
|
||||
- For `size` backfill, use `cli/hf-info.py` -> section `info` -> field `size` as the primary source of truth.
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
@@ -76,7 +80,7 @@ Priority 3 - assets and size backfill:
|
||||
7. Size backfill checks (`size: 0`)
|
||||
- Enumerate all entries with `"size": 0` across `data/reference*.json`.
|
||||
- For each Hugging Face repo-style path (`owner/name`), run `cli/hf-info.py`.
|
||||
- Parse `data.size` from tool output when present (format is MB string, e.g. `"23933.4MB"`).
|
||||
- Parse `info.data.size` from tool output when present (format is MB string, e.g. `"23933.4MB"`).
|
||||
- Convert MB to GB using deterministic rounding: `gb = round(mb / 1024, 2)`.
|
||||
- Update only the `size` field for resolvable records; do not modify unrelated fields.
|
||||
- If `cli/hf-info.py` returns `ok: false`, missing `data.size`, or non-repo paths, leave `size` unchanged and report as unresolved.
|
||||
|
||||
+1
-1
@@ -533,7 +533,7 @@ def check_diffusers():
|
||||
t_start = time.time()
|
||||
if args.skip_all:
|
||||
return
|
||||
target_commit = "9b0818cf87413b4b9ca2501bf49406eed6d881af" # diffusers commit hash == 0.39.0.dev0 == 06-03-2026 (adds Ideogram 4)
|
||||
target_commit = "79e408a705d6aff9762c58e4601b11e7ea00ea86" # diffusers commit hash == 0.39.0.dev0 == 06-03-2026 (adds Ideogram 4)
|
||||
# if args.use_rocm or args.use_zluda or args.use_directml:
|
||||
# sha = '043ab2520f6a19fce78e6e060a68dbc947edb9f9' # lock diffusers versions for now
|
||||
pkg = package_spec('diffusers')
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import os
|
||||
import transformers
|
||||
from transformers.models.qwen3_vl import Qwen3VLModel
|
||||
|
||||
|
||||
shared_te_map = {
|
||||
@@ -98,7 +97,7 @@ shared_te_map = {
|
||||
},
|
||||
|
||||
'Qwen3-VL 8B Base': {
|
||||
'cls': Qwen3VLModel,
|
||||
'cls': transformers.Qwen3VLModel,
|
||||
'target_repo': 'Qwen/Qwen3-VL-8B-Instruct',
|
||||
},
|
||||
}
|
||||
|
||||
@@ -42,7 +42,7 @@ class Ideogram4Pipeline(diffusers.Ideogram4Pipeline):
|
||||
self.text_encoder.to(devices.cpu)
|
||||
|
||||
|
||||
def pin_transformers_if_fit(transformer, unconditional_transformer) -> bool:
|
||||
def pin_transformers(transformer, unconditional_transformer) -> bool:
|
||||
"""Keep both transformers resident under balanced offload when they fit the budget.
|
||||
|
||||
Every denoise step runs both transformers, so balanced offload ping-pongs them across
|
||||
@@ -60,7 +60,7 @@ def pin_transformers_if_fit(transformer, unconditional_transformer) -> bool:
|
||||
if fits:
|
||||
transformer.offload_never = True
|
||||
unconditional_transformer.offload_never = True
|
||||
log.info(f'Load model: type=Ideogram4 offload=balanced transformers={size_gb:.1f} budget={budget_gb:.1f} action={"pin-resident" if fits else "offload"}')
|
||||
log.debug(f'Load model: type=Ideogram4 offload=balanced transformers={size_gb:.1f} budget={budget_gb:.1f} action={"pin" if fits else "default"}')
|
||||
return fits
|
||||
|
||||
|
||||
@@ -80,7 +80,7 @@ def load_ideogram4(checkpoint_info, diffusers_load_config=None):
|
||||
cls = diffusers.Ideogram4Transformer2DModel
|
||||
transformer = generic.load_transformer(repo_id, cls_name=cls, subfolder="transformer", load_config=diffusers_load_config)
|
||||
unconditional_transformer = generic.load_transformer(repo_id, cls_name=cls, subfolder="unconditional_transformer", load_config=diffusers_load_config)
|
||||
pin_transformers_if_fit(transformer, unconditional_transformer)
|
||||
pin_transformers(transformer, unconditional_transformer)
|
||||
# shared_te_map redirects to the shared Qwen3-VL repo (deduped with VQA + prompt-enhance);
|
||||
# the bundled text_encoder is the fallback when sharing is off. The vae, tokenizer, and
|
||||
# scheduler load from the repo via from_pretrained.
|
||||
|
||||
Reference in New Issue
Block a user