Revert design doc to complex v1.2 hierarchy design

2026-09-02 10:38:58 +00:00
parent 091457ae8a
commit 99746d59df
1 changed files with 295 additions and 272 deletions
@@ -1,354 +1,377 @@
 # Model Manager — Design Doc v1.2
 ## 1. Design Document
 ### 1.1 System Overview
-The Model Manager is a Windows application with a web-based UI served by a local Python/FastAPI server. It manages large AI model files (5–80 GB) downloaded from Hugging Face. Models are stored on one or more HDD (Level 3) storage locations, and can be manually promoted to an SSD (Level 2) cache folder for faster access. The application tracks metadata, supports family grouping (manual), checks for updates, and handles downloads with retry and checksum verification.
+The Model Manager is a Windows application with a web-based UI served by a local Python/FastAPI server. It manages large AI model files (5–80 GB) downloaded from Hugging Face. Content is organized as a hierarchy: **families contain models; models contain one or more repos; repos contain files (with quant/variant tags)**. Models are stored on one or more HDD (L3) roots and can be promoted to an SSD (L2) cache. Metadata includes per-file HF content hashes; families share files by hash-verified dedupe; update detection is per-file diff.
 ### 1.2 Goals & Non-Goals
 **Goals:**
- Download models from Hugging Face (public/private with token).
+- Download from Hugging Face (public/private with token), resumable across restarts
- Store on configurable HDD storage locations (multiple supported).
+- **Hierarchy: family → model → repo → files; multiple repos and multiple quants per model; non-standard variant tags supported (user-editable)** 
- Manual promotion to SSD cache via UI checkbox.
+- Multiple HDD roots; promotion to SSD cache; HDD integrity validated before demotion
- Manage model types: Text-Gen (GGUF), Image-Gen (safetensors, etc.), Embedding (GGUF).
+- Model types: text, image, embedding
- Support manual family grouping to share files (e.g., mmproj).
+- **Shared files at family or model scope (jinja at any level, mmproj typically model)** 
- Track download/upload timestamps and suggest updates.
+- Draft/speed-up models (MTP, DFlash) attachable **same-repo (role) or separate-repo (component) — user choice**
- Provide a responsive web UI (editable without recompilation).
+- Per-file update diff (commit hash as fast pre-check); timestamps tracked
- Handle graceful shutdown (finish downloads, flush database).
+- Web UI (editable without recompile), SSE live progress, Logs page
- Support checksum verification and exponential retry for failed downloads.
+- Graceful shutdown (5-min default wait, force-pause option); abort vs pause distinction
 - Retry in seconds + manual "Retry Now"; single-worker disk-copy queue
 **Non-Goals (current version):**
- Automatic family detection (manual only).
+- Automatic family/model detection (suggested from repo org/name, but user confirms)
- Automatic cache eviction (manual promotion only).
+- Automatic cache eviction; llama.cpp config generation (schema/roles ready; placeholder service)
- Full config generation for llama.cpp (placeholder only).
+- Automatic image-gen role detection; multi-user/LAN access
- Automatic image-gen file role detection (manual classification during download).
+- Local deep-hash verification on every operation (explicit command only)
 - Multi-user support, remote access (local only).
 - Pause/resume downloads (not required, but may be added later).
 ### 1.3 Functional Requirements
-#### 1.3.1 Storage Management
+#### 1.3.1 Content Model & Storage Layout
- **Level 3 (HDD)**: One or more root folders (e.g., `D:\Models`, `E:\Models`). User can add/remove storage locations in Settings. Each download is placed into a selected (or automatically chosen) HDD root.
+**Hierarchy:** `family → model → repo → files`.
- **Level 2 (SSD)**: A single cache folder (e.g., `C:\ModelCache`). When user checks "Cache to SSD" for a model, the entire model folder is copied from HDD to SSD cache. Original remains on HDD. Unchecking removes the cached copy.
+
- **File Organization**: Each model stored in its own subfolder (e.g., `{storage_root}/{model_type}/{hf_repo_name}/`). Within, files are kept as downloaded from HF (preserve repository structure).
+- **Family** (e.g., "Qwen3.5"): groups related models. Required; auto-suggested from repo org at Add New. May contain many models (27B, 35B, 9B…).
 - **Model** (e.g., "Qwen3.5-27B"): logical model of a type (`text`/`image`/`embedding`). May aggregate **multiple repos** (official + community quants); UI shows a merged per-model view of all variants across repos.
 - **Repo**: one HF repository, attached to exactly one model, with a `component` tag used as its folder segment: `main`, `draft`, `vae`, `lora`, `other` (default `main`).
 - **File**: one downloaded file, with `role`, `variant` (quant/variety tag), and `scope`.
 **Variant tags:** parsed from filenames (strip shard suffixes `-NNNNN-of-NNNNN`; match known patterns `Q\d+_…`, `IQ\d…`, `UD-…`; fallback = trailing name segments). **Heuristics will not catch every naming scheme (e.g., `APEX-Compact`, `I-Balanced`)** — the tag is always editable in the preview UI and in file details. Sharded GGUFs share one variant tag.
 **Draft/MTP/DFlash (user-selectable attachment):**
 - *Under repo:* draft files inside the main model's repo → role `speedup`, stored with the repo.
 - *Under model:* a separate HF repo → its own repo row with `component='draft'`, stored under `{model}/draft/`.
 - Either way, the config maker (future) can resolve a draft target for `--model-draft`.
 **Shared-scope files:** each file has `scope`:
 | scope | physical location | typical use |
 |---|---|---|
 | `repo` (default) | `{root}/{model_type}/{fam}/{component}/{model}/{repo}/…` | quants, VAE, CLIP, tokenizers, LoRA |
 | `model` | `{root}/{model_type}/{fam}/_shared/{model}/…` | **mmproj (default)**, model-level jinja |
 | `family` | `{root}/{model_type}/{fam}/_shared/…` | family-level jinja overrides |
 Files keep their source-repo provenance (FK to repo) even when stored at model/family scope. Defaults at download time: mmproj → `model`, everything else → `repo`; user can elevate jinja to model/family in the preview UI.
 **Folder structure:** `{storage_root}/{text|image|embed}/{family}/{component}/{model}/{repo_name}/{file}`
 (plus `{storage_root}/{model_type}/{fam}/_shared/…` and `{storage_root}/_incoming/{repo_id}/` temp areas).
 - Model type→folder map: text→`text`, image→`image`, embedding→`embed`.
 - Monolithic image repos (diffusers: `unet/`, `vae/`, `text_encoder/` inside one repo) → `component='main'`, **internal repo structure preserved** per file (`rel_file_path`).
 - Reserved folder names: `_shared`, `_incoming`; model/repo/family names sanitized (Windows-illegal chars, reserved device names, trailing dots/spaces, length caps). Repo folder = repo name; on collision append a short hash.
 **Storage roots:** multiple HDD roots, removable only when unreferenced. **L2 (SSD):** single cache root in settings; promote/demote operates on **repo folders** (promoting a "model" = queue promote for each of its repos). Demotion validates the HDD copy first (files present, sizes match DB). Shared-scope files are small and resolve from wherever they live; promoting them is a future option.
 **Path handling:** DB stores IDs + relative paths only; resolution joins root + segments and validates containment. Windows long paths enabled and documented; depth is bounded by the 6-level layout (TC-024).
 #### 1.3.2 Download Workflow
-1. User provides Hugging Face repo ID (e.g., `org/model-name`) or URL.
+1. User pastes **full URL or `org/name` shorthand**; URL cruft stripped; a branch/revision in the URL is pinned and stored. Recent repos remembered in a dropdown.
-2. App lists files in the repo using `huggingface_hub.list_repo_files`.
+2. Preview: `model_info(files_metadata=True)` → names, sizes, **LFS sha256 OIDs**.
-3. App classifies files by extension and filename patterns (GGUF, safetensors, etc.) and presents them to the user, grouped by inferred role (main, mmproj, VAE, etc.). Since detection is manual, the user can select which files to download and optionally assign roles.
+3. **Entity resolution:** app suggests family (from org) and model (from name, suffix-stripped); user can pick existing family/model or create new. For an existing model, the repo attaches to it; files whose OID matches any family file are flagged "available via family" and skipped by default (small non-LFS files never deduped).
-4. User selects variant(s) (e.g., specific GGUF quantization) and target HDD storage location.
+4. Files grouped by role and **variant**; user selects files, assigns roles/scopes/variant fixes, picks HDD root, optionally "Cache to SSD."
-5. App downloads selected files to a temporary folder within the target HDD. It uses `huggingface_hub.snapshot_download` or `hf_hub_download` with `allow_patterns` to filter. If token is provided, it is used.
+5. `families`/`models`/`repos` rows ensured (repo `status='downloading'`), `jobs` row created; download targets `{root}/_incoming/{repo_id}/` on the destination volume.
-6. After download completes, checksums are verified (if available from HF or computed locally).
+6. `snapshot_download(local_dir=<incoming>, allow_patterns, revision, token)`. Direct-to-target volume; no `~/.cache` duplication (asserted at runtime); `hf_transfer` optional toggle.
-7. The temp folder is atomically renamed to the final model folder.
+7. Resumable; retry 5s→15s→45s→2m, cap 5m, `max_retries` default 5, manual "Retry Now." Retries exhausted → `paused`, partials kept, auto-resume offered on next launch.
-8. Database entry is created with metadata: repo ID, local path, file list with roles, HF commit hash, timestamps, etc.
+8. Success: files moved into final locations per scope (temp+rename throughout; cross-volume dedupe → copy, same-volume → hardlink), OIDs/sizes recorded, `status='complete'`.
-9. If "Cache to SSD" was checked, a background copy to SSD cache is started.
+9. **Cancel** → `_incoming` deleted, repo `cancelled`. **Pause** → partials kept. Crash → startup sweep resumes.
 10. "Cache to SSD" → copy job enqueued (§1.3.7); SSD copy also temp-then-rename.
-#### 1.3.3 Family Management
+#### 1.3.3 Family Sharing & Scope Rules
- User can create a **family** (e.g., "Llama-3-Vision") with a name and description.
+- Dedupe identity = **LFS sha256 OID**, within a family, across models/repos. Matched files are materialized by hardlink (same volume) or copy (cross volume).
- When downloading a model, user can assign it to an existing family. The app will compare the new model's files with already downloaded files from the same family.
+- **Shared files are immutable: never rewritten in place** — updates create new files and swap via rename, so hardlink partners are unaffected.
- If a file with the same filename and role already exists locally (from another model in the family), the app can avoid re-downloading it by creating a **hard link** or **copy** (configurable) from the existing file to the new model's folder. This is especially useful for shared `mmproj` or `jinja` files.
+- Small non-LFS files (jinja, configs, tokenizers) are always re-downloaded, never deduped.
- Manual override: user can force download even if file exists.
+- User override: force re-download per file.
 #### 1.3.4 Update Check
- The app stores the HF repository commit hash at download time.
+Per repo (commit hash + per-file OIDs stored at download). Pre-check commit; if changed, per-file OID diff → added/changed/removed list → user selects → apply job re-downloads changed files to `_incoming`, swaps per file via rename (scope-aware), updates rows + revision. Model-level UI aggregates update state across its repos.
 - On demand (UI button or scheduled), the app queries the HF API for the current commit hash of the repo.
 - If the hash differs, the UI shows an "Update available" indicator. The user can initiate a re-download of changed files.
 - For family-shared files, update check can be done per file if desired, but initial version uses repo-level hash.
 #### 1.3.5 Web UI
- **Pages**: Overview (dashboard), Models (list/filter by type), Add New, Settings, Logs.
+- **Overview:** counts by type, disk usage per root, SSD usage, active jobs, pending updates.
- UI assets (HTML/CSS/JS) are stored in a `web/` folder next to the application. They can be edited without recompiling the Python code.
+- **Models:** grouped Family → Model → Repos; per-model merged variant list; per-repo cache status.
- All interactions go through a REST API; frontend is a simple SPA using Vue.js (CDN) or vanilla JS.
+- **Add New:** URL/shorthand input, preview, family/model resolver, file selection with roles/scopes/variants, recent repos.
 - **Settings:** roots, SSD cache root, token, concurrency, retries.
 - **Logs:** live tail (SSE) + rotating file log.
 - Static `web/` folder, vanilla JS, SSE (`/api/events`, `/api/logs/stream`). No recompile to edit.
 #### 1.3.6 Graceful Shutdown
- The console process captures `CTRL_C_EVENT` or receives a shutdown command from the UI.
+Ctrl+C or UI quit → stop accepting jobs → wait up to **5 min** (configurable) for active tasks; UI shows "waiting for N tasks"; force-pause keeps partials for next launch; final DB checkpoint (WAL; writes are continuous anyway).
- It stops accepting new API requests, waits for active background tasks (downloads, copies) to complete (with a configurable timeout), then closes database connections and exits.
+
- In-progress downloads are allowed to finish; if timeout exceeded, they are aborted and database remains consistent.
+#### 1.3.7 Copy Queue
 Single worker serializes all disk-to-disk ops (promote, demote, update swaps, scope materialization). Network pool separate (default 2). Copies cancelable with try/finally temp cleanup.
 #### 1.3.8 Startup Recovery
 Sweep: `repos.status='downloading'` ↔ `_incoming/{repo_id}/` reconciled (resume or clean); orphaned `_incoming` dirs and SSD `.tmp` folders deleted with log entries. 
 ### 1.4 Non-Functional Requirements
- **Performance**: Downloads use streaming with progress; UI remains responsive via background threads. Multiple concurrent downloads limited (configurable, default 2).
+Performance (SSE progress, 2 network workers, 1 copy worker, optional `hf_transfer`); reliability (same-volume atomic renames, DB written at job start/end, WAL); security (loopback-only, `keyring` token, path validation, log redaction); Windows (long paths, short roots, name sanitization).
 - **Reliability**: Atomic file moves, database transactions, checksum verification, exponential retry with configurable max attempts.
 - **Security**: Local-only server (bind 127.0.0.1), token stored encrypted using Windows DPAPI, input validation to prevent path traversal.
 - **Maintainability**: Modular services, decoupled UI, clear separation of concerns.
 ### 1.5 Technology Stack
- **Backend**: Python 3.11, FastAPI, Uvicorn, SQLAlchemy (SQLite), `huggingface_hub`, `pydantic`, `cryptography` (for DPAPI encryption).
+Python 3.11, FastAPI, Uvicorn, SQLAlchemy (SQLite/WAL), `huggingface_hub` (>= 0.23, pinned), `pydantic`, `keyring`, optional `hf_transfer`; static `web/` + vanilla JS; `ThreadPoolExecutor` pools; `filelock`; `shutil`/`robocopy`.
 - **Frontend**: Static HTML/JS/CSS served by FastAPI; optional Vue.js via CDN.
 - **Background Tasks**: FastAPI `BackgroundTasks` or a custom thread pool for long operations.
 - **File Operations**: `shutil` for copy/move, `os` for hard links, `filelock` for safe concurrent access.
-### 1.6 Data Model (Database Schema)
+### 1.6 Data Model
 ```sql
 -- families
 CREATE TABLE families (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    name TEXT UNIQUE NOT NULL,
+    name TEXT UNIQUE NOT NULL,          -- sanitized; reserved names rejected
-    description TEXT
+    description TEXT,
    created_at DATETIME
 );
 CREATE TABLE models (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    family_id INTEGER NOT NULL,
    name TEXT NOT NULL,                 -- "Qwen3.5-27B" (sanitized)
    model_type TEXT NOT NULL CHECK(model_type IN ('text','image','embedding')),
    description TEXT,
    created_at DATETIME,
    UNIQUE(family_id, name),
    FOREIGN KEY(family_id) REFERENCES families(id)
 );
 CREATE TABLE repos (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    model_id INTEGER NOT NULL,
    hf_repo_id TEXT UNIQUE NOT NULL,    -- "org/name"; one row per HF repo
    component TEXT NOT NULL DEFAULT 'main'
        CHECK(component IN ('main','draft','vae','lora','other')), -- folder {component}
    revision TEXT,                      -- commit hash at download
    status TEXT NOT NULL DEFAULT 'downloading'
        CHECK(status IN ('downloading','complete','paused','error','cancelled')),
    storage_location_id INTEGER NOT NULL,
    rel_path TEXT NOT NULL,             -- {model_type}/{fam}/{component}/{model}/{repo}
    hf_last_modified DATETIME,
    download_started DATETIME,
    download_completed DATETIME,
    is_cached INTEGER DEFAULT 0,
    cache_path TEXT,                    -- relative to SSD root
    UNIQUE(storage_location_id, rel_path),
    FOREIGN KEY(model_id) REFERENCES models(id),
    FOREIGN KEY(storage_location_id) REFERENCES storage_locations(id)
 );
 -- Model status is DERIVED from its repos (UI aggregate). Downloads/caching operate per repo.
 CREATE TABLE model_files (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    repo_id INTEGER NOT NULL,           -- provenance (source repo)
    rel_file_path TEXT NOT NULL,        -- path within its scope folder
    role TEXT NOT NULL DEFAULT 'other' CHECK(role IN (
        'main','mmproj','speedup','jinja','vae','clip','text_encoder',
        'lora','tokenizer','other')),
    variant TEXT,                       -- quant/variety tag; shared by shards; user-editable
    scope TEXT NOT NULL DEFAULT 'repo' CHECK(scope IN ('family','model','repo')),
    size_bytes INTEGER,
    lfs_sha256 TEXT,                    -- HF OID; NULL for small non-LFS files
    UNIQUE(repo_id, rel_file_path),
    FOREIGN KEY(repo_id) REFERENCES repos(id)
 );
 -- CHANGED: polymorphic target (downloads/caches target repos; checks may target models)
 CREATE TABLE jobs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    job_type TEXT NOT NULL CHECK(job_type IN
        ('download','cache_promote','cache_demote','update_check','update_apply','delete')),
    target_type TEXT NOT NULL CHECK(target_type IN ('repo','model','family')),
    target_id INTEGER NOT NULL,
    status TEXT NOT NULL CHECK(status IN
        ('queued','running','paused','completed','failed','cancelled')),
    detail TEXT,                        -- JSON
    progress REAL,
    created_at DATETIME, started_at DATETIME, ended_at DATETIME
 );
 -- storage_locations (HDD roots)
 CREATE TABLE storage_locations (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    path TEXT UNIQUE NOT NULL,
+    path TEXT UNIQUE NOT NULL,          -- HDD roots only
-    type TEXT DEFAULT 'hdd',  -- 'hdd' or 'ssd'
+    label TEXT,
    is_active INTEGER DEFAULT 1
 );
-- models
+CREATE TABLE settings (key TEXT PRIMARY KEY, value TEXT);
-CREATE TABLE models (
+-- Keys: ssd_cache_root, max_concurrent_downloads, retry_schedule, max_retries,
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
+-- hf_transfer_enabled, recent_repos, shutdown_wait_seconds
-    hf_repo_id TEXT NOT NULL,
+-- HF token lives in Credential Manager (keyring), never here.
    local_path TEXT NOT NULL UNIQUE,  -- absolute path to model folder on HDD
    model_type TEXT NOT NULL,  -- 'text', 'image', 'embedding'
    family_id INTEGER,
    storage_location_id INTEGER NOT NULL,
    download_timestamp DATETIME,
    hf_commit_hash TEXT,
    upload_timestamp DATETIME,  -- last commit date from HF
    is_cached INTEGER DEFAULT 0,  -- 1 if cached on SSD
    cache_path TEXT,  -- absolute path to cached folder
    FOREIGN KEY(family_id) REFERENCES families(id),
    FOREIGN KEY(storage_location_id) REFERENCES storage_locations(id)
 );
 -- model_files
 CREATE TABLE model_files (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    model_id INTEGER NOT NULL,
    filename TEXT NOT NULL,  -- relative path within model folder
    role TEXT,  -- 'main', 'mmproj', 'vae', 'clip', 'text_encoder', 'lora', 'other'
    size_bytes INTEGER,
    sha256 TEXT,
    FOREIGN KEY(model_id) REFERENCES models(id)
 );
 -- settings (key-value)
 CREATE TABLE settings (
    key TEXT PRIMARY KEY,
    value TEXT
 );
 ```
 **Delete guardrails:** repo delete requires confirm (removes folder + files rows); model delete blocked while repos exist; family delete blocked while models exist. **Same HF repo cannot attach to two models** (global UNIQUE) — flagged as a v1 limitation.
 ### 1.7 Architecture Overview
-
+Unchanged components: console entry (signals) → FastAPI (API+SSE+static) → Job Scheduler → Download Manager (network pool) + Copy Queue (1 worker) → Storage Manager (path resolution now hierarchy-aware, temp+rename, hardlink/copy, space/integrity checks) → HF Service → DB → Startup Recovery → Config Service (placeholder).
 The application consists of:
 - **Console Entry Point**: Starts Uvicorn server, handles signals.
 - **FastAPI Application**: Serves API and static UI files.
 - **Background Task Manager**: Uses a `ThreadPoolExecutor` to run downloads, copies, and update checks.
 - **Download Manager**: Handles HF interactions, retries, checksum verification.
 - **Storage Manager**: Manages HDD roots, SSD cache, file copying/moving, hard links.
 - **HF Service**: Wraps `huggingface_hub` functions for listing files, downloading, getting metadata.
 - **Database Layer**: SQLAlchemy models and CRUD.
 - **Config Service**: Placeholder for future llama.cpp config generation.
 ### 1.8 Key Design Decisions
-
+- **Hierarchy over flat repos:** families/models are cheap metadata; repos are the unit of download/caching; files carry role+variant+scope. Enables multi-repo models, multi-quant repos, and model/family-shared files without duplication. *(User decision, v1.2.)*
- **Download Method**: Use `huggingface_hub.snapshot_download` with `allow_patterns` to download selected files. This automatically verifies checksums if provided by HF. For additional verification, compute SHA256 after download and store.
+- **Variant = per-file tag**, heuristic-parsed, user-editable; shards grouped by tag.
- **Retry Mechanism**: Implement in Download Manager: on failure (network error, checksum mismatch), retry after delay: 5 min, 15 min, 45 min, 2 hours, etc., up to `max_retries` (default 5). Only retry if error is transient (e.g., network, HTTP 5xx).
+- **Scope controls placement**, provenance stays on the repo; shared files immutable (rename-swap updates).
- **Family Sharing**: When assigning a model to a family, after download, for each file, check if an identical file (by relative path and role) exists in another model of the same family. If yes, replace the downloaded file with a hard link to the existing file (to save space). This is safe because models are read-only.
+- **Draft attachment is a classification choice** (role in-repo vs component='draft' separate repo), not a schema fork.
- **Multiple HDD Storage**: Settings page allows adding multiple storage locations. When downloading, user selects which HDD to use (or app chooses one with sufficient free space).
+- Direct-to-target downloads; OID-based dedupe; resumable transfers; state machine (B4); seconds-scale retry + manual retry; single copy queue; startup sweep. (Carried from v1.1.)
 - **Cache Promotion**: Copy entire model folder from HDD to SSD cache. Use `shutil.copytree` with progress callback. Store `cache_path`. On demotion, delete the SSD copy.
 ### 1.9 Risks & Mitigations
- **Large File Copy**: Use efficient copy (robocopy or `shutil.copy2` with buffer). Show progress.
+Carried from v1.1 (large copies → queue; space pre-checks; capped pools; token+backoff; WAL; orphan sweep; long paths; pinned hub version) **plus:** variant mislabeling from heuristics → editable tags + preview review; name collisions/illegal chars → sanitization + reserved-name checks; deep folder nesting → bounded layout + long-path enablement.
 - **Disk Space**: Check available space before download/copy; warn user.
 - **Concurrency**: Limit simultaneous downloads/copies to avoid disk thrash.
 - **HF API Rate Limits**: Use token to increase limits; implement retry with backoff.
 - **Database Corruption**: Use SQLite WAL mode; backup on exit.
 ---
 ## 2. Architecture Diagrams
-### 2.1 High-Level Component Diagram
+### 2.0 Entity Relationships
 ```mermaid
-flowchart TD
+erDiagram
-    subgraph Console Application
+    FAMILIES ||--o{ MODELS : contains
-        A[main.py] --> B[Uvicorn/FastAPI Server]
+    MODELS  ||--o{ REPOS   : aggregates
-    end
+    REPOS   ||--o{ MODEL_FILES : contains
-
+    STORAGE_LOCATIONS ||--o{ REPOS : hosts
-    subgraph Web Browser
+    REPOS   ||--o{ JOBS    : targeted-by
-        C[UI - HTML/JS/CSS]
+    MODELS  ||--o{ JOBS    : targeted-by
-    end
+    FAMILIES ||--o{ JOBS   : targeted-by
-
+    MODEL_FILES }o--|| REPOS : "scope folder may live under _shared/"
    B <--> C
    subgraph Background Processing
        D[Background Task Manager<br>Thread Pool]
        E[Download Manager]
        F[Storage Manager]
    end
    B --> D
    D --> E
    D --> F
    subgraph External
        G[Hugging Face Hub API]
    end
    E <--> G
    subgraph Storage
        H[HDD Level 3<br>Multiple Roots]
        I[SSD Level 2<br>Cache]
    end
    F --> H
    F --> I
    subgraph Database
        J[SQLite]
    end
    B --> J
    D --> J
 ```
-### 2.2 Download Flow Sequence
+### 2.1 Components
-
+
-```mermaid
+### 2.2 Download Flow
 sequenceDiagram
    participant U as User (Browser)
    participant API as FastAPI
    participant BW as Background Worker
    participant HF as Hugging Face API
    participant ST as Storage (HDD/SSD)
    participant DB as Database
    U->>API: POST /api/download (repo_id, file selection, options)
    API->>BW: Create background task
    API-->>U: Return task ID
    BW->>HF: list_repo_files(repo_id)
    HF-->>BW: File list with metadata
    BW->>BW: Classify files (manual via UI prior)
    BW->>HF: snapshot_download(allow_patterns, token)
    HF-->>BW: Download files to temp folder
    BW->>BW: Verify checksums
    BW->>ST: Move temp folder to final HDD path (atomic rename)
    BW->>DB: Insert model record + files
    alt if cache requested
        BW->>ST: Copy folder to SSD cache
        BW->>DB: Update is_cached=1, cache_path
    end
    BW-->>U: (via WebSocket/SSE) Progress and completion
 ```
 ### 2.3 Cache Promotion Flow
 ```mermaid
 sequenceDiagram
    autonumber
    participant U as User
    participant API as FastAPI
-    participant BW as Background Worker
+    participant JS as Job Scheduler
-    participant ST as Storage
+    participant DM as Download Mgr
-    participant DB as Database
+    participant HF as Hugging Face
    participant CQ as Copy Queue
    participant ST as Storage Mgr
    participant DB as SQLite
-    U->>API: POST /api/model/{id}/cache (checkbox)
+    U->>API: GET /api/repo/preview (url or org/name)
-    API->>BW: Queue cache/demote task
+    API->>HF: model_info(files_metadata=true)
-    API-->>U: Accepted
+    HF-->>API: files, sizes, LFS sha256 OIDs
-
+    API->>DB: family OID lookup, suggest family/model
-    alt Cache to SSD
+    API-->>U: grouped list (role, variant, scope, dedupe flags)
-        BW->>ST: Check available space on SSD
+    U->>API: POST /api/repos (family, model, files, roles, scopes, variants, root, cache)
-        BW->>ST: Copy model folder from HDD to SSD cache
+    API->>DB: ensure family/model rows
-        BW->>DB: Update is_cached=1, cache_path
+    API->>DB: create repo (status=downloading) and job row
-    else Demote from SSD
+    API-->>U: job id
-        BW->>ST: Delete cached folder
+    JS->>DM: run download job
-        BW->>DB: Update is_cached=0, cache_path=NULL
+    DM->>HF: snapshot_download(local_dir=_incoming, allow_patterns, revision, token)
    HF-->>DM: resumable stream into _incoming
    loop until download completes
        DM-->>U: SSE progress (bytes, speed)
    end
    DM->>DM: verify sizes and OIDs
    DM->>ST: place files per scope (repo folder or _shared) via temp+rename
    DM->>ST: dedupe by OID (hardlink same volume, copy cross volume)
    DM->>DB: insert file rows, repo (status=complete)
    alt cache requested
        JS->>CQ: enqueue promote
        CQ->>ST: copy repo folder to SSD .tmp
        CQ->>ST: rename .tmp to final
        CQ->>DB: is_cached = 1
    end
    BW-->>U: Completion notification
 ```
-### 2.4 Update Check Flow
+### 2.3 Promote / Demote
 ```mermaid
 sequenceDiagram
    autonumber
    participant U as User
    participant API as FastAPI
-    participant HF as Hugging Face API
+    participant JS as Job Scheduler
-    participant DB as Database
+    participant CQ as Copy Queue
    participant ST as Storage Mgr
    participant DB as SQLite
-    U->>API: POST /api/model/{id}/check-update
+    U->>API: POST /api/repos/123/cache (promote or demote)
-    API->>DB: Retrieve stored HF commit hash
+    API->>DB: create job row (queued)
-    API->>HF: get_repo_info(repo_id)
+    API->>JS: enqueue on copy queue
-    HF-->>API: Current commit hash, last modified
+    API-->>U: accepted (job id)
-    API->>API: Compare hashes
+    alt promote
-    alt hash differs
+        JS->>CQ: run promote
-        API-->>U: Update available (new hash)
+        CQ->>ST: free-space check on SSD
-    else same
+        alt insufficient space
-        API-->>U: Up-to-date
+            CQ->>DB: job (failed)
            CQ-->>U: SSE error (insufficient SSD space)
        else space ok
            CQ->>ST: copy repo folder to SSD .tmp
            CQ->>ST: rename .tmp to final
            CQ->>DB: is_cached = 1, set cache_path
            CQ-->>U: SSE completion
        end
    else demote
        JS->>CQ: run demote
        CQ->>ST: validate HDD copy (files present, sizes match DB)
        alt HDD invalid
            CQ->>DB: job (failed, demote blocked)
            CQ-->>U: SSE error (HDD copy missing or incomplete)
        else HDD valid
            CQ->>ST: delete SSD folder
            CQ->>DB: is_cached = 0, cache_path = null
            CQ-->>U: SSE completion
        end
    end
 ```
 ### 2.4 Update Check
 ```mermaid
 sequenceDiagram
    autonumber
    participant U as User
    participant API as FastAPI
    participant HF as Hugging Face
    participant DB as SQLite
    U->>API: POST /api/repos/123/check-update
    API->>DB: load stored revision (commit hash)
    API->>HF: repo info (current sha)
    HF-->>API: current commit
    API->>API: compare commits
    alt same commit
        API-->>U: up to date
    else commit differs
        API->>HF: model_info(files_metadata=true)
        HF-->>API: current files, sizes, OIDs
        API->>DB: diff per-file OIDs against stored
        API-->>U: list added, changed, removed
        U->>API: POST /api/repos/123/apply-update (selected files)
        API->>DB: create update_apply job
        API-->>U: accepted (job id)
        Note over API,DB: apply job downloads changed files to _incoming, swaps per file via temp+rename (scope-aware), updates rows and revision
    end
    Note over U,DB: Models page aggregates update state across all repos of a model
 ```
 ---
 ## 3. Test Plan & Test Cases
-### 3.1 Test Strategy
+Strategy unchanged (unit / mock-HF integration / system / performance; retry delays configurable to seconds). TC-001…TC-029 from v1.1 carry over with target renames (model→repo where applicable: TC-016, TC-018, TC-019, TC-026). New/changed:
 - **Unit Tests**: For file role classification (basic extension-based), retry logic, family sharing decision, database operations.
 - **Integration Tests**: Downloading from a mock HF server (local HTTP), storage operations on temporary directories (tmpfs).
 - **System Tests**: End-to-end through API and UI using sample repositories (may be network-dependent; mocks used for speed).
 - **Performance Tests**: Simulate large file downloads (using dummy files) to ensure UI responsiveness.
-### 3.2 Test Cases
+| ID | Scenario |
-
+|----|----------|
-| ID | Test Scenario | Preconditions | Steps | Expected Result |
+| TC-030 | Model with 3 repos (official + 2 community) — merged per-model variant list correct; repos distinct folders |
-|----|---------------|---------------|-------|-----------------|
+| TC-031 | Non-standard variant names (`APEX-Compact`, `I-Balanced`) — parsed or defaulted, editable, persisted; shards grouped under one tag |
-| TC-001 | List files from HF repo | Valid repo ID, mock HF API | Call `/api/repo/{repo_id}/files` | Returns list with filenames, sizes, roles inferred |
+| TC-032 | Draft MTP attached (a) as role=speedup in main repo, (b) as separate repo component='draft' — both resolvable as draft target for future config maker |
-| TC-002 | Download a GGUF text model | Mock HF repo with GGUF file | User selects file, triggers download | File downloaded to chosen HDD, DB entry created, checksum verified |
+| TC-033 | mmproj scope=model — downloaded once; second repo of same model reuses via OID link/copy; stored in `{fam}/_shared/{model}/` |
-| TC-003 | Download image model with multiple safetensors | Mock repo with main, VAE, CLIP files | User selects all files, assigns roles manually | All files downloaded, roles stored correctly |
+| TC-034 | jinja scope=family — stored in `{fam}/_shared/`, shared across the family's models |
-| TC-004 | Retry on transient failure | Simulate network error on first attempt, then success | Trigger download | First attempt fails, retry after delay (configurable), second succeeds |
+| TC-035 | Add New suggests family=org, model=name; user can map to existing family/model instead |
-| TC-005 | Checksum mismatch | Corrupt file after download | Download | App detects mismatch, deletes temp file, retries download |
+| TC-036 | Delete guards: model with repos blocked; family with models blocked; repo delete removes folder+rows |
-| TC-006 | Promote model to SSD cache | Model on HDD | Check "Cache to SSD" in UI | Model folder copied to SSD, DB updated, original intact |
+| TC-037 | Reserved names rejected (`_shared`, `_incoming`, `CON`, trailing dots) for family/model/repo names |
-| TC-007 | Demote from SSD cache | Model cached | Uncheck "Cache to SSD" | SSD copy deleted, DB updated |
+| TC-038 | Two orgs, same repo name, under one model — second gets hash suffix, no overwrite |
-| TC-008 | Family sharing | Two models in same family; shared mmproj already exists | Download second model | Shared file is hard-linked instead of downloaded again |
+| TC-039 | Same HF repo submitted for a second model — rejected cleanly (UNIQUE) with clear message |
-| TC-009 | Update check | Model downloaded with older commit hash; HF has newer | Trigger update check | App reports update available |
+| TC-040 | URL with branch (`/tree/dev`) — revision pinned and stored; shorthand `org/name` accepted |
 | TC-010 | Graceful shutdown during download | Active download | Press Ctrl+C | Download completes (if within timeout), DB consistent, process exits |
 | TC-011 | Multiple HDD storage | Two HDD roots configured | Download model, choose second root | Model stored in selected root |
 | TC-012 | Token protected download | Private repo, valid token | Download with token | Download succeeds; token not logged |
 | TC-013 | Invalid repo ID | Nonexistent repo | Provide bad ID | Error message returned, no crash |
 | TC-014 | Low disk space | HDD nearly full | Start download | Warning displayed, download aborted before copying |
 | TC-015 | Hard link creation on different drives | Attempt to hard link across different drives | Family sharing across storage locations | App falls back to copy (or disallows family sharing if drives differ) |
 ### 3.3 Test Environment
 - Windows 10/11 machine with Python 3.11.
 - Use temporary directories to simulate HDD and SSD (different drives if possible; else use folders).
 - Mock Hugging Face server (e.g., `responses` library or a local Flask app) to avoid network dependency.
 - For UI tests, use browser automation (Selenium or Playwright) to test basic interactions.
 ---
 ## 4. Security & Privacy Assessment
-### 4.1 Assets and Threats
+Carried from v1.1 (loopback-only + Host-header check; `keyring` token; repo-ID regex + relative-path containment; try/finally temp cleanup; pinned deps + `pip-audit`; SQLite perms/WAL; log redaction; deep-verify as explicit command). **Added for v1.2:** folder-name sanitization treats all HF-derived strings (org, repo, filename) as untrusted — illegal chars, reserved device names, trailing dots/spaces, reserved app names (`_shared`, `_incoming`) — before any path is constructed; delete operations require explicit confirmation (destructive scope grew with hierarchy cascades). Privacy unchanged: no telemetry; HF metadata + transfers only; no content inspection beyond names/sizes/hashes.
 - **Assets**: Model files (potentially proprietary or sensitive), Hugging Face token, application configuration, database metadata.
 - **Threats**:
  - Unauthorized access to web UI if bound to non-localhost interface.
  - Token leakage through logs or configuration files.
  - Path traversal attacks via malicious repo IDs or filenames.
  - Supply chain attacks via compromised HF dependencies.
  - Data loss due to race conditions or improper shutdown.
 ### 4.2 Security Measures
 - **Network Binding**: By default, Uvicorn binds to `127.0.0.1` only. Settings can change to `0.0.0.0` with a warning, but no authentication is provided. For this single-user local tool, it's acceptable.
 - **Token Storage**: Hugging Face token is stored encrypted using Windows Data Protection API (DPAPI). The encryption key is tied to the Windows user account. Implementation uses `cryptography` with `win32crypt` or `dpapi` package. The token is never logged.
 - **Input Validation**: Repo IDs are validated against a strict pattern (e.g., `^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$`). File paths are constructed using `os.path.join` and checked to be within the allowed storage root using `os.path.commonpath`.
 - **File Operations**: All file moves/copies are performed within known directories. Temporary files are deleted on failure. Use `filelock` to prevent concurrent modifications.
 - **Dependency Management**: Use `pip-audit` in CI to scan for known vulnerabilities. Regularly update dependencies.
 - **Database**: SQLite database file permissions set to user-only (Windows default). WAL mode ensures durability.
 - **Logging**: Logs exclude tokens and full file paths if sensitive. Log level configurable.
 ### 4.3 Privacy Considerations
 - **Data Collection**: The application does not collect or transmit any personal data. It only communicates with Hugging Face API for download and metadata.
 - **User Activity**: The UI may store user preferences (e.g., last selected storage location) locally; no analytics.
 - **Model Files**: Model files are user-provided content; the app does not inspect their contents beyond file names and sizes.
 ### 4.4 Compliance
 - Not subject to GDPR or similar regulations as no personal data is processed. If private models are used, the token is the only sensitive credential; handled securely.
 ---
-## Summary
+## Open items (v1.2 decisions applied — veto freely)
-These documents provide a comprehensive design for the Model Manager as per your requirements. The system is modular, Windows-focused, and easy to extend. If you need further elaboration on any section or adjustments based on additional thoughts, please let me know.
+1. **Family required**, auto-suggested from org (TC-035) — alternative would be a `_none` pseudo-family.
 2. **Monolithic image repos stay whole** under `component='main'` with internal structure preserved (not split into vae/… type folders).
 3. **Variant parsing is heuristic + editable** — no perfect auto-detection for arbitrary naming schemes.
 4. **Promote unit = repo folder**; shared-scope files (mmproj/jinja) resolve from their live location (L3 is fine — they're small). Promoting `_shared` contents is a future option.
 5. **One HF repo = one model attachment** (global UNIQUE) — a repo truly serving two models isn't representable in v1.
 6. Repo folder = repo name (+hash on collision); `_shared`/`_incoming` reserved.
 7. Config maker still awaiting your details — the schema now gives it everything it needs (main GGUF + variant, mmproj path via scope, draft target via role or component='draft').
 Natural next steps: (a) implementation plan — module skeleton + build order, or (b) spec the llama.cpp config maker now that the data model can feed it. Which do you want?