doc: Added design doc

doc/DESIGN-DOC.md

@@ -0,0 +1,300 @@
+# Model Manager — Design Doc v1.2
+
+## 1. Design Document
+
+### 1.1 System Overview
+The Model Manager is a Windows application with a web-based UI served by a local Python/FastAPI server. It manages large AI model files (5–80 GB) downloaded from Hugging Face. Content is organized as a hierarchy: **families contain models; models contain one or more repos; repos contain files (with quant/variant tags)**. Models are stored on one or more HDD (L3) roots and can be promoted to an SSD (L2) cache. Metadata includes per-file HF content hashes; families share files by hash-verified dedupe; update detection is per-file diff.
+
+### 1.2 Goals & Non-Goals
+**Goals:**
+- Download from Hugging Face (public/private with token), resumable across restarts
+- **Hierarchy: family → model → repo → files; multiple repos and multiple quants per model; non-standard variant tags supported (user-editable)** 
+- Multiple HDD roots; promotion to SSD cache; HDD integrity validated before demotion
+- Model types: text, image, embedding
+- **Shared files at family or model scope (jinja at any level, mmproj typically model)** 
+- Draft/speed-up models (MTP, DFlash) attachable **same-repo (role) or separate-repo (component) — user choice**
+- Per-file update diff (commit hash as fast pre-check); timestamps tracked
+- Web UI (editable without recompile), SSE live progress, Logs page
+- Graceful shutdown (5-min default wait, force-pause option); abort vs pause distinction
+- Retry in seconds + manual "Retry Now"; single-worker disk-copy queue
+
+**Non-Goals (current version):**
+- Automatic family/model detection (suggested from repo org/name, but user confirms)
+- Automatic cache eviction; llama.cpp config generation (schema/roles ready; placeholder service)
+- Automatic image-gen role detection; multi-user/LAN access
+- Local deep-hash verification on every operation (explicit command only)
+
+### 1.3 Functional Requirements
+
+#### 1.3.1 Content Model & Storage Layout
+**Hierarchy:** `family → model → repo → files`.
+
+- **Family** (e.g., "Qwen3.5"): groups related models. Required; auto-suggested from repo org at Add New. May contain many models (27B, 35B, 9B…).
+- **Model** (e.g., "Qwen3.5-27B"): logical model of a type (`text`/`image`/`embedding`). May aggregate **multiple repos** (official + community quants); UI shows a merged per-model view of all variants across repos.
+- **Repo**: one HF repository, attached to exactly one model, with a `component` tag used as its folder segment: `main`, `draft`, `vae`, `lora`, `other` (default `main`).
+- **File**: one downloaded file, with `role`, `variant` (quant/variety tag), and `scope`.
+
+**Variant tags:** parsed from filenames (strip shard suffixes `-NNNNN-of-NNNNN`; match known patterns `Q\d+_…`, `IQ\d…`, `UD-…`; fallback = trailing name segments). **Heuristics will not catch every naming scheme (e.g., `APEX-Compact`, `I-Balanced`)** — the tag is always editable in the preview UI and in file details. Sharded GGUFs share one variant tag.
+
+**Draft/MTP/DFlash (user-selectable attachment):**
+- *Under repo:* draft files inside the main model's repo → role `speedup`, stored with the repo.
+- *Under model:* a separate HF repo → its own repo row with `component='draft'`, stored under `{model}/draft/`.
+- Either way, the config maker (future) can resolve a draft target for `--model-draft`.
+
+**Shared-scope files:** each file has `scope`:
+| scope | physical location | typical use |
+|---|---|---|
+| `repo` (default) | `{root}/{model_type}/{fam}/{component}/{model}/{repo}/…` | quants, VAE, CLIP, tokenizers, LoRA |
+| `model` | `{root}/{model_type}/{fam}/_shared/{model}/…` | **mmproj (default)**, model-level jinja |
+| `family` | `{root}/{model_type}/{fam}/_shared/…` | family-level jinja overrides |
+
+Files keep their source-repo provenance (FK to repo) even when stored at model/family scope. Defaults at download time: mmproj → `model`, everything else → `repo`; user can elevate jinja to model/family in the preview UI.
+
+**Folder structure:** `{storage_root}/{text|image|embed}/{family}/{component}/{model}/{repo_name}/{file}`
+(plus `{storage_root}/{model_type}/{fam}/_shared/…` and `{storage_root}/_incoming/{repo_id}/` temp areas).
+- Model type→folder map: text→`text`, image→`image`, embedding→`embed`.
+- Monolithic image repos (diffusers: `unet/`, `vae/`, `text_encoder/` inside one repo) → `component='main'`, **internal repo structure preserved** per file (`rel_file_path`).
+- Reserved folder names: `_shared`, `_incoming`; model/repo/family names sanitized (Windows-illegal chars, reserved device names, trailing dots/spaces, length caps). Repo folder = repo name; on collision append a short hash.
+
+**Storage roots:** multiple HDD roots, removable only when unreferenced. **L2 (SSD):** single cache root in settings; promote/demote operates on **repo folders** (promoting a "model" = queue promote for each of its repos). Demotion validates the HDD copy first (files present, sizes match DB). Shared-scope files are small and resolve from wherever they live; promoting them is a future option.
+
+**Path handling:** DB stores IDs + relative paths only; resolution joins root + segments and validates containment. Windows long paths enabled and documented; depth is bounded by the 6-level layout (TC-024).
+
+#### 1.3.2 Download Workflow
+1. User pastes **full URL or `org/name` shorthand**; URL cruft stripped; a branch/revision in the URL is pinned and stored. Recent repos remembered in a dropdown.
+2. Preview: `model_info(files_metadata=True)` → names, sizes, **LFS sha256 OIDs**.
+3. **Entity resolution:** app suggests family (from org) and model (from name, suffix-stripped); user can pick existing family/model or create new. For an existing model, the repo attaches to it; files whose OID matches any family file are flagged "available via family" and skipped by default (small non-LFS files never deduped).
+4. Files grouped by role and **variant**; user selects files, assigns roles/scopes/variant fixes, picks HDD root, optionally "Cache to SSD."
+5. `families`/`models`/`repos` rows ensured (repo `status='downloading'`), `jobs` row created; download targets `{root}/_incoming/{repo_id}/` on the destination volume.
+6. `snapshot_download(local_dir=<incoming>, allow_patterns, revision, token)`. Direct-to-target volume; no `~/.cache` duplication (asserted at runtime); `hf_transfer` optional toggle.
+7. Resumable; retry 5s→15s→45s→2m, cap 5m, `max_retries` default 5, manual "Retry Now." Retries exhausted → `paused`, partials kept, auto-resume offered on next launch.
+8. Success: files moved into final locations per scope (temp+rename throughout; cross-volume dedupe → copy, same-volume → hardlink), OIDs/sizes recorded, `status='complete'`.
+9. **Cancel** → `_incoming` deleted, repo `cancelled`. **Pause** → partials kept. Crash → startup sweep resumes.
+10. "Cache to SSD" → copy job enqueued (§1.3.7); SSD copy also temp-then-rename.
+
+#### 1.3.3 Family Sharing & Scope Rules
+- Dedupe identity = **LFS sha256 OID**, within a family, across models/repos. Matched files are materialized by hardlink (same volume) or copy (cross volume).
+- **Shared files are immutable: never rewritten in place** — updates create new files and swap via rename, so hardlink partners are unaffected.
+- Small non-LFS files (jinja, configs, tokenizers) are always re-downloaded, never deduped.
+- User override: force re-download per file.
+
+#### 1.3.4 Update Check
+Per repo (commit hash + per-file OIDs stored at download). Pre-check commit; if changed, per-file OID diff → added/changed/removed list → user selects → apply job re-downloads changed files to `_incoming`, swaps per file via rename (scope-aware), updates rows + revision. Model-level UI aggregates update state across its repos.
+
+#### 1.3.5 Web UI
+- **Overview:** counts by type, disk usage per root, SSD usage, active jobs, pending updates.
+- **Models:** grouped Family → Model → Repos; per-model merged variant list; per-repo cache status.
+- **Add New:** URL/shorthand input, preview, family/model resolver, file selection with roles/scopes/variants, recent repos.
+- **Settings:** roots, SSD cache root, token, concurrency, retries.
+- **Logs:** live tail (SSE) + rotating file log.
+- Static `web/` folder, vanilla JS, SSE (`/api/events`, `/api/logs/stream`). No recompile to edit.
+
+#### 1.3.6 Graceful Shutdown
+Ctrl+C or UI quit → stop accepting jobs → wait up to **5 min** (configurable) for active tasks; UI shows "waiting for N tasks"; force-pause keeps partials for next launch; final DB checkpoint (WAL; writes are continuous anyway).
+
+#### 1.3.7 Copy Queue
+Single worker serializes all disk-to-disk ops (promote, demote, update swaps, scope materialization). Network pool separate (default 2). Copies cancelable with try/finally temp cleanup.
+
+#### 1.3.8 Startup Recovery
+Sweep: `repos.status='downloading'` ↔ `_incoming/{repo_id}/` reconciled (resume or clean); orphaned `_incoming` dirs and SSD `.tmp` folders deleted with log entries. 
+
+### 1.4 Non-Functional Requirements
+Performance (SSE progress, 2 network workers, 1 copy worker, optional `hf_transfer`); reliability (same-volume atomic renames, DB written at job start/end, WAL); security (loopback-only, `keyring` token, path validation, log redaction); Windows (long paths, short roots, name sanitization).
+
+### 1.5 Technology Stack
+Python 3.11, FastAPI, Uvicorn, SQLAlchemy (SQLite/WAL), `huggingface_hub` (>= 0.23, pinned), `pydantic`, `keyring`, optional `hf_transfer`; static `web/` + vanilla JS; `ThreadPoolExecutor` pools; `filelock`; `shutil`/`robocopy`.
+
+### 1.6 Data Model
+
+```sql
+CREATE TABLE families (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    name TEXT UNIQUE NOT NULL,          -- sanitized; reserved names rejected
+    description TEXT,
+    created_at DATETIME
+);
+
+CREATE TABLE models (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    family_id INTEGER NOT NULL,
+    name TEXT NOT NULL,                 -- "Qwen3.5-27B" (sanitized)
+    model_type TEXT NOT NULL CHECK(model_type IN ('text','image','embedding')),
+    description TEXT,
+    created_at DATETIME,
+    UNIQUE(family_id, name),
+    FOREIGN KEY(family_id) REFERENCES families(id)
+);
+
+CREATE TABLE repos (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    model_id INTEGER NOT NULL,
+    hf_repo_id TEXT UNIQUE NOT NULL,    -- "org/name"; one row per HF repo
+    component TEXT NOT NULL DEFAULT 'main'
+        CHECK(component IN ('main','draft','vae','lora','other')), -- folder {component}
+    revision TEXT,                      -- commit hash at download
+    status TEXT NOT NULL DEFAULT 'downloading'
+        CHECK(status IN ('downloading','complete','paused','error','cancelled')),
+    storage_location_id INTEGER NOT NULL,
+    rel_path TEXT NOT NULL,             -- {model_type}/{fam}/{component}/{model}/{repo}
+    hf_last_modified DATETIME,
+    download_started DATETIME,
+    download_completed DATETIME,
+    is_cached INTEGER DEFAULT 0,
+    cache_path TEXT,                    -- relative to SSD root
+    UNIQUE(storage_location_id, rel_path),
+    FOREIGN KEY(model_id) REFERENCES models(id),
+    FOREIGN KEY(storage_location_id) REFERENCES storage_locations(id)
+);
+-- Model status is DERIVED from its repos (UI aggregate). Downloads/caching operate per repo.
+
+CREATE TABLE model_files (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    repo_id INTEGER NOT NULL,           -- provenance (source repo)
+    rel_file_path TEXT NOT NULL,        -- path within its scope folder
+    role TEXT NOT NULL DEFAULT 'other' CHECK(role IN (
+        'main','mmproj','speedup','jinja','vae','clip','text_encoder',
+        'lora','tokenizer','other')),
+    variant TEXT,                       -- quant/variety tag; shared by shards; user-editable
+    scope TEXT NOT NULL DEFAULT 'repo' CHECK(scope IN ('family','model','repo')),
+    size_bytes INTEGER,
+    lfs_sha256 TEXT,                    -- HF OID; NULL for small non-LFS files
+    UNIQUE(repo_id, rel_file_path),
+    FOREIGN KEY(repo_id) REFERENCES repos(id)
+);
+
+-- CHANGED: polymorphic target (downloads/caches target repos; checks may target models)
+CREATE TABLE jobs (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    job_type TEXT NOT NULL CHECK(job_type IN
+        ('download','cache_promote','cache_demote','update_check','update_apply','delete')),
+    target_type TEXT NOT NULL CHECK(target_type IN ('repo','model','family')),
+    target_id INTEGER NOT NULL,
+    status TEXT NOT NULL CHECK(status IN
+        ('queued','running','paused','completed','failed','cancelled')),
+    detail TEXT,                        -- JSON
+    progress REAL,
+    created_at DATETIME, started_at DATETIME, ended_at DATETIME
+);
+
+CREATE TABLE storage_locations (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    path TEXT UNIQUE NOT NULL,          -- HDD roots only
+    label TEXT,
+    is_active INTEGER DEFAULT 1
+);
+
+CREATE TABLE settings (key TEXT PRIMARY KEY, value TEXT);
+-- Keys: ssd_cache_root, max_concurrent_downloads, retry_schedule, max_retries,
+-- hf_transfer_enabled, recent_repos, shutdown_wait_seconds
+-- HF token lives in Credential Manager (keyring), never here.
+```
+
+**Delete guardrails:** repo delete requires confirm (removes folder + files rows); model delete blocked while repos exist; family delete blocked while models exist. **Same HF repo cannot attach to two models** (global UNIQUE) — flagged as a v1 limitation.
+
+### 1.7 Architecture Overview
+Unchanged components: console entry (signals) → FastAPI (API+SSE+static) → Job Scheduler → Download Manager (network pool) + Copy Queue (1 worker) → Storage Manager (path resolution now hierarchy-aware, temp+rename, hardlink/copy, space/integrity checks) → HF Service → DB → Startup Recovery → Config Service (placeholder).
+
+### 1.8 Key Design Decisions
+- **Hierarchy over flat repos:** families/models are cheap metadata; repos are the unit of download/caching; files carry role+variant+scope. Enables multi-repo models, multi-quant repos, and model/family-shared files without duplication. *(User decision, v1.2.)*
+- **Variant = per-file tag**, heuristic-parsed, user-editable; shards grouped by tag.
+- **Scope controls placement**, provenance stays on the repo; shared files immutable (rename-swap updates).
+- **Draft attachment is a classification choice** (role in-repo vs component='draft' separate repo), not a schema fork.
+- Direct-to-target downloads; OID-based dedupe; resumable transfers; state machine (B4); seconds-scale retry + manual retry; single copy queue; startup sweep. (Carried from v1.1.)
+
+### 1.9 Risks & Mitigations
+Carried from v1.1 (large copies → queue; space pre-checks; capped pools; token+backoff; WAL; orphan sweep; long paths; pinned hub version) **plus:** variant mislabeling from heuristics → editable tags + preview review; name collisions/illegal chars → sanitization + reserved-name checks; deep folder nesting → bounded layout + long-path enablement.
+
+---
+
+## 2. Architecture Diagrams
+
+### 2.0 Entity Relationships (new)
+```mermaid
+erDiagram
+    FAMILIES ||--o{ MODELS : contains
+    MODELS  ||--o{ REPOS   : aggregates
+    REPOS   ||--o{ MODEL_FILES : contains
+    STORAGE_LOCATIONS ||--o{ REPOS : hosts
+    REPOS   ||--o{ JOBS    : targeted-by
+    MODELS  ||--o{ JOBS    : targeted-by
+    FAMILIES ||--o{ JOBS   : targeted-by
+    MODEL_FILES }o--|| REPOS : "scope folder may live under _shared/"
+```
+
+### 2.1 Components — unchanged from v1.1 (Storage Manager now hierarchy/path-scope aware).
+
+### 2.2 Download Flow
+```mermaid
+sequenceDiagram
+    participant U as User
+    participant API as FastAPI
+    participant JS as Job Scheduler
+    participant DM as Download Mgr
+    participant HF as Hugging Face
+    participant ST as Storage
+    participant DB as SQLite
+
+    U->>API: GET /api/repo/preview (url or org/name)
+    API->>HF: model_info(files_metadata=true)
+    HF-->>API: files + sizes + LFS OIDs
+    API->>DB: family OID lookup + suggest family/model
+    API-->>U: grouped list (role, variant, scope, dedupe flags)
+    U->>API: POST /api/repos (family/model targets, files, roles, scopes, variants, root, cache?)
+    API->>DB: ensure family/model rows; repo(status=downloading) + job row
+    API-->>U: job id
+    JS->>DM: run
+    DM->>HF: snapshot_download(local_dir=_incoming/{repo_id}, allow_patterns, revision, token)
+    HF-->>DM: resumable stream
+    DM->>ST: place files per scope (repo folder / _shared/{model} / _shared) via temp+rename; dedupe by OID (link/copy)
+    DM->>DB: files rows + repo(status=complete)
+    alt cache requested
+        JS->>ST: enqueue promote (copy queue)
+        ST->>ST: copy → SSD .tmp → rename
+        ST->>DB: is_cached=1
+    end
+    DM--)U: SSE progress
+```
+
+### 2.3 Promote / Demote — as v1.1, unit = repo folder; demote gated on HDD validation.
+
+### 2.4 Update Check — as v1.4 flow, per repo; model page aggregates repo results.
+
+---
+
+## 3. Test Plan & Test Cases
+
+Strategy unchanged (unit / mock-HF integration / system / performance; retry delays configurable to seconds). TC-001…TC-029 from v1.1 carry over with target renames (model→repo where applicable: TC-016, TC-018, TC-019, TC-026). New/changed:
+
+| ID | Scenario |
+|----|----------|
+| TC-030 | Model with 3 repos (official + 2 community) — merged per-model variant list correct; repos distinct folders |
+| TC-031 | Non-standard variant names (`APEX-Compact`, `I-Balanced`) — parsed or defaulted, editable, persisted; shards grouped under one tag |
+| TC-032 | Draft MTP attached (a) as role=speedup in main repo, (b) as separate repo component='draft' — both resolvable as draft target for future config maker |
+| TC-033 | mmproj scope=model — downloaded once; second repo of same model reuses via OID link/copy; stored in `{fam}/_shared/{model}/` |
+| TC-034 | jinja scope=family — stored in `{fam}/_shared/`, shared across the family's models |
+| TC-035 | Add New suggests family=org, model=name; user can map to existing family/model instead |
+| TC-036 | Delete guards: model with repos blocked; family with models blocked; repo delete removes folder+rows |
+| TC-037 | Reserved names rejected (`_shared`, `_incoming`, `CON`, trailing dots) for family/model/repo names |
+| TC-038 | Two orgs, same repo name, under one model — second gets hash suffix, no overwrite |
+| TC-039 | Same HF repo submitted for a second model — rejected cleanly (UNIQUE) with clear message |
+| TC-040 | URL with branch (`/tree/dev`) — revision pinned and stored; shorthand `org/name` accepted |
+
+---
+
+## 4. Security & Privacy Assessment
+
+Carried from v1.1 (loopback-only + Host-header check; `keyring` token; repo-ID regex + relative-path containment; try/finally temp cleanup; pinned deps + `pip-audit`; SQLite perms/WAL; log redaction; deep-verify as explicit command). **Added for v1.2:** folder-name sanitization treats all HF-derived strings (org, repo, filename) as untrusted — illegal chars, reserved device names, trailing dots/spaces, reserved app names (`_shared`, `_incoming`) — before any path is constructed; delete operations require explicit confirmation (destructive scope grew with hierarchy cascades). Privacy unchanged: no telemetry; HF metadata + transfers only; no content inspection beyond names/sizes/hashes.
+
+---
+
+## Open items (v1.2 decisions applied — veto freely)
+
+1. **Family required**, auto-suggested from org (TC-035) — alternative would be a `_none` pseudo-family.
+2. **Monolithic image repos stay whole** under `component='main'` with internal structure preserved (not split into vae/… type folders).
+3. **Variant parsing is heuristic + editable** — no perfect auto-detection for arbitrary naming schemes.
+4. **Promote unit = repo folder**; shared-scope files (mmproj/jinja) resolve from their live location (L3 is fine — they're small). Promoting `_shared` contents is a future option.
+5. **One HF repo = one model attachment** (global UNIQUE) — a repo truly serving two models isn't representable in v1.
+6. Repo folder = repo name (+hash on collision); `_shared`/`_incoming` reserved.
+7. Config maker still awaiting your details — the schema now gives it everything it needs (main GGUF + variant, mmproj path via scope, draft target via role or component='draft').
+
+Natural next steps: (a) implementation plan — module skeleton + build order, or (b) spec the llama.cpp config maker now that the data model can feed it. Which do you want?