diff --git a/doc/DESIGN-DOC.md b/doc/DESIGN-DOC.md index 48d2a3a..a9710d3 100644 --- a/doc/DESIGN-DOC.md +++ b/doc/DESIGN-DOC.md @@ -208,7 +208,7 @@ Carried from v1.1 (large copies → queue; space pre-checks; capped pools; token ## 2. Architecture Diagrams -### 2.0 Entity Relationships (new) +### 2.0 Entity Relationships ```mermaid erDiagram FAMILIES ||--o{ MODELS : contains @@ -221,43 +221,120 @@ erDiagram MODEL_FILES }o--|| REPOS : "scope folder may live under _shared/" ``` -### 2.1 Components — unchanged from v1.1 (Storage Manager now hierarchy/path-scope aware). +### 2.1 Components ### 2.2 Download Flow + ```mermaid sequenceDiagram + autonumber participant U as User participant API as FastAPI participant JS as Job Scheduler participant DM as Download Mgr participant HF as Hugging Face - participant ST as Storage + participant CQ as Copy Queue + participant ST as Storage Mgr participant DB as SQLite U->>API: GET /api/repo/preview (url or org/name) API->>HF: model_info(files_metadata=true) - HF-->>API: files + sizes + LFS OIDs - API->>DB: family OID lookup + suggest family/model + HF-->>API: files, sizes, LFS sha256 OIDs + API->>DB: family OID lookup, suggest family/model API-->>U: grouped list (role, variant, scope, dedupe flags) - U->>API: POST /api/repos (family/model targets, files, roles, scopes, variants, root, cache?) - API->>DB: ensure family/model rows; repo(status=downloading) + job row + U->>API: POST /api/repos (family, model, files, roles, scopes, variants, root, cache) + API->>DB: ensure family/model rows + API->>DB: create repo (status=downloading) and job row API-->>U: job id - JS->>DM: run - DM->>HF: snapshot_download(local_dir=_incoming/{repo_id}, allow_patterns, revision, token) - HF-->>DM: resumable stream - DM->>ST: place files per scope (repo folder / _shared/{model} / _shared) via temp+rename; dedupe by OID (link/copy) - DM->>DB: files rows + repo(status=complete) - alt cache requested - JS->>ST: enqueue promote (copy queue) - ST->>ST: copy → SSD .tmp → rename - ST->>DB: is_cached=1 + JS->>DM: run download job + DM->>HF: snapshot_download(local_dir=_incoming, allow_patterns, revision, token) + HF-->>DM: resumable stream into _incoming + loop until download completes + DM-->>U: SSE progress (bytes, speed) + end + DM->>DM: verify sizes and OIDs + DM->>ST: place files per scope (repo folder or _shared) via temp+rename + DM->>ST: dedupe by OID (hardlink same volume, copy cross volume) + DM->>DB: insert file rows, repo (status=complete) + alt cache requested + JS->>CQ: enqueue promote + CQ->>ST: copy repo folder to SSD .tmp + CQ->>ST: rename .tmp to final + CQ->>DB: is_cached = 1 end - DM--)U: SSE progress ``` -### 2.3 Promote / Demote — as v1.1, unit = repo folder; demote gated on HDD validation. +### 2.3 Promote / Demote -### 2.4 Update Check — as v1.4 flow, per repo; model page aggregates repo results. +```mermaid +sequenceDiagram + autonumber + participant U as User + participant API as FastAPI + participant JS as Job Scheduler + participant CQ as Copy Queue + participant ST as Storage Mgr + participant DB as SQLite + + U->>API: POST /api/repos/123/cache (promote or demote) + API->>DB: create job row (queued) + API->>JS: enqueue on copy queue + API-->>U: accepted (job id) + alt promote + JS->>CQ: run promote + CQ->>ST: free-space check on SSD + alt insufficient space + CQ->>DB: job (failed) + CQ-->>U: SSE error (insufficient SSD space) + else space ok + CQ->>ST: copy repo folder to SSD .tmp + CQ->>ST: rename .tmp to final + CQ->>DB: is_cached = 1, set cache_path + CQ-->>U: SSE completion + end + else demote + JS->>CQ: run demote + CQ->>ST: validate HDD copy (files present, sizes match DB) + alt HDD invalid + CQ->>DB: job (failed, demote blocked) + CQ-->>U: SSE error (HDD copy missing or incomplete) + else HDD valid + CQ->>ST: delete SSD folder + CQ->>DB: is_cached = 0, cache_path = null + CQ-->>U: SSE completion + end + end +``` + +### 2.4 Update Check + +```mermaid +sequenceDiagram + autonumber + participant U as User + participant API as FastAPI + participant HF as Hugging Face + participant DB as SQLite + + U->>API: POST /api/repos/123/check-update + API->>DB: load stored revision (commit hash) + API->>HF: repo info (current sha) + HF-->>API: current commit + API->>API: compare commits + alt same commit + API-->>U: up to date + else commit differs + API->>HF: model_info(files_metadata=true) + HF-->>API: current files, sizes, OIDs + API->>DB: diff per-file OIDs against stored + API-->>U: list added, changed, removed + U->>API: POST /api/repos/123/apply-update (selected files) + API->>DB: create update_apply job + API-->>U: accepted (job id) + Note over API,DB: apply job downloads changed files to _incoming, swaps per file via temp+rename (scope-aware), updates rows and revision + end + Note over U,DB: Models page aggregates update state across all repos of a model +``` ---