Doc: Updated Design Doc

This commit is contained in:
2026-08-28 14:22:26 +02:00
parent 6531fb87c2
commit ae42966acf
+96 -19
View File
@@ -208,7 +208,7 @@ Carried from v1.1 (large copies → queue; space pre-checks; capped pools; token
## 2. Architecture Diagrams
### 2.0 Entity Relationships (new)
### 2.0 Entity Relationships
```mermaid
erDiagram
FAMILIES ||--o{ MODELS : contains
@@ -221,43 +221,120 @@ erDiagram
MODEL_FILES }o--|| REPOS : "scope folder may live under _shared/"
```
### 2.1 Components — unchanged from v1.1 (Storage Manager now hierarchy/path-scope aware).
### 2.1 Components
### 2.2 Download Flow
```mermaid
sequenceDiagram
autonumber
participant U as User
participant API as FastAPI
participant JS as Job Scheduler
participant DM as Download Mgr
participant HF as Hugging Face
participant ST as Storage
participant CQ as Copy Queue
participant ST as Storage Mgr
participant DB as SQLite
U->>API: GET /api/repo/preview (url or org/name)
API->>HF: model_info(files_metadata=true)
HF-->>API: files + sizes + LFS OIDs
API->>DB: family OID lookup + suggest family/model
HF-->>API: files, sizes, LFS sha256 OIDs
API->>DB: family OID lookup, suggest family/model
API-->>U: grouped list (role, variant, scope, dedupe flags)
U->>API: POST /api/repos (family/model targets, files, roles, scopes, variants, root, cache?)
API->>DB: ensure family/model rows; repo(status=downloading) + job row
U->>API: POST /api/repos (family, model, files, roles, scopes, variants, root, cache)
API->>DB: ensure family/model rows
API->>DB: create repo (status=downloading) and job row
API-->>U: job id
JS->>DM: run
DM->>HF: snapshot_download(local_dir=_incoming/{repo_id}, allow_patterns, revision, token)
HF-->>DM: resumable stream
DM->>ST: place files per scope (repo folder / _shared/{model} / _shared) via temp+rename; dedupe by OID (link/copy)
DM->>DB: files rows + repo(status=complete)
alt cache requested
JS->>ST: enqueue promote (copy queue)
ST->>ST: copy → SSD .tmp → rename
ST->>DB: is_cached=1
JS->>DM: run download job
DM->>HF: snapshot_download(local_dir=_incoming, allow_patterns, revision, token)
HF-->>DM: resumable stream into _incoming
loop until download completes
DM-->>U: SSE progress (bytes, speed)
end
DM->>DM: verify sizes and OIDs
DM->>ST: place files per scope (repo folder or _shared) via temp+rename
DM->>ST: dedupe by OID (hardlink same volume, copy cross volume)
DM->>DB: insert file rows, repo (status=complete)
alt cache requested
JS->>CQ: enqueue promote
CQ->>ST: copy repo folder to SSD .tmp
CQ->>ST: rename .tmp to final
CQ->>DB: is_cached = 1
end
DM--)U: SSE progress
```
### 2.3 Promote / Demote — as v1.1, unit = repo folder; demote gated on HDD validation.
### 2.3 Promote / Demote
### 2.4 Update Check — as v1.4 flow, per repo; model page aggregates repo results.
```mermaid
sequenceDiagram
autonumber
participant U as User
participant API as FastAPI
participant JS as Job Scheduler
participant CQ as Copy Queue
participant ST as Storage Mgr
participant DB as SQLite
U->>API: POST /api/repos/123/cache (promote or demote)
API->>DB: create job row (queued)
API->>JS: enqueue on copy queue
API-->>U: accepted (job id)
alt promote
JS->>CQ: run promote
CQ->>ST: free-space check on SSD
alt insufficient space
CQ->>DB: job (failed)
CQ-->>U: SSE error (insufficient SSD space)
else space ok
CQ->>ST: copy repo folder to SSD .tmp
CQ->>ST: rename .tmp to final
CQ->>DB: is_cached = 1, set cache_path
CQ-->>U: SSE completion
end
else demote
JS->>CQ: run demote
CQ->>ST: validate HDD copy (files present, sizes match DB)
alt HDD invalid
CQ->>DB: job (failed, demote blocked)
CQ-->>U: SSE error (HDD copy missing or incomplete)
else HDD valid
CQ->>ST: delete SSD folder
CQ->>DB: is_cached = 0, cache_path = null
CQ-->>U: SSE completion
end
end
```
### 2.4 Update Check
```mermaid
sequenceDiagram
autonumber
participant U as User
participant API as FastAPI
participant HF as Hugging Face
participant DB as SQLite
U->>API: POST /api/repos/123/check-update
API->>DB: load stored revision (commit hash)
API->>HF: repo info (current sha)
HF-->>API: current commit
API->>API: compare commits
alt same commit
API-->>U: up to date
else commit differs
API->>HF: model_info(files_metadata=true)
HF-->>API: current files, sizes, OIDs
API->>DB: diff per-file OIDs against stored
API-->>U: list added, changed, removed
U->>API: POST /api/repos/123/apply-update (selected files)
API->>DB: create update_apply job
API-->>U: accepted (job id)
Note over API,DB: apply job downloads changed files to _incoming, swaps per file via temp+rename (scope-aware), updates rows and revision
end
Note over U,DB: Models page aggregates update state across all repos of a model
```
---