Doc: Updated Design Doc
This commit is contained in:
+96
-19
@@ -208,7 +208,7 @@ Carried from v1.1 (large copies → queue; space pre-checks; capped pools; token
|
||||
|
||||
## 2. Architecture Diagrams
|
||||
|
||||
### 2.0 Entity Relationships (new)
|
||||
### 2.0 Entity Relationships
|
||||
```mermaid
|
||||
erDiagram
|
||||
FAMILIES ||--o{ MODELS : contains
|
||||
@@ -221,43 +221,120 @@ erDiagram
|
||||
MODEL_FILES }o--|| REPOS : "scope folder may live under _shared/"
|
||||
```
|
||||
|
||||
### 2.1 Components — unchanged from v1.1 (Storage Manager now hierarchy/path-scope aware).
|
||||
### 2.1 Components
|
||||
|
||||
### 2.2 Download Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant U as User
|
||||
participant API as FastAPI
|
||||
participant JS as Job Scheduler
|
||||
participant DM as Download Mgr
|
||||
participant HF as Hugging Face
|
||||
participant ST as Storage
|
||||
participant CQ as Copy Queue
|
||||
participant ST as Storage Mgr
|
||||
participant DB as SQLite
|
||||
|
||||
U->>API: GET /api/repo/preview (url or org/name)
|
||||
API->>HF: model_info(files_metadata=true)
|
||||
HF-->>API: files + sizes + LFS OIDs
|
||||
API->>DB: family OID lookup + suggest family/model
|
||||
HF-->>API: files, sizes, LFS sha256 OIDs
|
||||
API->>DB: family OID lookup, suggest family/model
|
||||
API-->>U: grouped list (role, variant, scope, dedupe flags)
|
||||
U->>API: POST /api/repos (family/model targets, files, roles, scopes, variants, root, cache?)
|
||||
API->>DB: ensure family/model rows; repo(status=downloading) + job row
|
||||
U->>API: POST /api/repos (family, model, files, roles, scopes, variants, root, cache)
|
||||
API->>DB: ensure family/model rows
|
||||
API->>DB: create repo (status=downloading) and job row
|
||||
API-->>U: job id
|
||||
JS->>DM: run
|
||||
DM->>HF: snapshot_download(local_dir=_incoming/{repo_id}, allow_patterns, revision, token)
|
||||
HF-->>DM: resumable stream
|
||||
DM->>ST: place files per scope (repo folder / _shared/{model} / _shared) via temp+rename; dedupe by OID (link/copy)
|
||||
DM->>DB: files rows + repo(status=complete)
|
||||
alt cache requested
|
||||
JS->>ST: enqueue promote (copy queue)
|
||||
ST->>ST: copy → SSD .tmp → rename
|
||||
ST->>DB: is_cached=1
|
||||
JS->>DM: run download job
|
||||
DM->>HF: snapshot_download(local_dir=_incoming, allow_patterns, revision, token)
|
||||
HF-->>DM: resumable stream into _incoming
|
||||
loop until download completes
|
||||
DM-->>U: SSE progress (bytes, speed)
|
||||
end
|
||||
DM->>DM: verify sizes and OIDs
|
||||
DM->>ST: place files per scope (repo folder or _shared) via temp+rename
|
||||
DM->>ST: dedupe by OID (hardlink same volume, copy cross volume)
|
||||
DM->>DB: insert file rows, repo (status=complete)
|
||||
alt cache requested
|
||||
JS->>CQ: enqueue promote
|
||||
CQ->>ST: copy repo folder to SSD .tmp
|
||||
CQ->>ST: rename .tmp to final
|
||||
CQ->>DB: is_cached = 1
|
||||
end
|
||||
DM--)U: SSE progress
|
||||
```
|
||||
|
||||
### 2.3 Promote / Demote — as v1.1, unit = repo folder; demote gated on HDD validation.
|
||||
### 2.3 Promote / Demote
|
||||
|
||||
### 2.4 Update Check — as v1.4 flow, per repo; model page aggregates repo results.
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant U as User
|
||||
participant API as FastAPI
|
||||
participant JS as Job Scheduler
|
||||
participant CQ as Copy Queue
|
||||
participant ST as Storage Mgr
|
||||
participant DB as SQLite
|
||||
|
||||
U->>API: POST /api/repos/123/cache (promote or demote)
|
||||
API->>DB: create job row (queued)
|
||||
API->>JS: enqueue on copy queue
|
||||
API-->>U: accepted (job id)
|
||||
alt promote
|
||||
JS->>CQ: run promote
|
||||
CQ->>ST: free-space check on SSD
|
||||
alt insufficient space
|
||||
CQ->>DB: job (failed)
|
||||
CQ-->>U: SSE error (insufficient SSD space)
|
||||
else space ok
|
||||
CQ->>ST: copy repo folder to SSD .tmp
|
||||
CQ->>ST: rename .tmp to final
|
||||
CQ->>DB: is_cached = 1, set cache_path
|
||||
CQ-->>U: SSE completion
|
||||
end
|
||||
else demote
|
||||
JS->>CQ: run demote
|
||||
CQ->>ST: validate HDD copy (files present, sizes match DB)
|
||||
alt HDD invalid
|
||||
CQ->>DB: job (failed, demote blocked)
|
||||
CQ-->>U: SSE error (HDD copy missing or incomplete)
|
||||
else HDD valid
|
||||
CQ->>ST: delete SSD folder
|
||||
CQ->>DB: is_cached = 0, cache_path = null
|
||||
CQ-->>U: SSE completion
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### 2.4 Update Check
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant U as User
|
||||
participant API as FastAPI
|
||||
participant HF as Hugging Face
|
||||
participant DB as SQLite
|
||||
|
||||
U->>API: POST /api/repos/123/check-update
|
||||
API->>DB: load stored revision (commit hash)
|
||||
API->>HF: repo info (current sha)
|
||||
HF-->>API: current commit
|
||||
API->>API: compare commits
|
||||
alt same commit
|
||||
API-->>U: up to date
|
||||
else commit differs
|
||||
API->>HF: model_info(files_metadata=true)
|
||||
HF-->>API: current files, sizes, OIDs
|
||||
API->>DB: diff per-file OIDs against stored
|
||||
API-->>U: list added, changed, removed
|
||||
U->>API: POST /api/repos/123/apply-update (selected files)
|
||||
API->>DB: create update_apply job
|
||||
API-->>U: accepted (job id)
|
||||
Note over API,DB: apply job downloads changed files to _incoming, swaps per file via temp+rename (scope-aware), updates rows and revision
|
||||
end
|
||||
Note over U,DB: Models page aggregates update state across all repos of a model
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user