refactor(caption): code review fixes for offload, inference, and maintainability

Comprehensive review of modules/caption/ addressing memory management,
consistency, and code quality:

Inference correctness:
- Add devices.inference_context() to _qwen(), _smol(), _sa2() handlers
- Remove redundant @torch.no_grad() decorator from joycaption predict()
- Remove dead dtype=torch.bfloat16 kwarg from Florence loader

Memory management:
- Bound moondream3 image cache with LRU eviction (max 8 entries)
- Replace fragile id(image) cache keys with content-based md5 hash
- Add devices.torch_gc() after model loading in deepseek
- Move deepbooru model to CPU before dropping reference on unload
- Add external handler delegation to VQA.unload() (moondream3,
  joycaption, joytag, deepseek)
- Protect batch offload mutation with try/finally

Code deduplication:
- Extract strip_think_xml_tags() shared helper for Qwen/Gemma/SmolVLM
- Extract save_tags_to_file() into tagger.py from deepbooru and
  waifudiffusion

Documentation and clarity:
- Document deepseek global monkey-patches (LlamaFlashAttention2, attrdict)
- Document Florence task="task" as intentional design choice
- Add vendored-code comment to joytag.py
- Document openclip direct .to() usage vs sd_models.move_model
- Comment model.eval() calls that are required (trust_remote_code,
  custom loaders) vs removed where redundant (standard from_pretrained)

API robustness:
- Add HTTP 422 error response for VQA caption error strings in API
  endpoints (post_vqa, _dispatch_vlm)
This commit is contained in:
CalamitousFelicitousness
2026-01-29 01:28:38 +00:00
parent bf7a72f12e
commit 443a73b740
10 changed files with 205 additions and 178 deletions
+4
View File
@@ -252,6 +252,8 @@ def post_vqa(req: models.ReqVQA):
thinking_mode=req.thinking_mode,
generation_kwargs=generation_kwargs if generation_kwargs else None
)
if isinstance(answer, str) and answer.startswith('Error:'):
raise HTTPException(status_code=422, detail=answer)
# Return annotated image if requested and available
annotated_b64 = None
if req.include_annotated:
@@ -445,6 +447,8 @@ def _dispatch_vlm(req: models.ReqCaptionVLM) -> models.ResCaptionDispatch:
thinking_mode=req.thinking_mode,
generation_kwargs=generation_kwargs if generation_kwargs else None
)
if isinstance(answer, str) and answer.startswith('Error:'):
raise HTTPException(status_code=422, detail=answer)
annotated_b64 = None
if req.include_annotated:
annotated_img = vqa.get_last_annotated_image()