fix(caption): safetensors-only downloads, model load fixes, UI default, prefill tests

- Add use_safetensors=True to all 16 model from_pretrained calls to
  avoid downloading redundant .bin files alongside safetensors
- Add device property to JoyTag VisionModel so move_model can relocate
  it to CUDA (fixes 'ViT object has no attribute device')
- Fix Pix2Struct dtype mismatch by casting float inputs to model dtype
  while preserving integer tensor types
- Patch AutoConfig.register with exist_ok=True during Ovis loading to
  handle duplicate aimv2 registration on model reload
- Detect Qwen VL fine-tune architecture from config model_type instead
  of repo name, fixing ToriiGate and similar third-party fine-tunes
- Change UI default task from Short Caption to Normal Caption, and
  preserve it on model switch instead of resetting to Use Prompt
- Add dual-prefill testing across 5 VQA test methods using a shared
  _check_prefill helper
- Fix pre-existing ruff W605 in strip_think_xml_tags docstring
This commit is contained in:
CalamitousFelicitousness
2026-01-29 04:21:01 +00:00
parent 57659ab642
commit e2cdbe47fa
7 changed files with 88 additions and 14 deletions
+3 -2
View File
@@ -3,7 +3,7 @@ from modules import shared, ui_common, generation_parameters_copypaste
from modules.caption import openclip
default_task = "Short Caption"
default_task = "Normal Caption"
def vlm_caption_wrapper(question, system_prompt, prompt, image, model_name, prefill, thinking_mode):
"""Wrapper for vqa.caption that handles annotated image display."""
@@ -19,7 +19,8 @@ def update_vlm_prompts_for_model(model_name):
"""Update the task dropdown choices based on selected model."""
from modules.caption import vqa
prompts = vqa.get_prompts_for_model(model_name)
return gr.update(choices=prompts, value=prompts[0] if prompts else default_task)
value = default_task if default_task in prompts else (prompts[0] if prompts else default_task)
return gr.update(choices=prompts, value=value)
def update_vlm_prompt_placeholder(question):