392c969754
add offload state_dict
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-08-17 08:52:46 +02:00
76941efc7e
strict typechecking
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-08-16 12:45:50 +02:00
12bf006b00
improve state management
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-08-15 19:21:51 +02:00
0ac89385a0
timers reporting
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-08-14 09:18:24 +02:00
5123bfd96b
cleanup minimax
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-08-11 15:42:29 +02:00
c1741bc7f0
cleanup
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-08-11 11:27:50 +02:00
ed6eb2ea0a
Merge pull request #5016 from vladmandic/feat/offload-engine
...
feat(offload): per-component group offload engine and memory observability
2026-08-11 11:26:56 +02:00
0608bdfbb1
log torch.accelerator
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-08-11 11:26:27 +02:00
5a66d9eabe
fix(offload): reclaim model memory across switches
...
Model switches kept most of the previous model resident, and the next
load could stall in kernel reclaim while the freed memory was still held.
- strip group offload hooks in disable_offload so the meta move at unload
actually frees component weights; hook removal resolves wrapper
components that carry hooks on the inner model
- flush the torch pinned host cache in torch_gc so freed streaming
buffers return to the OS instead of staying cached in-process
- skip the pipe-level accelerator move for group-managed pipes: the
offload engine already placed every component, and the move only
dragged on-demand components to the accelerator for the trailing
eviction to undo
2026-08-10 22:10:42 +01:00
6aa5b45b80
remove directml
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-08-10 13:43:45 +02:00
2250d8052a
remove sdnq_dequantize_compile option
...
- advanced users should use the env var SDNQ_USE_TORCH_COMPILE instead
2026-08-10 13:30:56 +03:00
a83795273e
prompt enhance and caption use llm-context
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-08-04 14:21:53 +02:00
55ba386941
torch reset dynamo cache on model reload
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-08-04 10:58:32 +02:00
7214ee9d42
triton/dynamo/inductor cache location and timer stats
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-07-15 15:06:40 +02:00
42c2c6382a
update torch==2.13.0+cu132
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-07-15 09:00:36 +02:00
dfa713e6fc
Add SDNQ attention
2026-06-23 08:10:02 +03:00
da3907a17c
Set has_triton to True for CPU backend
2026-06-21 14:25:16 +03:00
2b81eadecf
add force dtype on load
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-06-20 09:38:50 +02:00
5e99dee3c2
full codespell coverage
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-06-04 12:36:10 +02:00
3c2a742f40
cleanup logging
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-06-02 08:03:27 +02:00
d2d9f7caea
captioning improvements and cleanup
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2026-05-29 21:10:56 +02:00
8070e51777
openvino backend
...
Signed-off-by: vladmandic <mandic00@live.com >
2026-04-12 19:07:20 +02:00
d1a40b3ba7
openvino device selecrtion
...
Signed-off-by: vladmandic <mandic00@live.com >
2026-04-12 19:07:20 +02:00
49b0fa360f
error handler for same-device
...
Signed-off-by: vladmandic <mandic00@live.com >
2026-04-12 19:06:09 +02:00
481d974b91
RUF013 update
2026-03-24 03:40:15 -07:00
cb943919eb
fix(devices): triton version detection always reporting false
...
triton_version was initialized to False instead of None, causing
the `is None` checks to be skipped and torch_info to report triton=false
even when triton is functional
2026-03-18 20:14:02 +00:00
fb2f9ea650
new server info panel
...
Signed-off-by: vladmandic <mandic00@live.com >
2026-03-18 15:37:57 +01:00
feffbe0851
Update subprocess.run usage
2026-03-05 20:06:09 -08:00
bfe014f5da
modernize typing
2026-02-19 09:15:37 +01:00
e9eadd06db
remove dependency on pkg_resources and refactor cmd_args
2026-02-19 08:30:40 +01:00
139e331d80
style(caption): fix lint warnings across caption module
...
- Rename shadowing import in waifudiffusion batch to avoid F823/E0606
- Fix import order in cli/api-caption.py (stdlib before third-party)
- Rename local variable shadowing function name in cli/api-caption.py
- Remove unnecessary global statement in devices.bypass_sdpa_hijacks
2026-02-11 02:50:06 +00:00
17b03ed8e4
feat(caption): add Florence detection parsing, SDPA bypass, and offload support
...
- Add parse_florence_detections() and format_florence_response() to
vqa_detection for handling Florence-2 detection output formats
- Add bypass_sdpa_hijacks() context manager to devices.py for models
incompatible with SageAttention or other SDPA hijacks
- Add OpenCLIP model offload support when caption_offload is enabled
2026-02-11 02:48:11 +00:00
32b8b082e2
cleanup logging
...
Signed-off-by: vladmandic <mandic00@live.com >
2026-01-16 10:36:02 +01:00
85332594fc
triton test reduce verbosity
...
Signed-off-by: vladmandic <mandic00@live.com >
2026-01-10 10:32:13 +01:00
49965dfda8
get_hip_arch_name -> get_hip_agent, use amdhip64_7.dll served within rocm package
2026-01-03 21:00:36 +09:00
b9c18452f2
unify hip get arch name
...
Signed-off-by: vladmandic <mandic00@live.com >
2026-01-03 08:22:19 +01:00
0b1e6d2d3c
improve offloading
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2025-12-25 10:24:02 +00:00
e2fb70d4a1
detailer draw segmentation overlays
...
Signed-off-by: vladmandic <mandic00@live.com >
2025-12-17 10:03:17 +01:00
3a3c984411
Merge pull request #4388 from vladmandic/kanvas
...
merge kanvas to dev
2025-11-09 07:57:54 -05:00
f4ee9c7052
Add Flex attention
2025-11-09 00:14:38 +03:00
f491955991
Merge pull request #4383 from vladmandic/dev
...
refresh branch
2025-11-08 15:43:28 -05:00
69180202d3
kanvas integration
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2025-11-08 15:41:52 -05:00
2bbbb684cc
Rename CK Flash attention to just Flash attention
2025-11-08 23:24:40 +03:00
a93715e0da
Don't expose AMD Triton Flash Atten for non AMD
2025-11-08 23:20:55 +03:00
56026c4e61
refactor attention handling
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2025-11-08 10:55:41 -05:00
155ee7f84c
fix sage-attention checks on sm86
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2025-11-08 08:47:21 -05:00
5ffbca9377
cleanup and update changelog
...
Signed-off-by: Vladimir Mandic <mandic00@live.com >
2025-11-05 12:57:30 -05:00
bdc477d252
Refactor GPU backend selection for sage attention
...
Removed hot path, now everything is defined at setup
Also, passing device to get_device_capability so that it works properly with multi-gpu setups.
2025-11-05 16:31:13 +00:00
4c791fb795
Remove model check logic for SA2 workaround
2025-11-05 10:53:07 +00:00
18676996d0
Sage Attention 2 + Triton workaround Qwen-Image
...
Workaround to prevent black images generated with Qwen-Image models when Sage Attention 2 is enabled with Triton as backend on devices with compute capability 8.0 and 8.6.
Simply switches back to Cuda backend for these models only.
Proof of concept, feel free to close if this is not appropriate.
2025-11-04 23:31:14 +00:00
Vladimir Mandic
CalamitousFelicitousness
Dity0
awsr
Seunghoon Lee