Vulkan k-quant mmq and ggml-backend offload functionality (#6155)

Browse Source

* Fix Vulkan no kv offload incoherence

* Add k-quant mul mat mat shaders

* Rework working buffer allocation, reduces vram use noticeably

Clean up cpu assist code, replaced with ggml-backend offload function

* Default to all dedicated GPUs

* Add fallback for integrated GPUs if no dedicated GPUs are found

* Add debug info which device is allocating memory

* Fix Intel dequant issue

Fix validation issue

* Fix Vulkan GGML_OP_GET_ROWS implementation

* Clean up merge artifacts

* Remove Vulkan warning

...

This commit is contained in:

0cc4m

2024-03-29 17:29:21 +01:00

committed by GitHub

parent d48ccf3ad4

commit ba0c7c70ab

7 changed files with 25616 additions and 3192 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

ggml-vulkan.cpp

+327 -306

File diff suppressed because it is too large Load Diff