mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2026-09-06 21:11:24 +02:00
Max Krasnyansky
192067b72d
* hexagon: use non-host bufs by default and make the backend fully async * hex-hb: remove optional hostbuf support and fix async copy * hex-unary: relax supported unary check * hex-bufs: use same get_alignment for host bufs * snapdragon: bump android_platform to 34 * hex-rows: super hacky get/set rows for q8_0 * hex-get-rows: fix q8_0 * hex-get-rows: supprot for f16 and cleanup for q8_0 * hex-get-rows: generic macros and specialized thread funcs * hex-get-rows: add DMA pipeline, vtcm_layout and kernel params * hex-set-rows: fix q8_0 support, add dma and tracing * hex-tests: override nmse threshold for HTP of Q8_0 quants * hex-fa: add support for Q8_0 with inplace dequantizers * hex-get-rows: simplify type dispatch * hex-rows: simplify GET/SET_ROWS DMA pipeline * hex-async: add events, set/get-tensor-async and rest of the async api support * hex-repack: use slice instead of expert in repack functions * hex-cpy: update event/async-cpy logging * hex-set-rows: optimize smaller tensors * hex-geglu: fix perf regression with larger tensors * hex-get-rows: add missing header * hex-set-rows: add missing header * hex-bufs: ressurect GGML_HEXAGON_HOSTBUF but disable it by default * hexagon: do not reject ops with non-heaxon buffers * hex-get-rows: apply >=32 restriction only for q8_0 * hex-res: bump vtcm acquire timeout to 10 seconds * hex-bufs: add support for cloning buffers between sessions to speed up tensor copies * hex-async: rework event recording and batch flushing and integrate with meta backend * hex-bufs: improved handling of repacked tensors * hex-repack: handle get_tensor_2d offsets * hex-dev: add support for devices with multiple NPUs * hex-sync: add support for sync tokens to synchronize npu devices for async splits * hex-mmap: cleanup mmap calls and add a retry for robustness * hex-sync: add failsafe if sync wait gets stuck * hex-sync: use sync_seq to check for completed events * hex-sync: rotate tokens for extra robustness * hex-devs: add supprot for legacy device names for now * hex-bufs: add support for auto-cloning buffers from diff sessions * hex-fusion: simplify and optimize htp-opnode fusion handling * hex-sync: override opnode name so that it shows up in the profiles * hex-trace: update scripts to handle multiple devices * hex-sync: bump the size of the opbatch queue and number of sync tokens * hex-cpy-sync: do not explicitly flush opbatches in cpy_tensor_async and add support for cpy-dma * hex-sync: add graph-flush threshold to avoid single op batches * hex-sync: add sync_peer so that we can flush peers we depend on during cross-device ops * hex-bufs: introduce tensor->extra and shadow_bufs for repacking * hex-l2: flush tiny tensors inline * hex-sync: use explicit l2flush for sync tokens * hex-extra: track weight flags via tensor extra * hex-fence: rename sync to fence * hex-repack: proper handling of set-tensor-2d in the shadow_buf * hex-trace: remove obsolete opstage mask that we used for profiling * hex-env: remove obsolete use_hmx variable * hexagon: new unified run.py and build.py and updated docs * snapdragon: update run script to auto-escapt test-backend-op -p argument * hex-scripts: fix trailing spaces * hex-scripts: fix flake8 warnings * snapdragon: cleanup dst lib/bin dirs before copying new build * hex-ops: add support for allreduce * hex-ar: improved allreduce with dma pipeline * hex-ar: align macros * hex-ar: consistent use of fence_seq * hex-ar: add AR_SELECT env var to select ALLREDUCE kernel or fallback * hex-ar: add proper synchronize handling for ALLREDUCE * hex-opbatch: looks like we now just rely on backend.synchronise to flush the batches, no need to flush them by threshold * hex-ar: bump block size to improve dma efficiency * hex-ar: fused ALLREDUCE+ADD * hex-ar: cleaner fence buffer management * hex-ar: futher allreduce tweaking to remove race conditions * hex-ar: add simple solver and remove non-dma kernels * hex-ar: add row-broadcast to fuse with bias ADD * hex-fence: pass seq numbers via op_params * hex-ar: allow for both entry/exit seq for completing entry wait * hex-ar: align macros * hex-ar: do not refetch broadcast row * hex-fusion: move all fusion into opbatch::add_op for consistency with ALLREDUCE and things * hex-fusion: fix incorrect MUL_MAT reordering * hex-mm: make fused 2x and 3x matmuls more generic * hex-fusion: move tensor fusion tagging to graph_compute * hexagon: make sure to copy tensor->extra by value * hex-get-rows: fix offset calc with row-chunking * hex-repack: get_tensor_2d fixes for non-zero offsets * snapdragon: make profile/trace scripts more robust and donot mix stdout/stderr by default * hex-devices: use legacy device nameing by default to ease the transition * hex-devices: hardcode CDSP domain IDs for current devices for now * hex-optrace: improve multi-NPU timestamp alignment and overall handling of cycle values * hex-optrace: more robust handling of the fence events
178 lines
6.0 KiB
Markdown
178 lines
6.0 KiB
Markdown
# Snapdragon-based Windows devices
|
|
|
|
## Tool Dependencies
|
|
|
|
Native Windows 11 arm64 builds have the following tool dependencies:
|
|
- MS Visual Studio 2026 (Community Edition or Pro)
|
|
- MSVC arm64 standard and runtime libraries
|
|
- UCRT and Driver Kit
|
|
- LLVM core libraries and Clang compiler (winget)
|
|
- CMake, Git, Python (winget)
|
|
- Hexagon SDK Community Edition 6.6 or later (see below)
|
|
- OpenCL SDK 2.3 or later (see below)
|
|
|
|
Note: The rest of the **Windows** build process assumes that you're running natively in Powershell.
|
|
|
|
## Overview
|
|
|
|
The document covers procedures for installing the latest GPU and NPU drivers, and OpenCL and Hexagon SDKs.
|
|
|
|
|
|
In order to use Hexagon NPU on Snapdragon Windows devices the underlying HTP Ops libraries (e.g libggml-htp-v73.so)
|
|
must be included in the .cat file digitally signed with a trusted certificate.
|
|
|
|
This document covers details on how to generate personal certificate files (.pfx) and how to configure the system
|
|
to allow for test signatures (aka test-signing).
|
|
|
|
## Install the latest Adreno OpenCL SDK
|
|
|
|
Either use the trimmed down version (optimized for CI) from
|
|
|
|
https://github.com/snapdragon-toolchain/opencl-sdk/releases/download/v2.3.2/adreno-opencl-sdk-v2.3.2-arm64-wos.tar.xz
|
|
|
|
Or download the complete official version from
|
|
|
|
https://softwarecenter.qualcomm.com/catalog/item/Adreno_OpenCL_SDK?version=2.3.2
|
|
|
|
Unzip/untar the archive into
|
|
```
|
|
c:\Qualcomm\OpenCL_SDK\2.3.2
|
|
```
|
|
|
|
## Install the latest Hexagon SDK Community Edition
|
|
|
|
Either use the trimmed down version (optimized for CI) from
|
|
|
|
https://github.com/snapdragon-toolchain/hexagon-sdk/releases/download/v6.6.0.0/hexagon-sdk-v6.6.0.0-arm64-wos.tar.xz
|
|
|
|
Or download the complete official version from
|
|
|
|
https://softwarecenter.qualcomm.com/catalog/item/Hexagon_SDK?version=6.6.0.0
|
|
|
|
Unzip/untar the archive into
|
|
```
|
|
c:\Qualcomm\Hexagon_SDK\6.6.0.0
|
|
```
|
|
|
|
## Install the latest Adreno GPU driver
|
|
|
|
Download the driver from
|
|
|
|
https://softwarecenter.qualcomm.com/catalog/item/Windows_Graphics_Driver
|
|
|
|
After the automated installation and reboot please make sure that the GPU device shows up in the `Device Manager` (under 'Display Adapters`)
|
|
|
|
## Install the latest Qualcomm NPU driver
|
|
|
|
Download the driver from
|
|
|
|
https://softwarecenter.qualcomm.com/catalog/item/Qualcomm_HND
|
|
|
|
After the automated installation and reboot please make sure that the Hexagon NPU device shows up in the `Device Manager`
|
|
(under `Neural Processors`).
|
|
|
|
If the device is not available you can try installing all components (`qcnspmcdm8380`, `qcnspmcdm8380_ext`) manually.
|
|
The components are extracted into
|
|
```
|
|
c:\QCDrivers\qcnspmcdm...
|
|
```
|
|
|
|
## Enable NPU driver test signatures
|
|
|
|
Please note that the following steps are required only for the Hexagon NPU.
|
|
Adreno GPU backend does not require test signatures.
|
|
|
|
### Enable testsigning
|
|
|
|
Use `bcdedit` to enable test-signing
|
|
```
|
|
> bcdedit /set TESTSIGNING ON
|
|
```
|
|
(Secure Boot may need to be disabled for this to work)
|
|
|
|
Make sure test-signing is enabled after reboot
|
|
```
|
|
> bcdedit /enum
|
|
...
|
|
testsigning Yes
|
|
...
|
|
```
|
|
For additional details see Microsoft guide at
|
|
|
|
https://learn.microsoft.com/en-us/windows-hardware/drivers/install/the-testsigning-boot-configuration-option
|
|
|
|
### Create personal certificate
|
|
|
|
The tools required for this procedure are available as part of Windows SDK and Windows Driver Kit which should be
|
|
installed as part of the MS Visual Studio.
|
|
They are typically located at
|
|
```
|
|
c:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0
|
|
```
|
|
(replace 10.0.26100.0 with correct version).
|
|
|
|
To create personal self-signed certificate run the following commands (either from cmd or power-shell):
|
|
```
|
|
> cd c:\Users\MyUser
|
|
> mkdir Certs
|
|
> cd Certs
|
|
> makecert -r -pe -ss PrivateCertStore -n CN=GGML.HTP.v1 -eku 1.3.6.1.5.5.7.3.3 -sv ggml-htp-v1.pvk ggml-htp-v1.cer
|
|
> pvk2pfx.exe -pvk ggml-htp-v1.pvk -spc ggml-htp-v1.cer -pfx ggml-htp-v1.pfx
|
|
```
|
|
(replace `MyUser` with your username).
|
|
|
|
Add this certificate to `Trusted Root Certification Authorities` and `Trusted Publishers` stores.
|
|
This can be done using `certlm` Certificate Manager tool.
|
|
Right click on the certificate store, select `All Tasks -> Import` and follow the prompts to import the certificate from the
|
|
PFX file you created above.
|
|
|
|
For additional details see Microsoft guide at
|
|
|
|
https://learn.microsoft.com/en-us/windows-hardware/drivers/install/introduction-to-test-signing
|
|
|
|
Make sure to save the PFX file, you will need it for the build procedures.
|
|
Please note that the same certificate can be used for signing any number of builds.
|
|
|
|
## Build Hexagon backend with signed HTP ops libraries
|
|
|
|
The overall Hexagon backend build procedure for Windows on Snapdragon is the same as for other platforms.
|
|
However, additional settings are required for generating and signing HTP Ops libraries.
|
|
```
|
|
> $env:OPENCL_SDK_ROOT="C:\Qualcomm\OpenCL_SDK\2.3.2"
|
|
> $env:HEXAGON_SDK_ROOT="C:\Qualcomm\Hexagon_SDK\6.6.0.0"
|
|
> $env:HEXAGON_TOOLS_ROOT="C:\Qualcomm\Hexagon_SDK\6.6.0.0\tools\HEXAGON_Tools\19.0.07"
|
|
> $env:HEXAGON_HTP_CERT="c:\Users\MyUsers\Certs\ggml-htp-v1.pfx"
|
|
> $env:WINDOWS_SDK_BIN="C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0"
|
|
|
|
> cmake --preset arm64-windows-snapdragon-release -B build-wos
|
|
...
|
|
> cmake --install build-wos --prefix pkg-wos
|
|
```
|
|
|
|
Once the build is complete HTP ops libraries will be installed like this
|
|
```
|
|
> dir pkg-wos/lib
|
|
...
|
|
-a---- 1/22/2026 6:01 PM 187656 libggml-htp-v73.so
|
|
-a---- 1/22/2026 6:01 PM 191752 libggml-htp-v75.so
|
|
-a---- 1/22/2026 6:01 PM 187656 libggml-htp-v79.so
|
|
-a---- 1/22/2026 6:01 PM 187656 libggml-htp-v81.so
|
|
-a---- 1/22/2026 6:01 PM 4139 libggml-htp.cat
|
|
```
|
|
|
|
The .cat file, the signature and proper certificate installation can be verified with
|
|
|
|
```
|
|
> signtool.exe verify /v /pa .\pkg-wos\lib\libggml-htp.cat
|
|
Verifying: .\pkg-wos\lib\libggml-htp.cat
|
|
|
|
Signature Index: 0 (Primary Signature)
|
|
Hash of file (sha256): 9820C664DA59D5EAE31DBB664127FCDAEF59CDC31502496BC567544EC2F401CF
|
|
|
|
Signing Certificate Chain:
|
|
Issued to: GGML.HTP.v1
|
|
...
|
|
Successfully verified: .\pkg-wos\lib\libggml-htp.cat
|
|
...
|
|
```
|