llama.cpp/docs/backend/snapdragon/developer.md

Hexagon backend developer details

Backend libraries

The Hexagon backend consists of two parts:

• libggml-hexagon This is the regular CPU-side GGML backend library, either shared or statically linked.

• libggml-htp-vNN This is the NPU-side (HTP stands for Hexagon Tensor Processor) shared library that contains the Op dispatcher and kernels. The correct library is selected automatically at runtime based on the HW version.

Here is an example of the build artifacts:

~/src/llama.cpp$ ls -l pkg-adb/llama.cpp/lib/libggml*
pkg-adb/llama.cpp/lib/libggml-base.so
pkg-adb/llama.cpp/lib/libggml-cpu.so
pkg-adb/llama.cpp/lib/libggml-hexagon.so      <<< CPU library
pkg-adb/llama.cpp/lib/libggml-htp-v73.so      <<< HTP op/kernels for Hexagon v73
pkg-adb/llama.cpp/lib/libggml-htp-v75.so
pkg-adb/llama.cpp/lib/libggml-htp-v79.so
pkg-adb/llama.cpp/lib/libggml-htp-v81.so

Memory buffers

The Hexagon NPU backend takes advantage of Snapdragon unified memory where all DDR buffers are accessible by CPU, GPU, and NPU. The NPU has dedicated tightly-coupled memory called VTCM (Vector Tightly-Coupled Memory). VTCM is used for intermediate data (such as dynamically quantized activations) and streaming buffers (chunks of weight and activation tensors fetched via DMA).

Large model handling

Hexagon NPU sessions have a 32-bit virtual address space window of around 3.5GB. In llama.cpp/GGML, each Hexagon session is mapped to a single GGML backend device (e.g., HTP0:0, HTP0:1, etc. when using GGML_HEXAGON_DEVICES, or HTP0, HTP1 in legacy mode).

To support running models larger than 3.5GB on a single device, the Hexagon backend dynamically maps and unmaps buffers:

• Buffers are allocated in shared DDR (RPCMEM) via file descriptors (fastrpc_mmap using FASTRPC_MAP_FD_DELAYED).
• Pinned buffers (such as KV cache and active compute buffers) remain mapped throughout execution.
• Inactive weight buffers are dynamically mapped into the NPU session via HAP_mmap() during batch buffer preparation (prep_op_bufs() in htp/main.c) and unmapped via htp_iface_munmap() when no longer needed by the active batch.
• This dynamic sliding window allows a single NPU session to execute models that exceed the 3.5GB window.

Alternatively, users can partition and split the model across multiple virtual sessions or physical NPUs using layer-splitting, tensor-splitting, or row-splitting modes. For user-facing execution modes and examples, see the Snapdragon user guide.

Op and Kernel Development Guidelines

Writing high-performance operators for Hexagon requires following specific guidelines.

DDR -> DMA -> VTCM Execution Pipeline

• Strongly prefer the DDR -> DMA -> VTCM -> compute (HVX/HMX) -> VTCM -> DMA -> DDR data flow.

• Direct HVX reads/writes from/to DDR are less efficient and should only be used as a fallback.

• The DMA queue is a strict FIFO where operations must be pushed and popped in strict order.

• Follow the pipelined multi-buffering sequence properly (typically 2x to 16x buffering) so every push has a corresponding pop:

  1. In the prologue, push initial DDR -> VTCM transfers to prime the pipeline.
  2. In the loop body, wait for buffer N via DMA pop, launch HVX/HMX compute on buffer N, push VTCM -> DDR writeback of result N, and push DDR -> VTCM prefetch of buffer N+2.
  3. In the epilogue, pop all remaining in-flight transfers to drain the pipeline.

• Because every push must be matched by a pop, dma_queue_flush() is not required when the pipeline sequence is followed properly. Flushing is only used in rare exceptions where a batch of operations is pushed without individual pops.

• Use the DMA queue interface from dma-queue.h (dma_queue_push_ddr_to_vtcm, dma_queue_pop, dma_queue_push_vtcm_to_ddr). See cumsum-ops.c and act-ops.c for reference implementations.

Avoid Scalar Reads and Writes to VTCM

• Access VTCM data using DMA transfers or HVX/HMX vector instructions rather than scalar reads and writes.

Avoid Scalar Division in Inner Loops

• Hexagon cores do not have hardware division instructions.
• For recurring divisions across iterations or threads, use fastdiv from hex-fastdiv.h with precomputed divisors (such as octx->ctx->mdev.count_div or octx->n_threads_div).
• Do not call init_fastdiv_values() for single-use divisions; use standard compiler division (/) instead.

Host-Side Precomputation via kernel_params

• Precompute tensor shapes, strides, scale conversions, tiling layouts, and validation checks on the host CPU during graph preparation in ggml-hexagon.cpp.
• Pack precomputed parameters into the operator's fixed kernel_params structure in htp_op_node (such as htp_mm_kernel_params, htp_unary_kernel_params, htp_fa_kernel_params, htp_get_rows_kernel_params).
• The NPU executes directly using octx->kernel_params without redundant runtime metadata extraction or validation.
• Strict Host-Kernel Alignment:
  • Verify that parameters calculated by the host CPU are strictly honored by the NPU kernel.
  • Ensure the kernel does not ignore host-computed fields (for example, falling back to octx->n_threads instead of using kparams->n_threads, or ignoring precomputed tasks_per_thread and chunk counts).
  • Both human developers and coding agents must audit both sides of the interface: ensure fields populated in kernel_params in ggml-hexagon.cpp are actively and consistently utilized by the corresponding operator entry point and worker threads in htp/*-ops.c.

Tracing Instrumentation

• All kernels must include trace events for performance profiling and timeline visualization in Perfetto (hex-profile.h).
• Surround compute sections with htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) info) and htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) info).
• Use specific event types for major phases:
  • HTP_TRACE_EVT_HVX_COMP: Vector compute execution.
  • HTP_TRACE_EVT_DMA: DMA transfer wait or poll cycles.
  • HTP_TRACE_EVT_FENCE: Multi-device fence barrier synchronization.
  • HTP_TRACE_EVT_L2FLUSH: L2 cache cleaning operations.
• Pass meaningful progress metrics (such as row index, chunk index, or token index) in the 16-bit info parameter.

Work Queue and Threading

• Distribute parallel work across NPU worker threads using the thread pool work queue:

  work_queue_run(ctx->work_queue, worker_func, &op_ctx, n_threads);
  

• Keep worker functions independent and re-entrant. Worker threads should only operate on their designated chunk of rows or elements.

Avoid Redundant Defensive NULL Checks

• Do not add defensive NULL checks or assertions for internal framework pointers or required graph operands and outputs. Internal pointers include ctx, octx, local context structs like *ctx, kparams, and worker callback data.
• These pointers are architectural invariants during kernel execution and host-side graph preparation. Graph compute receives allocated nodes with valid required node->src[N] and node->data pointers.
• Do not turn an invariant violation into an unsupported operation or missed fusion. Checks such as if (!octx || !octx->ctx) clutter the code, obscure intent, and hide upstream errors.
• Distinction: octx->src[N] pointers can be NULL by design and must be checked when optional. Examples include attention masks, optional bias or weights in fused kernels, and frequency factors.

Multiline Macro Formatting

• Keep trailing backslashes in multiline #define macros cleanly aligned to a consistent column.

• Avoid trailing whitespace after macro backslashes.

• Use scripts/snapdragon/ggml-hexagon-align-macros.py to inspect, diff, or automatically align macro definitions across Hexagon kernel sources:

  # Check for misaligned macros
  python3 scripts/snapdragon/ggml-hexagon-align-macros.py ggml/src/ggml-hexagon/htp/
  
  # Fix misaligned macros in-place
  python3 scripts/snapdragon/ggml-hexagon-align-macros.py --fix ggml/src/ggml-hexagon/htp/
  

Multi-Device Partitioning (mdev)

Multi-device (mdev) mode enables row-level tensor parallel execution across multiple physical NPU cores or virtual NPU sessions.

128-Byte Cache Line Alignment

• Shared tensor buffers reside in DDR (RPCMEM) with a 128-byte cache line granularity (HEX_L2_LINE_SIZE = 128 bytes, HTP_TENSOR_MDEV_LINE_SIZE).
• Rule: Multi-device work partitions must align destination write regions to 128-byte cache line boundaries so distinct devices never share or overwrite the same cache line.

Partitioning Helpers in htp-tensor.h

Common partitioning logic is factored into reusable inline helpers in htp-tensor.h:

1. htp_tensor_mdev_rows_per_chunk: Determines the minimum number of rows per chunk so that the chunk byte size is a multiple of 128 bytes:

   rows_per_chunk = 128 / hex_gcd_u32(row_size, 128)
   

   If row stride nb[1] is already a multiple of 128 bytes, rows_per_chunk = 1. Returns false if the tensor cannot be safely row-partitioned (such as unaligned base pointer, permuted layout, or non-128-byte aligned outer strides).

2. htp_tensor_mdev_partition: Calculates the per-device work range struct htp_tensor_mdev_range { uint32_t start; uint32_t count; } given total_units, units_per_chunk, mdev_idx, mdev_count, and the precomputed mdev_count_div. Handles chunk distribution across devices, assigns remainder units to the last device, and automatically triggers single-device fallback when partitioning is unsafe.

Row-Partitioned Operators

For row-wise operators (such as activations in act-ops.c, binary ops in binary-ops.c, unary ops in unary-ops.c, and sameshape copies in cpy-ops.c):

const uint32_t total_rows   = ne01 * ne02 * ne03;
const size_t   dst_row_size = dst->ne[0] * elem_size;

uint32_t row_start = 0;
uint32_t nrows     = total_rows;

if (octx->ctx->mdev.count > 1) {
    uint32_t rows_per_chunk = 0;
    htp_tensor_mdev_rows_per_chunk(dst, elem_size, (uint32_t) dst_row_size, &rows_per_chunk);
    const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(
        total_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);
    row_start = range.start;
    nrows     = range.count;
}

if (nrows == 0) {
    return HTP_STATUS_OK;
}

Element-Partitioned Operators

For flat element-wise operations (such as reshape copies in cpy-ops.c):

• Partition total linear elements N = ne0 * ne1 * ne2 * ne3 in 128-byte cache line chunks (elems_per_line = (elem_size == 4) ? 32 : 64).
• Requires strict 1D contiguity: htp_tensor_is_contiguous(dst, elem_size) and 128-byte aligned destination pointer htp_tensor_mdev_data_aligned(dst).
• If contiguous and aligned, pass elems_per_line to htp_tensor_mdev_partition; otherwise pass 0 to trigger Device 0 fallback.

Single-Device Fallback (Device 0)

• Fallback to Device 0 (mdev.idx == 0) when partitioning would cause cache line tearing or when work cannot be evenly distributed.
• Triggers:
  1. Destination tensor cannot be safely partitioned (rows_per_chunk == 0 or non-contiguous/unaligned buffer).
  2. Total aligned chunks < mdev_count.
• Device 0 processes the entire tensor [0, total_units).
• Devices 1 ... N-1 receive count = 0 and return HTP_STATUS_OK immediately.

Flatten Outer Dimensions Globally

• Never partition solely on ne01 (dimension 1).
• Partitioning only on ne01 repeats the device boundary across every 2D slice (ne02, ne03). If each 2D slice is small, false sharing occurs repeatedly throughout the tensor.
• Always flatten outer dimensions globally: total_rows = ne01 * ne02 * ne03 and partition once across the combined row space.

Stateless Starting Coordinates

• Do not use incremental state variables across slices that assume the thread or device starts at index 0.
• Precompute starting multidimensional coordinates at r = row_start (or e = elem_start) once using fastdiv.
• In inner loops, step base pointers directly (ptr += stride) or reset/wrap coordinates explicitly (if (++i01 == ne01) { ... }).

Clean Range Encapsulation

• Initialize single-device default ranges at declaration:

  uint32_t row_start = 0;
  uint32_t nrows     = total_rows;
  

• Encapsulate all multi-device logic inside if (octx->ctx->mdev.count > 1). If the block is omitted or compiled out, the operator runs standard single-device execution untouched.

• Do not propagate mdev_ prefixes to worker functions or context structs. Worker threads are device-agnostic and should only receive standard range parameters (ctx.row_start, ctx.nrows).

• In worker threads, calculate row intervals using standard arithmetic:

  const uint32_t ir0 = ctx->row_start + dr * ith;
  const uint32_t ir1 = MIN(ir0 + dr, ctx->row_start + ctx->nrows);
  

  In single-device mode (row_start == 0), this naturally simplifies to dr * ith and MIN(ir0 + dr, ctx->nrows) with zero overhead.

Multi-Device Synchronization

Multi-device execution synchronizes worker sessions across devices using explicit barriers and tensor cache flushing.

Synchronization Fence Protocol

Multi-device execution synchronizes worker sessions through atomic fence slots and barriers defined in htp-fence.h:

[NPU Session 0]                              [NPU Session 1]
       |                                            |
  (Input Prep)                                 (Input Prep)
       |                                            |
  Pre-Op Barrier ----------------------------- Pre-Op Barrier
  (mdev_sync_fence)                            (mdev_sync_fence)
       |                                            |
  Kernel Execution                             Kernel Execution
  (Output Slice 0)                             (Output Slice 1)
       |                                            |
  Tensor Cache Flush                           Tensor Cache Flush
  (htp_tensor_flush_all)                       (htp_tensor_flush_all)
       |                                            |
  Post-Op/Batch Barrier ---------------------- Post-Op/Batch Barrier
  (htp_mdev_group_barrier)                     (htp_mdev_group_barrier)
       |                                            |
  Return Response to Host                      Return Response to Host

Atomic Fence Slots and Cache Invalidation

• Fence synchronization operates on dedicated RPCMEM shared memory mapped across all participating sessions (ctx->mdev.fence_base).

• Each device owns a dedicated 128-byte cache-line aligned fence slot:

  atomic_uint * my_fence = htp_mdev_fence_slot(fence_base, mdev_idx);
  

• Writing to fence (htp_fence_write): Stores seq and status, issues a syncht thread synchronization barrier, and flushes/invalidates the line using Q6_dccleaninva_A(fence).

• Reading from peer fence (htp_fence_read): Executes Q6_dccleaninva_A(fence) and syncht before reading atomic values to ensure fresh data from DDR.

Deterministic Monotonic Sequence Numbers

• Barrier fences use monotonically increasing sequence numbers:

  const uint32_t seq = ++ctx->mdev.fence_seq;
  

• Comparing sequence numbers with signed arithmetic (int32_t)(peer_seq - seq) >= 0 prevents race conditions or misaligned barrier arrivals across iterations.

• If any peer reports an error status (peer_status > HTP_STATUS_OK), the barrier propagates the error and unblocks immediately.

Tensor Cache Flush and Pipeline Completion

• In the kernel, ensure all pushed DMA operations have been popped in strict FIFO order to drain the queue.
• Use htp_tensor_flush_all() to flush specific dirty tensors back to DDR:
  • htp_tensor_flush_all() flushes only modified tensor address ranges, ensuring peer devices and the host CPU observe consistent data in DDR.
• Never signal completion before all DMA transfers are drained and dirty tensor flushes have completed.