tests : disable DOTS3NOTE arch test for WebGPU (#27654 )

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
convert: fix GLM regression in index_tensors (#27655 )
2026-08-24 13:42:33 +02:00 · 2026-08-24 13:39:31 +03:00 · 2026-08-24 13:21:00 +03:00 · 2026-08-24 12:35:08 +03:00 · 2026-08-24 10:07:12 +02:00 · 2026-08-24 09:59:04 +02:00
27 changed files with 442 additions and 139 deletions
@@ -21,68 +21,30 @@ inputs:
 runs:
  using: "composite"
  steps:
+    - name: Install GitHub CLI if missing
+      shell: bash
+      run: |
+        # e.g. in container jobs, where it is not preinstalled
+        if ! command -v gh >/dev/null 2>&1; then
+          echo "GitHub CLI not found, installing..."
+          if ! command -v curl >/dev/null 2>&1; then
+            apt-get update >/dev/null 2>&1 || true
+            apt-get install -y curl >/dev/null 2>&1 || true
+          fi
+          mkdir -p -m 755 /etc/apt/keyrings
+          curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg >/dev/null
+          chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg
+          echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" > /etc/apt/sources.list.d/github-cli.list
+          apt-get update >/dev/null 2>&1 || true
+          apt-get install -y gh || { echo "Failed to install GitHub CLI (gh)" >&2; exit 1; }
+        fi
+        command -v gh >/dev/null 2>&1 || { echo "GitHub CLI (gh) is required but could not be installed" >&2; exit 1; }
+
    - name: Clear caches
      shell: bash
-      env:
-        CLEAR_KEY: ${{ inputs.key }}
-        CLEAR_OLDER: ${{ inputs.older }}
-        CLEAR_MIN: ${{ inputs.min }}
-        CLEAR_DRY_RUN: ${{ inputs.dry-run }}
      run: |
-        # Convert a duration (e.g. 90m, 1h, 1d, plain seconds) to seconds
-        to_seconds() {
-          local val="$1"
-          [[ "$val" =~ ^[0-9]+$ ]] && { echo "$val"; return 0; }
-          local num="${val%?}" unit="${val: -1}" mult
-          [[ "$num" =~ ^[0-9]+$ ]] || return 1
-          case "$unit" in
-            s) mult=1 ;;
-            m) mult=60 ;;
-            h) mult=3600 ;;
-            d) mult=86400 ;;
-            *) return 1 ;;
-          esac
-          echo $((num * mult))
-        }
-
-        [[ "$CLEAR_MIN" =~ ^[0-9]+$ ]] || { echo "Invalid min value: $CLEAR_MIN" >&2; exit 1; }
-        [[ "$CLEAR_DRY_RUN" =~ ^(true|false)$ ]] || { echo "Invalid dry-run value: $CLEAR_DRY_RUN" >&2; exit 1; }
-
-        CACHES=$(gh cache list --key "ccache-$CLEAR_KEY" --json id,key,createdAt --jq '.[] | [.createdAt, .id, .key] | @tsv' 2>/dev/null | LC_ALL=C sort)
-        if [ -z "$CACHES" ]; then
-          echo "No caches found with key prefix: $CLEAR_KEY"
-          exit 0
-        fi
-
-        TOTAL=$(( $(wc -l <<< "$CACHES") ))
-
-        echo "Found $TOTAL cache(s) with key prefix: $CLEAR_KEY (oldest first):"
-        while IFS=$'\t' read -r CREATED ID KEY; do
-          printf '  %s  %s  %s\n' "$CREATED" "$ID" "$KEY"
-        done <<< "$CACHES"
-
-        CUTOFF=""
-        if [ -n "$CLEAR_OLDER" ]; then
-          OLDER_SECONDS=$(to_seconds "$CLEAR_OLDER") || { echo "Invalid older value: $CLEAR_OLDER (expected e.g. 90m, 1h, 1d)" >&2; exit 1; }
-          CUTOFF=$(( $(date +%s) - OLDER_SECONDS ))
-        fi
-
-        # Caches are sorted oldest first
-        DELETED=0
-        while IFS=$'\t' read -r CREATED ID KEY; do
-          if [ -n "$CUTOFF" ] && [ "$(date -d "$CREATED" +%s)" -ge "$CUTOFF" ]; then
-            echo "Rest are not older than $CLEAR_OLDER, stopping"
-            break
-          fi
-          if [ $((TOTAL - DELETED - 1)) -lt "$CLEAR_MIN" ]; then
-            echo "Keeping at least $CLEAR_MIN cache(s), stopping"
-            break
-          fi
-          if [ "$CLEAR_DRY_RUN" = "true" ]; then
-            echo "Would delete cache: $ID ($KEY)"
-          else
-            echo "Deleting cache: $ID ($KEY)"
-            gh cache delete "$ID"
-          fi
-          DELETED=$((DELETED + 1))
-        done <<< "$CACHES"
+        bash scripts/ccache-clear.sh \
+          --key "${{ inputs.key }}" \
+          --older "${{ inputs.older }}" \
+          --min "${{ inputs.min }}" \
+          ${{ inputs.dry-run == 'true' && '--dry-run' || '' }}
@@ -73,6 +73,16 @@ jobs:
          cd build
          ctest -L main -E "test-llama-archs" --verbose --timeout 900

+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: apple-arm64
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+
  macos-latest-x64:
    runs-on: macos-15-intel

@@ -109,6 +119,16 @@ jobs:
          cd build
          ctest -L main --verbose --timeout 900

+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: apple-x64
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+
  macos-latest-ios-xcode:
    runs-on: macos-latest

@@ -163,14 +183,6 @@ jobs:
        id: checkout
        uses: actions/checkout@v6

-      # TODO: this likely does not do anything - if yes, remove it
-      - name: ccache
-        uses: ggml-org/ccache-action@v1.2.21
-        with:
-          key: apple-tvos
-          evict-old-files: 1d
-          save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
-
      - name: Build
        id: cmake_build
        run: |
@@ -196,14 +208,6 @@ jobs:
        id: checkout
        uses: actions/checkout@v6

-      # TODO: this likely does not do anything - if yes, remove it
-      - name: ccache
-        uses: ggml-org/ccache-action@v1.2.21
-        with:
-          key: apple-visionos
-          evict-old-files: 1d
-          save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
-
      - name: Build
        id: cmake_build
        run: |
@@ -234,14 +238,6 @@ jobs:
        id: checkout
        uses: actions/checkout@v6

-      # TODO: this likely does not do anything - if yes, remove it
-      - name: ccache
-        uses: ggml-org/ccache-action@v1.2.21
-        with:
-          key: apple-swift
-          evict-old-files: 1d
-          save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
-
      - name: Download xcframework artifact
        uses: actions/download-artifact@v7
        with:
@@ -125,7 +125,7 @@ jobs:
          GH_TOKEN: ${{ github.token }}
        with:
          key: cpu-${{ matrix.os }}
-          older: 1h
+          older: 5m
          min: 1
          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}

@@ -215,3 +215,13 @@ jobs:
      #     cd build
      #     $env:LLAMA_SKIP_TESTS_SLOW_ON_EMULATOR = 1
      #     & $sde -future -- ctest -L main -C Release --verbose --timeout 900
+
+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: cpu-windows-2025-${{ matrix.build }}
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
@@ -72,6 +72,16 @@ jobs:
            -DGGML_CUDA_CUB_3DOT2=ON
          cmake --build build

+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: cuda-ubuntu-24.04-cuda
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+
  hip:
    runs-on: ubuntu-22.04
    container: rocm/dev-ubuntu-22.04:6.1.2
@@ -103,6 +113,16 @@ jobs:
            -DGGML_HIP=ON
          cmake --build build --config Release -j $(nproc)

+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: cuda-ubuntu-22.04-hip
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+
  musa:
    runs-on: ubuntu-22.04
    container: mthreads/musa:rc4.3.0-devel-ubuntu22.04-amd64
@@ -131,3 +151,13 @@ jobs:
          cmake -B build -S . \
            -DGGML_MUSA=ON
          time cmake --build build --config Release -j $(nproc)
+
+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: cuda-ubuntu-22.04-musa
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
@@ -80,3 +80,13 @@ jobs:
        run: |
          cmake -S . -B build -G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-llvm.cmake -DCMAKE_PREFIX_PATH="$env:RUNNER_TEMP/opencl-arm64-release" -DGGML_OPENCL=ON -DGGML_OPENCL_USE_ADRENO_KERNELS=ON -DLLAMA_BUILD_BORINGSSL=ON
          cmake --build build --config Release -j ${env:NUMBER_OF_PROCESSORS}
+
+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: opencl-windows-2025-x64
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
@@ -167,3 +167,13 @@ jobs:

          cd build
          ctest --test-dir ReleaseOV -L main -E "test-llama-archs|test-recurrent-state-rollback-nemotron-h" -C Release --verbose --timeout 3000
+
+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: openvino-windows-2022
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
@@ -96,6 +96,16 @@ jobs:
            -DGGML_SYCL_F16=${{ matrix.fp16 }}
          time cmake --build build --config Release -j $(nproc)

+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: sycl-ubuntu-24-${{ matrix.build }}
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+
  windows-latest-sycl:
    runs-on: windows-2022

@@ -139,3 +149,13 @@ jobs:
      - name: Build
        id: cmake_build
        run:  examples/sycl/win-build-sycl.bat
+
+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: sycl-windows-latest
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
@@ -55,7 +55,7 @@ jobs:
      - name: ccache
        uses: ggml-org/ccache-action@v1.2.21
        with:
-          key: vulkan-ubuntu-24.04-arm-new
+          key: vulkan-ubuntu-24.04-arm
          variant: ccache
          evict-old-files: 1d
          save: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
@@ -73,6 +73,16 @@ jobs:
        run: |
          time cmake --build build -j $(nproc)

+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: vulkan-ubuntu-24.04-arm
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+
  ubuntu-llvmpipe:
    runs-on: ubuntu-24.04

@@ -128,6 +138,16 @@ jobs:
          # test-backend-ops is too slow on llvmpipe, skip it
          ctest -L main -E test-backend-ops --verbose --timeout 900

+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: vulkan-ubuntu-24.04-llvmpipe
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+
  windows:
    runs-on: windows-2025

@@ -180,3 +200,13 @@ jobs:
        run: |
          cd build
          ctest -L main -C Release --verbose --timeout 900
+
+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: cpu-windows-2025-x64-vulkan
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
@@ -88,3 +88,13 @@ jobs:
            -DEMDAWNWEBGPU_DIR=emdawnwebgpu_pkg

          time cmake --build build-wasm --config Release --target test-backend-ops -j $(nproc)
+
+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: webgpu-ubuntu-24.04-arm-wasm
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
@@ -101,6 +101,16 @@ jobs:
          cd build
          ctest -L main --verbose --timeout 900

+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: webgpu-macos-latest
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+
  ubuntu:
    runs-on: ubuntu-24.04

@@ -153,3 +163,13 @@ jobs:
          # This is using llvmpipe and runs slower than other backends
          # test-backend-ops is too slow on llvmpipe, skip it
          ctest -L main -E test-backend-ops --verbose --timeout 900
+
+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: webgpu-ubuntu-24.04
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
@@ -84,3 +84,13 @@ jobs:
          cd build
          make -j $(nproc) 2>&1 | tee metrics.log | grep -v 'Rpass-analysis=kernel-resource-usage\|remark:\|^$'
          python3 ../scripts/hip/gcn-cdna-vgpr-check.py metrics.log
+
+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: hip-quality-check-ubuntu-22.04
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
@@ -128,6 +128,16 @@ jobs:
          export LLAMA_ARG_BACKEND_SAMPLING=1
          SLOW_TESTS=1 ./tests.sh

+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: server-ubuntu-24.04-arm
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
+
  windows:
    runs-on: windows-2025

@@ -181,3 +191,13 @@ jobs:
          cd tools/server/tests
          export SLOW_TESTS="1"
          ./tests.sh
+
+      - name: ccache-clear
+        uses: ./.github/actions/ccache-clear
+        env:
+          GH_TOKEN: ${{ github.token }}
+        with:
+          key: server-windows-2025-x64
+          older: 5m
+          min: 1
+          dry-run: ${{ github.event_name != 'push' || github.ref != 'refs/heads/master' }}
@@ -122,7 +122,9 @@ class Glm4MoeModel(TextModel):
            self.tensor_map = gguf.get_tensor_name_map(self.model_arch, self.block_count)

    def index_tensors(self, remote_hf_model_id: str | None = None):
-        type(self)._n_main_layers = self.hparams["num_hidden_layers"]
+        hparams = {**self.hparams, **self.hparams.get("text_config", {})}
+        key = next((k for k in ["n_layers", "num_hidden_layers", "n_layer", "num_layers"] if k in hparams), None)
+        type(self)._n_main_layers = hparams.get(key)
        return super().index_tensors(remote_hf_model_id=remote_hf_model_id)

    @classmethod
@@ -1724,6 +1724,19 @@ extern "C" {
            struct ggml_tensor  * a,
            int                   n_past);

+    GGML_API struct ggml_tensor * ggml_clamp(
+            struct ggml_context * ctx,
+            struct ggml_tensor  * a,
+            float                 min,
+            float                 max);
+
+    // in-place, returns view(a)
+    GGML_API struct ggml_tensor * ggml_clamp_inplace(
+            struct ggml_context * ctx,
+            struct ggml_tensor  * a,
+            float                 min,
+            float                 max);
+
    GGML_API struct ggml_tensor * ggml_soft_max(
            struct ggml_context * ctx,
            struct ggml_tensor  * a);
@@ -1990,14 +2003,6 @@ extern "C" {
            struct ggml_tensor  * a,
            int                   n_offs);

-    // clamp
-    // in-place, returns view(a)
-    GGML_API struct ggml_tensor * ggml_clamp(
-            struct ggml_context * ctx,
-            struct ggml_tensor  * a,
-            float                 min,
-            float                 max);
-
    // im2col
    // converts data into a format that effectively results in a convolution when combined with matrix multiplication
    GGML_API struct ggml_tensor * ggml_im2col(
@@ -40,6 +40,7 @@ bool ggml_op_can_inplace(enum ggml_op op) {
        case GGML_OP_SILU_BACK:
        case GGML_OP_RMS_NORM:
        case GGML_OP_RMS_NORM_BACK:
+        case GGML_OP_CLAMP:
        case GGML_OP_SOFT_MAX:
        case GGML_OP_SOFT_MAX_BACK:
            return true;
@@ -4611,8 +4611,8 @@ static std::string ggml_cuda_device_description(int device) {
    const ggml_cuda_device_info & info = ggml_cuda_info();
    std::string description = prop.name;
    if (info.device_count > info.physical_device_count) {
-        description += " (physical device " + std::to_string(info.devices[device].physical_device) +
-                       ", virtual device " + std::to_string(info.devices[device].virtual_index) + ")";
+        description += " (dev p" + std::to_string(info.devices[device].physical_device) +
+                       "/v" + std::to_string(info.devices[device].virtual_index) + ")";
    }
    return description;
 }
@@ -17,10 +17,10 @@ struct ggml_metal_device_deleter {

 typedef std::unique_ptr<ggml_metal_device, ggml_metal_device_deleter> ggml_metal_device_ptr;

-ggml_metal_device_t ggml_metal_device_get(int device) {
+ggml_metal_device_t ggml_metal_device_get(int device, int n_devices) {
    static std::vector<ggml_metal_device_ptr> devs;

-    devs.emplace_back(ggml_metal_device_init(device));
+    devs.emplace_back(ggml_metal_device_init(device, n_devices));

    return devs.back().get();
 }
@@ -259,6 +259,8 @@ enum ggml_metal_device_id {

 struct ggml_metal_device_props {
    int device;
+    int device_phys;
+    int device_virt;
    char name[128];
    char desc[128];

@@ -286,10 +288,10 @@ typedef struct ggml_metal_event * ggml_metal_event_t;
 void ggml_metal_event_encode_signal(ggml_metal_event_t ev, ggml_metal_cmd_buf_t cmd_buf);
 void ggml_metal_event_encode_wait  (ggml_metal_event_t ev, ggml_metal_cmd_buf_t cmd_buf);

-ggml_metal_device_t ggml_metal_device_init(int device);
+ggml_metal_device_t ggml_metal_device_init(int device, int n_devices);
 void ggml_metal_device_free(ggml_metal_device_t dev);

-ggml_metal_device_t ggml_metal_device_get(int device);
+ggml_metal_device_t ggml_metal_device_get(int device, int n_devices);

 void * ggml_metal_device_get_obj  (ggml_metal_device_t dev); // id<MTLDevice>
 void * ggml_metal_device_get_queue(ggml_metal_device_t dev); // id<MTLCommandQueue>
@@ -711,7 +711,7 @@ static enum ggml_metal_device_id ggml_metal_device_id_parse(const char * name) {
    return GGML_METAL_DEVICE_GENERIC;
 }

-ggml_metal_device_t ggml_metal_device_init(int device) {
+ggml_metal_device_t ggml_metal_device_init(int device, int n_devices) {
    ggml_metal_device_t dev = calloc(1, sizeof(struct ggml_metal_device));

    assert(dev != NULL);
@@ -728,6 +728,12 @@ ggml_metal_device_t ggml_metal_device_init(int device) {
            dev->addr_virt = 0x000000400ULL;

            dev->props.device = device;
+
+            // the Metal backend uses the system default device as the single physical device;
+            // additional (virtual) devices are emulated on top of it via GGML_METAL_DEVICES
+            dev->props.device_phys = 0;
+            dev->props.device_virt = device;
+
            dev->props.has_simdgroup_reduction  = [dev->mtl_device supportsFamily:MTLGPUFamilyApple7];
            dev->props.has_simdgroup_reduction |= [dev->mtl_device supportsFamily:MTLGPUFamilyMetal3_GGML];

@@ -891,7 +897,13 @@ ggml_metal_device_t ggml_metal_device_init(int device) {
            }

            snprintf(dev->props.name, sizeof(dev->props.name), "%s%d", "MTL", device);
-            snprintf(dev->props.desc, sizeof(dev->props.desc), "%s", [[dev->mtl_device name] UTF8String]);
+            const char * gpu_name = [[dev->mtl_device name] UTF8String];
+            if (n_devices > 1) {
+                snprintf(dev->props.desc, sizeof(dev->props.desc), "%s (dev p%d/v%d)",
+                         gpu_name, dev->props.device_phys, dev->props.device_virt);
+            } else {
+                snprintf(dev->props.desc, sizeof(dev->props.desc), "%s", gpu_name);
+            }

            dev->library = ggml_metal_library_init(dev);
            if (!dev->library) {
@@ -891,7 +891,7 @@ static ggml_backend_dev_t ggml_backend_metal_device_init(ggml_backend_reg_t reg,
    return new ggml_backend_device {
        /* .iface   = */ ggml_backend_metal_device_i,
        /* .reg     = */ reg,
-        /* .context = */ ggml_metal_device_get(device),
+        /* .context = */ ggml_metal_device_get(device, g_devices),
    };
 }

@@ -5,10 +5,9 @@ enable subgroups;
 enable chromium_experimental_subgroup_matrix;

 #define BYTE_HELPERS
-#include "common_decls.tmpl"
-
 #define FLASH_ATTN_SCALAR_KV
 #include "flash_attn_decls.tmpl"
+#include "common_decls.tmpl"

 // Default values
 // The actual values are defined in shader-lib.
@@ -2,8 +2,8 @@ enable f16;
 enable subgroups;

 #define BYTE_HELPERS
-#include "common_decls.tmpl"
 #include "flash_attn_decls.tmpl"
+#include "common_decls.tmpl"

 // Default values
 // The actual values are defined in shader-lib.
@@ -3,9 +3,9 @@ enable f16;
 enable subgroups;

 #define BYTE_HELPERS
-#include "common_decls.tmpl"
 #define FLASH_ATTN_VEC_SPLIT
 #include "flash_attn_decls.tmpl"
+#include "common_decls.tmpl"

 // Default values
 // The actual values are defined in shader-lib.
@@ -4042,6 +4042,41 @@ struct ggml_tensor * ggml_diag_mask_zero_inplace(
    return ggml_diag_mask_zero_impl(ctx, a, n_past, true);
 }

+// ggml_clamp
+
+static struct ggml_tensor * ggml_clamp_impl(
+        struct ggml_context * ctx,
+        struct ggml_tensor  * a,
+        float                 min,
+        float                 max,
+        bool                  inplace) {
+    struct ggml_tensor * result = inplace ? ggml_view_tensor(ctx, a) : ggml_dup_tensor(ctx, a);
+
+    float params[] = { min, max };
+    ggml_set_op_params(result, params, sizeof(params));
+
+    result->op     = GGML_OP_CLAMP;
+    result->src[0] = a;
+
+    return result;
+}
+
+struct ggml_tensor * ggml_clamp(
+    struct ggml_context * ctx,
+    struct ggml_tensor  * a,
+    float                 min,
+    float                 max) {
+    return ggml_clamp_impl(ctx, a, min, max, false);
+}
+
+struct ggml_tensor * ggml_clamp_inplace(
+    struct ggml_context * ctx,
+    struct ggml_tensor  * a,
+    float                 min,
+    float                 max) {
+    return ggml_clamp_impl(ctx, a, min, max, true);
+}
+
 // ggml_soft_max

 static struct ggml_tensor * ggml_soft_max_impl(
@@ -4438,25 +4473,6 @@ struct ggml_tensor * ggml_rope_set_offset(
    return a;
 }

-// ggml_clamp
-
-struct ggml_tensor * ggml_clamp(
-        struct ggml_context * ctx,
-        struct ggml_tensor  * a,
-        float                 min,
-        float                 max) {
-    // TODO: when implement backward, fix this:
-    struct ggml_tensor * result = ggml_view_tensor(ctx, a);
-
-    float params[] = { min, max };
-    ggml_set_op_params(result, params, sizeof(params));
-
-    result->op     = GGML_OP_CLAMP;
-    result->src[0] = a;
-
-    return result;
-}
-
 static int64_t ggml_calc_conv_output_size(int64_t ins, int64_t ks, int s, int p, int d) {
    return (ins + 2 * p - d * (ks - 1) - 1) / s + 1;
 }
@@ -0,0 +1,105 @@
+#!/bin/bash
+# Delete GitHub Actions caches matching a key prefix, oldest first.
+#
+# Usage: ccache-clear.sh --key KEY [--older DURATION] [--min N] [--dry-run]
+#   --key:       cache key prefix to match and delete (without the ccache- prefix)
+#   --older:     only delete caches created more than DURATION ago (e.g. 5m, 1h, 1d);
+#                by default all matching caches are deleted
+#   --min:       stop deleting if fewer than N caches would remain (default: 0)
+#   --dry-run:   only print the caches that would be deleted, without deleting them
+#
+# Env (when running in GitHub Actions):
+#   GH_TOKEN:          token for the gh CLI
+#   GITHUB_REPOSITORY: owner/repo of the caches to manage
+set -euo pipefail
+
+KEY=""
+OLDER=""
+MIN=0
+DRY_RUN=false
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --key)     [[ $# -ge 2 ]] || { echo "Missing value for $1" >&2; exit 1; }; KEY="$2"; shift 2 ;;
+        --older)   [[ $# -ge 2 ]] || { echo "Missing value for $1" >&2; exit 1; }; OLDER="$2"; shift 2 ;;
+        --min)     [[ $# -ge 2 ]] || { echo "Missing value for $1" >&2; exit 1; }; MIN="$2"; shift 2 ;;
+        --dry-run) DRY_RUN=true; shift ;;
+        *) echo "Unknown argument: $1"; exit 1 ;;
+    esac
+done
+
+command -v gh >/dev/null 2>&1 || { echo "Error: GitHub CLI (gh) is required" >&2; exit 1; }
+[[ -n "${GITHUB_REPOSITORY:-}" ]] || { echo "Error: GITHUB_REPOSITORY not set" >&2; exit 1; }
+[[ -n "$KEY" ]] || { echo "Error: --key is required" >&2; exit 1; }
+[[ "$MIN" =~ ^[0-9]+$ ]] || { echo "Invalid min value: $MIN" >&2; exit 1; }
+
+# Convert a duration (e.g. 90m, 1h, 1d, plain seconds) to seconds
+to_seconds() {
+  local val="$1"
+  [[ "$val" =~ ^[0-9]+$ ]] && { echo "$val"; return 0; }
+  local num="${val%?}" unit="${val: -1}" mult
+  [[ "$num" =~ ^[0-9]+$ ]] || return 1
+  case "$unit" in
+    s) mult=1 ;;
+    m) mult=60 ;;
+    h) mult=3600 ;;
+    d) mult=86400 ;;
+    *) return 1 ;;
+  esac
+  echo $((num * mult))
+}
+
+# Convert an ISO-8601 UTC timestamp (e.g. 2026-08-23T16:51:23.313693Z) to epoch seconds
+to_epoch() {
+  local val="$1" out
+  # GNU date (e.g. Linux)
+  if out=$(date -d "$val" +%s 2>/dev/null) && [[ "$out" =~ ^[0-9]+$ ]]; then
+    echo "$out"
+    return 0
+  fi
+  # BSD date (e.g. macOS); fractional seconds are not needed, TZ forces UTC
+  out=$(TZ=UTC date -j -f "%Y-%m-%dT%H:%M:%S" "${val:0:19}" +%s 2>/dev/null) || return 1
+  [[ "$out" =~ ^[0-9]+$ ]] || return 1
+  echo "$out"
+}
+
+CACHES=$(gh cache list --repo "$GITHUB_REPOSITORY" --key "ccache-$KEY" --json id,key,createdAt --jq '.[] | [.createdAt, .id, .key] | @tsv' | LC_ALL=C sort)
+if [[ -z "$CACHES" ]]; then
+    echo "No caches found with key prefix: $KEY"
+    exit 0
+fi
+
+TOTAL=$(( $(wc -l <<< "$CACHES") ))
+
+echo "Found $TOTAL cache(s) with key prefix: $KEY (oldest first):"
+while IFS=$'\t' read -r CREATED ID CACHE_KEY; do
+    printf '  %s  %s  %s\n' "$CREATED" "$ID" "$CACHE_KEY"
+done <<< "$CACHES"
+
+CUTOFF=""
+if [[ -n "$OLDER" ]]; then
+    OLDER_SECONDS=$(to_seconds "$OLDER") || { echo "Invalid older value: $OLDER (expected e.g. 90m, 1h, 1d)" >&2; exit 1; }
+    CUTOFF=$(( $(date +%s) - OLDER_SECONDS ))
+fi
+
+# Caches are sorted oldest first
+DELETED=0
+while IFS=$'\t' read -r CREATED ID CACHE_KEY; do
+    if [[ -n "$CUTOFF" ]]; then
+        CREATED_SECONDS=$(to_epoch "$CREATED") || { echo "Failed to parse date: $CREATED" >&2; exit 1; }
+        if [[ "$CREATED_SECONDS" -ge "$CUTOFF" ]]; then
+            echo "Rest are not older than $OLDER, stopping"
+            break
+        fi
+    fi
+    if (( TOTAL - DELETED - 1 < MIN )); then
+        echo "Keeping at least $MIN cache(s), stopping"
+        break
+    fi
+    if [[ "$DRY_RUN" == "true" ]]; then
+        echo "Would delete cache: $ID ($CACHE_KEY)"
+    else
+        echo "Deleting cache: $ID ($CACHE_KEY)"
+        gh cache delete --repo "$GITHUB_REPOSITORY" "$ID"
+    fi
+    DELETED=$((DELETED + 1))
+done <<< "$CACHES"
@@ -507,7 +507,7 @@ static bool arch_supported(const llm_arch arch) {
    }
    // FIXME: these hit scheduler/view-backed-output issues with WebGPU on CI.
 #ifdef GGML_USE_WEBGPU
-    if (arch == LLM_ARCH_DEEPSEEK32 || arch == LLM_ARCH_GLM_DSA) {
+    if (arch == LLM_ARCH_DEEPSEEK32 || arch == LLM_ARCH_GLM_DSA || arch == LLM_ARCH_DOTS3NOTE) {
        return false;
    }
 #endif // GGML_USE_WEBGPU
@@ -42,6 +42,11 @@
 #ifdef MTMD_VIDEO
 #include "sheredom/subprocess.h"
 #include <thread>
+#ifndef _WIN32
+#include <csignal>
+#include <fcntl.h>
+#include <pthread.h>
+#endif
 #endif

 //
@@ -522,7 +527,8 @@ struct mtmd_helper_video {
    // RAII wrapper for managing subprocess
    struct subprocess_handle {
        struct subprocess_s proc = {};
-        bool alive = false;
+        bool created = false; // process exists and must be cleaned up
+        bool alive   = false; // process can still give us data
        std::thread feeder;

        subprocess_handle() = default;
@@ -531,18 +537,27 @@ struct mtmd_helper_video {
        ~subprocess_handle() { stop(); }

        void stop() {
-            if (alive) {
-                subprocess_terminate(&proc);
+            // note: alive becomes false on stdout EOF, but the process still needs cleanup
+            if (!created) {
+                return;
            }
+            subprocess_terminate(&proc);
+#ifdef _WIN32
+            // no SIGPIPE on windows: a blocked feeder only gets a broken pipe once we close our read end of the child stdin
+            if (proc.hStdInput) {
+                CloseHandle(proc.hStdInput);
+                proc.hStdInput = nullptr;
+            }
+#endif
            // join before destroy: feeder holds a FILE* from subprocess_stdin;
            // subprocess_destroy closes it, so the thread must finish first
            if (feeder.joinable()) {
                feeder.join();
            }
-            if (alive) {
-                subprocess_destroy(&proc);
-                alive = false;
-            }
+            subprocess_join(&proc, nullptr); // reap the child, or else it stays a zombie
+            subprocess_destroy(&proc);
+            created = false;
+            alive   = false;
        }

        FILE * stdout_pipe() {
@@ -552,10 +567,21 @@ struct mtmd_helper_video {
        // buf is tied to lifetime of mtmd_helper_video, so it's guaranteed to outlive the feeder thread
        void start_feeder(const std::vector<uint8_t> & buf) {
            feeder = std::thread([this, &buf]() {
+#ifndef _WIN32
+                // ffmpeg can exit before it reads all the input, for example when ffprobe already got the metadata.
+                // the write below must then fail with EPIPE, instead of killing the process with SIGPIPE
+                sigset_t sigpipe_set;
+                sigemptyset(&sigpipe_set);
+                sigaddset(&sigpipe_set, SIGPIPE);
+                pthread_sigmask(SIG_BLOCK, &sigpipe_set, nullptr); // linux sends the signal to the writing thread
+#endif
                FILE * f = subprocess_stdin(&proc);
                if (!f) {
                    return;
                }
+#ifdef F_SETNOSIGPIPE
+                fcntl(fileno(f), F_SETNOSIGPIPE, 1); // macos/bsd send it to the process, so turn it off per fd
+#endif
                fwrite(buf.data(), 1, buf.size(), f);
                fclose(f);
                proc.stdin_file = nullptr; // prevent double-close in subprocess_destroy
@@ -601,7 +627,8 @@ struct mtmd_helper_video {
            LOG_ERR("%s: failed to launch ffprobe\n", __func__);
            return false;
        }
-        probe_sp.alive = true;
+        probe_sp.created = true;
+        probe_sp.alive   = true;

        if (is_buf_input()) {
            probe_sp.start_feeder(input_buf);
@@ -673,6 +700,11 @@ struct mtmd_helper_video {
        }

        cmd.push_back("-nostdin");
+        if (is_buf_input()) {
+            // remove the 64KB read-ahead limit of cache:, or else ffmpeg cannot reach a moov atom at end of file
+            cmd.push_back("-read_ahead_limit");
+            cmd.push_back("-1");
+        }
        cmd.push_back("-i");
        // cache:pipe:0 wraps stdin with a seekable in-memory cache, letting ffmpeg seek
        // backwards for container headers (e.g. MP4 moov atom at end of file)
@@ -711,7 +743,8 @@ struct mtmd_helper_video {
            subprocess_option_search_user_path | subprocess_option_inherit_environment,
            &sp.proc);

-        sp.alive = (ret == 0);
+        sp.created = (ret == 0);
+        sp.alive   = (ret == 0);
        LOG_DBG("%s: subprocess_create ret=%d proc_alive=%d\n", __func__, ret, (int)sp.alive);

        if (sp.alive && is_buf_input()) {
Author	SHA1	Message	Date
fairydreaming	7584430716	tests : disable DOTS3NOTE arch test for WebGPU (#27654 ) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>	2026-08-24 13:39:31 +03:00
jacekpoplawski	71cc86fa41	convert: fix GLM regression in index_tensors (#27655 )	2026-08-24 13:21:00 +03:00
Georgi Gerganov	a14dba686a	ggml : shorten virtual device naming in CUDA and Metal (#27608 ) * ggml : shorten virtual device naming in CUDA and Metal Assisted-by: llama.cpp:DeepSeek-V4-Flash-0731 * ggml-metal : build device description at init Assisted-by: llama.cpp:DeepSeek-V4-Flash-0731 * cont : naming	2026-08-24 12:35:08 +03:00
fairydreaming	c1c766da59	webgpu : reorder includes since V that appears in common_decls.tmpl may be defined as K in flash_attn_decls.tmpl if KV_OVERLAP (#27545 ) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>	2026-08-24 10:07:12 +02:00
Xuan-Son Nguyen	160c6b0bdd	mtmd: video: fix moov atom at the end of file (#27596 ) * mtmd: video: fix moov at the end of file Co-authored-by: rkfg <rkfg@rkfg.me> * fix SIGPIPE * windows: handle broken pipe case --------- Co-authored-by: rkfg <rkfg@rkfg.me>	2026-08-24 09:59:04 +02:00
Georgi Gerganov	985b14912b	ci : apply ccache-clear with older/min/dry-run to all ccache jobs (#27602 ) * ci : apply ccache-clear with older/min/dry-run to all ccache jobs Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : install gh in ccache-clear if missing (container jobs) The ccache-clear action relies on the gh CLI, which is not present in container-based jobs. Install it on demand so those jobs can clear caches. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : install gh via apt repo in ccache-clear The install.sh script used previously is no longer served (404). Switch to the official GitHub CLI apt repository, which is still available. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : pass --repo to gh cache commands in ccache-clear In container jobs gh cannot auto-detect the repository from git, so gh cache list/delete fail with 'failed to run git: not a git repository'. Pass the repository explicitly via --repo using GITHUB_REPOSITORY. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : drop -new suffix from vulkan ccache key The -new suffix was only needed to force a fresh cache. With ccache-clear now evicting stale caches, the original key can be used again. The old ccache-vulkan-ubuntu-24.04-arm-new entries still match the ccache-clear key prefix and are cleaned up automatically. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : fix ccache-clear date parsing on macOS (BSD date) macOS ships BSD date, which has no -d option. The older cutoff check was silently disabled there: 'date: illegal option -- d' errors in the log and the loop was only stopped by the min limit, risking deletion of caches not older than the cutoff (e.g. saved by a concurrent job). Parse the ISO-8601 timestamps with GNU date when available and fall back to BSD date otherwise (TZ=UTC, fractional seconds dropped). Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : extract ccache-clear logic into scripts/ccache-clear.sh The composite action now consists of a dedicated step that installs the GitHub CLI when missing (e.g. in container jobs) and a thin step that calls the new script. The script follows the make-release-checks.sh conventions (usage/env header, set -euo pipefail, CLI flags) and only checks that gh is available. The action inputs are unchanged, so the workflow steps are untouched. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : remove unused apple ccaches	2026-08-24 10:49:20 +03:00
Georgi Gerganov	6036c635e2	ggml : fix ggml_clamp (#27644 ) * ggml : fix ggml_clamp * cont : update ggml-alloc	2026-08-24 10:43:04 +03:00