mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-08-27 15:41:19 +02:00
YiChen Lv
f280b26983
* metal : per-device tuned (Q, NE) for flash-attn vec (#25750)
* rebase Q-generic FA vec body from 01dc93607 (#23114)
* add 53 f16 (Q,NE) flash-attn vec instantiations (vec 80 -> 133)
* add FA vec (Q,NE) tuning table + dispatch wiring + SMEM cap fallback
* add FA vec (Q,NE) perf sweep
* fill tuning result
* fold family table into a per-family representative SKU
* refactor tuning result format
* extend FA vec tuning to quantized KV caches
* sync fa vec tuner bucketing with runtime, use pointwise tuning regret
* update tuned table
* format and cleanup
* prefix fa_vec tuning procs with ggml_backend_metal_tuning_, drop unused fa_vec_override_active
* add device id -> token lookup for the offline tuning tool
* add ggml-metal-tuning skeleton
* add op-agnostic perf cell + median timing for the tuner
* add FA-vec graph build + tensor init to the tuner
* tools : add FA-vec (Q,NE) sweep, compression and table emit
* cool down and re-measure the dirty window on thermal drift
* test-backend-ops : replace the FA vec tune mode with a bounded (Q,NE) slice
* tools : document the Metal tuner, point the table comment at it
* abort on unknown KV type, single-source fa_vec_legal_ne
* cleanup
* honor -o in the FA vec (Q,NE) slice
* retune FA-vec (Q, NE) under a pointwise no-harm gate
* cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
64 lines
3.2 KiB
Markdown
64 lines
3.2 KiB
Markdown
# ggml-metal-tuning
|
|
|
|
Offline kernel tuner for the Metal backend.
|
|
It sweeps a kernel's config grid on the machine it runs on and prints pasteable table rows for `ggml/src/ggml-metal/ggml-metal-tuning.cpp`.
|
|
|
|
This is not a test: it never reports pass/fail on performance.
|
|
A non-zero exit code means bad arguments or a wrong environment (no Metal device, missing proc bridges), never a perf result.
|
|
|
|
| tuner | tunes | table |
|
|
|---|---|---|
|
|
| `fa-vec` | flash-attn vec `(Q, NE)` per `(dtype, head size, KV depth, batch width)` | `fa_vec_tuned_table` |
|
|
|
|
## Adding a device to the FA-vec table
|
|
|
|
Build on the target machine:
|
|
|
|
```bash
|
|
cmake -B build -DGGML_METAL=ON
|
|
cmake --build build --target ggml-metal-tuning -j
|
|
cmake --build build --target test-backend-ops -j
|
|
```
|
|
|
|
Sweep the grid (6 dtypes x 10 head sizes x 4 KV depths x 9 batch widths; a few hours):
|
|
|
|
```bash
|
|
./build/bin/ggml-metal-tuning fa-vec > fa_vec_rows.txt 2> fa_vec_sweep.log
|
|
```
|
|
|
|
`fa_vec_rows.txt` holds nothing but table rows, ready to paste into `fa_vec_tuned_table`: the min-max-regret target, the aggregate benefit gate, the short-KV drop and the pointwise compression are already applied.
|
|
A config represents a bucket only if it is no slower than the baseline config at every point that bucket covers, so a config that wins on average but loses at one batch width leaves its bucket at baseline.
|
|
`fa_vec_sweep.log` holds the per-cell timings, bucket coverage, noise floor, any cooldown activity, and every config the no-harm rule refused together with the point that refused it.
|
|
Post both: the log is what makes the rows reviewable.
|
|
|
|
Long sweeps can be split.
|
|
`--dtype f16,q4_0` and `--dk 128,192` restrict the grid, and the emitted rows for one `(dtype, head size)` do not depend on the others.
|
|
Concatenating the shard outputs in the order the full grid would visit them gives the same rows a single run prints.
|
|
|
|
Then validate the numerics, where Metal is compared against the CPU reference:
|
|
|
|
```bash
|
|
./build/bin/test-backend-ops test -o FLASH_ATTN_EXT -b MTL0
|
|
```
|
|
|
|
This forces every legal `(Q, NE)` on `dk=128` and `dk=576`.
|
|
The tuner itself does no numerical checks, so the other head sizes have no automated numerical coverage.
|
|
|
|
If the device is not in `enum ggml_metal_device_id` yet, register it in `ggml/src/ggml-metal/ggml-metal-device.{h,m}` first.
|
|
The tuner emits whatever token the runtime reports for the machine, so an unregistered device emits `GGML_METAL_DEVICE_GENERIC` and its rows would apply to every unknown device.
|
|
|
|
## Thermal throttling
|
|
|
|
Long sweeps heat the GPU, and a throttled measurement is indistinguishable from a slow kernel.
|
|
The tuner re-measures a fixed baseline config every four candidates as an anchor.
|
|
When the anchor drifts more than `--cool-drift` (10% by default) from the coolest anchor seen in that cell, the tuner:
|
|
|
|
1. discards every candidate measured since the last clean anchor,
|
|
2. sleeps with exponential backoff until the anchor comes back within `--cool-eps` (3%),
|
|
3. re-measures the discarded candidates.
|
|
|
|
If it cannot cool down within `--cool-max-wait` seconds, or a cell needs more than `--cool-max-retry` rounds, that cell is dropped from the table and reported on stderr.
|
|
|
|
`--no-cooldown` only warns on drift and keeps the measurement.
|
|
Use it to reproduce a sweep taken without cooling.
|