Piotr Wilkin
16db737a1c
Add an optional per-op / per-copy profiler to the ggml scheduler that records timed events across all backends of a split graph, so a single run can be inspected end to end (compute kernels, host<->device copies, fusion names, tensor shapes/strides/types, op params). - ggml-profiler.h/.cpp: ggml_profile_record, per-backend profiler interface (enable/reset/get_records), JSON export - ggml-backend.cpp: scheduler-level collection, copy events, backend attribution, mul_mat_id stats, throughput stat, concurrent-mode fix, auto-export via GGML_PROFILE env var - Backend profilers: CPU, CUDA/HIP/MUSA (event-based timing), Vulkan (timestamp queries), BLAS, Metal (tentative); stubs for the remaining backends - llama: expose profiler enable/export; --profile, --profile-output, --with-backends args in common; hooks in server, completion and the debug example - tools/profiler/profiler.py: analysis tool (per-op / per-backend summaries, Chrome trace export) - test-backend-ops / test-export-graph-ops: run perf tests with exactly the tensor shapes recorded in a profile (converged with export-graph-ops) - docs/cross-profiler.md - ggml-cuda: avoid ROCm_Host compute on HIP integrated GPUs Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ney1sm8n1bSjeA3DrrW5ah
llama.cpp/examples/debug
This is a utility intended to help debug a model by registering a callback that logs GGML operations and tensor data. It can also store the generated logits or embeddings as well as the prompt and token ids for comparison with the original model.
Usage
llama-debug \
--hf-repo ggml-org/models \
--hf-file phi-2/ggml-model-q4_0.gguf \
--model phi-2-q4_0.gguf \
--prompt hello \
--save-logits \
--verbose
The tensor data is logged as debug and required the --verbose flag. The reason
for this is that while useful for a model with many layers there can be a lot of
output. You can filter the tensor names using the --tensor-filter option.
A recommended approach is to first run without --verbose and see if the
generated logits/embeddings are close to the original model. If they are not,
then it might be required to inspect tensor by tensor and in that case it is
useful to enable the --verbose flag along with --tensor-filter to focus on
specific tensors.
Options
This example supports all standard llama.cpp options and also accepts the
following options:
$ llama-debug --help
...
----- example-specific params -----
--save-logits save final logits to files for verification (default: false)
--logits-output-dir PATH directory for saving logits output files (default: data)
--tensor-filter REGEX filter tensor names for debug output (regex pattern, can be specified multiple times)
Output Files
When --save-logits is enabled, the following files are created in the output
directory:
llamacpp-<model>[-embeddings].bin- Binary output (logits or embeddings)llamacpp-<model>[-embeddings].txt- Text output (logits or embeddings, one per line)llamacpp-<model>[-embeddings]-prompt.txt- Prompt text and token IDsllamacpp-<model>[-embeddings]-tokens.bin- Binary token IDs for programmatic comparison
These files can be compared against the original model's output to verify the converted model.