Public/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-09-20 01:31:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

ggml: add cross-backend profiler

Browse Source 
Add an optional per-op / per-copy profiler to the ggml scheduler that
records timed events across all backends of a split graph, so a single
run can be inspected end to end (compute kernels, host<->device copies,
fusion names, tensor shapes/strides/types, op params).

- ggml-profiler.h/.cpp: ggml_profile_record, per-backend profiler
  interface (enable/reset/get_records), JSON export
- ggml-backend.cpp: scheduler-level collection, copy events, backend
  attribution, mul_mat_id stats, throughput stat, concurrent-mode fix,
  auto-export via GGML_PROFILE env var
- Backend profilers: CPU, CUDA/HIP/MUSA (event-based timing), Vulkan
  (timestamp queries), BLAS, Metal (tentative); stubs for the remaining
  backends
- llama: expose profiler enable/export; --profile, --profile-output,
  --with-backends args in common; hooks in server, completion and the
  debug example
- tools/profiler/profiler.py: analysis tool (per-op / per-backend
  summaries, Chrome trace export)
- test-backend-ops / test-export-graph-ops: run perf tests with exactly
  the tensor shapes recorded in a profile (converged with
  export-graph-ops)
- docs/cross-profiler.md
- ggml-cuda: avoid ROCm_Host compute on HIP integrated GPUs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ney1sm8n1bSjeA3DrrW5ah
This commit is contained in:
Piotr Wilkin
2026-09-07 13:42:53 +02:00
parent 5202104b59
commit 16db737a1c
45 changed files with 3890 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files
common/arg.cpp
+22
View File
@@ -1483,6 +1483,21 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.server_base = value;
}
).set_examples({LLAMA_EXAMPLE_CLI}));
add_opt(common_arg(
{"--profile"},
"enable cross-backend profiling (CPU, BLAS, CUDA)",
[](common_params & params) {
params.profiling = true;
}
).set_examples({LLAMA_EXAMPLE_CLI, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_COMPLETION, LLAMA_EXAMPLE_DEBUG}));
add_opt(common_arg(
{"--profile-output"}, "FNAME",
"write profiling JSON output to FNAME (default: stdout)",
[](common_params & params, const std::string & value) {
params.profiling = true;
params.profiling_output = value;
}
).set_examples({LLAMA_EXAMPLE_CLI, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_COMPLETION, LLAMA_EXAMPLE_DEBUG}));
add_opt(common_arg(
{"--verbose-prompt"},
string_format("print a verbose prompt before generation (default: %s)", params.verbose_prompt ? "true" : "false"),
@@ -3177,6 +3192,13 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
}
).set_examples({LLAMA_EXAMPLE_IMATRIX, LLAMA_EXAMPLE_CVECTOR_GENERATOR, LLAMA_EXAMPLE_EXPORT_LORA, LLAMA_EXAMPLE_TTS, LLAMA_EXAMPLE_FINETUNE,
LLAMA_EXAMPLE_RESULTS, LLAMA_EXAMPLE_EXPORT_GRAPH_OPS, LLAMA_EXAMPLE_CLI}));
add_opt(common_arg(
{"--with-backends"},
"export graph ops with backend assignments (default: CPU only)",
[](common_params & params) {
params.with_backends = true;
}
).set_examples({LLAMA_EXAMPLE_EXPORT_GRAPH_OPS}));
add_opt(common_arg(
{"-ofreq", "--output-frequency"}, "N",
string_format("output the imatrix every N iterations (default: %d)", params.n_out_freq),