Public/koboldcpp
1
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-09-06 13:01:21 +02:00
Code Issues Packages Projects Releases Wiki Activity

ggml : adjust logic for offloading ops to weight's backend (#25832)

Browse Source 
* ggml : adjust logic for offloading ops to weight's backend

* llama : dsv4 graph fixes
This commit is contained in:
Georgi Gerganov
2026-07-27 14:54:46 +03:00
committed by GitHub
parent 7ef790f90a
commit dee2a846b8
3 changed files with 34 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/llama-context.cpp
+3 -2
View File
@@ -2473,11 +2473,12 @@ llm_graph_cb llama_context::graph_get_cb() const {
ggml_set_name(cur, name);
}
// norm may be automatically assigned to the backend of the previous layer, increasing data transfer between backends
// - norm may be automatically assigned to the backend of the previous layer, increasing data transfer between backends
// - force the last op of the layer on the specified backend to avoid running it on the backend of the next layer due to scheduling
// FIXME: fix in ggml_backend_sched
const bool full_offload = model.n_gpu_layers() > model.hparams.n_layer_all;
if (ubatch.n_tokens < 32 || full_offload) {
if (il != -1 && strcmp(name, "norm") == 0) {
if (il != -1 && (strcmp(name, "norm") == 0 || strcmp(name, "l_last") == 0)) {
const auto & dev_layer = model.dev_layer(il);
for (const auto & backend : backends) {
if (ggml_backend_get_device(backend.get()) == dev_layer) {