swap worker <--> main thread design

This commit is contained in:
Xuan Son Nguyen
2026-08-15 16:00:56 +02:00
parent 51a4f6303b
commit 9871d51fd7
3 changed files with 63 additions and 40 deletions
+2 -2
View File
@@ -3580,8 +3580,8 @@ private:
has_output |= batch.tokens[i].output;
}
// decode on the worker thread, so we can still handle metrics tasks while waiting
// note: the sync is done here too, so that the wait also happens off the main thread
// yield to the queue, so we can still handle metrics tasks while decoding
// note: the sync is done here too, so that the wait is also covered by the yield
int ret = 0;
queue_tasks.yield_to_queue([&]() {
ret = llama_decode(ctx_tgt, batch_view);