Public/koboldcpp
1
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-09-02 11:01:20 +02:00
Code Issues Packages Projects Releases Wiki Activity

server + ui: fix stream routes for model names containing a slash (#26137)

Browse Source 
* server + ui: refactor resumable stream routes to query string conv_id

The conversation id can embed a model name containing slashes
(ggml-org/...) in router mode, which the decoded path splits before the
:conv_id param is captured, so stop and resume never matched the
session. Move the id to the conv_id query string on the public routes
and on the internal router -> child hop, where slashes survive
encoding. Handlers are unchanged since query and path params land in
the same map. Add a regression test with a slashed model name.

* server: move stream route docs to server-stream.h

Address review: ngxson wants the main server.cpp registration code kept
clean and simple, with route-level explanations living in the header.
Move the query string rationale and the lookup ownership note next to
the handler declarations in server-stream.h, and shorten the wiring
comment to a pointer.

* server: cancel a pending request when its stream is stopped during model load

The conversation was registered in the conv map only after the blocking
autoload wait, so a stop issued while the model loaded found nothing to
cancel and the request went on to generate an orphan once the load
ended. Register the conversation before the wait and give the entry a
ticket: a stop erases the entry, and the parked request checks its
ticket after the wait and aborts with 400 instead of starting. A newer
request on the same conversation replaces the entry, so only the
stopped request is cancelled. Add a regression test that stops during
the load window.

* server + ui: resume a stream after a page reload during model load

A pending request died with the client socket when the page was
reloaded while its model was loading, so no session ever existed and
the conversation had nothing to recover. A session request that waited
for a load now detaches from the client socket and reaches the child
regardless, the session buffer receives the generation, and the resume
route answers 503 while the owner is loading so the client retries
instead of dropping its state. The WebUI persists the pending stream at
send time, quietly polls on 503, and attaches once the session exists.
Add a regression test that drops the client during the load window.

* ui: show the model load progress again after a page refresh

The resume wait was invisible, so a conversation refreshed while its
model was loading showed nothing until the first byte. On a 503 from
the resume probe, mark the conversation as loading again so the
assistant row persisted at send time renders the processing info, and
target the model frozen in the persisted stream state for the
progress, since the row has no model yet and the dropdown may not be
restored.

* fix CI

* fix CI bis
This commit is contained in:
Pascal
2026-07-27 07:34:47 +02:00
committed by GitHub
parent 88b47a755c
commit d73c1d6b22
11 changed files with 345 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/server/server-models.cpp
+48 -19
View File
@@ -1172,7 +1172,7 @@ bool server_models::ensure_model_ready(const std::string & name) {
return true;
}
server_http_res_ptr server_models::proxy_request(const server_http_req & req, const std::string & method, const std::string & name, bool update_last_used) {
server_http_res_ptr server_models::proxy_request(const server_http_req & req, const std::string & method, const std::string & name, bool update_last_used, bool detached) {
auto meta = get_meta(name);
if (!meta.has_value()) {
throw std::runtime_error("model name=" + name + " is not found");
@@ -1198,7 +1198,10 @@ server_http_res_ptr server_models::proxy_request(const server_http_req & req, co
req.headers,
req.body,
req.files,
req.should_stop,
// a detached request belongs to a replay session that outlives the client socket:
// it reaches the child even when the downstream died during the load wait, the
// session buffer is the recipient and DELETE remains the stop
detached ? std::function<bool()>([]() { return false; }) : req.should_stop,
base_params.timeout_read,
base_params.timeout_write
);
@@ -1469,13 +1472,9 @@ static bool router_validate_model(std::string & name, server_models & models, bo
}
// resolve alias to canonical model name
name = meta->name;
if (models_autoload) {
models.ensure_model_ready(name);
} else {
if (!meta->is_running()) {
res_err(res, format_error_response("model is not loaded", ERROR_TYPE_INVALID_REQUEST));
return false;
}
if (!models_autoload && !meta->is_running()) {
res_err(res, format_error_response("model is not loaded", ERROR_TYPE_INVALID_REQUEST));
return false;
}
return true;
}
@@ -1568,6 +1567,9 @@ void server_models_routes::init_routes() {
if (!router_validate_model(name, models, autoload, error_res)) {
return error_res;
}
if (autoload) {
models.ensure_model_ready(name);
}
return models.proxy_request(req, method, name, false);
};
@@ -1581,12 +1583,23 @@ void server_models_routes::init_routes() {
return error_res;
}
// remember which child serves this conversation so the stream routes can route straight
// to it without polling, keyed on the exact conv id from the header
// to it without polling, keyed on the exact conv id from the header. registered before
// the load wait so a stop issued while the model loads can erase the entry and cancel
// this request instead of leaving an orphan generation
std::string conv_id = server_stream_conv_id_from_headers(req.headers);
if (!conv_id.empty()) {
models.conv_models.remember(conv_id, name);
uint64_t ticket = models.conv_models.remember(conv_id, name);
bool waited = autoload && models.ensure_model_ready(name);
if (ticket != 0 && !models.conv_models.alive(conv_id, ticket)) {
SRV_INF("request for conv_id=%s cancelled while model name=%s was loading\n",
conv_id.c_str(), name.c_str());
res_err(error_res, format_error_response(
"request cancelled by a stop while the model was loading", ERROR_TYPE_INVALID_REQUEST));
return error_res;
}
return models.proxy_request(req, method, name, true); // update last usage for POST request only
// a session request that waited for a load detaches from the client socket: the
// client may have dropped during the wait (page reload) and the session buffer must
// still receive the generation for a later resume
return models.proxy_request(req, method, name, true, waited && ticket != 0); // update last usage for POST request only
};
this->post_router_models_load = [this](const server_http_req & req) {
@@ -1779,7 +1792,7 @@ void server_models_routes::init_routes() {
};
this->router_stream_get = [this](const server_http_req & req) {
// GET /v1/stream/<conv_id>?from=N. resolve the owning child from the conv_id -> model
// GET /v1/stream?conv_id=<id>&from=N. resolve the owning child from the conv_id -> model
// map, 404 when nothing maps
auto res = std::make_unique<server_http_res>();
std::string conv_id = req.get_param("conv_id");
@@ -1789,13 +1802,24 @@ void server_models_routes::init_routes() {
}
std::optional<server_model_meta> owner = resolve_child_for_conv(models, conv_id);
if (!owner.has_value()) {
res_err(res, format_error_response("Stream not found or expired", ERROR_TYPE_NOT_FOUND));
// a registered conv whose model is still loading earns a retry: the session appears
// once the load ends and the pending request reaches the child
auto tracked = models.conv_models.lookup(conv_id);
auto meta = tracked.has_value() ? models.get_meta(*tracked) : std::nullopt;
bool transient = meta.has_value() && (meta->status == SERVER_MODEL_STATUS_LOADING ||
meta->status == SERVER_MODEL_STATUS_DOWNLOADING ||
meta->status == SERVER_MODEL_STATUS_DOWNLOADED);
if (transient) {
res_err(res, format_error_response("Stream owner model is loading, retry later", ERROR_TYPE_UNAVAILABLE));
} else {
res_err(res, format_error_response("Stream not found or expired", ERROR_TYPE_NOT_FOUND));
}
return res;
}
std::string from = req.get_param("from");
std::string child_path = "/v1/stream/" + encode_qs(conv_id);
std::string child_path = "/v1/stream?conv_id=" + encode_qs(conv_id);
if (!from.empty()) {
child_path += "?from=" + from;
child_path += "&from=" + from;
}
SRV_TRC("proxying stream resume to model %s on port %d, path=%s\n",
owner->name.c_str(), owner->port, child_path.c_str());
@@ -1875,7 +1899,7 @@ void server_models_routes::init_routes() {
};
this->router_stream_delete = [this](const server_http_req & req) {
// DELETE /v1/stream/<conv_id>. resolve the owning child via the map and forward only to
// DELETE /v1/stream?conv_id=<id>. resolve the owning child via the map and forward only to
// it, evict_and_cancel is idempotent on the child
auto res = std::make_unique<server_http_res>();
std::string conv_id = req.get_param("conv_id");
@@ -1883,7 +1907,7 @@ void server_models_routes::init_routes() {
res_err(res, format_error_response("Missing conversation id in path", ERROR_TYPE_INVALID_REQUEST));
return res;
}
std::string child_path = "/v1/stream/" + encode_qs(conv_id);
std::string child_path = "/v1/stream?conv_id=" + encode_qs(conv_id);
auto owner = resolve_child_for_conv(models, conv_id);
if (owner.has_value()) {
httplib::Client cli(CHILD_ADDR, owner->port);
@@ -1892,6 +1916,11 @@ void server_models_routes::init_routes() {
cli.set_write_timeout(0, STREAM_LOOKUP_TIMEOUT_MS * 1000);
auto resp = cli.Delete(child_path.c_str());
(void) resp; // the child logs its own miss when the session is unknown there
} else if (auto tracked = models.conv_models.lookup(conv_id); tracked.has_value()) {
// the entry exists but its model is still loading: the forget below erases it,
// which cancels the request parked in proxy_post before the generation starts
SRV_INF("router stop for conv_id=%s while model name=%s is loading, cancelling the pending request\n",
conv_id.c_str(), tracked->c_str());
} else {
SRV_WRN("router stop for unknown conv_id=%s, no owning child in the conv map\n",
conv_id.c_str());