Xuan-Son Nguyen 160bd031b2 server: fix LRU hang on multiple requests same model (#28539) ...

* server: fix LRU hang on multiple requests same model

* server: keep a queued model out of the victim pool until its waiters leave

A waiter that gave up while its model was still loading left the
model idle with no request behind it, and nothing recounted the free
slots, so a second request queued behind it stayed queued forever.
tick() was only driven by requests: join, claim and the end of a
proxied request.

Keep the queue entry alive after a successful claim so the model
coming up is never picked as a victim before its waiters use it, and
recount the slots on every status change and whenever a waiter
abandons the queue. The model is then evicted as soon as it comes up
with nobody left to serve.

---------

Co-authored-by: Pascal <admin@serveurperso.com>

2026-09-07 15:50:46 +02:00

..

test_basic.py

server: remove all internal mentions about "webui" (#24817)

2026-06-19 22:12:46 +02:00

test_chat_completion.py

server: add "verbose" field to schema (#24864)

2026-06-21 13:03:14 +02:00

test_compat_anthropic.py

server : use pytest-xdist for server tests (#28298)

2026-09-03 15:04:30 +02:00

test_compat_gcp.py

server: support Vertex AI compatible API (#22545)

2026-05-08 15:23:04 +02:00

test_compat_oai_responses.py

server : add timings and progress to /responses API stream (#25348)

2026-07-07 16:13:03 +02:00

test_completion.py

server : add missing task parameters(adaptive_target, adaptive_decay) in generation_settings (#25830)

2026-07-25 20:53:08 +02:00

test_ctx_shift.py

memory : remove KV cache size padding (#16812)

2025-10-28 20:19:44 +02:00

test_embedding.py

llama : fix pooling assertion crash in chunked GDN detection path (#20468)

2026-03-13 20:53:42 +02:00

test_ignore_eos.py

server: respect the ignore eos flag (#21203)

2026-04-08 17:12:15 +02:00

test_infill.py

server : support unified cache across slots (#16736)

2025-11-02 18:14:04 +02:00

test_kv_keep_only_active.py

server: rename debug tags to match --cache-idle-slots naming (#22292)

2026-04-24 09:28:44 +03:00

test_lora.py

server : disable context shift by default (#15416)

2025-08-19 16:46:37 +03:00

test_mcp_servers.py

server : use pytest-xdist for server tests (#28298)

2026-09-03 15:04:30 +02:00

test_metrics.py

server: refactor + correctness fixes for metrics (#26920)

2026-08-13 10:02:01 +02:00

test_proxy.py

server: fix empty response for /cors-proxy (#26656)

2026-08-06 15:07:22 +02:00

test_rerank.py

server / ranking : add sorting and management of top_n (#16403)

2025-10-11 16:39:04 +03:00

test_router.py

server: fix LRU hang on multiple requests same model (#28539)

2026-09-07 15:50:46 +02:00

test_security.py

server : make models endpoints private when authentication is enabled (#26347)

2026-08-19 20:44:42 +02:00

test_sleep.py

server: refactor sleep handling, allow access /metrics during sleep (#27376)

2026-08-19 20:48:09 +02:00

test_slot_save.py

server : use pytest-xdist for server tests (#28298)

2026-09-03 15:04:30 +02:00

test_speculative.py

spec: Add benchmark-only synthetic speculative acceptance options (#27711)

2026-08-27 13:53:42 +03:00

test_stream.py

server + ui: fix stream routes for model names containing a slash (#26137)

2026-07-27 07:34:47 +02:00

test_template.py

tests : use reasoning instead of reasoning_budget in server tests (#20432)

2026-03-12 13:41:01 +01:00

test_tokenize.py

server : disable context shift by default (#15416)

2025-08-19 16:46:37 +03:00

test_tool_call.py

server : use pytest-xdist for server tests (#28298)

2026-09-03 15:04:30 +02:00

test_tools_builtin.py

server : use pytest-xdist for server tests (#28298)

2026-09-03 15:04:30 +02:00

test_vision_api.py

server : accept data: URLs for input_video and input_audio (#27735)

2026-09-02 22:24:31 +02:00