mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-09-19 09:15:04 +02:00
kv-cache: clear the cache once when restoring a whole context
state_read walks the streams of the cache in turn, and for a whole-context restore each stream went through state_read_meta, which starts by calling clear(). clear() resets every stream at once, so each stream after the first threw away the streams already restored, and the K/V buffers with them. A non-unified cache holds one stream per sequence, so a context saved with N sequences in it came back with only the sequence in the last stream that carried any cells - the highest sequence id. A unified cache has one stream and never showed it. The cache is now emptied once, before the loop, which is what a whole-context restore means. A blob whose streams are all empty now empties the cache as well, where before it left the old contents in place.
This commit is contained in:
@@ -2071,6 +2071,13 @@ const slot_info_vec_t * sinfos_in) {
|
||||
throw std::runtime_error("n_stream mismatch");
|
||||
}
|
||||
|
||||
// a whole-context restore replaces every stream, so the cache is emptied once here. clear()
|
||||
// resets all streams at once, so doing this per stream below would throw away the streams
|
||||
// already read and leave only the last one
|
||||
if (seq_id == -1) {
|
||||
clear(true);
|
||||
}
|
||||
|
||||
for (uint32_t s = 0; s < n_stream; ++s) {
|
||||
uint32_t cell_count;
|
||||
io.read(&cell_count, sizeof(cell_count));
|
||||
@@ -2345,8 +2352,6 @@ bool llama_kv_cache::state_read_meta(llama_io_read_i & io, uint32_t strm, uint32
|
||||
return false;
|
||||
}
|
||||
|
||||
clear(true);
|
||||
|
||||
for (uint32_t i = 0; i < cell_count; ++i) {
|
||||
llama_pos pos;
|
||||
uint32_t n_seq_id;
|
||||
|
||||
Reference in New Issue
Block a user