llama: make the qwen4exp PLE n-gram history per context and serialise it

The PLE hash of a token mixes in the ple_ngram_size - 1 tokens before it, which
a decode ubatch does not carry, so they were remembered in a map on
llama_model_qwen4exp. That is the wrong owner twice over.

A llama_model is shared by every context that loads it, and the map was keyed
only by llama_seq_id, so two contexts running the same sequence id - two server
instances on one model, or a draft/target pair - overwrote each other's window.
The next_pos guard turned that into EOS padding instead of a crash, so it
degraded quality silently.

The map was also in no state blob: grep found ple_hist in neither
llama-kv-cache.cpp nor llama-memory-*.cpp nor llama-context.cpp. A restored
context therefore failed the next_pos check on its first ubatch and hashed the
first tokens after the restore against EOS padding. This is why a session blob
round-tripped byte for byte while the restored context computed different
logits: the state was never in the bytes.

It moves to llama_memory_hybrid_idx, which is per context, is the memory type
qwen4exp always builds, and already does the per-sequence bookkeeping this
needs. Every sequence operation now carries the window with it:

  seq_rm   a rewind (p1 < 0) truncates the window to the surviving prefix and
           moves next_pos to p0, so a rollback keeps exact context; a hole
           punched in the middle leaves the window non-contiguous, so it is
           dropped
  seq_cp   the destination inherits the source's window, truncated to the
           copied position range - a copied sequence continues with the same
           n-grams the source would have used
  seq_keep every other sequence's window is dropped, like its cells
  seq_add  a shift that moves the whole window keeps it and moves next_pos with
           it, which is the context-shift case; one that cuts through it drops
           it
  seq_div  positions stop being consecutive, so an overlapping window is
           dropped
  clear    everything is dropped

Dropping means next_pos = -1, which set_input turns into full EOS padding: the
same thing a fresh sequence gets, and the same thing this code did before it
followed the sequence operations at all, so no case is worse than before.

The state payload is a self-delimiting list, u32 count then per entry
{ i32 seq_id, i32 next_pos, u32 n_toks, i32 toks[n_toks] }, so a whole-context
save and a single-sequence save share one format and a single-sequence restore
can retarget the window at its destination seq_id. It is written after the
indexer section, last, for the same reason that one is: as a pure suffix an
older reader stops early instead of parsing these bytes as something else.

Unlike the indexer section it is not under LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY.
The window is recurrent state - it is the input the PLE convolution's own
recurrent state is derived from - and the recurrent cache beside it is written
for partial checkpoints too. Gating it would leave the server's speculative
decoding checkpoints restoring the conv state without the window that produced
it.

No further version bump: LLAMA_SESSION_VERSION 10 and LLAMA_STATE_SEQ_VERSION 3
were introduced for the indexer section in the same unreleased series, and both
changes are qwen4exp-only additions to the same blob layout.

Also fixes the padding of a short window. set_input pads a window shorter than
ngram_size - 1 up to that length, but prev() indexes the snapshot with the most
recent token last, and resize() pads at the back, so the filler EOS landed where
the immediately preceding token belongs. It now pads at the front. A window is
short at a sequence start after a one-token prefill, and after a seq_rm rewind,
which the new bookkeeping makes common.

Every architecture other than qwen4exp builds llama_memory_hybrid rather than
llama_memory_hybrid_idx, has no PLE table and never asks for a history, so
nothing about its graph, its sequence operations or its state bytes changes.

(cherry picked from commit de170364c052c68fcf63285cc0028095edb9f23c)

src/llama-memory-hybrid-idx.cpp

@@ -2,11 +2,14 @@
 
 #include "llama-impl.h"
 #include "llama-batch.h"
+#include "llama-io.h"
 #include "llama-model.h"
 
 #include <algorithm>
 #include <cassert>
 #include <cmath>
+#include <iterator>
+#include <stdexcept>
 
 //
 // llama_memory_hybrid_idx
@@ -139,6 +142,9 @@ void llama_memory_hybrid_idx::clear(bool data) {
     if (mem_idx) {
         mem_idx->clear(data);
     }
+
+    // [TAG_PLE_HISTORY] every sequence is gone, so no window is trusted any more
+    ple_hist.clear();
 }
 
 bool llama_memory_hybrid_idx::seq_rm(llama_seq_id seq_id, llama_pos p0, llama_pos p1) {
@@ -152,6 +158,8 @@ bool llama_memory_hybrid_idx::seq_rm(llama_seq_id seq_id, llama_pos p0, llama_po
         mem_idx->seq_rm(seq_id, p0, p1);
     }
 
+    ple_hist_rm(seq_id, p0, p1);
+
     return get_mem_attn()->seq_rm(seq_id, p0, p1);
 }
 
@@ -161,6 +169,8 @@ void llama_memory_hybrid_idx::seq_cp(llama_seq_id seq_id_src, llama_seq_id seq_i
     if (mem_idx) {
         mem_idx->seq_cp(seq_id_src, seq_id_dst, p0, p1);
     }
+
+    ple_hist_cp(seq_id_src, seq_id_dst, p0, p1);
 }
 
 void llama_memory_hybrid_idx::seq_keep(llama_seq_id seq_id) {
@@ -169,6 +179,8 @@ void llama_memory_hybrid_idx::seq_keep(llama_seq_id seq_id) {
     if (mem_idx) {
         mem_idx->seq_keep(seq_id);
     }
+
+    ple_hist_keep(seq_id);
 }
 
 void llama_memory_hybrid_idx::seq_add(llama_seq_id seq_id, llama_pos p0, llama_pos p1, llama_pos shift) {
@@ -177,6 +189,8 @@ void llama_memory_hybrid_idx::seq_add(llama_seq_id seq_id, llama_pos p0, llama_p
     if (mem_idx) {
         mem_idx->seq_add(seq_id, p0, p1, shift);
     }
+
+    ple_hist_add(seq_id, p0, p1, shift);
 }
 
 void llama_memory_hybrid_idx::seq_div(llama_seq_id seq_id, llama_pos p0, llama_pos p1, int d) {
@@ -185,6 +199,8 @@ void llama_memory_hybrid_idx::seq_div(llama_seq_id seq_id, llama_pos p0, llama_p
     if (mem_idx) {
         mem_idx->seq_div(seq_id, p0, p1, d);
     }
+
+    ple_hist_div(seq_id, p0, p1);
 }
 
 std::map<ggml_backend_buffer_type_t, size_t> llama_memory_hybrid_idx::memory_breakdown() const {
@@ -199,6 +215,256 @@ std::map<ggml_backend_buffer_type_t, size_t> llama_memory_hybrid_idx::memory_bre
     return mb;
 }
 
+//
+// [TAG_PLE_HISTORY] per-sequence PLE n-gram history
+//
+// The window is only meaningful while it is contiguous with the position the sequence is
+// about to decode, so every operation below either rewrites it exactly or invalidates it.
+// Invalidating means next_pos = -1, which set_input turns into full EOS padding - the same
+// thing a fresh sequence gets, and the same thing this code did before it followed the
+// sequence operations at all. It is therefore never worse than the previous behaviour, and
+// it is exact in the cases that matter (a rewind to a prefix, a copied sequence, a context
+// shift).
+//
+
+llama_memory_hybrid_idx::ple_history & llama_memory_hybrid_idx::ple_hist_get(llama_seq_id seq_id) const {
+    return ple_hist[seq_id];
+}
+
+// first position still remembered by h
+static llama_pos ple_hist_beg(const llama_memory_hybrid_idx::ple_history & h) {
+    return h.next_pos - (llama_pos) h.toks.size();
+}
+
+static void ple_hist_invalidate(llama_memory_hybrid_idx::ple_history & h) {
+    h.next_pos = -1;
+    h.toks.clear();
+}
+
+// drop everything at position >= p, so the sequence now ends just before p
+static void ple_hist_truncate(llama_memory_hybrid_idx::ple_history & h, llama_pos p) {
+    const llama_pos beg = ple_hist_beg(h);
+
+    if (p <= beg) {
+        h.toks.clear();
+    } else if (p < h.next_pos) {
+        h.toks.resize((size_t) (p - beg));
+    }
+
+    h.next_pos = p;
+}
+
+void llama_memory_hybrid_idx::ple_hist_rm(llama_seq_id seq_id, llama_pos p0, llama_pos p1) {
+    if (seq_id < 0) {
+        for (auto & it : ple_hist) {
+            ple_hist_rm(it.first, p0, p1);
+        }
+        return;
+    }
+
+    auto it = ple_hist.find(seq_id);
+    if (it == ple_hist.end() || it->second.next_pos < 0) {
+        return;
+    }
+    auto & h = it->second;
+
+    if (p0 <= 0 && p1 < 0) {
+        // the whole sequence is gone
+        ple_hist.erase(it);
+        return;
+    }
+
+    if (p1 < 0) {
+        // a rewind: the sequence now ends at p0 and the surviving prefix of the window is
+        // still contiguous with it, which is the case a session rollback actually hits
+        if (p0 < h.next_pos) {
+            ple_hist_truncate(h, p0);
+        }
+        return;
+    }
+
+    // a hole punched somewhere in the middle. seq_rm does not renumber what follows, so a
+    // window that overlaps the hole is no longer a run of consecutive positions
+    if (p1 > ple_hist_beg(h) && p0 < h.next_pos) {
+        ple_hist_invalidate(h);
+    }
+}
+
+void llama_memory_hybrid_idx::ple_hist_cp(llama_seq_id seq_id_src, llama_seq_id seq_id_dst, llama_pos p0, llama_pos p1) {
+    if (seq_id_src == seq_id_dst) {
+        return;
+    }
+
+    // whatever the destination had is replaced by the copied range, exactly as its cells are
+    ple_hist.erase(seq_id_dst);
+
+    auto it = ple_hist.find(seq_id_src);
+    if (it == ple_hist.end() || it->second.next_pos < 0) {
+        return;
+    }
+
+    ple_history h = it->second;
+
+    if (p1 >= 0 && p1 < h.next_pos) {
+        ple_hist_truncate(h, p1);
+    }
+
+    // positions below p0 were not copied, so for the destination they are before the start
+    // of the sequence, which the hash already reads as EOS
+    const llama_pos lo = p0 < 0 ? 0 : p0;
+    if (lo > ple_hist_beg(h)) {
+        const llama_pos drop = std::min<llama_pos>(lo - ple_hist_beg(h), (llama_pos) h.toks.size());
+        h.toks.erase(h.toks.begin(), h.toks.begin() + (size_t) drop);
+    }
+
+    ple_hist[seq_id_dst] = std::move(h);
+}
+
+void llama_memory_hybrid_idx::ple_hist_keep(llama_seq_id seq_id) {
+    for (auto it = ple_hist.begin(); it != ple_hist.end(); ) {
+        it = it->first == seq_id ? std::next(it) : ple_hist.erase(it);
+    }
+}
+
+void llama_memory_hybrid_idx::ple_hist_add(llama_seq_id seq_id, llama_pos p0, llama_pos p1, llama_pos shift) {
+    if (seq_id < 0) {
+        for (auto & it : ple_hist) {
+            ple_hist_add(it.first, p0, p1, shift);
+        }
+        return;
+    }
+
+    auto it = ple_hist.find(seq_id);
+    if (it == ple_hist.end() || it->second.next_pos < 0) {
+        return;
+    }
+    auto & h = it->second;
+
+    const llama_pos beg = ple_hist_beg(h);
+    const llama_pos lo  = p0 < 0 ? 0 : p0;
+
+    if (p1 >= 0 && p1 <= beg) {
+        // entirely below the window: the tokens we remember keep their positions
+        return;
+    }
+    if (lo >= h.next_pos) {
+        // entirely above the window: nothing we remember moves
+        return;
+    }
+    if (lo <= beg && (p1 < 0 || p1 >= h.next_pos)) {
+        // the whole window moves as one, so it stays a run of consecutive positions.
+        // this is the context-shift case
+        if (beg + shift < 0) {
+            ple_hist_invalidate(h);
+        } else {
+            h.next_pos += shift;
+        }
+        return;
+    }
+
+    // the shift cuts through the window and breaks its contiguity
+    ple_hist_invalidate(h);
+}
+
+void llama_memory_hybrid_idx::ple_hist_div(llama_seq_id seq_id, llama_pos p0, llama_pos p1) {
+    if (seq_id < 0) {
+        for (auto & it : ple_hist) {
+            ple_hist_div(it.first, p0, p1);
+        }
+        return;
+    }
+
+    auto it = ple_hist.find(seq_id);
+    if (it == ple_hist.end() || it->second.next_pos < 0) {
+        return;
+    }
+    auto & h = it->second;
+
+    const llama_pos lo = p0 < 0 ? 0 : p0;
+
+    // dividing positions makes them non-consecutive, so any overlap ends the window. no
+    // caller in tree divides positions of an architecture that has a PLE table, but leaving
+    // this out would silently keep a window whose positions no longer line up
+    if ((p1 < 0 || p1 > ple_hist_beg(h)) && lo < h.next_pos) {
+        ple_hist_invalidate(h);
+    }
+}
+
+// Serialised as a self-delimiting list so that seq_id == -1 (whole context) and a single
+// sequence share one format:
+//   u32 n_entries
+//   n_entries * { i32 seq_id, i32 next_pos, u32 n_toks, i32 toks[n_toks] }
+// n_toks is at most ple_ngram_size - 1, so an entry is a handful of bytes.
+void llama_memory_hybrid_idx::ple_hist_state_write(llama_io_write_i & io, llama_seq_id seq_id) const {
+    uint32_t n_entries = 0;
+    for (const auto & it : ple_hist) {
+        if ((seq_id < 0 || it.first == seq_id) && it.second.next_pos >= 0) {
+            ++n_entries;
+        }
+    }
+
+    io.write(&n_entries, sizeof(n_entries));
+
+    for (const auto & it : ple_hist) {
+        if ((seq_id >= 0 && it.first != seq_id) || it.second.next_pos < 0) {
+            continue;
+        }
+
+        const int32_t  id       = it.first;
+        const int32_t  next_pos = it.second.next_pos;
+        const uint32_t n_toks   = (uint32_t) it.second.toks.size();
+
+        io.write(&id,       sizeof(id));
+        io.write(&next_pos, sizeof(next_pos));
+        io.write(&n_toks,   sizeof(n_toks));
+        if (n_toks > 0) {
+            io.write(it.second.toks.data(), n_toks*sizeof(llama_token));
+        }
+    }
+}
+
+void llama_memory_hybrid_idx::ple_hist_state_read(llama_io_read_i & io, llama_seq_id seq_id) {
+    uint32_t n_entries = 0;
+    io.read(&n_entries, sizeof(n_entries));
+
+    // a single-sequence restore replaces only that sequence's window; a whole-context one
+    // replaces the lot, matching what the caches around it do
+    if (seq_id >= 0) {
+        ple_hist.erase(seq_id);
+    } else {
+        ple_hist.clear();
+    }
+
+    for (uint32_t i = 0; i < n_entries; ++i) {
+        int32_t  id       = 0;
+        int32_t  next_pos = 0;
+        uint32_t n_toks   = 0;
+
+        io.read(&id,       sizeof(id));
+        io.read(&next_pos, sizeof(next_pos));
+        io.read(&n_toks,   sizeof(n_toks));
+
+        // the window is never longer than ple_ngram_size - 1; anything else is a corrupt or
+        // mismatched blob, and reading it would size an allocation from the file
+        if (n_toks > 64) {
+            throw std::runtime_error("qwen4exp PLE history: implausible token count in state blob");
+        }
+
+        std::vector<llama_token> toks(n_toks);
+        if (n_toks > 0) {
+            io.read(toks.data(), n_toks*sizeof(llama_token));
+        }
+
+        // a single-sequence restore may target a different seq_id than the one it was saved
+        // from, so the destination wins over the stored id
+        const llama_seq_id dst = seq_id >= 0 ? seq_id : (llama_seq_id) id;
+
+        auto & h = ple_hist[dst];
+        h.next_pos = next_pos;
+        h.toks     = std::move(toks);
+    }
+}
+
 void llama_memory_hybrid_idx::state_write(llama_io_write_i & io, llama_seq_id seq_id, llama_state_seq_flags flags) const {
     llama_memory_hybrid::state_write(io, seq_id, flags);
 
@@ -216,6 +482,17 @@ void llama_memory_hybrid_idx::state_write(llama_io_write_i & io, llama_seq_id se
             mem_idx->state_write(io, seq_id, flags);
         }
     }
+
+    // [TAG_PLE_HISTORY]
+    // last again, for the same reason the indexer section is: a pure suffix, so a reader
+    //   that does not expect it stops early rather than parsing these bytes as something
+    //   else. This is written after the indexer section because it is the newer of the two.
+    // unlike the indexer this is NOT under the PARTIAL_ONLY gate. The n-gram window is
+    //   recurrent state, not a token-level cache - it is the input the PLE convolution's
+    //   own recurrent state is derived from - and the recurrent cache next to it is written
+    //   for partial checkpoints too. Skipping it would leave the server's speculative
+    //   decoding checkpoints restoring the conv state without the window that produced it.
+    ple_hist_state_write(io, seq_id);
 }
 
 void llama_memory_hybrid_idx::state_read(llama_io_read_i & io, llama_seq_id seq_id, llama_state_seq_flags flags) {
@@ -233,6 +510,9 @@ void llama_memory_hybrid_idx::state_read(llama_io_read_i & io, llama_seq_id seq_
             mem_idx->state_read(io, seq_id, flags);
         }
     }
+
+    // [TAG_PLE_HISTORY] must mirror the write order above
+    ple_hist_state_read(io, seq_id);
 }
 
 llama_kv_cache * llama_memory_hybrid_idx::get_mem_idx() const {
@@ -311,6 +591,12 @@ uint32_t llama_memory_hybrid_idx_context::get_n_stream() const {
     return ns_ubatch[i_cur];
 }
 
+llama_memory_hybrid_idx::ple_history & llama_memory_hybrid_idx_context::get_ple_hist(llama_seq_id seq_id) const {
+    GGML_ASSERT(mem != nullptr);
+
+    return mem->ple_hist_get(seq_id);
+}
+
 void llama_memory_hybrid_idx_context::set_input_qsa(
         ggml_tensor * cell_blk,
         ggml_tensor * blk_cells,

src/llama-memory-hybrid-idx.h

@@ -3,6 +3,7 @@
 #include "llama-memory-hybrid.h"
 
 #include <memory>
+#include <unordered_map>
 #include <vector>
 
 //
@@ -86,12 +87,54 @@ public:
 
     llama_kv_cache * get_mem_idx() const;   // nullptr when the model carries no indexer
 
+    // [TAG_PLE_HISTORY]
+    // The qwen4exp PLE hash of a token mixes in the ple_ngram_size - 1 tokens before it.
+    // A decode ubatch does not carry them, so they are remembered here (vLLM's
+    // ngram_context).
+    //
+    // This lives on the memory rather than on llama_model because it is per-context
+    // per-sequence state, not a model weight: a llama_model is shared by every context
+    // that loads it, so a map on the model keyed only by llama_seq_id let two contexts
+    // (two server instances on one model, or a draft/target pair) overwrite each other's
+    // history. It is also state that has to survive save/restore and to follow seq_rm,
+    // seq_cp, seq_keep, seq_add and seq_div, and this class already does exactly that
+    // bookkeeping for the caches next to it.
+    struct ple_history {
+        // position the next token of this sequence must have. -1 means "unknown": the
+        // window is not trusted and the hash falls back to EOS padding.
+        llama_pos next_pos = -1;
+
+        // the tokens at positions [next_pos - toks.size(), next_pos), oldest first.
+        // never longer than ple_ngram_size - 1, and may be shorter near a sequence start
+        // or after a rewind - callers pad the missing front with EOS.
+        std::vector<llama_token> toks;
+    };
+
+    // history for seq_id, default-constructed (and therefore untrusted) on first use.
+    // const + mutable because it is read and updated from set_input, which runs off a
+    // const memory context.
+    ple_history & ple_hist_get(llama_seq_id seq_id) const;
+
 private:
     // the indexer cache stores only one key head per layer, so it needs its own hparams
     // instance: llama_kv_cache keeps a reference to whatever it is given
     llama_hparams hparams_idx;
 
     const std::unique_ptr<llama_kv_cache> mem_idx;
+
+    // [TAG_PLE_HISTORY] empty for every architecture but qwen4exp, which is the only one
+    // whose graph asks for a history
+    mutable std::unordered_map<llama_seq_id, ple_history> ple_hist;
+
+    // the seq_* halves of the history bookkeeping, one per llama_memory_i operation
+    void ple_hist_rm  (llama_seq_id seq_id,                              llama_pos p0, llama_pos p1);
+    void ple_hist_cp  (llama_seq_id seq_id_src, llama_seq_id seq_id_dst, llama_pos p0, llama_pos p1);
+    void ple_hist_keep(llama_seq_id seq_id);
+    void ple_hist_add (llama_seq_id seq_id,                              llama_pos p0, llama_pos p1, llama_pos shift);
+    void ple_hist_div (llama_seq_id seq_id,                              llama_pos p0, llama_pos p1);
+
+    void ple_hist_state_write(llama_io_write_i & io, llama_seq_id seq_id) const;
+    void ple_hist_state_read (llama_io_read_i  & io, llama_seq_id seq_id);
 };
 
 class llama_memory_hybrid_idx_context : public llama_memory_hybrid_context {
@@ -137,6 +180,9 @@ public:
     // streams in the current slot info, matching get_k/get_v's `ns`. 1 if unified.
     uint32_t get_n_stream() const;
 
+    // [TAG_PLE_HISTORY] the owning memory's per-sequence n-gram history, for set_input
+    llama_memory_hybrid_idx::ple_history & get_ple_hist(llama_seq_id seq_id) const;
+
     // block-compressed sparse attention (qwen4exp QSA) over the indexer cache's cells.
     // blocks cut the *position* line, not the cell array, so nothing assumes a contiguous
     // layout:

src/models/models.h

@@ -2278,14 +2278,6 @@ struct llama_model_qwen35 : public llama_model_base {
 struct llama_model_qwen4exp : public llama_model_base {
     llama_model_qwen4exp(const struct llama_model_params & params) : llama_model_base(params) {}
 
-    // PLE predecessors are absent from a decode ubatch, so remember them here
-    // (vLLM's ngram_context). next_pos guards it: a mismatch means the sequence
-    // was reset or rewound, and the hash falls back to EOS padding.
-    struct ple_history {
-        llama_pos                next_pos = -1;
-        std::vector<llama_token> toks;
-    };
-    mutable std::unordered_map<llama_seq_id, ple_history> ple_hist;
     void load_arch_hparams(llama_model_loader & ml) override;
     void load_arch_tensors(llama_model_loader & ml) override;
 
@@ -2367,6 +2359,7 @@ struct llama_model_qwen4exp : public llama_model_base {
 
         ggml_tensor * build_ple(
              llm_graph_input_rs * inp,
+  const llama_memory_hybrid_idx_context * mctx_hyb,
                     ggml_tensor * hidden,
                             int   il);
 

src/models/qwen4exp.cpp

@@ -2,6 +2,8 @@
 #include "llama-memory-hybrid-idx.h"
 #include "llama-memory-recurrent.h"
 
+#include <algorithm>
+
 void llama_model_qwen4exp::load_arch_hparams(llama_model_loader & ml) {
     ml.get_key(LLM_KV_EXPERT_FEED_FORWARD_LENGTH,        hparams.n_ff_exp, false);
     ml.get_key(LLM_KV_EXPERT_SHARED_FEED_FORWARD_LENGTH, hparams.n_ff_shexp, false);
@@ -298,7 +300,7 @@ llama_model_qwen4exp::graph::graph(const llama_model & model, const llm_graph_pa
         res->t_layer_inp[il] = res_hc;
 
         if (hparams.is_ple(il)) {
-            res_hc = build_ple(inp->get_recr(), res_hc, il);
+            res_hc = build_ple(inp->get_recr(), mctx_hyb, res_hc, il);
         }
 
         ggml_tensor * inject = nullptr;
@@ -887,7 +889,8 @@ ggml_tensor * llama_model_qwen4exp::graph::build_layer_ffn(ggml_tensor * cur, co
 
 class llm_graph_input_ple : public llm_graph_input_i {
 public:
-    llm_graph_input_ple(const llama_model_qwen4exp & pmodel) : pmodel(pmodel) {}
+    llm_graph_input_ple(const llama_model_qwen4exp & pmodel,
+                        const llama_memory_hybrid_idx_context * mctx) : pmodel(pmodel), mctx(mctx) {}
     virtual ~llm_graph_input_ple() = default;
 
     void set_input(const llama_ubatch * ubatch) override;
@@ -895,6 +898,12 @@ public:
     ggml_tensor * rows = nullptr;   // I32 [ple_n_heads * n_tokens]
 
     const llama_model_qwen4exp & pmodel;
+
+    // [TAG_PLE_HISTORY] the token history lives on the memory, which is per context.
+    // On the model it was shared by every context that loaded the same weights, so two
+    // contexts running the same seq_id overwrote each other's window, and it took part in
+    // no state blob at all.
+    const llama_memory_hybrid_idx_context * mctx;
 };
 
 void llm_graph_input_ple::set_input(const llama_ubatch * ubatch) {
@@ -928,23 +937,35 @@ void llm_graph_input_ple::set_input(const llama_ubatch * ubatch) {
 
     // Missing predecessors come from per-sequence history (vLLM's ngram_context),
     // trusted only when contiguous with the incoming position, else EOS padding.
-    auto & hist_map = pmodel.ple_hist;
+    GGML_ASSERT(mctx != nullptr);
 
     // Snapshot the incoming history before touching it. Reading and updating in
     // the same pass would let a token near the start of the ubatch pick up an
     // earlier token of this same ubatch as if it were prior context.
+    //
+    // The snapshot is always exactly n_gram - 1 long, EOS-padded at the FRONT, because
+    // prev() below indexes it with the most recent token last. A history shorter than
+    // n_gram - 1 - a sequence that has decoded only one or two tokens, or one rewound by
+    // seq_rm - used to be padded at the back by resize(), which put the EOS filler where
+    // the immediately preceding token belongs and read the real token as older than it is.
     std::unordered_map<llama_seq_id, std::vector<llama_token>> snap;
     for (int64_t i = 0; i < n_tokens; ++i) {
         const llama_seq_id seq = ubatch->seq_id[i][0];
         if (snap.count(seq)) {
             continue;
         }
-        auto & h = hist_map[seq];
+        auto & h = mctx->get_ple_hist(seq);
         if (h.next_pos != ubatch->pos[i]) {
-            h.toks.assign(n_gram - 1, eos);
+            h.next_pos = ubatch->pos[i];
+            h.toks.clear();
         }
-        h.toks.resize(n_gram - 1, eos);
-        snap[seq] = h.toks;
+        if ((int64_t) h.toks.size() > n_gram - 1) {
+            h.toks.erase(h.toks.begin(), h.toks.end() - (n_gram - 1));
+        }
+
+        std::vector<llama_token> padded(n_gram - 1, (llama_token) eos);
+        std::copy(h.toks.begin(), h.toks.end(), padded.end() - (int64_t) h.toks.size());
+        snap[seq] = std::move(padded);
     }
 
     for (int64_t i = 0; i < n_tokens; ++i) {
@@ -995,7 +1016,7 @@ void llm_graph_input_ple::set_input(const llama_ubatch * ubatch) {
             }
         }
 
-        auto & h = hist_map[seq];
+        auto & h = mctx->get_ple_hist(seq);
         h.toks.push_back(tok_of(i));
         if ((int64_t) h.toks.size() > n_gram - 1) {
             h.toks.erase(h.toks.begin(), h.toks.end() - (n_gram - 1));
@@ -1066,6 +1087,7 @@ ggml_tensor * llama_model_qwen4exp::graph::build_conv_state_at(
 
 ggml_tensor * llama_model_qwen4exp::graph::build_ple(
         llm_graph_input_rs * inp,
+        const llama_memory_hybrid_idx_context * mctx_hyb,
         ggml_tensor *        hidden,
         int                  il) {
     GGML_UNUSED(inp);
@@ -1075,7 +1097,7 @@ ggml_tensor * llama_model_qwen4exp::graph::build_ple(
     const int64_t n_heads = hparams.ple_n_heads;
 
     auto ple_inp = std::make_unique<llm_graph_input_ple>(
-            static_cast<const llama_model_qwen4exp &>(model));
+            static_cast<const llama_model_qwen4exp &>(model), mctx_hyb);
 
     ple_inp->rows = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, n_heads * n_tokens);
     ggml_set_input(ple_inp->rows);