* dspark: support speculators-format checkpoints (SpecForge exports)
Speculators-format DSpark drafts (e.g. SpecForge exports for the
Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in
three ways:
- the config nests the backbone hparams under transformer_layer_config
and gives the extract layers as aux_hidden_state_layer_ids
- the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus
token, not a prediction slot. Written as dflash.bonus_anchor; such
drafts build the block and read the mask positions exactly like
DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence
sampling comes from DSpark
- the draft output vocab may be reduced (draft_vocab_size < vocab_size)
with a d2t remap table. The converter expands lm_head/markov_w2 back
to the full vocab and synthesizes an lm_head bias of -1e9 on the rows
the draft cannot produce, so the runtime needs no d2t remapping. Such
drafts ship their own (now optional) token_embd/output tensors instead
of sharing the target's
Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical
with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6).
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5
* dspark: fold the speculators draft class into DSparkModel
One class now covers every DSpark variant. What used to pick the class is
a single flag, because the arch name turns out to be the only thing that
separates the two families: SpecForge also exports a flat schema that
carries no speculators_* fields yet still uses the 1+N bonus-anchor block,
so keying on those fields would silently mis-read its drafts.
Also rename i0 to i_first_pred in the draft read loop and the Markov head,
and give the head a real bonus_anchor bool instead of testing i0 > 0.
Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts
produces byte-identical GGUFs. The one behaviour change is that the
markov_head_type check now also covers the DeepSpec checkpoints, which
previously skipped it.
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Opus 5
* dspark: address review comments
- rename bonus_anchor to sample_from_anchor (GGUF key and code), matching
the checkpoint config field; absent key still means anchor-first
- rework the reduced draft vocab to match EAGLE3: d2t is written as I64
absolute target ids and the logits are scattered at runtime, instead of
expanding lm_head/markov_w2 and synthesizing an output bias at conversion
- move the t2d skip to modify_tensors, like EAGLE3
- drop _is_specforge: the arch name only picks the sample_from_anchor
default, embed/lm_head sharing is decided by the draft vocab size
- deduplicate the tok_embd create_tensor left behind by the rebase
Verified with the RedHat gemma-4-31b speculator draft: greedy output is
byte-identical with and without the draft; acceptance 0.26 (n_max 7).
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5
* dspark: fold the sample_from_anchor read into the block_size block
* dspark: fix flake8 continuation indent
* clean up
* dspark: key the sample_from_anchor default off the export format
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
* dspark: drop t2d in filter_tensors
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
* dspark: map model.lm_head instead of bypassing the dflash prefix
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
---------
Co-authored-by: desovo7 <942845546@qq.com>
Co-authored-by: ruixiang63 <wangruixiang07@outlook.com>
bytes_to_unicode was removed from transformers.models.gpt2.tokenization_gpt2
in huggingface/transformers#40936, but it had already been copied into
transformers.convert_slow_tokenizer in huggingface/transformers#30334
(transformers 4.54.1), so import it directly from there.
Applies the same fix to chatglm.py.
* spec: add DSpark speculative decoding
DSpark (DeepSpec, 2026) on top of the merged DFlash drafter. It reuses the
DFlash encoder/decoder graph, target feature extraction and KV-cache injection,
and the verify/accept path unchanged; the draft model is a new "dspark" arch
adding a low-rank Markov head (markov_w1/w2) and an optional (unused here)
confidence head. No new public APIs.
The proposal is the only change: the block is anchor-first (position 0 already
predicts the first draft) and the decoder graph applies a semi-autoregressive,
previous-token conditioned logit bias in-graph, chained per block position:
logits'(i) = logits(i) + markov_w2 . markov_w1[prev(i)]
prev(0) = the block's anchor token, prev(i>0) = argmax(logits'(i-1))
vectorized across all blocks in the batch; the anchors are fed through a
dedicated graph input (token 0 of every block). Greedy stays lossless
(verify unchanged, same as DFlash).
- new arch "dspark" (llama_model_dspark : llama_model_dflash, reuses the graph,
loads the markov/confidence tensors; shares the target's embed/lm_head).
- Qwen3DSparkModel converter.
- new spec type "draft-dspark" (common_speculative_impl_draft_dspark :
common_speculative_impl_draft_dflash, overrides draft() only: submits whole
anchor-first blocks and greedily reads back the biased logits).
* spec: read draft block size in the dflash impl
* docs: add DSpark section to speculative.md
* spec: keep dspark block size read in the dspark impl
* dspark : add TODOs for incomplete parts
- confidence head is loaded but not used yet
- confidence-scheduled prefix pruning is not implemented
- the in-graph Markov chain is greedy-only
- only Qwen3 backbones are supported for now (also noted in docs)
* spec: fold DSpark into the DFlash arch
Address review: drop LLM_ARCH_DSPARK and the dspark.block_size /
markov_rank GGUF keys. A DSpark draft now converts to a DFlash GGUF;
the Markov head tensors are detected by presence (like eagle3 d2t),
block_size is read from the existing dflash.block_size key, and the
block anchors are taken as a strided view of the decoder's token
input instead of a separate graph input.
* spec: add confidence-based draft pruning for DSpark
The DSpark confidence head predicts per-position acceptance of the
drafted block. --spec-draft-conf-min truncates the block at the first
position below the threshold (default 0 = disabled).
* fold the dspark impl into dflash, selected by spec type
* address review comments
* dspark: clean up and improve naming
* update readme
* remove trailing whitespace
* dflash: draft full n_max blocks, defer dp.n_max to the central truncation
The DSpark markov head views the draft batch as a uniform [n_seqs x block]
grid, but the per-seq dp.n_max clamp could produce blocks of different
sizes, silently corrupting the strided views and the resulting logits.
Drop the clamp and always draft the full n_max block for every sequence:
dp.n_max is already enforced by the central truncation in
common_speculative_draft(), the same way eagle3 handles it.
Co-authored-by: Zaire404 <3147879462@qq.com>
* dflash: assert the markov head block-uniformity invariant, require the conf head
With the draft batch always submitting equal-size n_max blocks, a
non-divisible token count can only mean the batch was split across
ubatches or a caller broke the layout - fail loudly instead of silently
dropping the markov bias. The block_drafts > block_size early return
stays: worst-case graph reserve passes legitimately build with
n_seq_tokens > block_size.
Also make conf_proj required when the markov head is present: the
confidence head is part of the DSpark checkpoint format, and a missing
head would otherwise leave --spec-draft-conf-min silently reading stale
embeddings instead of confidences.
Co-authored-by: Zaire404 <3147879462@qq.com>
* dspark: fold conf_min into p_min
p_min and conf_min express the same thing - the minimum predicted
survival probability for a drafted position - differing only in how the
estimate is obtained: token probability for regular drafters, the
trained confidence head for DSpark. The DSpark readback never used
p_min, so reuse it for the confidence threshold and drop the separate
--spec-draft-conf-min flag. Both defaulted to 0 (disabled), so behavior
is unchanged.
Co-authored-by: Zaire404 <3147879462@qq.com>
* dflash: note the confidence broadcast workaround
Requested in review: the ggml_repeat only adapts the [1, n_tok]
confidences to the n_embd-wide embd_nextn transport so that
llama_get_embeddings_nextn can be reused - not a placeholder.
Co-authored-by: Zaire404 <3147879462@qq.com>
* cont : clarify
[no ci]
---------
Co-authored-by: Ruixiang Wang <wangruixiang07@outlook.com>
Co-authored-by: Zaire404 <3147879462@qq.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
- Add a supports_mtp_export capability to ModelBase so architectures can opt
into --mtp and --no-mtp without extending a central class allowlist.
- Enable the capability for the existing Qwen3.5/3.6 and Step3.5/3.7
implementations, and for HY V3, whose converter already supports
filtering the appended MTP layers.
* Refactored Compressed Tensors NVFP4 support for new base.py
* Support compressed-tensors NVFP4 conversion
* Moved Qwen MTP remap into filter_tensors
* simplify
* pathlib no longer used
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* spec: support MTP
* fix batch size
* rename files
* cont : simplify (#7)
* MTP: clean-up (#9)
* MTP: clean-up
* review: use llama_context_type instead of llama_graph_type
* review: remove llama_model_has_mtp
* review: fix convert issues
* convert: fix pycheck
* review: formatting
* use `mtp-` for identifying mtp models
* convert: fix mtp conversion
* mtp -> draft-mtp
* remove unused llama_arch
* add need_embd in speculative
* llama: allow partial seq_rm for GDN models for speculative decoding
Currently speculative checkpoint needs to restart from a checkpoint
after some draft tokens are not accepted, this leads to some wastage in
running the target again. This PR adds the ability to rollback upto
`draft_max` by storing the GDN intermediates.
* fix pending state
* vulkan: add GDN partial rollback
* meta: extend check to axis 1
* metal: add GDN partial rollback
Extend the gated delta net kernel to store intermediate states for
partial rollback support on the Metal backend.
- Add K (snapshot slot count) as a function constant
- Read input state from slot 0 of the 3D state tensor
- Write intermediate states to different slots during token loop
- For K=1, maintain backward-compatible single-slot behavior
Ref: https://github.com/ggml-org/llama.cpp/commit/8c05923630110223669f069af2000e9cf10c02bc
Assisted-by: llama.cpp:local pi
* delta_net_base: use ggml_pad instead of new_tensor
* review: add need_rs_seq
* review: rename part_bounded to n_rs
* review: deslop comments
* review: rename, add asserts
* server : adjust checkpoint logic (#11)
* server : adjust checkpoint logic
* cont : rm asserts
* server-context: fix early exit
* spec : fix compatibility with n-gram and add TODOs (#13)
* metal : cleanup
* llama : fix faulty bitwise check in recurrent memory
* server : disable RS-based MTP in combination with other spec types
* spec : add TODOs
* cont : fix comment
* cont : update comment
* common : fix logic for ngram + mtp compat
* llama-memory: enable checkpointing with partial rollback
* cont: add test-case for loading into a dirty ctx
* llama-memory-recurrent: clear rs_idx in clear
* download: fix mtp path
* llama-arch: fix enorm op
* docs: update docs
* conversion: fix type annotations
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* move conversion code to a dedicated conversion directory and split the files akin to the src/models architecture
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Tiwei Bie
Tarek Dakhran
王金旭
Xuan-Son Nguyen
lnigam
Caleb DeLeeuw
Masashi Yoshimura
Ruixiang Wang
Thiago Padilha
Sigbjørn Skjæret
Michael Wand
Incarnas
Aman Gupta
Piotr Wilkin (ilintar)