mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2026-09-04 20:11:17 +02:00
model : BailingMoE3 Support (#26608)
* Adding support for bailingmoe3
* Adds speculative decoding support
* Make BailingMoE3 safe gate metadata optional
* bailingmoe3: apply trained SwiGLU clamps
* common: fix Bailing V3 tool argument parsing
* llama-model-saver, instantiate float vector metadata writer
* bailingmoe3: support Q-LoRA (Ling-3.0-tiny)
Ling-3.0-flash sets q_lora_rank: None and projects Q directly, so the current
implementation loads a single ATTN_Q tensor. Ling-3.0-tiny sets q_lora_rank: 256
and routes Q through a LoRA bottleneck instead:
q_a_proj -> q_a_layernorm -> q_b_proj
Conversion therefore failed with:
ValueError: Can not map tensor 'model.layers.3.attention.q_a_layernorm.weight'
Add the missing path, mirroring the existing deepseek2 MLA implementation:
* constants.py - add ATTN_Q_A / ATTN_Q_B / ATTN_Q_A_NORM to BAILINGMOE3
* tensor_mapping.py - map model.layers.{bid}.attention.q_{a,b}_proj and
q_a_layernorm
* conversion - emit attention.q_lora_rank when the config has it
* bailingmoe3.cpp - read n_lora_q; create the Q-LoRA tensors and build Q
through the bottleneck when q_lora_rank > 0
Everything is gated on q_lora_rank > 0. Ling-3.0-flash's config has no
q_lora_rank, the converter only emits the key when present, hparams.n_lora_q
defaults to 0, and get_key(..., required=false) leaves the target untouched when
the key is absent - so flash keeps taking the existing direct-Q branch.
The LoRA path produces the same shape as the direct projection, so the
nope/rope split, RoPE application and wk_b absorption downstream are unchanged.
* small mtp change
* bailingmoe3: support separate MTP GGUF and Q-LoRA MTP
* gguf: remove duplicate add_kda_gate_lower_bound definition
---------
Co-authored-by: bloomer <bloomer@booper.brushtail.me>
Co-authored-by: Dyluhn <dylanranejohnston1@gmail.com>
This commit is contained in:
@@ -193,6 +193,14 @@ static std::vector<std::function<void(const common_chat_template & tmpl, autopar
|
||||
LOG_DBG(ANSI_ORANGE "[Patch: Laguna]\n" ANSI_RESET);
|
||||
}
|
||||
},
|
||||
// Bailing V3
|
||||
[](const common_chat_template & tmpl, autoparser & analysis) -> void {
|
||||
if (tmpl.src.find("Bailing V3 chat template") != std::string::npos) {
|
||||
analysis.tools.arguments.value_suffix = trim_whitespace(analysis.tools.arguments.value_suffix);
|
||||
analysis.tools.arguments.tolerate_intertag_whitespace = true;
|
||||
LOG_DBG(ANSI_ORANGE "[Patch: Bailing V3]\n" ANSI_RESET);
|
||||
}
|
||||
},
|
||||
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user