mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-09-19 17:24:57 +02:00
quantize: fall back to F16 for 32-block types with an odd ncols
tensor_type_fallback demotes a tensor whose ncols is not a multiple of the target's block size, but its switch only enumerates the 256-block types. A target that is already a 32-block type (iq4_nl, q4_0, q5_0, q8_0, ...) falls into default: and throws, even though the function already knows how to answer that case: the ncols check right below the switch resolves an unrepresentable shape to F16. Route those types into that check instead of throwing. Only paths that abort today change, so no quantization that currently succeeds is affected. Found on a 4-wide depthwise conv kernel. llama-quantize reported nothing but "failed to quantize model from ...", with no tensor name and no exception text, which made a quant recipe that had simply not pinned the tensor look like a corrupt model. It now names the tensor and continues.
This commit is contained in:
committed by
Daniel Han
parent
53c0f624ae
commit
570c26d3d6
@@ -401,6 +401,15 @@ static ggml_type tensor_type_fallback(quantize_state_impl & qs, const ggml_tenso
|
||||
case GGML_TYPE_Q5_K: return_type = GGML_TYPE_Q5_1; break;
|
||||
case GGML_TYPE_Q6_K: return_type = GGML_TYPE_Q8_0; break;
|
||||
default:
|
||||
if (qk_k <= 32) {
|
||||
// the target is already a 32-block type, so there is no smaller block to demote to
|
||||
// and the shape is simply not representable; the check below turns it into F16, the
|
||||
// same answer 256-block types reach when their fallback does not fit. Getting here
|
||||
// means ncols is not a multiple of 32, e.g. a conv kernel a recipe forgot to pin;
|
||||
// throwing gave no tensor name or message, making a recipe gap look like corruption.
|
||||
return_type = target_type;
|
||||
break;
|
||||
}
|
||||
throw std::runtime_error(format("no tensor type fallback is defined for type %s",
|
||||
ggml_type_name(target_type)));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user