quantize: fall back to F16 for 32-block types with an odd ncols

tensor_type_fallback demotes a tensor whose ncols is not a multiple of the
target's block size, but its switch only enumerates the 256-block types. A
target that is already a 32-block type (iq4_nl, q4_0, q5_0, q8_0, ...) falls
into default: and throws, even though the function already knows how to answer
that case: the ncols check right below the switch resolves an unrepresentable
shape to F16.

Route those types into that check instead of throwing. Only paths that abort
today change, so no quantization that currently succeeds is affected.

Found on a 4-wide depthwise conv kernel. llama-quantize reported nothing but
"failed to quantize model from ...", with no tensor name and no exception text,
which made a quant recipe that had simply not pinned the tensor look like a
corrupt model. It now names the tensor and continues.
This commit is contained in:
danielhanchen
2026-08-25 23:18:12 +00:00
committed by Daniel Han
parent 53c0f624ae
commit 570c26d3d6
+9
View File
@@ -401,6 +401,15 @@ static ggml_type tensor_type_fallback(quantize_state_impl & qs, const ggml_tenso
case GGML_TYPE_Q5_K: return_type = GGML_TYPE_Q5_1; break;
case GGML_TYPE_Q6_K: return_type = GGML_TYPE_Q8_0; break;
default:
if (qk_k <= 32) {
// the target is already a 32-block type, so there is no smaller block to demote to
// and the shape is simply not representable; the check below turns it into F16, the
// same answer 256-block types reach when their fallback does not fit. Getting here
// means ncols is not a multiple of 32, e.g. a conv kernel a recipe forgot to pin;
// throwing gave no tensor name or message, making a recipe gap look like corruption.
return_type = target_type;
break;
}
throw std::runtime_error(format("no tensor type fallback is defined for type %s",
ggml_type_name(target_type)));
}