Public/automatic
1
0
Fork 0
mirror of https://github.com/vladmandic/automatic synced 2026-09-19 01:04:32 +02:00
Code Issues Packages Projects Releases Wiki Activity

SDNQ enable matmul support for float8_e5m2

Browse Source
This commit is contained in:
Disty0
2025-06-02 00:53:10 +03:00
parent 8f1a1d7311
commit e8588c91ea
2 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+1 -1
View File
@@ -26,7 +26,7 @@ Also unlike most traditional methods, its also applicable to nearly all model ty
- `INT4_SYM` -> `int4`
- `INT4` -> `uint4`
- Add `float8_e4m3fn`, `float8_e5m2`, `float8_e4m3fnuz`, `float8_e5m2fnuz`, `int6`, `uint6`, `int2`, `uint2` and `uint1` support
- Add quantized matmul support for `float8_e4m3fn`
- Add quantized matmul support for `float8_e4m3fn` and `float8_e5m2`
- Set the default quant mode to `pre`
- Use per token input quant with int8 and fp8 quantized matmul
- Implement better layer hijacks
modules/model_quant_sdnq.py
+2 -2
View File
@@ -28,9 +28,9 @@ dtype_dict = {
"float8_e5m2fnuz": {"min": -57344, "max": 57344, "num_bits": 8, "target_dtype": CustomDtype.FP8, "torch_dtype": torch.float8_e5m2fnuz, "storage_dtype": torch.float8_e5m2fnuz, "is_unsigned": False, "is_integer": False},
}
quantized_matmul_dtypes = ("int8", "int6", "int4", "int2", "float8_e4m3fn")
quantized_matmul_dtypes = ("int8", "int6", "int4", "int2", "float8_e4m3fn", "float8_e5m2")
if devices.backend in {"cpu", "openvino"}:
quantized_matmul_dtypes += ("float8_e5m2", "float8_e4m3fnuz", "float8_e5m2fnuz")
quantized_matmul_dtypes += ("float8_e4m3fnuz", "float8_e5m2fnuz")
linear_types = ("Linear",)
conv_types = ("Conv1d", "Conv2d", "Conv3d")