Public/koboldcpp
1
0
Fork 0
mirror of https://github.com/LostRuins/koboldcpp.git synced 2026-08-27 07:30:49 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
6166fdfde4b118bfd74c0ee4fddff8a6d4f61b9a
koboldcpp/examples/quantize
T
History
Concedo 9c0fbf9f73 Merge commit 'ad3a0505e3b6cd777259ee35e61d428357ffc565' into concedo_experimental 
# Conflicts:
#	.github/workflows/build.yml
#	.github/workflows/close-issue.yml
#	.github/workflows/code-coverage.yml
#	.github/workflows/docker.yml
#	.github/workflows/editorconfig.yml
#	.github/workflows/nix-ci-aarch64.yml
#	.github/workflows/nix-ci.yml
#	.github/workflows/python-check-requirements.yml
#	.github/workflows/python-lint.yml
#	.github/workflows/server.yml
#	.github/workflows/zig-build.yml
#	.gitignore
#	CMakeLists.txt
#	Makefile
#	README-sycl.md
#	README.md
#	build.zig
#	common/CMakeLists.txt
#	llama.cpp
#	tests/CMakeLists.txt
#	tests/test-backend-ops.cpp
2024-04-06 18:32:57 +08:00
..
CMakeLists.txt
build : link against build info instead of compiling against it (#3879)
2023-11-02 08:50:16 +02:00
quantize.cpp
Merge commit 'ad3a0505e3b6cd777259ee35e61d428357ffc565' into concedo_experimental
2024-04-06 18:32:57 +08:00
README.md
readme : add some recent perplexity and bpw measurements to READMES, link for k-quants (#3340)
2023-09-27 18:30:36 +03:00

README.md

quantize

TODO

Llama 2 7B

Quantization Bits per Weight (BPW)
Q2_K 3.35
Q3_K_S 3.50
Q3_K_M 3.91
Q3_K_L 4.27
Q4_K_S 4.58
Q4_K_M 4.84
Q5_K_S 5.52
Q5_K_M 5.68
Q6_K 6.56

Llama 2 13B

Quantization Bits per Weight (BPW)
Q2_K 3.34
Q3_K_S 3.48
Q3_K_M 3.89
Q3_K_L 4.26
Q4_K_S 4.56
Q4_K_M 4.83
Q5_K_S 5.51
Q5_K_M 5.67
Q6_K 6.56

Llama 2 70B

Quantization Bits per Weight (BPW)
Q2_K 3.40
Q3_K_S 3.47
Q3_K_M 3.85
Q3_K_L 4.19
Q4_K_S 4.53
Q4_K_M 4.80
Q5_K_S 5.50
Q5_K_M 5.65
Q6_K 6.56
Reference in New Issue View Git Blame Copy Permalink