6a4becb731
dry is still buggy because token indexes are wrong
2024-08-21 00:59:26 +08:00
50addec9a5
[SYCL] fallback mmvq ( #9088 )
...
* fallback mmvq to mul_mat
* mmvq in cuda path
* Update ggml/src/ggml-sycl.cpp
Co-authored-by: Alberto Cabrera Pérez <alberto.cabrera@codeplay.com >
---------
Co-authored-by: Alberto Cabrera Pérez <alberto.cabrera@codeplay.com >
2024-08-20 23:50:17 +08:00
4f8d19ff17
[SYCL] Fix SYCL im2col and convert Overflow with Large Dims ( #9052 )
...
* sycl: fix im2col overflow and sync with cuda
Signed-off-by: zhentaoyu <zhentao.yu@intel.com >
* sycl: fix convert overflow
Signed-off-by: zhentaoyu <zhentao.yu@intel.com >
* sycl: fix convert and dequantize
Signed-off-by: zhentaoyu <zhentao.yu@intel.com >
* sycl: fix ib in dmmv
Signed-off-by: zhentaoyu <zhentao.yu@intel.com >
* sycl:refine convert
Signed-off-by: zhentaoyu <zhentao.yu@intel.com >
* sycl: move downsample global_range into common
Signed-off-by: zhentaoyu <zhentao.yu@intel.com >
* test: add im2col and convert test cases
Signed-off-by: zhentaoyu <zhentao.yu@intel.com >
* test: make new cases only in sycl
Signed-off-by: zhentaoyu <zhentao.yu@intel.com >
* test: comment new test_cases for only local testing
Signed-off-by: zhentaoyu <zhentao.yu@intel.com >
---------
Signed-off-by: zhentaoyu <zhentao.yu@intel.com >
2024-08-20 23:06:51 +08:00
db6ef8d1e1
revert dry state reset
2024-08-20 22:22:21 +08:00
c1ae350e5b
fixed race condition when generating
2024-08-20 20:17:55 +08:00
7ee359a59b
on multigpu setups, pick lowest free mem instead of highest for auto layers
2024-08-20 19:02:16 +08:00
90db8146d5
tests : add missing comma in grammar integration tests ( #9099 )
...
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com >
2024-08-20 12:09:55 +03:00
cfac111e2b
cann: add doc for cann backend ( #8867 )
...
Co-authored-by: xuedinge233 <damow890@gmail.com >
Co-authored-by: hipudding <huafengchun@gmail.com >
2024-08-19 16:46:38 +08:00
1b6ff90ff8
rpc : print error message when failed to connect endpoint ( #9042 )
2024-08-19 10:11:45 +03:00
18eaf29f4c
rpc : prevent crashes on invalid input ( #9040 )
...
Add more checks which prevent RPC server from crashing if invalid input
is received from client
2024-08-19 10:10:21 +03:00
3bd70d75ea
fix segfault, kcpp is now debuggable
2024-08-19 13:50:49 +08:00
1fbf21eec4
Revert "fix out of bounds access"
...
This reverts commit 3ac183633a
2024-08-19 13:06:39 +08:00
3ac183633a
fix out of bounds access
2024-08-19 00:30:54 +08:00
554b049068
flake.lock: Update ( #9068 )
2024-08-18 07:43:32 -07:00
04166d20a4
better quant clip
2024-08-18 22:15:59 +08:00
b3b00750b7
update lite
2024-08-18 18:23:21 +08:00
2339a0be1c
tests : add integration test for lora adapters ( #8957 )
...
* Add printing to check weights match torch version
* minor code style changes
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co >
2024-08-18 11:58:04 +02:00
e9eb6fe51a
move chat compl to models tab
2024-08-18 14:56:10 +08:00
314a620e96
added readme for macos
2024-08-18 13:11:49 +08:00
06476b8247
Merge branch 'upstream' into concedo_experimental
2024-08-18 12:11:14 +08:00
98dff80b9c
update lite
2024-08-18 12:00:06 +08:00
e2e6d892b4
fix declaration order
2024-08-18 02:15:34 +08:00
d71b5477c5
update lite, cleanup, fix interrogate format
2024-08-18 00:48:53 +08:00
2fb9267887
Fix incorrect use of ctx_split for bias tensors ( #9063 )
2024-08-17 15:34:21 +02:00
1edf83761a
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/bench.yml.disabled
# Makefile
# README.md
# ggml/CMakeLists.txt
# ggml/src/CMakeLists.txt
# ggml/src/ggml-vulkan.cpp
2024-08-17 16:21:14 +08:00
8b3befc0e2
server : refactor middleware and /health endpoint ( #9056 )
...
* server : refactor middleware and /health endpoint
* move "fail_on_no_slot" to /slots
* Update examples/server/server.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* fix server tests
* fix CI
* update server docs
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2024-08-16 17:19:05 +02:00
d565bb2fd5
llava : support MiniCPM-V-2.6 ( #8967 )
...
* init
* rename
* add run android for termux in readme
* add android readme
* add instructions in readme
* change name in readme
* Update README.md
* fixed line
* add result in readme
* random pos_embed
* add positions index
* change for ollama
* change for ollama
* better pos_embed in clip
* support ollama
* updata cmakelist
* updata cmakelist
* rename wrapper
* clear code
* replace and organize code
* add link
* sync master
* fix warnings
* fix warnings
* fix bug in bicubic resize when need resize iamge smaller
* receive review comments and modify
* receive review comments and modify
* put all code into llava dir
* fix quality problem in pr code
* change n_layer
* add space in "-1"
* imitate reshape bug of python code
* fix bug in clip
* fix issues for merging
* fix llama-minicpmv-cli in cmake file
* change pr readme
* fix code review
* remove in line 33 directory in the /cmakelists.txt (not in example, in the main dir
* fix cmakefile
* add warn
* fix KEY_HAS_MINICPMV_PROJ
* remove load_image_size into clip_ctx
* remove the extern "C", MINICPMV_API
* fix uhd code for review comment
* delete minicpmv-wrapper in pr
* remove uhd_image_embed
* Modify 2 notes
* support minicpmv2.6
* modify convert script of minicpmv
* modify convert
* modify convert
* add readme
* add resampler of v2.6
* modify clip
* modify readme
* fix type-check
* fix type-check
* fix type-check
* fix type-check
* modify convert script and readme
* fix convert script and readme
* fix convert
* fix num in convert
* fix type-check
---------
Co-authored-by: Hongji Zhu <fireyoucan@gmail.com >
Co-authored-by: harvestingmoon <leewenyeong@gmail.com >
2024-08-16 16:34:41 +03:00
ee2984bdaf
py : fix wrong input type for raw_dtype in ggml to gguf scripts ( #8928 )
...
Co-authored-by: farbod <farbod.bjary82@gmail.com >
2024-08-16 13:36:30 +03:00
c8ddce8560
Fix inference example lacks required parameters ( #9035 )
...
Signed-off-by: Aisuko <urakiny@gmail.com >
2024-08-16 11:08:59 +02:00
23fd453544
gguf-py : bump version from 0.9.1 to 0.10.0 ( #9051 )
2024-08-16 09:36:11 +03:00
c679e0cb5c
llama : add EXAONE model support ( #9025 )
...
* add exaone model support
* add chat template
* fix whitespace
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* add ftype
* add exaone pre-tokenizer in `llama-vocab.cpp`
Co-Authored-By: compilade <113953597+compilade@users.noreply.github.com >
* fix lint
Co-Authored-By: compilade <113953597+compilade@users.noreply.github.com >
* add `EXAONE` to supported models in `README.md`
* fix space
Co-authored-by: compilade <git@compilade.net >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
Co-authored-by: compilade <113953597+compilade@users.noreply.github.com >
Co-authored-by: compilade <git@compilade.net >
2024-08-16 09:35:18 +03:00
fb487bb567
common : add support for cpu_get_num_physical_cores() on Windows ( #8771 )
...
* Add support for cpu_get_num_phsical_cores() on Windows
* fix build bug on msys2-clang64 and ucrt64
* avoid adding new function
* add new macros to avoid windows+mingw64
* Add error checking to return default value
2024-08-16 09:23:12 +03:00
2a24c8caa6
Add Nemotron/Minitron GGUF Conversion & Inference Support ( #8922 )
...
* Add nemotron GGUF conversion & inference support
* Fix formatting issues
* Remove unnecessary write_tensors()
* Update convert_hf_to_gguf.py
Co-authored-by: compilade <git@compilade.net >
* Update src/llama.cpp
Co-authored-by: compilade <git@compilade.net >
* Address comments by @compilade
* Replace ggml_mul_mat()->llm_build_lora_mm()
* Remove mutable variable
* Use for bias tensors
* Cover corner case for role_scaling not in config.json
---------
Co-authored-by: compilade <git@compilade.net >
2024-08-16 04:23:33 +02:00
e3f6fd56b1
ggml : dynamic ggml_sched_max_splits based on graph_size ( #9047 )
...
* ggml : Dynamic ggml_sched_max_splits based on graph_size
* Fixed and readded debug code for causes
2024-08-16 04:22:55 +02:00
4b9afbbe90
retrieval : fix memory leak in retrieval query handling ( #8955 )
...
* retrieval
* Reuse querybatch to reduce frequent memory allocation
* delete unused white space
2024-08-15 10:40:12 +03:00
37501d9c79
server : fix duplicated n_predict key in the generation_settings ( #8994 )
2024-08-15 10:28:05 +03:00
4af8420afb
common : remove duplicate function llama_should_add_bos_token ( #8778 )
2024-08-15 10:23:23 +03:00
6bda7ce6c3
llama : add pre-tokenizer regexes for BLOOM and gpt3-finnish ( #8850 )
2024-08-15 10:17:12 +03:00
d5492f0525
ci : disable bench workflow ( #9010 )
2024-08-15 10:11:11 +03:00
234b30676a
server : init stop and error fields of the result struct ( #9026 )
...
Signed-off-by: Jiri Podivin <jpodivin@redhat.com >
2024-08-15 09:21:57 +03:00
5fd89a70ea
Vulkan Optimizations and Fixes ( #8959 )
...
* Optimize Vulkan REPEAT performance
* Use Vulkan GLSL fused multiply-add instruction where possible
* Add GGML_VULKAN_PERF option to output performance data per operator
* Rework and fix Vulkan descriptor set and descriptor pool handling
* Fix float32 concat f16 shader validation error
* Add Vulkan GROUP_NORM eps parameter
* Fix validation error with transfer queue memory barrier flags
* Remove trailing whitespaces
2024-08-14 18:32:53 +02:00
e8de0af3ec
Merge branch 'upstream' into concedo_experimental
...
# Conflicts:
# .github/workflows/bench.yml
# .github/workflows/build.yml
# .github/workflows/python-check-requirements.yml
# README.md
# docs/backend/SYCL.md
# flake.lock
# ggml/CMakeLists.txt
# ggml/src/kompute-shaders/op_rope_f16.comp
# ggml/src/kompute-shaders/op_rope_f32.comp
# ggml/src/kompute-shaders/rope_common.comp
2024-08-14 22:25:43 +08:00
2c108ab17e
correct phrasing
2024-08-14 21:55:53 +08:00
e12ab53488
force clear some DRY state vars on new generation - not sure if this helps
2024-08-14 21:35:39 +08:00
98a532d474
server : fix segfault on long system prompt ( #8987 )
...
* server : fix segfault on long system prompt
* server : fix parallel generation with very small batch sizes
* server : fix typo in comment
2024-08-14 09:51:02 +03:00
43bdd3ce18
cmake : remove unused option GGML_CURL ( #9011 )
2024-08-14 09:14:49 +03:00
06943a69f6
ggml : move rope type enum to ggml.h ( #8949 )
...
* ggml : move rope type enum to ggml.h
This commit moves the `llama_rope_type` enum from `llama.h` to
`ggml.h` and changes its name to `ggml_rope_type`.
The motivation for this change is to address the TODO in `llama.h` and
use the enum in ggml.
Note: This commit does not change the `mode` parameter to be of type
`enum ggml_rope_type`. The name `mode` and its usage suggest that it
might be more generic and possibly used as a bit field for multiple
flags. Further investigation/discussion may be needed to determine
if `mode` should be restricted to RoPE types.
* squash! ggml : move rope type enum to ggml.h
This commit removes GGML_ROPE_TYPE_NONE and GGML_ROPE_TYPE_GLM from
ggml.h, and back the llama_rope_type enum.
I've kept the assert for GGML_ROPE_TYPE_GLM as I'm not sure if it is
safe to remove it yet.
* squash! ggml : move rope type enum to ggml.h
This commit removes the enum ggml_rope_type from ggml.h and replaces it
with a define (GGML_ROPE_TYPE_NEOX). This define is used in the code to
check if the mode is set to GPT-NeoX. Also the enum llama_rope_type has
been updated to reflect this change.
* squash! ggml : move rope type enum to ggml.h
This commit contains a suggestion enable the GGML_ROPE_TYPE_NEOX
macro/define to be passed to the shader compiler.
* squash! ggml : move rope type enum to ggml.h
This commit fixes the editorconfig-checker warnings.
* squash! ggml : move rope type enum to ggml.h
Update comment for ggml_rope function.
* Revert "squash! ggml : move rope type enum to ggml.h"
This reverts commit 6261222bd0dc0efd51f0fb0435ad3f16a5b52fd6.
* squash! ggml : move rope type enum to ggml.h
Add GGML_ROPE_TYPE_NEOX to rope_common.comp.
* remove extra line
---------
Co-authored-by: slaren <slarengh@gmail.com >
2024-08-13 21:13:15 +02:00
828d6ff7d7
export-lora : throw error if lora is quantized ( #9002 )
2024-08-13 11:41:14 +02:00
fc4ca27b25
ci : fix github workflow vulnerable to script injection ( #9008 )
...
Signed-off-by: Diogo Teles Sant'Anna <diogoteles@google.com >
2024-08-12 19:28:23 +03:00
1f67436c5e
ci : enable RPC in all of the released builds ( #9006 )
...
ref: #8912
2024-08-12 19:17:03 +03:00
Concedo
Meng, Hengyu
zhentaoyu
fairydreaming
wangshuai09
Radoslav Gerganov
Georgi Gerganov
ltoniazzi
Yoshi Suhara
Xuan Son Nguyen
tc-mb
Farbod Bijary
Aisuko
compilade
Minsoo Cheong
Liu Jia
Yoshi Suhara
Nico Bosshard
gtygo
Riceball LEE
Zhenwei Jin
Esko Toivonen
Jiří Podivín
0cc4m
Daniel Bevenius
Diogo Teles Sant'Anna