581c4a0239
unicode : try fix windows
2024-04-27 18:36:00 +03:00
91eaa414bf
unicode : support \p{N}, \p{L} and \p{P} natively
2024-04-27 17:48:38 +03:00
ce5485aee0
unicode : always use std::wregex
2024-04-27 17:11:34 +03:00
2affd0b221
unicode : set bomb
2024-04-27 11:56:02 +03:00
a22645c2a7
unicode : set bomb
2024-04-27 11:48:24 +03:00
4434c9d6c2
minor
2024-04-27 11:33:16 +03:00
ad929833cb
llama : adapt punctuation regex + add llama 3 regex
2024-04-27 11:06:08 +03:00
96965f67e6
models : add llama v3 vocab file
2024-04-27 11:05:12 +03:00
c160818ec0
wip
2024-04-27 00:28:36 +03:00
a774d7084e
make : add test-tokenizer-0-llama-v3
2024-04-26 21:25:36 +03:00
8791e94e3c
lint : fix
2024-04-26 21:12:05 +03:00
1b9b79dd14
convert : fix pre-tokenizer type writing
2024-04-26 20:55:14 +03:00
43e12ce8e5
llama : use new pre-tokenizer type
2024-04-26 20:08:57 +03:00
9b4d63ae53
convert : add "tokenizer.ggml.pre" GGUF KV (wip)
2024-04-26 19:21:55 +03:00
e3f6dc7409
Merge branch 'master' into gg/bpe-preprocess
2024-04-26 18:08:40 +03:00
e2764cd7ca
gguf : fix mismatch between alloc and free functions ( #6929 )
2024-04-26 18:07:42 +03:00
4b1c3c98b4
llamafile : use 64-bit integers in sgemm ( #6928 )
2024-04-26 17:05:33 +03:00
e9891769ff
unicode : first try custom implementations
2024-04-26 15:09:07 +03:00
e8c206be61
unicode : shot in the dark to fix tests on Windows
2024-04-26 14:57:12 +03:00
4907e41aa7
llama : towards llama3 tokenization support (wip)
2024-04-26 14:55:37 +03:00
ed42711b90
gguf-py : reader prints warnings on duplicate keys
2024-04-26 14:32:22 +03:00
e1b2bf783e
tests : add sample usage
2024-04-26 13:43:54 +03:00
aeafb43ed7
tests : remove and rename tokenizer test scripts
2024-04-26 13:39:03 +03:00
d999cf65c5
unicode : remove redundant headers
2024-04-26 13:29:48 +03:00
bbe3c6e761
ci: server: fix python installation ( #6925 )
2024-04-26 12:27:25 +02:00
7a44e44342
tests : add tokenizer tests for numbers
2024-04-26 13:21:28 +03:00
7f5ff558ee
server: stop generation at n_ctx_train if n_predict is not set ( #6638 )
...
* server: cap n_predict if not set to n_ctx_train
* server: fix infinite loop
* server: infinite loop, move in process_token
server: infinite loop: set stop limit to true
* minor: spaces
* minor: spaces
* server: include prompt tokens in the EOS limit
2024-04-26 12:15:30 +02:00
c56e19db4b
lint : fix whitespaces
2024-04-26 12:58:07 +03:00
06d3e693db
unicode : fix? unicode_wstring_to_utf8
2024-04-26 12:55:11 +03:00
9e4e077ec5
ci: server: fix python installation ( #6922 )
2024-04-26 11:11:51 +02:00
36d983262e
Fixed issue with gpt2 regex custom preprocessor
2024-04-26 11:43:29 +03:00
753580360b
Fixed issues
2024-04-26 11:43:29 +03:00
feeaf4f39c
Added needed functionality, testing remains
2024-04-26 11:43:29 +03:00
7e308ed212
Adding unicode regex function
2024-04-26 11:43:29 +03:00
a5710a4101
Adding unicode regex mappings
2024-04-26 11:43:29 +03:00
4c3e882a85
Refactored code
2024-04-26 11:43:29 +03:00
c8e7d9521d
Updated/merged the deepseek coder pr
2024-04-26 11:43:29 +03:00
4056dc5b1e
added and refactored unicode_regex_split and related functions
2024-04-26 11:43:28 +03:00
1c924e4b35
Resolved issues
2024-04-26 11:43:28 +03:00
54f93eb50b
Moved header files
2024-04-26 11:43:28 +03:00
d2cfc2225f
Moved regex patterns to unicode.cpp and updated unicode.h
2024-04-26 11:43:28 +03:00
6fbab2dbc8
merged the changes from deepseeker models to main branch
2024-04-26 11:43:08 +03:00
83b72cb086
Merge pull request from GHSA-p5mv-gjc5-mwqv
...
* always use calloc
clamp n_kv on failure to read a kv
* ggml : alternative ctx->header.n_kv update
---------
Co-authored-by: slaren <slarengh@gmail.com >
2024-04-26 10:41:53 +03:00
d4a9afc100
ci: server: fix python installation ( #6918 )
2024-04-26 09:27:49 +02:00
7d641c26ac
ci: fix concurrency for pull_request_target ( #6917 )
2024-04-26 09:26:59 +02:00
5790c8dac1
bench: server add stop word for PHI-2 ( #6916 )
2024-04-26 09:26:16 +02:00
46e12c4692
llava : add support for moondream vision language model ( #6899 )
...
* add support for moondream vision language model
This required making the following changes to the CLIP model:
1. Support for patch embedding bias.
2. Make class embedding and pre-layernorm optional.
3. Add support for post-layernorm.
* Update examples/llava/clip.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2024-04-25 22:38:31 +03:00
dba497e0c1
cmake : restore LLAMA_LLAMAFILE_DEFAULT
2024-04-25 21:37:27 +03:00
fa0b4ad252
cmake : remove obsolete ANDROID check
2024-04-25 18:59:51 +03:00
d6e1d44f16
llama : synchronize before get/set session data ( #6911 )
2024-04-25 17:59:03 +02:00
Georgi Gerganov
slaren
Justine Tunney
Pierrick Hymbert
Kazim Abrar Mahi
Jaggzh
vik