unicode : include '~' in collapsed symbol class (#26972)

The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.

Assisted-by: Codex
This commit is contained in:
Thiago Padilha
2026-08-18 10:15:22 -03:00
committed by GitHub
parent 169e4a7ff2
commit afd439df1f
3 changed files with 27 additions and 1 deletions
+24
View File
@@ -0,0 +1,24 @@
#include "../src/unicode.h"
#include <cstdio>
#include <string>
#include <vector>
int main() {
const std::vector<std::string> regex_exprs = {
"[~][A-Za-z]+| ?[\\p{S}]+|\\s+",
};
const std::vector<std::string> expected = { " ~", "foo" };
const auto actual = unicode_regex_split(" ~foo", regex_exprs, false);
if (actual != expected) {
fprintf(stderr, "unexpected split:");
for (const auto & piece : actual) {
fprintf(stderr, " [%s]", piece.c_str());
}
fprintf(stderr, "\n");
return 1;
}
return 0;
}