diff --git a/docs/speculative.md b/docs/speculative.md index 29da332875..fb6ef03067 100644 --- a/docs/speculative.md +++ b/docs/speculative.md @@ -33,18 +33,18 @@ An example to use this approach can be the rewriting of source code by a LLM. This implementation looks for the last n-gram in history that matches the current n-gram and creates a draft using the m tokens following the matched n-gram. It is the simplest self-speculative approach with minimal overhead. ``` -llama-server [...] --spec-type ngram-simple --draft-max 64 +llama-server [...] --spec-type ngram-simple --spec-draft-n-max 64 ``` #### n-gram Map Key (`ngram-map-k`) -This implementation looks for the current n-gram of size n (called the _key_) in the token history. If the key n-gram is followed by the same m tokens (called the _mgram_) multiple times, it creates a draft using these m tokens. This approach requires a minimum number of occurrences (argument `--spec-ngram-min-hits`, default is 1) before generating drafts. +This implementation looks for the current n-gram of size n (called the _key_) in the token history. If the key n-gram is followed by the same m tokens (called the _mgram_) multiple times, it creates a draft using these m tokens. This approach requires a minimum number of occurrences (argument `--spec-ngram-map-k-min-hits`, default is 1) before generating drafts. The number of accepted tokens is stored for each used n-gram. **Example:** ``` -llama-server [...] --spec-type ngram-map-k --draft-max 64 +llama-server [...] --spec-type ngram-map-k --spec-draft-n-max 64 ``` #### n-gram Map Key-4-Values (`ngram-map-k4v`) @@ -55,7 +55,7 @@ The number of accepted tokens is stored for each used n-gram. **Example:** Server options to be used if there are a lot of longer repetitions. ``` -llama-server [...] --spec-type ngram-map-k4v --spec-ngram-size-n 8 --spec-ngram-size-m 8 --spec-ngram-min-hits 2 --draft-max 64 +llama-server [...] --spec-type ngram-map-k4v --spec-ngram-map-k4v-size-n 8 --spec-ngram-map-k4v-size-m 8 --spec-ngram-map-k4v-min-hits 2 --spec-draft-n-max 64 ``` ### n-gram Mod (`ngram-mod`) @@ -80,9 +80,9 @@ Currently, a single hash pool is shared across all server slots, so different re # notes: # - small `n` are not recommended # - MoEs require long drafts -# - dense models: can reduce `--draft-min` and `--draft-max` +# - dense models: can reduce `--spec-ngram-mod-n-min` and `--spec-ngram-mod-n-max` -llama-server ... --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 48 --draft-max 64 +llama-server ... --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 ``` Applications: @@ -105,21 +105,90 @@ Example Video: If a draft model is combined with a draftless decoding the draftless decoding has higher precedence. +### General Speculative Parameters + ``` ---draft, --draft-n, --draft-max N number of tokens to draft for speculative decoding (default: 16) - (env: LLAMA_ARG_DRAFT_MAX) ---draft-min, --draft-n-min N minimum number of draft tokens to use for speculative decoding - (default: 0) - (env: LLAMA_ARG_DRAFT_MIN) -[...] --spec-type [none|ngram-cache|ngram-simple|ngram-map-k|ngram-map-k4v|ngram-mod] type of speculative decoding to use when no draft model is provided (default: none) ---spec-ngram-size-n N ngram size N for ngram-simple/ngram-map speculative decoding, length - of lookup n-gram (default: 12) ---spec-ngram-size-m N ngram size M for ngram-simple/ngram-map speculative decoding, length - of draft m-gram (default: 48) ---spec-ngram-min-hits N minimum hits for ngram-map speculative decoding (default: 1) + (env: LLAMA_ARG_SPEC_TYPE) +--spec-default use default speculative decoding +``` + +### Draft Model Parameters + +``` +--spec-draft-model, -md, --model-draft FNAME + draft model for speculative decoding (default: unused) + (env: LLAMA_ARG_SPEC_DRAFT_MODEL) +--spec-draft-hf, -hfd, -hfrd, --hf-repo-draft /[:quant] + HuggingFace repository for the draft model +--spec-draft-n-max N + number of tokens to draft for speculative decoding (default: 16) + (env: LLAMA_ARG_SPEC_DRAFT_N_MAX) +--spec-draft-n-min N + minimum number of draft tokens to use for speculative decoding (default: 0) + (env: LLAMA_ARG_SPEC_DRAFT_N_MIN) +--spec-draft-p-split, --draft-p-split P + speculative decoding split probability (default: 0.10) + (env: LLAMA_ARG_SPEC_DRAFT_P_SPLIT) +--spec-draft-p-min, --draft-p-min P + minimum speculative decoding probability (greedy) (default: 0.75) + (env: LLAMA_ARG_SPEC_DRAFT_P_MIN) +--spec-draft-ctx-size, -cd, --ctx-size-draft N + size of the prompt context for the draft model (default: 0, 0 = loaded from model) + (env: LLAMA_ARG_SPEC_DRAFT_CTX_SIZE) +--spec-draft-ngl, -ngld, --gpu-layers-draft, --n-gpu-layers-draft N + max. number of draft model layers to store in VRAM, either an exact number, 'auto', or 'all' (default: auto) + (env: LLAMA_ARG_N_GPU_LAYERS_DRAFT) +--spec-draft-device, -devd, --device-draft + comma-separated list of devices to use for offloading the draft model +--spec-draft-replace, --spec-replace TARGET DRAFT + translate the string in TARGET into DRAFT if the draft model and main model are not compatible +``` + +### n-gram Mod Parameters + +``` +--spec-ngram-mod-n-match N + ngram-mod lookup length (default: 24) +--spec-ngram-mod-n-min N + minimum number of ngram tokens to use for ngram-based speculative decoding (default: 48) +--spec-ngram-mod-n-max N + maximum number of ngram tokens to use for ngram-based speculative decoding (default: 64) +``` + +### n-gram Simple Parameters + +``` +--spec-ngram-simple-size-n N + ngram size N for ngram-simple speculative decoding, length of lookup n-gram (default: 12) +--spec-ngram-simple-size-m N + ngram size M for ngram-simple speculative decoding, length of draft m-gram (default: 48) +--spec-ngram-simple-min-hits N + minimum hits for ngram-simple speculative decoding (default: 1) +``` + +### n-gram Map Key Parameters + +``` +--spec-ngram-map-k-size-n N + ngram size N for ngram-map-k speculative decoding, length of lookup n-gram (default: 12) +--spec-ngram-map-k-size-m N + ngram size M for ngram-map-k speculative decoding, length of draft m-gram (default: 48) +--spec-ngram-map-k-min-hits N + minimum hits for ngram-map-k speculative decoding (default: 1) +``` + +### n-gram Map Key-4-Values Parameters + +``` +--spec-ngram-map-k4v-size-n N + ngram size N for ngram-map-k4v speculative decoding, length of lookup n-gram (default: 12) +--spec-ngram-map-k4v-size-m N + ngram size M for ngram-map-k4v speculative decoding, length of draft m-gram (default: 48) +--spec-ngram-map-k4v-min-hits N + minimum hits for ngram-map-k4v speculative decoding (default: 1) ``` ### `--spec-type TYPE` @@ -140,21 +209,40 @@ Specifies a type of speculative decoding without draft model. ./llama-server [...] --spec-type ngram-simple ``` -### `--spec-ngram-size-n N` +### `--spec-ngram-*-size-n N` Sets the size N of the lookup n-gram for n-gram map based speculative decoding. The n-gram size N determines how many tokens in a row to look back when searching for matching patterns. -### `--spec-ngram-size-m M` +Each n-gram implementation has its own parameter: + +- `--spec-ngram-simple-size-n` for `ngram-simple` +- `--spec-ngram-map-k-size-n` for `ngram-map-k` +- `--spec-ngram-map-k4v-size-n` for `ngram-map-k4v` +- `--spec-ngram-mod-n-match` for `ngram-mod` + +### `--spec-ngram-*-size-m M` Sets the size M of the draft m-gram for n-gram map based speculative decoding. The m-gram size determines how many tokens to draft when a match is found. Larger values can provide more speedup but may reduce acceptance rate. -### `--spec-ngram-min-hits H` +Each n-gram implementation has its own parameter: + +- `--spec-ngram-simple-size-m` for `ngram-simple` +- `--spec-ngram-map-k-size-m` for `ngram-map-k` +- `--spec-ngram-map-k4v-size-m` for `ngram-map-k4v` + +### `--spec-ngram-*-min-hits H` This option defines how often a key has to appear in the token history to be used as a draft (default is 1). +Each n-gram implementation has its own parameter: + +- `--spec-ngram-simple-min-hits` for `ngram-simple` +- `--spec-ngram-map-k-min-hits` for `ngram-map-k` +- `--spec-ngram-map-k4v-min-hits` for `ngram-map-k4v` + ## Statistics Each speculative decoding implementation prints statistics. @@ -180,4 +268,3 @@ statistics ngram_map_k: #calls(b,g,a) = 6 1690 26, #gen drafts = 26, #acc drafts - `#gen tokens`: number of tokens generated by this implementation (including rejected tokens) - `#acc tokens`: number of tokens accepted by the main model - `dur(b,g,a): durations of begin (new prompt), generation and accumulation (process acceptance). -