mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-08-30 00:51:19 +02:00
Xuan-Son Nguyen
6e62ba5384
* adapt the api * text model ok * working impl, need verify and clean up * mtmd: build the pocket-tts transposed convolutions as GEMM + col2im ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample was built as one convolution and one concat per channel, which floods the graph with small nodes and makes kernel launches dominate the decoder. Fold both cases into the column form the seanet decoder already needs: the general case reshapes the kernel to [IC, K * OC] and matmuls it with the input, the depthwise case batches a matmul over the channels so a step scales its own kernel. A single col2im_1d then scatter-adds the columns back to the signal, with the same shape as before, so the overlap-add tail, the streaming state and the bias are untouched. Generation time per frame drops by 80% on CUDA and by 50% on CPU. The output matches the previous implementation sample for sample, with a correlation of 0.999994 and identical frame counts. * flow_temp + frames_after_eos * chunking * mtmd: carry the remaining pocket-tts per-pack settings The language packs also tune the end-of-speech padding and the padding of short prompts, next to the temperature already carried in the mmproj: french_24l asks for 8 tail frames instead of the guessed 3, english_2026-01 asks for short prompts to be padded with spaces. Write both in the mmproj as clip.gen.audio.frames_after_eos and clip.gen.audio.pad_short_text, keyed on the pack in the conversion script like the temperature. The loader keeps them optional, so a mmproj without them behaves as before. Map semicolons to commas for every pack instead, the reference only asks for it on three of them and it costs nothing elsewhere. Existing mmproj files must be converted again to carry the two keys. On a long french text the port now lands within 2% of the reference: 22.96s against 23.44s, with the same peak level and the same amount of silence. * clip.gen.audio.model_variant * clean up code comments * nit: drop the dead flow_temp hparam, the pack table holds the default * update docs * address security problems * less invasive base.py * lint * add mtmd_gen_inp_default * add docs * rm gen_flow_temp --------- Co-authored-by: Pascal <admin@serveurperso.com>
60 lines
2.0 KiB
Markdown
60 lines
2.0 KiB
Markdown
# llama.cpp TTS
|
|
|
|
This is a tool to demonstrate audio generation capability in llama.cpp via `libmtmd`. It was added via PR [#26254](https://github.com/ggml-org/llama.cpp/pull/26254)
|
|
|
|
Note: this tool used to serve as a demo for OuteTTS, but it was converted to a more model-agnostic tool.
|
|
|
|
## Common usage
|
|
|
|
Simple usage:
|
|
|
|
```sh
|
|
llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF -p "Hello world" --output out.wav
|
|
```
|
|
|
|
Common params:
|
|
- Sampling params such as `--top-k`, `--top-p`, `--temp`, etc.
|
|
- `-n <number_of_frames>` limits the output length, e.g. `-n 500`. Note that how many milliseconds each frame represents varies by model
|
|
- Core inference params such as `-ngl`, `-b`, `-ub`, etc.
|
|
|
|
## Qwen3-TTS
|
|
|
|
Available params:
|
|
- `--tts-lang` can be `zh`, `en`, `de`, `it`, `pt`, `es`, `ja`, `ko`, `fr`, `ru` (default: `en`)
|
|
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3)
|
|
|
|
Example usage:
|
|
|
|
```sh
|
|
llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \
|
|
-p "Hello world" \
|
|
--tts-lang english \
|
|
--tts-speaker-file speaker.mp3 \
|
|
--output out.wav
|
|
```
|
|
|
|
## Pocket TTS
|
|
|
|
Available params:
|
|
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it
|
|
- Note: `lang` is not used, the language is a property of the weights
|
|
|
|
Example usage:
|
|
|
|
```sh
|
|
llama-tts -m pocket-tts.gguf \
|
|
-mm mmproj-pocket-tts.gguf \
|
|
-p "Hello world" \
|
|
--tts-speaker-file speaker.mp3 \
|
|
--output out.wav
|
|
```
|
|
|
|
**Note for GGUF conversion:**
|
|
|
|
The [upstream repository](https://huggingface.co/kyutai/pocket-tts) holds one complete model per language under `languages/`, next to a set of shared files at the root. Convert one of the `languages/<name>` directories, **not** the root directory:
|
|
|
|
```sh
|
|
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf
|
|
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf
|
|
```
|