Public/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-09-20 01:31:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

update dev docs

Browse Source
This commit is contained in:
Xuan Son Nguyen
2026-08-01 15:07:28 +02:00
parent eae9c2c3f5
commit 86020c467a
1 changed files with 15 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/mtmd/README-dev.md
+15 -10
View File
@@ -33,18 +33,17 @@ A typical pipeline of the core libmtmd is as follows:
Audio generation is added to mtmd in PR [#26254](https://github.com/ggml-org/llama.cpp/pull/26254)
Currently, we support the 3-stage pipeline below which should cover most TTS models:
1. (Optional) an audio encoder model that converts reference voice into codes or features
2. A backbone model that accepts text prompt and reference voice as input
3. A feature generator model that takes the hidden state from backbone and generate audio features (usually as audio codes or mel-spectrogram)
4. A model that converts audio features to the final PCM waveform
- Stage 1: Backbone / Semantic Stage: Backbone model accepts text prompt and reference voice as input
- Stage 2: Acoustic Detail Generator: A model takes the hidden state from backbone and generate audio details (usually as audio codes or mel-spectrogram)
- Stage 3: Waveform Reconstruction: Convert the semantic and acoustic data from previous stages to the final waveform
For example, Qwen3-TTS:
1. Reference voice is encoded using ECAPA-TDNN speaker encoder (`speaker_encoder`)
2. Text prompt and reference voice are processed via a backbone (`talker.model`)
3. A model converts sampled semantic token and hidden state from stage 2 into a list of 15 acoustic codes (`talker.code_predictor`)
4. 16 generated codes are converted into waveform (`code2wav`)
- Reference voice is encoded using ECAPA-TDNN speaker encoder (`speaker_encoder`)
- Text prompt and reference voice are processed via a backbone (`talker.model`)
- A model converts sampled semantic token and hidden state from stage 2 into a list of 15 acoustic codes (`talker.code_predictor`)
- 16 generated codes are converted into waveform (`code2wav`)
### API design constraint
### API design constraints
Due to wide variety of audio generation pipelines, the `mtmd_gen_audio` system is designed to be flexible and reusable by new models.
@@ -55,13 +54,19 @@ Due to wide variety of audio generation pipelines, the `mtmd_gen_audio` system i
### Checklist for porting new audio generation models to mtmd
1. Establish a list of reusable and missing components from the current mtmd implementation.
2. Sidecar models (code2wav, bigvgan, etc) must live inside the same GGUF file (but can be in different `clip_context` if necessary)
2. For GGUF conversion:
- Backbone model should be converted to a normal text model (loadable via `libllama`)
- If model used hard-coded embedding row ID, append them to token embeddings and assign token name for them (see `qwen3tts.py`)
- If model have a specific output logits head for audio codes (usually semantic code), keep the head as-is and pad the logits at inference time (see `src/models/qwen3vl.cpp`)
- Sidecar models (code2wav, bigvgan, etc) must live inside the mmproj GGUF (but can be in different `clip_context` if necessary)
3. Make sure most of the changes happen inside `mtmd-helper-gen.cpp`. A good PR looks like this:
- 10-20% changes is to add new backbone (text) model and conversion
- 60% changes inside `mtmd-helper-gen.cpp`
- 10% changes inside `libmtmd` and `clip.cpp` systems
- The rest downstream code (CLI, server) should have no changes at all
IMPORTANT: if the model requires any changes that doesn't fit into the existing infrastructure, **open an issue** first for discussion. Any new components must be verified to respect the API design constraints stated above
## Helper
We provide a set of helper functions via `mtmd_helper` to make using libmtmd easier. The helper provides: