update docs

This commit is contained in:
Xuan Son Nguyen
2026-08-01 18:26:59 +02:00
parent 4a9e349f63
commit 026d35cfc8
3 changed files with 5 additions and 0 deletions
+1
View File
@@ -133,6 +133,7 @@ Note:
- To debug the multimodal preprocessor and encoder, you can use [llama-mtmd-debug](tools/mtmd/debug/mtmd-debug.cpp).
- Adding a model-specific API or CLI is an anti-pattern in `libmtmd`. The goal of `libmtmd` is to provide an easy-to-use, model-agnostic library for multimodal pipeline.
- In most cases, `llama-mtmd-cli` should not be modified. If a model requires a specific prompt, either let the user provide it or bake it into the Jinja chat template.
- For audio generation models, see `tools/mtmd/README-dev.md`
## Tips and tricks
+1
View File
@@ -118,6 +118,7 @@ Public API changes carry a higher bar than internal ones (`CONTRIBUTING.md`). Re
- In most cases, `build_vit` should be enough to build the transformer graph for vision models. Do not add a loop to build the transformer graph manually, unless you have a very good reason to do so. If you do, please explain why in the PR description.
- If you need a dedicated preprocessor, there is a high chance that it can be a derived class from one of the existing preprocessors. Check carefully before adding a new preprocessor class.
- If the model need a new public API in `mtmd.h`, open a discussion first.
- For audio generation models, see `tools/mtmd/README-dev.md`
## General (always)
+3
View File
@@ -65,11 +65,14 @@ Due to wide variety of audio generation pipelines, the `mtmd_gen_audio` system i
- If model used hard-coded embedding row ID, append them to token embeddings and assign token name for them (see `qwen3tts.py`)
- If model have a specific output logits head for audio codes (usually semantic code), keep the head as-is and pad the logits at inference time (see `src/models/qwen3vl.cpp`)
- Sidecar models (code2wav, bigvgan, etc) must live inside the mmproj GGUF (but can be in different `clip_context` if necessary)
- Note: it should use `ggml_build_forward_select` to select graphs if multiple graphs living in the same context
- Reuse existing GGUF metadata key name and tensor name whenever possible; think twice before adding extensive changes to GGUF writer. For example, Qwen3-TTS hard-code part of the hparams to `clip.cpp` as they won't likely to change.
3. Make sure most of the changes happen inside `mtmd-helper-gen.cpp`. A good PR looks like this:
- 10-20% changes is to add new backbone (text) model and conversion
- 60% changes inside `mtmd-helper-gen.cpp`
- 10% changes inside `libmtmd` and `clip.cpp` systems
- The rest downstream code (CLI, server) should have no changes at all
4. Update usage documentation in `tools/tts/README.md`
IMPORTANT: If your model needs changes that don't fit the existing infrastructure, **open an issue first for discussion**.