Files
koboldcpp/tools/mtmd/CMakeLists.txt
T
Daniel Bevenius 7e1e28cae3 mtmd : add Nemotron 3 Nano Omni support (parakeet) (#22520)
* mtmd : add Nemotron 3 Nano Omni support (parakeet)

This commit adds support for the subsampling and encoder part of
Nemotron Nemo 3 omni model.

The Parakeet subsampling/encoder were taken from parakeet.cpp which
is currently a pull request against whisper.cpp. I've tried to copy the
code a close as possible to hopefully enable easy patching between the
these two project later.

Refs: https://github.com/ggml-org/whisper.cpp/pull/3735

* mtmd : generate rel pos tensor in graph instead of in conversion [no ci]

This commit removes the generation of the relative positional tensor in
the model conversion script and instead computes it in the encoder
graph. This is only done for the window of positions required for the
current audio sample.

* mtmd : add clip_get_model to clip API [no ci]

This commit adds a function to get access to the clip_model. It also
removes the two functions clip_get_mel_filter_tensor, and
clip_get_window_tensor(const struct clip_ctx * ctx) which can now use
clip_get_model to access the model tensors that it needs.

* mtmd : read mel_filters and window into hparams

* mtmd : use set_input_f32 lambda [no ci]

* mtmd : add better asserts for mel_filters and hann window [no ci]

* mtmd : add missing size_t cast

* mtmd : change type of pad to size_t

* mtmd : zero initialize samples_padded

* mtmd : remove unsued ctx member from parakeet preprocessor

* mtmd : make log_mel_spectrogram_parakeet_worker_thread private static

* mtmd : sync/update parakeeet impl with latest whisper.cpp

This commit updates the parakeet code in mtmd to reflect the latest
updates to parakeet.cpp in whisper.cpp.

A follow up commit will address the currently hardcoded dw_pad and see
if we can add n_conv_kernel as a model metadata field.

* mtmd : add audio_conv_kernel_size to model conversion

This commit updates the model conversion to read the conv_kernel_size
field from the sound_config section of the models config.json file.
It then uses this field instead of the hardcoded values in parakeet.cpp.

* mtmd : cleanup [no ci]

* conversion : call super().filter_tensors [no ci]

* do not discard result of super filter_tensors

* mtmd : use build_mm instead of ggml_mul_mat

* mtmd : use build_ffn

* mtmd : move and reuse get_vector lambda

* mtmd : use build_inp_raw for parakeet

* mtmd : throw exception in get_scalar instead of assert

* mtmd : fix std::min call

* mtmt : use .c_str in throw clause in get_vector

* mtmd : check for F32 type and non-empty tensor in get_vector

The get_vector lambda is used by get_scalar but also standalone to read
in the mel_filters and the window data. Therefor we are not checking
for 1D tensors but allowing multiple dimensions. We do have a check in
get_scalar to verify the size of the vector.

* mtmd : replace hardcoded 1101 for n_tokens_real

* mtmd : assert subsampling_factor is 8

This commit adds an assert of the parakeet subsampling factor to check
that it is 8.

The motivation for this is that this model currently has three
convolutions with a stride of 2. If the underlying model updates the
subsampling factor these convolution operations will need to be updated
and this will produce and error if this occurs.

* mtmd : remove unused ggml_tensors attn_pos_w and mm_norm_w

* mtmd : remove single thread path

This commit removes the single thread path which was a left over from
the original parakeet.cpp where n_threads is configurable.

* fix some security issues

---------

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-07-28 17:20:25 +02:00

153 lines
5.0 KiB
CMake

# mtmd
set(MTMD_VIDEO_HELP "enable video support in mtmd (requires ffmpeg binary in PATH)")
set(MTMD_VIDEO ON CACHE BOOL "${MTMD_VIDEO_HELP}")
# TODO: add MTMD_VIDEO_METHOD in the future to select between ffmpeg and other backends
if (MTMD_VIDEO AND NOT LLAMA_SUBPROCESS)
message(STATUS "Disabling MTMD_VIDEO because LLAMA_SUBPROCESS is OFF")
set(MTMD_VIDEO OFF CACHE BOOL "${MTMD_VIDEO_HELP}" FORCE)
endif()
find_package(Threads REQUIRED)
add_library(mtmd
mtmd.cpp
mtmd-audio.cpp
mtmd-image.cpp
mtmd.h
mtmd-helper.cpp
mtmd-helper.h
clip.cpp
clip.h
clip-impl.h
clip-model.h
clip-graph.h
models/models.h
models/cogvlm.cpp
models/conformer.cpp
models/dotsocr.cpp
models/exaone4_5.cpp
models/gemma4a.cpp
models/gemma4v.cpp
models/gemma4ua.cpp
models/gemma4uv.cpp
models/glm4v.cpp
models/granite-speech.cpp
models/granite4-vision.cpp
models/hunyuanvl.cpp
models/internvl.cpp
models/kimivl.cpp
models/kimik25.cpp
models/nemotron-v2-vl.cpp
models/llama4.cpp
models/llava.cpp
models/minicpmv.cpp
models/paddleocr.cpp
models/pixtral.cpp
models/qwen2vl.cpp
models/minimax-m3.cpp
models/qwen3vl.cpp
models/mimovl.cpp
models/qwen3a.cpp
models/mimo-audio.cpp
models/step3vl.cpp
models/siglip.cpp
models/whisper-enc.cpp
models/deepseekocr.cpp
models/deepseekocr2.cpp
models/mobilenetv5.cpp
models/youtuvl.cpp
models/yasa2.cpp
models/parakeet.cpp
)
set_target_properties(mtmd PROPERTIES
VERSION ${LLAMA_INSTALL_VERSION}
SOVERSION 0
MACHO_CURRENT_VERSION 0 # keep macOS linker from seeing oversized version number
)
target_link_libraries (mtmd PUBLIC ggml llama)
target_link_libraries (mtmd PRIVATE Threads::Threads)
target_include_directories(mtmd PUBLIC .)
target_include_directories(mtmd PRIVATE ../..)
target_include_directories(mtmd PRIVATE ../../vendor)
target_compile_features (mtmd PRIVATE cxx_std_17)
if (MTMD_VIDEO)
target_compile_definitions(mtmd PRIVATE MTMD_VIDEO)
endif()
if (BUILD_SHARED_LIBS)
set_target_properties (mtmd PROPERTIES POSITION_INDEPENDENT_CODE ON)
target_compile_definitions(mtmd PRIVATE LLAMA_BUILD)
target_compile_definitions(mtmd PUBLIC LLAMA_SHARED)
endif()
set(MTMD_PUBLIC_HEADERS
${CMAKE_CURRENT_SOURCE_DIR}/mtmd.h
${CMAKE_CURRENT_SOURCE_DIR}/mtmd-helper.h
)
set_target_properties(mtmd
PROPERTIES
PUBLIC_HEADER "${MTMD_PUBLIC_HEADERS}")
set_target_properties(mtmd
PROPERTIES
PRIVATE_HEADER debug/mtmd-debug.h)
install(TARGETS mtmd LIBRARY PUBLIC_HEADER)
if (NOT MSVC)
# for stb_image.h and miniaudio.h
target_compile_options(mtmd PRIVATE -Wno-cast-qual)
endif()
if (ANDROID)
# miniaudio.h defines ma_android_sdk_version() without a prior prototype
target_compile_options(mtmd PRIVATE -Wno-missing-prototypes)
endif()
if (TARGET BUILD_INFO)
add_dependencies(mtmd BUILD_INFO)
add_dependencies(mtmd-helper BUILD_INFO)
endif()
# if mtmd is linked against llama-common, we throw an error
if (TARGET mtmd)
get_target_property(libs mtmd LINK_LIBRARIES)
if (libs AND "llama-common" IN_LIST libs)
message(FATAL_ERROR "mtmd is designed to be a public library.\n"
"It must not link against llama-common")
endif()
endif()
# Gate CLI binaries on LLAMA_BUILD_TOOLS so that standalone library-only
# builds (LLAMA_BUILD_MTMD=ON with LLAMA_BUILD_TOOLS=OFF — e.g. Apple
# XCFramework packaging) skip the executables entirely. LLAMA_BUILD_COMMON
# defaults to ON in standalone builds, so we cannot rely on it for gating.
if (LLAMA_BUILD_TOOLS)
add_executable(llama-llava-cli deprecation-warning.cpp)
add_executable(llama-gemma3-cli deprecation-warning.cpp)
add_executable(llama-minicpmv-cli deprecation-warning.cpp)
add_executable(llama-qwen2vl-cli deprecation-warning.cpp)
set(TARGET llama-mtmd-cli)
add_executable (${TARGET} mtmd-cli.cpp)
set_target_properties (${TARGET} PROPERTIES OUTPUT_NAME llama-mtmd-cli)
if(LLAMA_TOOLS_INSTALL)
install(TARGETS ${TARGET} RUNTIME)
endif()
target_link_libraries (${TARGET} PRIVATE llama-common mtmd Threads::Threads)
target_compile_features(${TARGET} PRIVATE cxx_std_17)
# mtmd-debug tool
add_executable(llama-mtmd-debug debug/mtmd-debug.cpp)
set_target_properties(llama-mtmd-debug PROPERTIES OUTPUT_NAME llama-mtmd-debug)
target_link_libraries(llama-mtmd-debug PRIVATE llama-common mtmd Threads::Threads)
target_compile_features(llama-mtmd-debug PRIVATE cxx_std_17)
endif()