mirror of
https://github.com/LostRuins/koboldcpp.git
synced 2026-08-30 00:50:49 +02:00
Pedro Cuenca
62bf73d25c
* Get started with Onyx * Add architecture * Skip keys handled in super() * Loading tensors * Shorten * Graph * Apply suggestion from @pcuenca * Remove norm now embedding in transformers weights * Add eot * Explicit output_multiplier * Handle post_norm_eps * No super call; unhardcode eot. The pattern `self._set_vocab_gpt2()` seems preferred throughout the codebase, and it allows `set_vocab()` to be called from a different part of the Python class hierarchy: the drafter model converter that we may need eventually. * Register for drafting * DFlash: inherit rope type from the linked target. Another option would be to store it in the gguf file itself. * mmproj conversion Note: some fields to be renamed after the implementation works. We are keeping compatibility with the reference Meta gguf for testing purposes. * "clip" header declarations * Load mmproj * Pre-processing * Graph * Go back to using delimiters. Otherwise our generations are worse. Transformers does not use them. We need to trace inputs to verify whether they are equivalent. * downsample_factor -> merge_size * Add vision graph lol, forgot from a previous commit * Additional renames, align with llama.cpp / transformers * Prefer _size instead of independent _h and _w * Fix token layout Co-authored-by: Young Han <younghan@fb.com> * onyx: bring the chat parser onto the onyx branch common/chat.cpp on this branch has no Onyx handling, so a converted model serves malformed chat: the assistant preamble leaks into content ("to=self<|message|>...") and tool calls fail with HTTP 500 "The model produced output that does not match the expected peg-native format" common_chat_params_init_onyx exists on onyx-fair-patch, added there by 8bb73dd3d. It was never on this branch, so this is not a regression -- the two lines developed independently. The code here is taken verbatim from that commit. It is the clean side of `git merge origin/onyx-fair-patch`: chat.cpp is one of the files that merges without conflict. The full merge is not viable -- it produces 13 conflicts, including add/add on conversion/onyx.py and src/models/onyx.cpp where the q_norm-folding and metadata-scale approaches contradict each other, and #4/#7 are stacked on this branch's side of that. Verified on this branch: builds with 0 errors, converts an Onyx checkpoint, and serving it gives "4" for "What is 2+2?" plus a correct get_weather {"city":"Paris"} tool call, where the unported branch gives the two failures above. No converter or runtime changes are included, so this should not interact with the q_norm work. Co-authored-by: Beto de Paola <betodepaola@meta.com> * Less params, bilinear pos-emb interpolation as a graph op instead of CPU * Map to symbolic V_MMPROJ instead of strings * Make a couple params explicit * Patchify via build_inp() * No param for rope_theta * Small cleanup * Restore blank line * Unpermute, to adapt to the latest transformers checkpoint * Apply norm after token embeddings This follows the latest transformers approach. * Remove duplicated function * build_vit * onyx: use the model rope theta on sliding-window layers * DFlash: conversion from transformers drafter * Revert rope_type derivation from target NOTE: this breaks compatibility with Meta's distributed DFlash GGUFs, as the Q/K are stored in "NEOX" (rotated half) format, like in transformers. * Apply suggestion from @pcuenca * Set model type * Remove comment that will become obsolete * Hardcode post_norm_rms_eps instead of new param * Derive SWA+RoPE pattern from gguf array or scalar * Fix model type <-> number of layers * Reorder * Rename * Fix typo * DFlash: seed the draft KV cache from multimodal embedding batches `common_speculative_impl_draft_dflash::process()` returned early on any batch carrying embeddings, so an image prefill never had its target-layer features fused through the DFlash encoder and injected into the draft's KV cache. That left a hole spanning the image's positions, and the next injection at a post-image position failed to initialize its batch: ``` decoding image batch 1/1, n_tokens_batch = 256 decode: failed to initialize batch llama_decode: failed to decode, ret = -1 process: llama_decode(ctx_dft) failed rc=-1 (n_tokens=17, offset=0) srv decode: failed to process speculative batch ``` Every image request with `--spec-type draft-dflash` failed with HTTP 500. Text-only was unaffected, since those batches carry token ids and were let through. Restore the earlier condition, which admits a batch that is either tokens or embeddings and skips only the degenerate neither/both cases. The rest of `process()` is already layout-agnostic -- it gathers features via `llama_get_embeddings_layer_inp()` and indexes `batch_in.pos[]` / `batch_in.seq_id[]`, none of which assume token ids -- so this is the whole fix. Validated against `muse-glimmer-30B-bf16.gguf` + `mmproj-muse-glimmer-30B-bf16.gguf` + a DFlash draft head, on an image describe-the-shapes request: - before: HTTP 500, `failed to process speculative batch` - after: HTTP 200, draft acceptance 0.34012 (167 accepted / 491 generated), mean len 3.04 Output equivalence holds, which is the property that matters: at temperature 0 the drafted response is byte-identical to the same request served with no draft attached (1213/1213 chars), so the draft is drafting correctly through the image context rather than merely not crashing. * Conversion: prefer rewrite to mapping * Revert "Conversion: prefer rewrite to mapping" This reverts commit a92d0ac584d315e876741e85b6dad3dbc8b23bf7. * fix lint * sliding_window metadata is not optional * disable state save/load * Apply suggestion from @pcuenca --------- Co-authored-by: Young Han <younghan@fb.com> Co-authored-by: Beto de Paola <betodepaola@meta.com> Co-authored-by: Daniel Han <michaelhan2050@gmail.com> Co-authored-by: ruanrms <ruanslv@gmail.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
378 lines
14 KiB
Python
378 lines
14 KiB
Python
from __future__ import annotations
|
|
|
|
from .base import (
|
|
ModelBase, TextModel, MmprojModel, ModelType, SentencePieceTokenTypes,
|
|
logger, _mistral_common_installed, _mistral_import_error_msg,
|
|
get_model_architecture, LazyTorchTensor,
|
|
)
|
|
from typing import Type
|
|
|
|
|
|
__all__ = [
|
|
"ModelBase", "TextModel", "MmprojModel", "ModelType", "SentencePieceTokenTypes",
|
|
"get_model_architecture", "LazyTorchTensor", "logger",
|
|
"_mistral_common_installed", "_mistral_import_error_msg",
|
|
"get_model_class", "print_registered_models", "load_all_models",
|
|
]
|
|
|
|
|
|
TEXT_MODEL_MAP: dict[str, str] = {
|
|
"AfmoeForCausalLM": "afmoe",
|
|
"LagunaForCausalLM": "laguna",
|
|
"ApertusForCausalLM": "llama",
|
|
"ArceeForCausalLM": "llama",
|
|
"ArcticForCausalLM": "arctic",
|
|
"AudioFlamingo3ForConditionalGeneration": "qwen",
|
|
"BaiChuanForCausalLM": "baichuan",
|
|
"BaichuanForCausalLM": "baichuan",
|
|
"BailingMoeForCausalLM": "bailingmoe",
|
|
"BailingMoeV2ForCausalLM": "bailingmoe",
|
|
"BambaForCausalLM": "granite",
|
|
"BertForMaskedLM": "bert",
|
|
"BertForSequenceClassification": "bert",
|
|
"BertModel": "bert",
|
|
"BitnetForCausalLM": "bitnet",
|
|
"BitNetForCausalLM": "bitnet",
|
|
"BloomForCausalLM": "bloom",
|
|
"BloomModel": "bloom",
|
|
"CamembertModel": "bert",
|
|
"ChameleonForCausalLM": "chameleon",
|
|
"ChameleonForConditionalGeneration": "chameleon",
|
|
"ChatGLMForConditionalGeneration": "chatglm",
|
|
"ChatGLMModel": "chatglm",
|
|
"CodeShellForCausalLM": "codeshell",
|
|
"CogVLMForCausalLM": "cogvlm",
|
|
"Cohere2MoeForCausalLM": "command_r",
|
|
"Cohere2ForCausalLM": "command_r",
|
|
"CohereForCausalLM": "command_r",
|
|
"DbrxForCausalLM": "dbrx",
|
|
"DeciLMForCausalLM": "deci",
|
|
"DeepseekForCausalLM": "deepseek",
|
|
"DeepseekOCRForCausalLM": "deepseek",
|
|
"DeepseekV2ForCausalLM": "deepseek",
|
|
"DeepseekV3ForCausalLM": "deepseek",
|
|
"DeepseekV32ForCausalLM": "deepseek",
|
|
"DFlashDraftModel": "qwen",
|
|
"Qwen3DSparkModel": "qwen",
|
|
"DeepseekV4ForCausalLM": "deepseek",
|
|
"DeepseekV4DSparkModel": "deepseek",
|
|
"DistilBertForMaskedLM": "bert",
|
|
"DistilBertForSequenceClassification": "bert",
|
|
"DistilBertModel": "bert",
|
|
"Dots1ForCausalLM": "dots1",
|
|
"DotsOCRForCausalLM": "qwen",
|
|
"DreamModel": "dream",
|
|
"Ernie4_5ForCausalLM": "ernie",
|
|
"Ernie4_5_ForCausalLM": "ernie",
|
|
"Ernie4_5_MoeForCausalLM": "ernie",
|
|
"EuroBertModel": "bert",
|
|
"Exaone4_5_ForConditionalGeneration": "exaone",
|
|
"Exaone4ForCausalLM": "exaone",
|
|
"ExaoneForCausalLM": "exaone",
|
|
"ExaoneMoEForCausalLM": "exaone",
|
|
"ExaoneMoeForCausalLM": "exaone",
|
|
"FalconForCausalLM": "falcon",
|
|
"FalconH1ForCausalLM": "falcon_h1",
|
|
"FalconMambaForCausalLM": "mamba",
|
|
"GPT2LMHeadModel": "gpt2",
|
|
"GPTBigCodeForCausalLM": "starcoder",
|
|
"GPTNeoXForCausalLM": "gptneox",
|
|
"GPTRefactForCausalLM": "refact",
|
|
"Gemma2ForCausalLM": "gemma",
|
|
"Gemma3ForCausalLM": "gemma",
|
|
"Gemma3ForConditionalGeneration": "gemma",
|
|
"Gemma3TextModel": "gemma",
|
|
"Gemma3nForCausalLM": "gemma",
|
|
"Gemma3nForConditionalGeneration": "gemma",
|
|
"Gemma4AssistantForCausalLM": "gemma",
|
|
"Gemma4ForConditionalGeneration": "gemma",
|
|
"Gemma4ForCausalLM": "gemma",
|
|
"Gemma4UnifiedForConditionalGeneration": "gemma",
|
|
"Gemma4UnifiedAssistantForCausalLM": "gemma",
|
|
"GemmaForCausalLM": "gemma",
|
|
"Glm4ForCausalLM": "glm",
|
|
"Glm4MoeForCausalLM": "glm",
|
|
"Glm4MoeLiteForCausalLM": "glm",
|
|
"Glm4vForConditionalGeneration": "glm",
|
|
"Glm4vMoeForConditionalGeneration": "glm",
|
|
"GlmForCausalLM": "chatglm",
|
|
"GlmMoeDsaForCausalLM": "glm",
|
|
"GlmOcrForConditionalGeneration": "glm",
|
|
"GptOssForCausalLM": "gpt_oss",
|
|
"GraniteForCausalLM": "granite",
|
|
"GraniteMoeForCausalLM": "granite",
|
|
"GraniteMoeHybridForCausalLM": "granite",
|
|
"GraniteMoeSharedForCausalLM": "granite",
|
|
"GraniteSwitchForCausalLM": "granite",
|
|
"GraniteSpeechForConditionalGeneration": "granite",
|
|
"GraniteSpeechPlusForConditionalGeneration": "granite",
|
|
"Grok1ForCausalLM": "grok",
|
|
"GrokForCausalLM": "grok",
|
|
"GroveMoeForCausalLM": "grovemoe",
|
|
"HunYuanDenseV1ForCausalLM": "hunyuan",
|
|
"HunYuanMoEV1ForCausalLM": "hunyuan",
|
|
"HunYuanVLForConditionalGeneration": "hunyuan",
|
|
"HYV3ForCausalLM": "hunyuan",
|
|
"IQuestCoderForCausalLM": "llama",
|
|
"InternLM2ForCausalLM": "internlm",
|
|
"InternLM3ForCausalLM": "internlm",
|
|
"JAISLMHeadModel": "jais",
|
|
"Jais2ForCausalLM": "jais",
|
|
"JambaForCausalLM": "jamba",
|
|
"JanusForConditionalGeneration": "januspro",
|
|
"JinaBertForMaskedLM": "bert",
|
|
"JinaBertModel": "bert",
|
|
"JinaEmbeddingsV5Model": "bert",
|
|
"KORMoForCausalLM": "qwen",
|
|
"KimiK25ForConditionalGeneration": "deepseek",
|
|
"KimiLinearForCausalLM": "kimi_linear",
|
|
"KimiLinearModel": "kimi_linear",
|
|
"KimiVLForConditionalGeneration": "deepseek",
|
|
"LFM2ForCausalLM": "lfm2",
|
|
"LLaDAMoEModel": "llada",
|
|
"LLaDAMoEModelLM": "llada",
|
|
"LLaDAModelLM": "llada",
|
|
"LLaMAForCausalLM": "llama",
|
|
"Lfm25AudioTokenizer": "lfm2",
|
|
"Lfm2BidirectionalModel": "lfm2",
|
|
"Lfm2ForCausalLM": "lfm2",
|
|
"Lfm2Model": "lfm2",
|
|
"Lfm2MoeForCausalLM": "lfm2",
|
|
"Llama4ForCausalLM": "llama",
|
|
"Llama4ForConditionalGeneration": "llama",
|
|
"LlamaBidirectionalModel": "llama",
|
|
"LlamaForCausalLM": "llama",
|
|
"LlamaModel": "llama",
|
|
"Eagle3DraftModel": "llama",
|
|
"Eagle3Speculator": "llama",
|
|
"Eagle3LlamaForCausalLM": "llama",
|
|
"LlamaForCausalLMEagle3": "llama",
|
|
"LlavaForConditionalGeneration": "llama",
|
|
"LlavaStableLMEpochForCausalLM": "stablelm",
|
|
"MPTForCausalLM": "mpt",
|
|
"MT5ForConditionalGeneration": "t5",
|
|
"MaincoderForCausalLM": "maincoder",
|
|
"Mamba2ForCausalLM": "mamba",
|
|
"MambaForCausalLM": "mamba",
|
|
"MambaLMHeadModel": "mamba",
|
|
"MellumForCausalLM": "mellum",
|
|
"MiMoV2FlashForCausalLM": "mimo",
|
|
"MiMoV2ForCausalLM": "mimo",
|
|
"MiniCPM3ForCausalLM": "minicpm",
|
|
"MiniCPMForCausalLM": "minicpm",
|
|
"MiniCPMV4_6ForConditionalGeneration": "minicpm",
|
|
"MiniMaxM2ForCausalLM": "minimax",
|
|
"MiniMaxM3SparseForCausalLM": "minimax",
|
|
"MiniMaxM3SparseForConditionalGeneration": "minimax",
|
|
"Ministral3ForCausalLM": "mistral3",
|
|
"Mistral3ForConditionalGeneration": "mistral3",
|
|
"MistralForCausalLM": "llama",
|
|
"MixtralForCausalLM": "llama",
|
|
"ModernBertForMaskedLM": "bert",
|
|
"ModernBertForSequenceClassification": "bert",
|
|
"ModernBertModel": "bert",
|
|
"NanbeigeForCausalLM": "nanbeige",
|
|
"NemotronForCausalLM": "nemotron",
|
|
"NemotronHForCausalLM": "nemotron",
|
|
"NeoBERT": "bert",
|
|
"NeoBERTForSequenceClassification": "bert",
|
|
"NeoBERTLMHead": "bert",
|
|
"NomicBertModel": "bert",
|
|
"OLMoForCausalLM": "olmo",
|
|
"Olmo2ForCausalLM": "olmo",
|
|
"Olmo3ForCausalLM": "olmo",
|
|
"OlmoForCausalLM": "olmo",
|
|
"OlmoeForCausalLM": "olmo",
|
|
"MuseGlimmerAssistantModel": "muse_glimmer",
|
|
"MuseGlimmerForConditionalGeneration": "muse_glimmer",
|
|
"OpenELMForCausalLM": "openelm",
|
|
"OrionForCausalLM": "orion",
|
|
"PLMForCausalLM": "plm",
|
|
"PLaMo2ForCausalLM": "plamo",
|
|
"PLaMo3ForCausalLM": "plamo",
|
|
"PaddleOCRVLForConditionalGeneration": "ernie",
|
|
"PanguEmbeddedForCausalLM": "pangu",
|
|
"Phi3ForCausalLM": "phi",
|
|
"Phi4ForCausalLMV": "phi",
|
|
"PhiForCausalLM": "phi",
|
|
"PhiMoEForCausalLM": "phi",
|
|
"Plamo2ForCausalLM": "plamo",
|
|
"Plamo3ForCausalLM": "plamo",
|
|
"PlamoForCausalLM": "plamo",
|
|
"QWenLMHeadModel": "qwen",
|
|
"Qwen2AudioForConditionalGeneration": "qwen",
|
|
"Qwen2ForCausalLM": "qwen",
|
|
"Qwen2Model": "qwen",
|
|
"Qwen2MoeForCausalLM": "qwen",
|
|
"Qwen2VLForConditionalGeneration": "qwenvl",
|
|
"Qwen2VLModel": "qwenvl",
|
|
"Qwen2_5OmniModel": "qwenvl",
|
|
"Qwen2_5_VLForConditionalGeneration": "qwenvl",
|
|
"Qwen3ASRForConditionalGeneration": "qwen3vl",
|
|
"Qwen3ForCausalLM": "qwen",
|
|
"Qwen3Model": "qwen",
|
|
"Qwen3MoeForCausalLM": "qwen",
|
|
"Qwen3NextForCausalLM": "qwen",
|
|
"Qwen3OmniMoeForConditionalGeneration": "qwen3vl",
|
|
"Qwen3TTSForConditionalGeneration": "qwen3tts",
|
|
"Qwen3VLForConditionalGeneration": "qwen3vl",
|
|
"Qwen3VLMoeForConditionalGeneration": "qwen3vl",
|
|
"Qwen3_5ForCausalLM": "qwen",
|
|
"Qwen3_5ForConditionalGeneration": "qwen",
|
|
"Qwen3_5MoeForCausalLM": "qwen",
|
|
"Qwen3_5MoeForConditionalGeneration": "qwen",
|
|
"RND1": "qwen",
|
|
"RWForCausalLM": "falcon",
|
|
"RWKV6Qwen2ForCausalLM": "rwkv",
|
|
"RWKV7ForCausalLM": "rwkv",
|
|
"RobertaForSequenceClassification": "bert",
|
|
"RobertaModel": "bert",
|
|
"RuGPT3XLForCausalLM": "gpt2",
|
|
"Rwkv6ForCausalLM": "rwkv",
|
|
"Rwkv7ForCausalLM": "rwkv",
|
|
"RwkvHybridForCausalLM": "rwkv",
|
|
"Sarashina2VisionForCausalLM": "sarashina2",
|
|
"SarvamMoEForCausalLM": "bailingmoe",
|
|
"SeedOssForCausalLM": "olmo",
|
|
"SmallThinkerForCausalLM": "smallthinker",
|
|
"SmolLM3ForCausalLM": "llama",
|
|
"SolarOpenForCausalLM": "glm",
|
|
"StableLMEpochForCausalLM": "stablelm",
|
|
"StableLmForCausalLM": "stablelm",
|
|
"Starcoder2ForCausalLM": "starcoder",
|
|
"Step3p5ForCausalLM": "step3",
|
|
"StepVLForConditionalGeneration": "step3",
|
|
"Step3p7ForConditionalGeneration": "step3",
|
|
"T5EncoderModel": "t5",
|
|
"T5ForConditionalGeneration": "t5",
|
|
"T5WithLMHeadModel": "t5",
|
|
"TalkieForCausalLM": "talkie",
|
|
"UMT5ForConditionalGeneration": "t5",
|
|
"UMT5Model": "t5",
|
|
"UltravoxModel": "ultravox",
|
|
"UnlimitedOCRForCausalLM": "deepseek",
|
|
"VLlama3ForCausalLM": "llama",
|
|
"VoxtralForConditionalGeneration": "llama",
|
|
"WavTokenizerDec": "wavtokenizer",
|
|
"XLMRobertaForSequenceClassification": "bert",
|
|
"XLMRobertaModel": "bert",
|
|
"XverseForCausalLM": "xverse",
|
|
"YoutuForCausalLM": "deepseek",
|
|
"YoutuVLForConditionalGeneration": "deepseek",
|
|
"modeling_grove_moe.GroveMoeForCausalLM": "grovemoe",
|
|
"modeling_sarvam_moe.SarvamMoEForCausalLM": "bailingmoe",
|
|
}
|
|
|
|
|
|
MMPROJ_MODEL_MAP: dict[str, str] = {
|
|
"AudioFlamingo3ForConditionalGeneration": "ultravox",
|
|
"CogVLMForCausalLM": "cogvlm",
|
|
"DeepseekOCR2ForCausalLM": "deepseek",
|
|
"DeepseekOCRForCausalLM": "deepseek",
|
|
"DotsOCRForCausalLM": "dotsocr",
|
|
"Exaone4_5_ForConditionalGeneration": "exaone",
|
|
"Gemma3ForConditionalGeneration": "gemma",
|
|
"Gemma3nForConditionalGeneration": "gemma",
|
|
"Gemma4ForConditionalGeneration": "gemma",
|
|
"Gemma4UnifiedForConditionalGeneration": "gemma",
|
|
"Glm4vForConditionalGeneration": "qwen3vl",
|
|
"Glm4vMoeForConditionalGeneration": "qwen3vl",
|
|
"Glm5vForConditionalGeneration": "kimivl",
|
|
"GlmOcrForConditionalGeneration": "qwen3vl",
|
|
"GlmasrModel": "ultravox",
|
|
"Granite4VisionForConditionalGeneration": "granite",
|
|
"GraniteSpeechForConditionalGeneration": "granite",
|
|
"GraniteSpeechPlusForConditionalGeneration": "granite",
|
|
"HunYuanVLForConditionalGeneration": "hunyuan",
|
|
"Idefics3ForConditionalGeneration": "smolvlm",
|
|
"InternVisionModel": "internvl",
|
|
"JanusForConditionalGeneration": "januspro",
|
|
"KimiK25ForConditionalGeneration": "kimivl",
|
|
"KimiVLForConditionalGeneration": "kimivl",
|
|
"Lfm2AudioForConditionalGeneration": "lfm2",
|
|
"Lfm2VlForConditionalGeneration": "lfm2",
|
|
"LightOnOCRForConditionalGeneration": "lighton_ocr",
|
|
"Llama4ForConditionalGeneration": "llama4",
|
|
"LlavaForConditionalGeneration": "llava",
|
|
"MERaLiON2ForConditionalGeneration": "ultravox",
|
|
"MiMoV2ForCausalLM": "mimo",
|
|
"MiniMaxM3SparseForConditionalGeneration": "minimax",
|
|
"MiniCPMV4_6ForConditionalGeneration": "minicpm",
|
|
"Mistral3ForConditionalGeneration": "llava",
|
|
"NemotronH_Nano_VL_V2": "nemotron",
|
|
"MuseGlimmerForConditionalGeneration": "muse_glimmer",
|
|
"PaddleOCRVisionModel": "ernie",
|
|
"Phi4ForCausalLMV": "phi",
|
|
"Qwen2AudioForConditionalGeneration": "ultravox",
|
|
"Qwen2VLForConditionalGeneration": "qwenvl",
|
|
"Qwen2VLModel": "qwenvl",
|
|
"Qwen2_5OmniModel": "qwenvl",
|
|
"Qwen2_5_VLForConditionalGeneration": "qwenvl",
|
|
"Qwen3ASRForConditionalGeneration": "qwen3vl",
|
|
"Qwen3OmniMoeForConditionalGeneration": "qwen3vl",
|
|
"Qwen3TTSForConditionalGeneration": "qwen3tts",
|
|
"Qwen3VLForConditionalGeneration": "qwen3vl",
|
|
"Qwen3VLMoeForConditionalGeneration": "qwen3vl",
|
|
"Qwen3_5ForConditionalGeneration": "qwen3vl",
|
|
"Qwen3_5MoeForConditionalGeneration": "qwen3vl",
|
|
"RADIOModel": "nemotron",
|
|
"Sarashina2VisionForCausalLM": "sarashina2",
|
|
"SmolVLMForConditionalGeneration": "smolvlm",
|
|
"StepVLForConditionalGeneration": "step3",
|
|
"Step3p7ForConditionalGeneration": "step3",
|
|
"UltravoxModel": "ultravox",
|
|
"UnlimitedOCRForCausalLM": "deepseek",
|
|
"VoxtralForConditionalGeneration": "ultravox",
|
|
"YoutuVLForConditionalGeneration": "youtuvl",
|
|
}
|
|
|
|
|
|
_TEXT_MODEL_MODULES = sorted(set(TEXT_MODEL_MAP.values()))
|
|
_MMPROJ_MODEL_MODULES = sorted(set(MMPROJ_MODEL_MAP.values()))
|
|
|
|
|
|
_loaded_text_modules: set[str] = set()
|
|
_loaded_mmproj_modules: set[str] = set()
|
|
|
|
|
|
def load_all_models() -> None:
|
|
"""Import all model modules to trigger @ModelBase.register() decorators."""
|
|
if len(_loaded_text_modules) != len(_TEXT_MODEL_MODULES):
|
|
for module_name in _TEXT_MODEL_MODULES:
|
|
if module_name not in _loaded_text_modules:
|
|
try:
|
|
__import__(f"conversion.{module_name}")
|
|
_loaded_text_modules.add(module_name)
|
|
except Exception as e:
|
|
logger.warning(f"Failed to load model module {module_name}: {e}")
|
|
|
|
if len(_loaded_mmproj_modules) != len(_MMPROJ_MODEL_MODULES):
|
|
for module_name in _MMPROJ_MODEL_MODULES:
|
|
if module_name not in _loaded_mmproj_modules:
|
|
try:
|
|
__import__(f"conversion.{module_name}")
|
|
_loaded_mmproj_modules.add(module_name)
|
|
except Exception as e:
|
|
logger.warning(f"Failed to load model module {module_name}: {e}")
|
|
|
|
|
|
def get_model_class(name: str, mmproj: bool = False) -> Type[ModelBase]:
|
|
"""Dynamically import and return a model class by its HuggingFace architecture name."""
|
|
relevant_map = MMPROJ_MODEL_MAP if mmproj else TEXT_MODEL_MAP
|
|
if name not in relevant_map:
|
|
raise NotImplementedError(f"Architecture {name!r} not supported!")
|
|
module_name = relevant_map[name]
|
|
__import__(f"conversion.{module_name}")
|
|
model_type = ModelType.MMPROJ if mmproj else ModelType.TEXT
|
|
return ModelBase._model_classes[model_type][name]
|
|
|
|
|
|
def print_registered_models() -> None:
|
|
load_all_models()
|
|
logger.error("TEXT models:")
|
|
for name in sorted(TEXT_MODEL_MAP.keys()):
|
|
logger.error(f" - {name}")
|
|
logger.error("MMPROJ models:")
|
|
for name in sorted(MMPROJ_MODEL_MAP.keys()):
|
|
logger.error(f" - {name}")
|