transformers 5 defaults tokenizer.add_tokens() to normalized=True, so the CLIP
tokenizer lowercases added embedding names. tokenizer.tokenize() then returns the
lowercased surface, maybe_convert_prompt never matches mixed-case names in
added_tokens_encoder, and multi-vector expansion is skipped. Every mixed-case
multi-vector embedding collapsed to its first vector and looked ignored.
Add embedding tokens as AddedToken(name, normalized=False) so they stay
case-sensitive and tokenize() surfaces them verbatim. Valid on transformers 4.x
and 5.x; convert_tokens_to_ids and encoding are unaffected.
CalamitousFelicitousness
Vladimir Mandic
awsr