jina-embeddings-v5-omni-small-retrieval-NVFP4

NVFP4-quantized version of jinaai/jina-embeddings-v5-omni-small-retrieval, produced using NVIDIA Model Optimizer (nvidia-modelopt 0.44.0).

Quantization Details

What is and isn't quantized

The audio path was explicitly excluded from quantization because the model's vllm_qwen3vl_audio.py constructs its audio components with plain (non-quantization-aware) nn.Linear layers. Quantizing those layers produces NVFP4-packed safetensors that don't fit the BF16 parameter shapes at load time. Keeping audio in BF16 costs ~30 MB on a ~1.1 GB model (roughly 3% size overhead) and lets the model serve out of the box with vLLM's --quantization modelopt_fp4 flag.

Hardware Requirements

• Quantized on: NVIDIA Thor (GB10, Blackwell CC 11.0)
• Inference requires: Blackwell GPU (B100, B200, GB10, GB200, etc.)
• Minimum CUDA: 13.0 / Driver 580.00+

Usage

vLLM (recommended)

vllm serve georgelpreput/jina-embeddings-v5-omni-small-retrieval-NVFP4 \
  --trust-remote-code \
  --runner pooling \
  --quantization modelopt_fp4

Use Jina v5's asymmetric retrieval prefixes for queries and documents:

import requests

r = requests.post("http://localhost:8000/v1/embeddings", json={
    "model": "georgelpreput/jina-embeddings-v5-omni-small-retrieval-NVFP4",
    "input": [
        "Query: What is the capital of France?",
        "Document: Paris is the capital city of France.",
    ],
})

Required pre-step: stage vllm_qwen3vl_audio.py

The model ships a custom vllm_qwen3vl_audio.py that self-registers Qwen3VLAudioModel with vLLM. In NGC vLLM 26.04-py3 (vLLM 0.19.0+nv26.04) this registration silently fails — even when loading via the HF Hub repo ID — because HuggingFace's trust_remote_code loader copies only the file listed in config.json's auto_map (modeling_qwen3vl_audio.py) into the transformers_modules/<SHA>/ cache namespace. The companion vllm_qwen3vl_audio.py lives in the snapshot dir but never gets copied into the import namespace, so modeling_qwen3vl_audio.py's from .vllm_qwen3vl_audio import _register_vllm raises ModuleNotFoundError. vLLM then falls back to its generic transformers backend with the wrong pooling, producing degenerate embeddings (every text → near-identical vector).

The fix is a one-time pre-stage. Run this once (or wrap vllm serve in a script that runs it on every start):

python3 - <<'PY'
import shutil
from pathlib import Path
from huggingface_hub import snapshot_download

snap = Path(snapshot_download(
    repo_id="georgelpreput/jina-embeddings-v5-omni-small-retrieval-NVFP4",
    allow_patterns=["*.py", "config.json"],
))
sha = snap.name
hf_home = Path.home() / ".cache" / "huggingface"
dst = hf_home / "modules" / "transformers_modules" / sha
dst.mkdir(parents=True, exist_ok=True)
(dst / "__init__.py").touch()
for src in snap.glob("*.py"):
    shutil.copy2(src, dst / src.name)
    print(f"staged {src.name}")
PY

This may be unnecessary on newer upstream vLLM versions (>= 0.20.1 per the Jina model card claims native support); we have not tested those builds.

Limitations

• NVFP4 inference is only supported on Blackwell GPUs (not Ampere, Hopper, or Ada)
• The CC BY-NC 4.0 license of the base model carries through — commercial use is not permitted
• Audio and video modalities were not included in the calibration set (text-only calibration)
• Audio tower and audio projector remain in BF16 (see "What is and isn't quantized" above)

License

CC BY-NC 4.0 (inherited from base model — non-commercial use only).