jina embeddings v5 omni small retrieval NVFP4 NVFP4 quantized version of jinaai/jina embeddings v5 omni small retrieval, produced using NVIDIA Model Optimizer ( nvidia modelopt 0.44.0 ). Quantization Details Property Value Method Post Training Quantization (PTQ) Format NVFP4 (W4A4) Config mtq.NVFP4 DEFAULT CFG with audio tower and audio projector excluded Calibration cnn dailymail 3.0.0, 128 samples, max seq len=512 Base Precision BF16 (non quantized layers remain BF16) GPU Required NVIDIA Blackwell (Compute Capability ≥ 10.0) Tool nvidia modelopt 0.44.0 What is and isn't quantized Component Precision Why Language model (Qwen3) NVFP4 W4A4 vLLM's Qwen3ForCausalLM is quantization aware Vision tower (Qwen3VL) NVFP4 W4A4 vLLM's Qwen3 VisionTransformer is quantization aware Audio tower BF16 (unquantized) transformers.Qwen2 5OmniAudioEncoder uses plain nn.Linear Audio projector BF16 (unquantized) Plain nn.Linear in vllm qwen3vl audio.py The audio path was explicitly excluded from quantization because the model's vllm qwen3vl audio.py constructs its audio components with plain (non quantization aware) nn.Linear layers. Quantizing those layers produces NVFP4 packed safetensors that don't f…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy