Qwen3 Reranker 4B W4A16 G128 GPTQ Quantized Qwen/Qwen3 Reranker 4B with Ultrachat, THUIR/T2Ranking and m a p/COIG CQIA for calibration set. What's the benefit? VRAM Usage: 17430M 11000M (w/o FA2, according to Embedding model's result). What's the cost? I think <5% accuracy, further evaluation on the way... The Embedding one shows ~0.7% . How to use it? pip install compressed tensors optimum and auto gptq / gptqmodel , then goto the official usage guide.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy