olmOCR 2 7B 1025 GPTQ W4A16 INT4 weight only quantization of allenai/olmOCR 2 7B 1025 using GPTQ with group activation ordering. Benchmark results Evaluated on olmOCR bench sample data (23 pages) with vLLM 0.18.0: Aggregate absent baseline math order present table This model (GPTQ W4A16) 0.949 0.875 1.000 0.833 0.923 0.733 0.950 Base BF16 0.909 0.875 0.952 0.750 1.000 0.700 0.650 The quantized model exceeds the BF16 baseline (+4.4%) on this benchmark, primarily from stronger table and math handling. Inference time for 23 pages at concurrency 4: 144s vs 201s for the official FP8 model (with cuBLAS fallback). Quantization details Parameter Value Method GPTQ Weight dtype INT4 Activation dtype BF16 (weight only, no activation quantization) Group size 128 Symmetric Yes Activation ordering Group ( actorder="group" ) Format pack quantized (compressed tensors) Excluded layers lm head , re:model.visual. (entire visual encoder kept in BF16) Activation ordering actorder="group" reorders weight columns by descending Hessian diagonal magnitude within each group before packing. This allocates quantization precision to the most sensitive weight dimensions and consistently improves perplexity over…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy