Model Details This model is an int4 model with group size 128 and symmetric quantization of Qwen/Qwen2.5 0.5B Instruct generated by intel/auto round. Load the model with revision="7cac2d1" to use AutoGPTQ format ⚠️ Important: This model is used for internal testing with VLLM. Please do not delete or modify without approval. How To Use INT4 Inference(CPU/HPU/CUDA) CPU requires auto round version 0.3.1 Evaluate the model pip3 install lm eval==0.4.5 Metric BF16 INT4 : : : : : Avg 0.4229 0.4124 leaderboard mmlu pro 5 shots 0.1877 0.1678 leaderboard ifeval inst level strict acc 0.3501 0.3441 leaderboard ifeval prompt level strict acc 0.2107 0.2218 mmlu 0.4582 0.4434 cmmlu 0.5033 0.4542 ceval valid 0.5327 0.4918 gsm8k 5 shots 0.2146 0.2267 lambada openai 0.4968 0.4692 hellaswag 0.4062 0.3927 winogrande 0.5541 0.5675 piqa 0.7051 0.7035 truthfulqa mc1 0.2693 0.2815 openbookqa 0.2400 0.2200 boolq 0.6783 0.6471 arc easy 0.6566 0.6595 arc challenge 0.3020 0.3072 Generate the model Here is the sample command to generate the model. We observed a larger accuracy drop in Chinese tasks and recommend using a high quality Chinese dataset for calibration or smaller group size like 32. Ethical Conside…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy