Mistral Small 3.1 24B Instruct 2503 GPTQ 4b 128g Model Overview This model was obtained by quantizing the weights of Mistral Small 3.1 24B Instruct 2503 to INT4 data type. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights of the linear operators within language model transformers blocks are quantized. Vision model and multimodal projection are kept in original precision. Weights are quantized using a symmetric per group scheme, with group size 128. The GPTQ algorithm is applied for quantization. Model checkpoint is saved in compressed tensors format. Evaluation This model was evaluated on the OpenLLM v1 benchmarks. Model outputs were generated with the vLLM engine. Model ArcC GSM8k Hellaswag MMLU TruthfulQA mc2 Winogrande Average Recovery : : : : : : : : : : : : : : : : Mistral Small 3.1 24B Instruct 2503 0.7125 0.8848 0.8576 0.8107 0.6409 0.8398 0.7910 1.0000 Mistral Small 3.1 24B Instruct 2503 INT4 (this) 0.7073 0.8711 0.8530 0.8062 0.6252 0.8256 0.7814 0.9878 Reproduction The results were obtained using the following commands: Usage To use the model in transformers up…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy