[!IMPORTANT] This repository is a community driven quantized version of the original model mistralai/Mixtral 8x7B Instruct v0.1 which is the BF16 half precision official version released by Mistral AI. Model Information The Mixtral 8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts. The Mixtral 8x7B outperforms Llama 2 70B on most benchmarks they tested. For full details of this model please read our release blog post. This repository contains mistralai/Mixtral 8x7B Instruct v0.1 quantized using AutoAWQ from FP16 down to INT4 using the GEMM kernels performing zero point quantization with a group size of 128. Model Usage [!NOTE] In order to run the inference with Mixtral 8x7B Instruct AWQ in INT4, around 24 GiB of VRAM are needed only for loading the model checkpoint, excluding the KV cache and/or the CUDA graphs, meaning that there should be a bit over that VRAM available. In order to use the current quantized model, support is offered for different solutions as transformers , autoawq , or text generation inference . 🤗 Transformers In order to run the inference with Mixtral 8x7B Instruct AWQ in INT4, you need to install the following packages: To…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy