Llama 3.2 1B Instruct FP8 Model Overview Model Architecture: Llama 3 Input: Text Output: Text Model Optimizations: Activation quantization: FP8 Weight quantization: FP8 Intended Use Cases: Intended for commercial and research use multiple languages. Similarly to Llama 3.2 1B Instruct, this models is intended for assistant like chat. Out of scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Release Date: 9/25/2024 Version: 1.0 License(s): Llama3.2 Model Developers: Neural Magic Quantized version of Llama 3.2 1B Instruct. It achieves scores within 1.0% of the scores of the unquantized model for MMLU, ARC Challenge, GSM 8k, Hellaswag, Winogrande and TruthfulQA. Model Optimizations This model was obtained by quantizing the weights of Llama 3.2 1B Instruct to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%. Only weights and activations of the linear operators withi…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy