Meta Llama 3.1 70B Instruct FP8 KV Introduction This model was created by applying Quark with calibration samples from Pile dataset. Quantization Stragegy Quantized Layers : All linear layers excluding "lm head" Weight : FP8 symmetric per tensor Activation : FP8 symmetric per tensor KV Cache : FP8 symmetric per tensor Quick Start 1. Download and install Quark 2. Run the quantization script in the example folder using the following command line: Deployment Quark has its own export format and allows FP8 quantized models to be efficiently deployed using the vLLM backend(vLLM compatible). Evaluation Quark currently uses perplexity(PPL) as the evaluation metric for accuracy loss before and after quantization.The specific PPL algorithm can be referenced in the quantize quark.py. The quantization evaluation results are conducted in pseudo quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only. Evaluation scores Benchmark Meta Llama 3.1 70B Instruct Meta Llama 3.1 70B Instruct FP8 KV(this model) Perplexity wikitext2 3.7797 3.8561 License Modifications copyright(c) 2024 Advanced Micro Devices,Inc. All rights r…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy