SeerAttention Llama 3.1 8B AttnGates This repo only contains the AttnGates' weights for Llama 3.1 8B Instruct Model. SeerAttention introduces learnable AttnGate modules to accelerate the computationally intensive prefill stage of long context large language models (LLMs) via dynamic block level sparsity. The AttnGates are trained in a parameter efficient self distillation framework, where they learn to mimic the 2D max pooled attention patterns of the original frozen model, preserving its integrity while avoiding costly retraining. During inference, these gates generate block sparse binary masks by applying threshold/TopK to their learned soft scores, enabling efficient computation through a custom block sparse FlashAttention kernel. Original Github Repo https://github.com/microsoft/SeerAttention. Evaluation Results Perplexity on PG19 Density 8192 16384 32768 65536 131072 1.00 10.03 9.88 9.92 9.97 10.03 0.50 10.04 9.89 9.92 9.99 10.05 0.40 10.06 9.89 9.93 9.99 10.07 0.30 10.09 9.91 9.95 10.01 10.15 0.20 10.19 9.94 9.97 10.04 10.37 0.10 10.61 10.08 10.04 10.09 10.88 LongBench With threshold set to 2e 3. Task 0 4k 4 8k 8k+ 2wikimqa 51.1 47.85 33.36 gov report 35.03 35.05 34.57 hotpot…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy