KVzap KVzap is a fast, adaptive, and faithful KV cache pruning method aiming to accelerate LLM inference in both prefilling and decoding. It applies a lightweight model to the hidden states to predict importance scores for every KV pair and prunes the ones with a score below a given threshold, following the Dynamic Memory Sparsification (DMS) inference strategy. The method was introduced in the paper KVzap: Fast, Adaptive, and Faithful KV Cache Pruning. KVzap is trained as a fast approximation of KVzip+, using 1.2M samples from Nemotron Pretraining Dataset sample. Training code is available in the kvpress repository. Usage KVzap can be used with the kvpress library, through the custom KVPressTextGenerationPipeline , which is automatically registered as a transformers pipeline with the name kv press text generation when kvpress is imported: Citation If you use KVzap in your research, please cite the following paper:
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy