EAGLE EAGLE EAGLE 2 EAGLE 3 Blog EAGLE (Extrapolation Algorithm for Greater Language model Efficiency) is a new baseline for fast decoding of Large Language Models (LLMs) with provable performance maintenance. This approach involves extrapolating the second top layer contextual feature vectors of LLMs, enabling a significant boost in generation efficiency. EAGLE is: certified by the third party evaluation as the fastest speculative method so far. achieving 2x speedup on gpt fast . 3x faster than vanilla decoding (13B). 2x faster than Lookahead (13B). 1.6x faster than Medusa (13B). provably maintaining the consistency with vanilla decoding in the distribution of generated texts. trainable (within 1 2 days) and testable on 8x RTX 3090 GPUs. So even the GPU poor can afford it. combinable with other parallelled techniques such as vLLM, DeepSpeed, Mamba, FlashAttention, quantization, and hardware optimization. EAGLE 2 uses the confidence scores from the draft model to approximate acceptance rates, dynamically adjusting the draft tree structure, which further enhances performance. EAGLE 2 is: 4x faster than vanilla decoding (13B). 1.4x faster than EAGLE 1 (13B). EAGLE 3 removes the…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy