Block Diffusion Interpolates Between Autoregressive and Diffusion Language Models (ICLR 2025 Oral) By Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, Volodymyr Kuleshov We introduce BD3 LMs , a family of B lock D iscrete D enoising D iffusion L anguage M odels that achieve SOTA likelihoods among diffusion models and enable generation of arbitrary length sequences. BD3 LMs combine the strengths of autoregressive and diffusion language models by decomposing a token sequence into blocks and performing discrete diffusion within each block. By tuning the block size, we interpolate between autoregressive and diffusion models which introduces a trade off between quality and sample efficiency. We propose a recipe of building effective BD3 LMs that includes an efficient training algorithm, estimators of gradient variance, and data driven noise schedules to minimize the variance. Model Description BD3 LMs are Block Discrete Denoising Diffusion Language Models. They combine the strengths of autoregressive and diffusion language models by decomposing a token sequence into blocks and performing discrete diffusion within each block. How t…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy