Scaling Behavior of Discrete Diffusion Language Models Dimitri von Rütte, Janis Fluri, Antonio Orvieto, Omead Pooladzandi, Bernhard Schölkopf, Thomas Hofmann This repository contains the model checkpoints from the paper "Scaling Behavior of Discrete Diffusion Language Models". In our paper, we investigate the scaling behavior of discrete diffusion language models (DLMs) for different noise types (masking, uniform, and hybrid noise), finding that all of them scale well in compute bound settings and especially in token bound settings, with uniform noise coming out on top for the latter. To confirm these findings, we train scaled up models to compute optimality. Specifically, we train two 3B models (masked and uniform diffusion) as well as a 10B parameter uniform diffusion model, which, to the best of our knowledge, is the largest public uniform diffusion model to date. Model Size Train. PPL Diffusion type HuggingFace Link : : : : : gidd unif 10b 10B 9.15 uniform https://huggingface.co/dvruette/gidd unif 10b gidd mask 3b 3B 11.3 masked https://huggingface.co/dvruette/gidd mask 3b gidd unif 3b 3B 11.7 uniform https://huggingface.co/dvruette/gidd unif 3b Quick Start Training This model…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy