Tech Report HuggingFace Megatron(coming soon) Abstract Recently, the Muon optimizer has demonstrated strong results in training small scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: Weight Decay : Critical for scaling to larger models Consistent RMS Updates : Enforcing a consistent root mean square on model updates These techniques allow Muon to work out of the box on large scale training without the need of hyper parameter tuning. Scaling law experiments indicate that Muon is $\sim2\times$ more sample efficient than Adam with compute optimal training. Based on these improvements, we introduce Moonlight , a 3B/16B parameter Mixture of Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open source our Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction tuned, and intermediate checkpoints to support future research. Our code is available at MoonshotAI/Moonlight. Key Ingredients Our work builds…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy