DeepScaleR 1.5B Preview 🚀 Democratizing Reinforcement Learning for LLMs 🌟 DeepScaleR Overview DeepScaleR 1.5B Preview is a language model fine tuned from DeepSeek R1 Distilled Qwen 1.5B using distributed reinforcement learning (RL) to scale up to long context lengths. The model achieves 43.1% Pass@1 accuracy on AIME 2024, representing a 15% improvement over the base model (28.8%) and surpassing OpenAI's O1 Preview performance with just 1.5B parameters. Data Our training dataset consists of approximately 40,000 unique problem answer pairs compiled from: AIME problems (1984 2023) AMC problems (prior to 2023) Omni MATH dataset Still dataset Training Recipe We employ Deepseek's Group Relative Policy Optimization (GRPO), a simplified RL algorithm that extends PPO by: Normalizing advantage function over all samples generated from the same prompt. Applying KL divergence regularization on top of PPO's surrogate loss to prevent significant policy drift. Reward Function : Our reward function is simple but effective: 1 for correct answers passing LaTeX/Sympy checks 0 for incorrect or improperly formatted answers Note: No partial rewards (such as PRMs) or intermediate feedback. Iterative Con…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy