SmolLM3 Checkpoints We are releasing intermediate checkpoints of SmolLM3 to enable further research. For more details, check the SmolLM GitHub repo with the end to end training and evaluation code: ✓ Pretraining scripts (nanotron) ✓ Post training code SFT + APO (TRL/alignment handbook) ✓ Evaluation scripts to reproduce all reported metrics Pre training We release checkpoints every 40,000 steps, which equals 94.4B tokens. The GBS (Global Batch Size) in tokens for SmolLM3 3B is 2,359,296. To calculate the number of tokens from a given step: Training Stages Stage 1: Steps 0 to 3,450,000 (86 checkpoints) config Stage 2: Steps 3,450,000 to 4,200,000 (19 checkpoints) config Stage 3: Steps 4,200,000 to 4,720,000 (13 checkpoints) config Long Context Extension For the additional 2 stages that extend the context length to 64k, we sample checkpoints every 4,000 steps (9.4B tokens) for a total of 10 checkpoints: Long Context 4k to 32k config Long Context 32k to 64k config Post training We release checkpoints at every step of our post training recipe: Mid training, SFT, APO soup, and LC expert. How to Load a Checkpoint License Apache 2.0 Citation
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy