Model Summary OLMoE 1B 7B is a Mixture of Experts LLM with 1B active and 7B total parameters released in January 2025 (0125) that is 100% open source. It is an improved version of OLMoE 09 24, see the paper appendix for details. This information and more can also be found on the OLMoE GitHub repository . Paper : arxiv.org/abs/2409.02060 Pretraining Checkpoints, Code, Data and Logs. SFT (Supervised Fine Tuning) Checkpoints, Code, Data and Logs. DPO/KTO (Direct Preference Optimization/Kahneman Tversky Optimization) , Checkpoints, Preference Data, DPO code, KTO code and Logs. Use Install transformers (version 4.45.0 or greater) & torch and run: You can list all revisions/branches by installing huggingface hub & running: Important branches: step1200000 tokens5033B : Pretraining checkpoint used for annealing. There are a few more checkpoints after this one but we did not use them. main : Checkpoint annealed from step1200000 tokens5033B for an additional 100B tokens (23,842 steps). We use this checkpoint for our adaptation (https://huggingface.co/allenai/OLMoE 1B 7B 0125 SFT & https://huggingface.co/allenai/OLMoE 1B 7B 0125 Instruct). fp32 : FP32 version of main . The model weights were…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy