Model Summary OLMoE 1B 7B is a Mixture of Experts LLM with 1B active and 7B total parameters released in September 2024 (0924). It yields state of the art performance among models with a similar cost (1B) and is competitive with much larger models like Llama2 13B. OLMoE is 100% open source. This information and more can also be found on the OLMoE GitHub repository . Paper : https://arxiv.org/abs/2409.02060 Pretraining Checkpoints, Code, Data and Logs. SFT (Supervised Fine Tuning) Checkpoints, Code, Data and Logs. DPO/KTO (Direct Preference Optimization/Kahneman Tversky Optimization) , Checkpoints, Preference Data, DPO code, KTO code and Logs. Use Install transformers from source until a release after this PR & torch and run: You can list all revisions/branches by installing huggingface hub & running: Important branches: step1200000 tokens5033B : Pretraining checkpoint used for annealing. There are a few more checkpoints after this one but we did not use them. main : Checkpoint annealed from step1200000 tokens5033B for an additional 100B tokens (23,842 steps). We use this checkpoint for our adaptation (https://huggingface.co/allenai/OLMoE 1B 7B 0924 SFT & https://huggingface.co/alle…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy