Model Summary OLMoE 1B 7B Instruct is a Mixture of Experts LLM with 1B active and 7B total parameters released in September 2024 (0924) that has been adapted via SFT and DPO from OLMoE 1B 7B. It yields state of the art performance among models with a similar cost (1B) and is competitive with much larger models like Llama2 13B Chat. OLMoE is 100% open source. This information and more can also be found on the OLMoE GitHub repository . Paper : https://arxiv.org/abs/2409.02060 Pretraining Checkpoints, Code, Data and Logs. SFT (Supervised Fine Tuning) Checkpoints, Code, Data and Logs. DPO/KTO (Direct Preference Optimization/Kahneman Tversky Optimization) , Checkpoints, Preference Data, DPO code, KTO code and Logs. Use Install transformers from source until a release after this PR & torch and run: Branches: main : Preference tuned via DPO model of https://hf.co/allenai/OLMoE 1B 7B 0924 SFT ( main branch) load balancing : Ablation with load balancing loss during DPO starting from the load balancing branch of https://hf.co/allenai/OLMoE 1B 7B 0924 SFT non annealed : Ablation starting from the non annealed branch of https://hf.co/allenai/OLMoE 1B 7B 0924 SFT which is an SFT of the pretrain…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy