Mistral7B PairRM SPPO ExPO The extrapolated (ExPO) model based on UCLA AGI/Mistral7B PairRM SPPO and mistralai/Mistral 7B Instruct v0.2 , as in the "Weak to Strong Extrapolation Expedites Alignment" paper. Specifically, we obtain this model by extrapolating (alpha = 0.3) from the weights of the SFT and DPO/RLHF checkpoints, achieving superior alignment with human preference. This extrapolated model achieves the 35.4% win rate and 31.8% LC win rate on AlpacaEval 2.0 , outperforming the original Mistral7B PairRM SPPO 's 32.2% and 30.5%, respectively. Evaluation Results Evaluation results on the AlpacaEval 2.0 benchmark (you can find the evaluation outputs on the official GitHub repo): Win Rate (Ori) LC Win Rate (Ori) Win Rate (+ ExPO) LC Win Rate (+ ExPO) HuggingFaceH4/zephyr 7b alpha 6.7% 10.0% 10.6% 13.6% HuggingFaceH4/zephyr 7b beta 10.2% 13.2% 11.1% 14.0% berkeley nest/Starling LM 7B alpha 15.0% 18.3% 18.2% 19.5% Nexusflow/Starling LM 7B beta 26.6% 25.8% 29.6% 26.4% snorkelai/Snorkel Mistral PairRM 24.7% 24.0% 28.8% 26.4% RLHFlow/LLaMA3 iterative DPO final 29.2% 36.0% 32.7% 37.8% internlm/internlm2 chat 1.8b 3.8% 4.0% 5.2% 4.3% internlm/internlm2 chat 7b 20.5% 18.3% 28.1% 22.7% i…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy