UNA SimpleSmaug 34b v1beta Scoring 04 February 2024 1 34B model, outperforming its original base model Smaug 34B v0.1 with 77.41 😎 Oh, btw.. this one went thru SFT so the abacus inside Smaug is back to normal.. so you can further train/dpo him .. RESET!.. UPDATES March : Stills undisputed 34B King Smaug 70B stills undisputed 70B King ==== And people wonders.. why there is no UNA of Hermes or Smaug 70B? << i dont think is worth the time to spend on a model that is widely known for not being too useful, likely UNA can fix some of the internal mess.. for Hermes, we spoke chitchat quick a couple times but nothing solid, but we would like to make a reborn of excellent models using UNA, just liek we did with UNA Dolphin where we saw relevant performance is short time. === Applied UNA only on the Attention, not on the MLP's Is based on Smaug SimpleMath dataset It was trained on Axolotl Experiment The thing here is to understand whats the impact of SimpleMath applied at the attention layer during a SFT session and how it impacts on the neural network overall. Results: Improving mathematican and reasoning capabilities without degrading and presserving previous training sessions. And enjoy…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy