Smol SmalTalk This is a subset of SmolTalk dataset adapted for smol models with less than 1B parameters. We used it to build SmolLM2 360M Instruct and SmolLM2 135M Instruct. We do SFT on this dataset and then DPO on UltraFeedback. Compared to SmolTalk: The conversations from Smol Magpie Ultra are shorter in this dataset We include less task specific data compared to SmolTalk (e.g no function calling and less rewriting and summarization examples) since these smaller models have limited capacity We don't include any advanced math datasets Citation
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy