UltraFeedback Binarized using the Average of Preference Ratings (Cleaned) KTO A KTO signal transformed version of the highly loved UltraFeedback Binarized Preferences Cleaned, the preferred dataset by Argilla to use from now on when fine tuning on UltraFeedback This dataset represents a new iteration on top of argilla/ultrafeedback binarized preferences , and is the recommended and preferred dataset by Argilla to use from now on when fine tuning on UltraFeedback . Read more about Argilla's approach towards UltraFeedback binarization at argilla/ultrafeedback binarized preferences/README.md . Why KTO? The KTO paper states: KTO matches or exceeds DPO performance at scales from 1B to 30B parameters.1 That is, taking a preference dataset of n DPO pairs and breaking it up into 2n examples for KTO can yield better generations, despite the model ostensibly learning from a weaker signal. KTO can handle extreme data imbalances, matching DPO performance while using up to 90% fewer desirable examples (i.e., examples of good generations). Its success thus cannot be ascribed to the alignment data being sourced from a preference dataset. When the pretrained model is sufficiently good, one can ski…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy