ClimbMix About 🧗 A more convenient ClimbMix (https://arxiv.org/abs/2504.13161) Description Unfortunately, the original ClimbMix (https://huggingface.co/datasets/nvidia/ClimbMix) has four main inconveniences: 1) It is in GPT2 tokens, meaning you have to detokenize it to inspect it or use it with another tokenizer. 2) It contains all of the 20 clusters in order together (in the same "subset"), so you have to load the whole dataset in memory (\~1TB) and shuffle it (\~3h) before using it. 3) It does not mention the precise cluster ratios (Figure 6 does not show decimal digits, e.g. 0% instead of 0.23%), therefore you cannot subsample it while maintaining the original ratios. 4) It does not come with topic annotation (similar to Table 4, which corresponds to another dataset, with 21 clusters) https://huggingface.co/datasets/OptimalScale/ClimbMix fixes (1), and this is my attempt to solve (2), (3) and (4) as follows: (2): I separate the clusters in different "subsets", selectable by the name argument of load dataset(). I also shuffle them for good measure (in case they contained subclusters in order). (3): I use len() in each subset, and divide by the total to get the ratio (4): Similar…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy