ClimbMix

About

🧗 A more convenient ClimbMix (https://arxiv.org/abs/2504.13161)

Description

Unfortunately, the original ClimbMix (https://huggingface.co/datasets/nvidia/ClimbMix) has four main inconveniences:

1. It is in GPT2 tokens, meaning you have to detokenize it to inspect it or use it with another tokenizer.
2. It contains all of the 20 clusters in order together (in the same "subset"), so you have to load the whole dataset in memory (~1TB) and shuffle it (~3h) before using it.
3. It does not mention the precise cluster ratios (Figure 6 does not show decimal digits, e.g. 0% instead of 0.23%), therefore you cannot subsample it while maintaining the original ratios.
4. It does not come with topic annotation (similar to Table 4, which corresponds to another dataset, with 21 clusters)

https://huggingface.co/datasets/OptimalScale/ClimbMix fixes (1), and this is my attempt to solve (2), (3) and (4) as follows:

(2): I separate the clusters in different "subsets", selectable by the name argument of load_dataset(). I also shuffle them for good measure (in case they contained subclusters in order).

(3): I use len() in each subset, and divide by the total to get the ratio

(4): Similarly to the paper, I use gpt-4.1-mini and 100 samples from every cluster to extract the main topics

Usage

import datasets

dataset = datasets.load_dataset("gvlassis/ClimbMix", "cluster_id=1", split="train")