As datasets puts limits on the number of calls to huggingface, downloading SlimPajama 627B is problematic as it's composed of a ton of small files. I have reuploaded it here as larger chunks to easily download the dataset without having to do anything hacky. The original dataset can be found here https://huggingface.co/datasets/cerebras/SlimPajama 627B
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy