This dataset is the downloaded variant of Spawning/PD12M. More specifically, this dataset is compatible with webdataset . It was made public after obtaining permission from the original authors of the dataset. You can use the following to explore the dataset with webdataset : Additionally, this script provides a reference dataloader implementation. The dataset was downloaded by using the img2dataset tool. The following command was used to perform the download on a CPU cluster: Code The command above serializes the webdataset shards to an S3 bucket. Additionally, here is the wandb log of the run. pd12m full.parquet was obtained by collating all the parquet files from here into a single pandas dataframe. It's available here. To copy the files from the S3 bucket to this repository, the following script was used: Code
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy