CodeParrot 🦜 Dataset Cleaned What is it? A dataset of Python files from Github. This is the deduplicated version of the codeparrot. Processing The original dataset contains a lot of duplicated and noisy data. Therefore, the dataset was cleaned with the following steps: Deduplication Remove exact matches Filtering Average line length 0.25 Remove auto generated files (keyword search) For more details see the preprocessing script in the transformers repository here. Splits The dataset is split in a train and validation split used for training and evaluation. Structure This dataset has ~50GB of code and 5361373 files.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy