Dataset 1: TheStack - Rust - Cleaned
Description: This dataset is drawn from TheStack Corpus, an open-source code dataset with over 3TB of GitHub data covering 48 programming languages. We selected a small portion of this dataset to optimize smaller language models for Rust, a popular statically typed language.
Target Language: Rust
Dataset Size:
- Training: 900,000 files
- Validation: 50,000 files
- Test: 50,000 files
Preprocessing:
- Selected Rust as the target language due to its popularity on GitHub.
- Filtered out files with average line length > 100 characters, maximum line length > 1000 characters, and alphabet ratio < 25%.
- Split files into 90% training, 5% validation, and 5% test sets.
Tokenizer: Byte Pair Encoding (BPE) tokenizer with tab and whitespace tokens. GPT-2 vocabulary extended with special tokens.
Training Sequences: Sequences constructed by joining training data text to reach a context length of 2048 tokens (1024 tokens for full fine-tuning).