Dataset 1: TheStack Rust Cleaned Description : This dataset is drawn from TheStack Corpus, an open source code dataset with over 3TB of GitHub data covering 48 programming languages. We selected a small portion of this dataset to optimize smaller language models for Rust, a popular statically typed language. Target Language : Rust Dataset Size : Training: 900,000 files Validation: 50,000 files Test: 50,000 files Preprocessing : 1. Selected Rust as the target language due to its popularity on GitHub. 2. Filtered out files with average line length 100 characters, maximum line length 1000 characters, and alphabet ratio < 25%. 3. Split files into 90% training, 5% validation, and 5% test sets. Tokenizer : Byte Pair Encoding (BPE) tokenizer with tab and whitespace tokens. GPT 2 vocabulary extended with special tokens. Training Sequences : Sequences constructed by joining training data text to reach a context length of 2048 tokens (1024 tokens for full fine tuning).
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy