princeton nlp/prolong data 64K [Paper] [HF Collection] [Code] ProLong ( Pr incet o n long context language models) is a family of long context models that are continued trained and supervised fine tuned from Llama 3 8B, with a maximum context window of 512K tokens. Our main ProLong model is one of the best performing long context models at the 10B scale (evaluated by HELMET). To train this strong long context model, we conduct thorough ablations on the long context pre training data, SFT data, and numerous other design choices. We demonstrate our findings in our paper, How to Train Long Context Language Models (Effectively). Authors: Tianyu Gao\ , Alexander Wettig\ , Howard Yen, Danqi Chen ( equal contribution) Contact: {tianyug, awettig}@princeton.edu Dataset Loading This dataset contains 31B tokens, tokenzized with the Llama 3 tokenizer and packed to sequences of 65,536 tokens. The data is stored as MDS (Mosaic Data Shard) and requires mosaicml streaming to be loaded. Instead of datasets.load dataset , download the data by cloning the repository or the huggingface hub.snapshot download function. When loading the datasets with mosaicml streaming, each entry has the following field…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy