common-pile/comma_v0.1_training_datasetComma v0.1 dataset This repository contains the dataset used to train Comma v0.1 1T and Comma v0.1 2T. It is a slightly modified and consolidated version of the Common Pile v0.1 "filtered" data. If you are looknig for the raw Common Pile v0.1 data, please see this collection. You can learn more about Common Pile in our paper. Mixing rates and token counts The Comma v0.1 models were trained in two stages, a "main" stage and a "cooldown" stage. During each stage, we heuristically set mixing rates to up or downweight different sources. In the two tables below, we provide per source token count, repeat rate, and effective token count (after up/downweighting) for the main and cooldown stage of the Comma v0.1 1T training run. For the Comma v0.1 2T training run, all sources are repeated 2x as many times in both stages. Token counts are as provided by the Comma v0.1 tokenizer; using a different tokenizer may change these counts significantly. Main stage Tokens (B) Repeats Effective tokens (B) arxiv abstracts 0.57 6 3.4 arxiv papers 6.0 6 35.8 biodiversity heritage library 9.8 0.25 2.5 caselaw access project 19.7 1 19.7 cccc 15.2 6 91.4 data provenance initiative 0.92 6 5.5 doab 3.0 6 18.2…

← Back to common-pile/comma_v0.1_training_dataset

SQL Console

Runs entirely in your browser via DuckDB-Wasm — this dataset's real data file is loaded once, then queried locally. Nothing is sent to a server.

Loading dataset into an in-browser SQL engine…

We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy