Open Markdown Clean markdown from the web, ready for training and retrieval What is it? Open Markdown is a large scale web text dataset built from Common Crawl. Common Crawl is a non profit that crawls the web and freely provides its archives and datasets to the public — see their latest crawl announcement for details on the source data. Every page goes through a pipeline that extracts the main content from raw HTML, converts it to clean Markdown, and packages the result into Parquet files with useful WARC metadata for traceability. The dataset currently includes crawl CC MAIN 2026 21 with 46,322,157 documents across 2402 shards . Processed 6.2 TB of raw HTML into 371.9 GB of clean Markdown — a 94.2% reduction . We plan to add more snapshots over time. Live Progress Processing at 20.6 shards/hour — 2,402 of 100,000 done ( 2.40% ) Estimated completion: December 18, 2026 (197 days) Current server: 6 CPU cores, 12 GB RAM (7.5 GB available), 1 GB disk free Memory per session: avg 607 MB, peak 667 MB (measured via VmRSS) With 10 identical servers: 206 shards/hour → June 24, 2026 (20 days) Open Markdown is released under the Open Data Commons Attribution License (ODC By) v1.0 , the same…
Runs entirely in your browser via DuckDB-Wasm — this dataset's real data file is loaded once, then queried locally. Nothing is sent to a server.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy