Forecast News

Deduplicated daily news corpus used by forecast-sim and future-sim.

Snapshot

• 31,696,086 articles
• 3,401 daily partitions
• Coverage: 2016-08-26 through 2026-06-30
• Snapshot published: 2026-07-13
• Stored data size: approximately 168.8 GB

The Parquet files are the canonical complete representation. The repository also contains daily JSONL files where available and compact headline JSON files used by article-browsing workflows.

Layout

Files are partitioned by date:

YYYY/MM/DD/articles_b0000.parquet
YYYY/MM/DD/articles.jsonl
YYYY/MM/DD/headlines_b0000.json

Parquet columns:

id, title, source, date, date_publish, date_modify, url, content,
authors, description

A precomputed LanceDB search table for this corpus is published in shash42/forecast-news-embeddings.