Common Crawl Host Index v2 GitHub: https://github.com/commoncrawl/cc host index Each crawl, we generate a Host Index, which aggregates information about each web hosted visited during the crawl. The information is aggregated from the Common Crawl columnar index, web graph, and raw crawler logs. Quickstart Using duckdb: DuckDB can read directly from Huggingface: Using Hugging Face datasets library: "Streaming" mode ( streaming=True ) is recommended (see the datasets docs). This avoids the need to download all of the parquet files up front. Use Cases Example questions this index can answer What's our history of crawling a particular website, or group of websites? What popular websites have a lot of non English content? What popular websites seem to have so little content that we might need to execute javascript to crawl them? Example questions that we'll use to improve our crawl What's the full list of websites where more than half of the webpages are primarily not English? What popular websites end our crawls with most of their crawl budget left uncrawled? Example questions that future versions of this host index can answer What websites have a lot of content in particular languages…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy