INFINI NEWS FM Index 🔎 Live search API: these FM indexes power a public search service — full text search, n gram counts, and document retrieval in the browser or via a keyless REST API, without building the index yourself — at infini news.uni graz.at (API reference). Pre built FM indexes (Burrows–Wheeler Transform + suffix array, built with infini gram mini, Liu et al. 2025) over the ruggsea/infini news corpus parquets. Enables exact, byte level substring count and document retrieval over 1.36 B news articles in milliseconds, without scanning the corpus. At a glance Articles indexed 1 357 027 742 Distinct hostnames covered 133 565 Time range Aug 2016 – Apr 2026 Shards 117 (one per year=YYYY/month=MM ) Total size ~2.2 TB Median p50 latency (whole corpus) 6–13 ms Median p95 (whole corpus) < 10 ms License CC BY 4.0 Layout 4 files per shard: data.fm9 (text index, ~12–20 GB), meta.fm9 (metadata JSON index, ~3–10 GB), data offset / meta offset (row offsets, ~100 MB each). Same (year, month) partitioning as the companion corpus. Per year totals year shards months : 2016 5 Aug–Dec 2017–2025 12 each full years 2026 4 Jan–Apr total 117 117 (year, month) pairs Query — Python Whole corpus lo…
Runs entirely in your browser via DuckDB-Wasm — this dataset's real data file is loaded once, then queried locally. Nothing is sent to a server.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy