INFINI-NEWS FM-Index

🔎 Live search API: these FM-indexes power a public search service — full-text search, n-gram counts, and document retrieval in the browser or via a keyless REST API, without building the index yourself — at infini-news.uni-graz.at (API reference).

Pre-built FM-indexes (Burrows–Wheeler Transform + suffix array, built with infini-gram-mini, Liu et al. 2025) over the ruggsea/infini-news-corpus parquets. Enables exact, byte-level substring count and document retrieval over 1.36 B news articles in milliseconds, without scanning the corpus.

At a glance

Layout

ccnews_2016/shard_08/{data.fm9, meta.fm9, data_offset, meta_offset}
…
ccnews_2026/shard_04/{data.fm9, meta.fm9, data_offset, meta_offset}

4 files per shard: data.fm9 (text index, ~12–20 GB), meta.fm9 (metadata JSON index, ~3–10 GB), data_offset / meta_offset (row offsets, ~100 MB each). Same (year, month) partitioning as the companion corpus.

Per-year totals

Query — Python

import sys
sys.path.insert(0, "/path/to/infini-gram-mini/engine")
from src.engine import InfiniGramMiniEngine
from pathlib import Path

shards = sorted(str(p) for p in Path("/local/index/ccnews_2022").glob("shard_*"))
# All three kwargs are required. load_to_ram=False uses mmap (free,
# page-cache only); get_metadata=False skips loading the meta.fm9
# indexes — set True if you need get_doc_by_rank.
engine = InfiniGramMiniEngine(index_dirs=shards,
                              load_to_ram=False,
                              get_metadata=False)

# .count() returns {"count": int}, .find() returns
# {"cnt": int, "segment_by_shard": [[start, end], ...]}.
print(engine.count(query="Vladimir Putin"))  # {'count': 2_828_877}
print(engine.count(query="ChatGPT"))         # {'count': 22_783}
                                             # — released 2022-11-30, so
                                             # most hits are December 2022

Whole-corpus load (117 shards, mmap) takes ~8 s; subsequent counts run in single-digit milliseconds. No RAM cost beyond the OS page cache.

Dataset creation

Curation rationale

Counting occurrences of a phrase across 1.36 B articles by scanning the parquet is slow — sub-second-per-query exact substring search is what makes large-scale lexical analyses (information retrieval, language modelling diagnostics, corpus linguistics, media studies, n-gram statistics) practical on a single workstation. This index is the sibling artefact of the corpus: identical row coverage, one shard per (year, month).

Source data

ruggsea/infini-news-corpus — 1 357 027 742 CC-News articles, body extracted by trafilatura.

Processing pipeline

1. Read each monthly parquet partition; emit one byte stream per shard (texts) and one JSON-lines stream (metadata).
2. Build the BWT + suffix array with infini-gram-mini (Liu et al. 2025) — a memory- efficient FM-index optimised for very large text collections.
3. Persist data.fm9, meta.fm9, data_offset, meta_offset per shard; one shard per (year, month).

Full code: codeberg.org/ksolovev/infini-news.

Citation

@misc{lazzaroni2026infininews,
  author    = {Lazzaroni, Ruggero Marino and Lasser, Jana and Solovev, Kirill},
  title     = {{INFINI-NEWS Corpus}},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/ruggsea/infini-news-corpus},
  doi       = {10.57967/hf/8606}
}