arxiv-ai-ml-100k-papers

    license: other
    tags:
      - arxiv
      - ocr
      - machine-learning
    ---

    # obswork/arxiv-ai-ml-100k

    A 99,999-paper stratified subset of
    [`Rendra8631/arxiv-papers`](https://huggingface.co/datasets/Rendra8631/arxiv-papers)
    at revision `a2c6afb51332d2744b46308df6917697582f8cd4`, filtered to the primary
    subjects `cs.AI`, `cs.CV`, `cs.LG`, and `stat.ML`. Only papers submitted in `2023`-`2025` are included.

    This dataset is a build artifact of the OCR benchmark in
    `modal-model-experiments/arxiv_mirror/`. It exists so that the
    downstream benchmark corpus (rasterized page images) is built
    from a stable, well-scoped PDF pool.

    ## Contents

    - `metadata.parquet` - 99999 rows, one per paper. Carries
      the source schema plus derived `primary_code`, `submission_year`,
      and `target_filename` columns.
    - `pdfs/<primary_code>/<YYMM>/<arxiv_id>.pdf` - one file per
      paper, bucketed by primary arXiv subject code (e.g.
      `cs.CV`) and submission year-month. The YYMM layer keeps
      every directory under the HF 10k-files-per-dir cap and
      gives consumers a natural temporal slice.

    ## Filter and sampling

    - Primary subjects kept: `cs.AI`, `cs.CV`, `cs.LG`, `stat.ML`.
    - Submission year kept: `>= 2023` (derived from
      the arxiv_id `YYMM.NNNNN` prefix, falling back to a regex over
      `submission_date` for any pre-2007 old-format IDs).
    - Version dedup: for papers with multiple versions in the source
      metadata (`...v1`, `...v2`), only the latest version is kept.
    - Stratified random sample of 25000 papers per
      category, `numpy.random.default_rng(seed=20260416)`.
      Categories with fewer than 25000 available
      rows contribute all of their rows; any resulting shortfall is
      topped up uniformly from the remaining filtered pool.
    - Per-category final counts:

    - `cs.AI`: 11788 papers

• cs.CV: 43616 papers

• cs.LG: 39599 papers

• stat.ML: 4996 papers

    ## Licensing
  
    This dataset is a collection of individual scholarly works from
    arXiv.org. Each paper remains under the copyright and license
    chosen by its respective author(s) (typically Creative Commons).
    Users of this dataset are responsible for checking the specific
    license terms of any paper they download or redistribute. See the
    [source dataset](https://huggingface.co/datasets/Rendra8631/arxiv-papers)
    for upstream licensing notes.
  
    ## Reproducibility
  
    Source commit: `a2c6afb51332d2744b46308df6917697582f8cd4`.
    Build pipeline: `arxiv_mirror/{filter_plan,mirror,finalize}.py`.