arXiv Papers by Subject A reorganised version of the nick007x/arxiv papers dataset, partitioned by subject code, year, and month for efficient selective access. Dataset Description This dataset contains metadata for over 2.5 million arXiv papers, organised into a hierarchical directory structure that allows users to download only the specific subjects and time periods they need, rather than the entire dataset. Motivation The original nick007x/arxiv papers dataset is an excellent resource containing comprehensive arXiv paper metadata. However, its monolithic structure requires downloading the entire dataset even when only a subset of papers is needed. This derived dataset addresses that limitation by partitioning the data into small, focused parquet files organised by: 1. Subject code (e.g., cs.AI , astro ph.CO , math.NA ) 2. Year (1989–2025) 3. Month (01–12) This structure enables: Downloading only specific research domains Fetching data for particular time ranges Incremental updates as new papers are published Efficient caching and lazy loading Dataset Structure Subject Categories The dataset includes 148 arXiv subject categories spanning: Domain Example Categories Astrophysics as…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy