ReasonStack-Prime
A highly normalized, streaming-optimized Stack Exchange corpus engineered for LLM reasoning and instruction tuning.
CC BY-SA 4.0
~1M Rows
176 Parquet Shards
21 SE Sites
1. Executive Summary
ReasonStack-Prime is a large-scale, meticulously curated text dataset derived from the official Archive.org Stack Exchange data dump (Version 2021-12-07). Unlike raw XML dumps or poorly cleaned JSON exports, this dataset applies a rigorous streaming ETL pipeline to normalize HTML into clean Markdown, preserve complex code structures, and retain critical metadata.
The corpus contains 999,892 rows across 21 distinct Stack Exchange communities, totaling 3.36 GB in highly compressed Parquet format. It is specifically designed to serve as a high-quality pretraining or fine-tuning corpus for Large Language Models (LLMs) targeting complex reasoning, code generation, and technical question answering.
999,892
Total Instances
18.4 GB
Raw XML Processed
3.36 GB
Final Parquet Size
176
Optimized Shards
2. Pipeline Architecture & Data Curation
Processing 18+ GB of deeply nested, inconsistent XML requires significant memory management. This dataset was built using a custom streaming architecture (stackexchange-streaming-v1.0) designed to run efficiently on constrained hardware without OOM (Out of Memory) errors.
2.1 The ETL Process
1
Streaming XML Parsing: Utilized
xml.etree.ElementTree.iterparse to stream the 7z-compressed XML files directly from Archive.org. This avoids loading the entire DOM into memory, keeping RAM usage strictly bounded.
2
HTML to Markdown Sanitization: Implemented a custom AST-based converter. Unlike naive regex replacements, this step correctly handles nested blockquotes, preserves Stack Exchange's specific MathJax syntax (
$ and $$), and strictly protects <pre><code> blocks from being mangled by the Markdown parser.
3
Answer Aggregation & Ranking: For each question, all associated answers are joined. The pipeline identifies the
accepted_answer_id; if none exists, it falls back to the highest-scoring answer to populate the best_answer field.
4
Parquet Sharding (Dask/Polars): The normalized data is partitioned into 176 optimized Parquet shards. This specific shard count ensures that distributed training frameworks (like DeepSpeed or FSDP) can assign whole files to individual GPUs without requiring complex row-level slicing.
2.2 Filtering & Quality Control
To ensure high signal-to-noise ratio, the following heuristic filters were applied during the streaming phase:
- Deleted Content: Removed questions and answers marked with
deletedflags in the XML dump. - Spam & Abusive: Filtered out posts with severe spam flags.
- Empty Bodies: Dropped instances where the Markdown conversion resulted in an empty string (e.g., image-only questions without alt-text).
3. Comprehensive Data Schema
The dataset is provided in a flat, normalized structure optimized for Hugging Face datasets and analytical tools like Polars.
| Column | Type | Description & Edge Cases |
|---|---|---|
id | int64 | Unique Post ID from the Stack Exchange database. Guaranteed unique across the entire corpus. |
site | string | The FQDN of the source site (e.g., stackoverflow.com, math.stackexchange.com). |
title | string | Raw HTML-decoded title of the question. |
question_body_md | string | Clean Markdown representation of the question. MathJax is preserved as raw LaTeX. |
question_score | int32 | Net vote count (upvotes minus downvotes) at the time of the 2021 dump. |
question_tags | list[string] | Array of tags. Note: Stack Overflow limits to 5 tags; other sites may have fewer. |
answers | list[struct] | Array of all answers. Struct contains: id, body_md, score, is_accepted. |
best_answer | string | The text of the accepted answer. If no answer was accepted, this contains the highest-voted answer. Null if 0 answers exist. |
text | string | Primary Training Field. A pre-formatted string combining Title, Tags, Question Body, and Best Answer, optimized for direct causal language modeling (CLM). |
tokens_estimate | int32 | Estimated token count using standard BPE/GPT-2 tokenizers. Useful for context-window filtering. |
meta | dict | Provenance data: dump_version, parser_version, extraction_date, and original Archive.org URLs. |
4. Site Distribution & Sharding
The corpus covers 21 distinct communities, heavily weighted toward technical and scientific domains to maximize reasoning density.
| Domain | Shards | Domain Focus | Domain | Shards | Domain Focus |
|---|---|---|---|---|---|
| stackoverflow.com | 80 | Software Engineering | physics.stackexchange.com | 6 | Theoretical/Applied Physics |
| math.stackexchange.com | 10 | Pure & Applied Math | stats.stackexchange.com | 6 | Statistics & ML Theory |
| tex.stackexchange.com | 13 | LaTeX / Typesetting | softwareengineering | 5 | Architecture / Design |
| unix.stackexchange.com | 8 | Linux / CLI / Shell | dba.stackexchange.com | 5 | Database Admin / SQL |
| mathematica | 8 | Symbolic Computation | electronics | 4 | Circuit Design / EE |
| codereview | 8 | Code Optimization | security / crypto | 6 | InfoSec / Cryptography |
| gis.stackexchange.com | 7 | Geospatial / GIS | Others (cs, ai, devops...) | 14 | Specialized Computing |
5. Usage Guide
5.1 Streaming with Hugging Face datasets
Because the dataset is sharded, you can stream it without downloading the 3.36 GB payload to disk.
from datasets import load_dataset
# Stream the dataset directly
ds = load_dataset("AdhyanshVerma/ReasonStack-Prime", split="train", streaming=True)
# Iterate through high-quality software engineering questions
for row in ds:
if row['site'] == 'stackoverflow.com' and row['question_score'] > 20:
print(row['text'])
break