wikipedia-paragraphs

wikipedia-paragraphs is a dataset generated from Wikipedia, designed for natural language processing (NLP) research.

Each entry contains cleaned paragraph text and Wikilink information extracted from a Wikipedia page, along with useful metadata such as categories, templates, and the associated Wikidata QID.

Dataset structure

Configurations

The dataset is organized into multiple configurations, such as enwiki-20260301-v1.2.0. Each configuration corresponds to a specific Wikipedia dump from which the data was sourced. The version number denotes the version of the scripts used to process the data.

Data instances

Each instance in the dataset represents a single Wikipedia page, which can be an article, category, or template.

The following example from the enwiki-20260301-v1.2.0 configuration shows the data generated from the English Wikipedia article on Éclair.

{
  "id": "en-1980219-1336076184",
  "lang": "en",
  "page_id": 1980219,
  "revision_id": 1336076184,
  "page_type": "article",
  "title": "Éclair",
  "wikidata_qid": "Q273426",
  "paragraph_texts": [
    "An éclair (English: /ɪˈklɛər/ ⓘ ih-KLAIR or /eɪˈklɛər/ ay-KLAIR, French: [eklɛːʁ] ⓘ; lit. 'lightning') is a pastry made with choux dough filled with a cream and topped with a flavored icing. The dough, which is the same as that used for profiteroles, is typically piped into an oblong shape with a pastry bag and baked until it is crisp and hollow inside. Once cool, the pastry is filled with custard (crème pâtissière), whipped cream or chiboust cream, then iced with fondant icing. Other fillings include pistachio- and rum-flavored custard, fruit-flavored fillings or chestnut purée. When the icing is caramel, the dessert may be called a bâton de Jacob (lit. 'Jacob's staff'). A similar pastry in a round rather than oblong shape is called a religieuse.",
    "The word comes from the French éclair, meaning 'flash of lightning', so named because it is eaten quickly (in a flash); however some believe that the name is due to the glistening of the frosting resembling lightning.",
    "The éclair originated during the 19th century in Lyon, France where it was called pain à la Duchesse ('Duchess-style bread') or petite duchesse ('little duchess') until 1850. The word is first attested both in English and in French in the 1860s.",
    ...
  ],
  "paragraph_sections": [
    "__LEAD__",
    "Etymology",
    "History",
    ...
  ],
  "paragraph_wikilinks": [                                                                                                                             [229/346]
    [
      {
        "lang": "en",
        "title": "Pastry",
        "text": "pastry",
        "start": 108,
        "end": 114
      },
      {
        "lang": "en",
        "title": "Choux pastry",
        "text": "choux",
        "start": 125,
        "end": 130
      },
      {
        "lang": "en",
        "title": "Profiterole",
        "text": "profiteroles",
        "start": 237,
        "end": 249
      },
      ...
    ],
    [
      {
        "lang": "en",
        "title": "Lightning",
        "text": "lightning",
        "start": 57,
        "end": 66
      }
    ],
    [
      {
        "lang": "en",
        "title": "Lyon",
        "text": "Lyon",
        "start": 49,
        "end": 53
      }
    ],
    ...
  ],
  "paragraph_html_tags": [
    "p",
    "p",
    "p",
    ...
  ],
  "linked_articles": [
    "France",
    "French cuisine",
    "Pastry",
    ...
  ],
  "categories": [
    "Albanian cuisine",
    "Belgian desserts",
    "Custard desserts",
    ...
  ],
  "templates": [
    "Short description",
    "Use American English",
    "Use mdy dates",
    ...
  ],
  "num_inlinks": 69,
  "redirects": [
    "Éclair",
    "Chocolate eclair",
    "Eclairs",
    ...
  ]
}

Data Fields

• id (string): A unique ID for the instance, composed of lang, page_id and revision_id
• lang (string): The Wikipedia edition code of the page.
• page_id (int64): The page ID of the page.
• revision_id (int64): The revision ID of the page.
• page_type (string): The page type, one of: "article", "category", or "template".
• title (string): The page title.
• wikidata_qid (string, optional): The corresponding Wikidata QID, if available.
• paragraph_texts (list of string): A list of cleaned paragraph texts extracted from the page's HTML.
• paragraph_sections (list of string): A list of section titles corresponding to each paragraph. __LEAD__ denotes the page's lead section.
• paragraph_wikilinks (list of list): A nested list where each inner list contains the wikilinks for the corresponding paragraph. Each wikilink object has the following fields:

- **lang** (*string*): The Wikipedia edition code of the target page.
- **title** (*string*): The title of the target page.
- **text** (*string*): The anchor text of the wikilink.
- **start** (*int64*): The starting character index of the anchor text in the paragraph.
- **end** (*int64*): The ending character index of the anchor text in the paragraph.

• paragraph_html_tags (list of string): The HTML tag from which each paragraph was extracted, one of: "p", "li" "dt", "dd", or "blockquote"
• linked_articles (list of string): A list of article titles linked from the page.
• categories (list of string): A list of categories assigned to the page.
• templates (list of string): A list of templates used on the page.
• num_inlinks (int64): The number of pages linking to this page. The meaning depends on the page type:
  • For articles, it's the number of pages with wikilinks pointing to this page.
  • For categories, it's the number of pages assigned to this category.
  • For templates, it's the number of pages where this template is used.
• redirects (list of string): A list of page titles that redirect to this page.

Example usage

The following example shows how to use the dataset to build a corpus of lead paragraphs from articles.

from datasets import load_dataset

# Load the dataset in streaming mode to avoid downloading it all at once
dataset = load_dataset("singletongue/wikipedia-paragraphs", name="enwiki-20260301-v1.2.0", split="train", streaming=True)

# Define a function to filter for relevant article pages
def filter_example(example):
    # Keep only article pages
    if example["page_type"] != "article":
        return False
    
    # Exclude articles that are just lists
    if example["title"].startswith("List of"):
        return False
    
    # Exclude disambiguation pages
    if "Disambiguation" in example["templates"]:
        return False
    
    return True

dataset = dataset.filter(filter_example)

# Define a function to extract and combine all paragraphs from the lead section
def get_lead_paragraph_text(example):
    lead_paragraphs = []
    for text, section in zip(example["paragraph_texts"], example["paragraph_sections"]):
        if section == "__LEAD__":
            lead_paragraphs.append(text)
    
    # Join the paragraphs into a single string, replacing newlines with spaces
    return {"lead_paragraph_text": " ".join(lead_paragraphs).replace("\n", " ")}

dataset = dataset.map(get_lead_paragraph_text)

# Take the first 100 articles and save their lead paragraphs to a file
with open("corpus.txt", "w") as fo:
    for example in dataset.take(100):
        print(example["lead_paragraph_text"], file=fo)

Dataset creation process

This dataset is created from the Wikipedia CirrusSearch dumps and Wikimedia Enterprise HTML dumps. We also utilized the Wikimedia REST API to fetch HTML content that was missing from the dumps.

The code used for creating this dataset is available in the GitHub repository.

License

This dataset is available under the CC-BY-SA 4.0 and GFDL licenses.

Limitation

This dataset is created from specific snapshots of Wikipedia. It may contain outdated information or content that is no longer available on Wikipedia, as well as material that may be considered inappropriate.