Dataset Card for NewsWire

Dataset Description

• Homepage: Dell Research homepage
• Repository: Github repository
• Paper: arxiv submission
• Point of Contact: Melissa Dell

Dataset Summary

NewsWire contains 2.7 million unique public domain U.S. news wire articles, written between 1878 and 1977. Locations in these articles are georeferenced, topics are tagged using customized neural topic classification, named entities are recognized, and individuals are disambiguated to Wikipedia using a novel entity disambiguation model.

Languages

English (en)

Dataset Structure

Each year in the dataset is divided into a distinct file (eg. 1952_data_clean.json)

Data Instances

An example from the NewsWire dataset looks like:

{
    "year": 1880,
    "dates": ["Feb-23-1880"], 
    "article": "SENATE Washington, Feb. 23.--Bayard moved that in respect of the 
        memory of George Washington the senate adjourn ... ",
    "byline": "",
    "newspaper_metadata": [
        {
            "lccn": "sn92053943",
            "newspaper_title": "the rock island argus",
            "newspaper_city": "rock island",
            "newspaper_state": " illinois "
        },
        ...
    ],
    "antitrust": 0,
    "civil_rights": 0,
    "crime": 0,
    "govt_regulation": 1,
    "labor_movement": 0,
    "politics": 1,
    "protests": 0,
    "ca_topic": "Federal Government Operations",
    "ner_words": ["SENATE", "Washington", "Feb", "23", "Bayard", "moved", "that", 
        "in", "respect", "of", "the", "memory", "of", "George", "Washington", 
        "the", "senate", "adjourn", ... ],
    "ner_labels": ["B-ORG", "B-LOC", "O", "B-PER", "B-PER", "O", "O", "O", "O", 
        "O", "O", "O", "O", "B-PER", "I-PER", "O", "B-ORG", "O", ...],
    "wire_city": "Washington",
    "wire_state": "district of columbia",
    "wire_country": "United States",
    "wire_coordinates": [38.89511, -77.03637],
    "wire_location_notes": "",
    "people_mentioned": [
        {
            "wikidata_id": "Q23",
            "person_name": "George Washington",
            "person_gender": "man",
            "person_occupation": "politician"
        },
        ...
    ],
    "cluster_size": 8
}

Data Fields

• year: year of article publication.

• dates: list of dates on which this article was published, as strings in the form mmm-DD-YYYY.

• byline: article byline, if any.

• article: article text.

• newspaper_metadata: list of newspapers that carried the article. Each newspaper is represented as a list of dictionaries, where lccn is the newspaper's Library of Congress identifier, newspaper_title is the name of the newspaper, and newspaper_city and newspaper_state give the location of the newspaper.

• antitrust: binary variable. 1 if the article was classified as being about antitrust.

• civil_rights: binary variable. 1 if the article was classified as being about civil rights.

• crime: binary variable. 1 if the article was classified as being about crime.

• govt_regulation: binary variable. 1 if the article was classified as being about government regulation.

• labor_movement: binary variable. 1 if the article was classified as being about the labor movement.

• politics: binary variable. 1 if the article was classified as being about politics.

• protests: binary variable. 1 if the article was classified as being about protests.

• ca_topic: predicted Comparative Agendas topic of article.

• wire_city: City of wire service bureau that wrote the article.

• wire_state: State of wire service bureau that wrote the article.

• wire_country: Country of wire service bureau that wrote the article.

• wire_coordinates: Coordinates of city of wire service bureau that wrote the article.

• wire_location_notes: Contains wire dispatch location if it is not a geographic location. Can be one of Pacific Ocean (WWII)'', Supreme Headquarters Allied Expeditionary Force (WWII)'', North Africa'', War Front (WWI)'', War Front (WWII)'' or Johnson Space Center''.

• people_mentioned: list of disambiguated people mentioned in the article. Each disambiguated person is represented as a dictionary, where wikidata_id is their ID in Wikidata, person_name is their name on Wikipedia, person_gender is their gender from Wikidata and person_occupation is the first listed occupation on Wikidata.

• cluster_size: Number of newspapers that ran the wire article. Equals length of newspaper_metadata.

Accessing the Data

The whole dataset can be easily downloaded using the datasets library:

from datasets import load_dataset
dataset_dict = load_dataset("dell-research-harvard/newswire")

Specific files can be downloaded by specifying them:

from datasets import load_dataset
load_dataset(
    "dell-research-harvard/newswire", 
    data_files=["1929_data_clean.json", "1969_data_clean.json"]
)

Dataset Creation

Curation Rationale

The dataset was created to provide researchers with a large, high-quality corpus of historical news articles.
These texts provide a massive repository of information about historical topics and events - and which newspapers were covering them. The dataset will be useful to a wide variety of researchers including historians, other social scientists, and NLP practitioners.

Source Data

Initial Data Collection and Normalization

Dataset construction is described in the associated paper.

Who are the source language producers?

The source language was produced by people - by newspaper editors, columnists, and other sources.

Annotations

Annotation process

Not Applicable

Who are the annotators?

The dataset does not contain any additional annotations.

Personal and Sensitive Information

The dataset may contain information about individuals, to the extent that this is covered in news stories. However we make no additional information about individuals publicly available.

Considerations for Using the Data

Social Impact of Dataset

This dataset provides high-quality data that could be used for pre-training a large language model to achieve better understanding of historical English and historical world knowledge. The dataset could also be added to the external database of a retrieval-augmented language model to make historical information more widely accessible.

Discussion of Biases

This dataset contains unfiltered content composed by newspaper editors, columnists, and other sources. In addition to other potentially harmful content, the corpus may contain factual errors and intentional misrepresentations of news events. All content should be viewed as individuals' opinions and not as a purely factual account of events of the day.

Additional Information

Dataset Curators

Emily Silcock (Harvard), Abhishek Arora (Harvard), Luca D'Amico-Wong (Harvard), Melissa Dell (Harvard)

Licensing Information

The dataset has a CC-BY 4.0 license

Citation Information

You can cite this dataset using

@misc{silcock2024newswirelargescalestructureddatabase,
      title={Newswire: A Large-Scale Structured Database of a Century of Historical News}, 
      author={Emily Silcock and Abhishek Arora and Luca D'Amico-Wong and Melissa Dell},
      year={2024},
      eprint={2406.09490},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.09490}, 
}

Contributions

Coming Soon