The Data
CORNELL NEWSROOM is a large dataset for training and evaluating summarization systems. It contains 1.3 million articles and summaries written by authors and editors in the newsrooms of 38 major publications. The summaries are obtained from search and social metadata between 1998 and 2017 and use a variety of summarization strategies combining extraction and abstraction.
@inproceedings{N18-1065,
author = {Grusky, Max and Naaman, Mor and Artzi, Yoav},
title = {NEWSROOM: A Dataset of 1.3 Million Summaries
with Diverse Extractive Strategies},
booktitle = {Proceedings of the 2018 Conference of the
North American Chapter of the Association for
Computational Linguistics: Human Language Technologies},
publisher = {Association for Computational Linguistics},
pages = {708--719},
year = {2018},
month = {June},
address = {New Orleans, Louisiana},
url = {http://aclweb.org/anthology/N18-1065}
}
Getting Started
Use this site to explore the dataset and better understand the task of summarization as used by newsrooms around the Web.
Explore example summaries in the dataset across publications, time, and summarization strategies, analyze overall statistics in the dataset across these categories, and evaluate performance of existing summarization systems trained and tested on the unreleased NEWSROOM data.
The full dataset is available to download online with tools for extracting text and summaries from Archive.org, analyzing summary extractiveness, and evaluating system performance.
Released Test Leaderboard
We are unable to maintain this table to exhaustively reflect the current state of the art summarization performance on the Newsroom dataset. We recommend consulting Google Scholar or Semantic Scholar for papers recently evaluating using Newsroom. Because of this, we are no longer updating this table.
| Date | System | R-1 | R-2 | R-L |
|---|
ABS Abstractive Systems
EXT Extractive Systems
MIX Mixed-Strategy Systems
The leaderboard ranks systems using unstemmed, untokenized ROUGE-1 F-score by default, in order to fully reflect the difficulty of the summarization task, account for generated summary length, and measure performance most comparably across systems.
Visit the evaluate page to explore system performance in depth and read example summaries.
* Evaluation performed in Grusky, et al. 2018.
NEWSROOM was developed as part of the Connected Experiences Lab.
This work is supported by Oath and by a Google Research award.
