DocHPLT: A Massively Multilingual Document Level Translation Dataset Existing document level machine translation resources are only available for a handful of languages, mostly high resourced ones. To facilitate the training and evaluation of document level translation and, more broadly, long context modeling for global communities, we create DocHPLT, the largest publicly available document level translation dataset to date. It contains 124 million aligned document pairs across 50 languages paired with English, comprising 4.26 billion sentences, with further possibility to provide 2500 bonus pairs not involving English. Unlike previous reconstruction based approaches that piece together documents from sentence level data, we modify an existing web extraction pipeline to preserve complete document integrity from the source, retaining all content including un aligned portions. After our preliminary experiments identify the optimal training context strategy for document level translation, we demonstrate that LLMs fine tuned on DocHPLT substantially outperform off the shelf instruction tuned baselines, with particularly dramatic improvements for under resourced languages. We open sourc…
Runs entirely in your browser via DuckDB-Wasm — this dataset's real data file is loaded once, then queried locally. Nothing is sent to a server.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy