natgillin/translations-raw
Frozen, canonical raw bitext consolidated from upstream alvations/mtdata-raw* snapshots (since deleted). This is the read-only source-of-truth for downstream quality-filtering pipelines.
- 31,663 parquet files (1566.8 GB)
- 49 language pairs under
data/<src-tgt>/ - Schema: 5 columns — see below
- Read-only for downstream pipelines. Do not delete or modify.
Schema
Each parquet has 5 columns:
| column | type | description |
|---|---|---|
source | string | source-language sentence |
target | string | target-language sentence |
source_lang | string | ISO-639-3 source language code |
target_lang | string | ISO-639-3 target language code |
origin | string | upstream OPUS / mtdata corpus tag (e.g. opus-nllb, opus-paracrawl) |
Layout
data/
aym-eng/
opus-nllb-mtdata-recovery-00000-of-00200.parquet
opus-globalvoices-v2018q4-mtdata-recovery-00000-of-00200.parquet
...
aze-eng/
opus-nllb-mtdata-recovery-00000-of-00200.parquet
...
bel-eng/
bul-eng/
ces-eng/
...
49 pairs total. File naming convention:
<origin>-NNNNN-of-NNNNN.parquet(original consolidation runs)<origin>-mtdata-recovery-NNNNN-of-NNNNN.parquet(re-sourced from mtdata)
Provenance
- Originally consolidated from
alvations/mtdata-raw,alvations/mtdata-raw-2,alvations/mtdata-raw-hashed(now deleted). - Files marked
-mtdata-recovery-were re-pulled directly from the mtdata library on 2026-06-09.
Downstream filtered versions
natgillin/translations— rows that scored> 0.8on Claude Haiku 4.5 translation-quality evaluationnatgillin/translations-rejected— rows that scored<= 0.8