Chess Lakehouse

Is it a dataset, a data warehouse or a data lake? It's a lakehouse 🏠🏞️🛶

Versions

• Lichess's openings dataset: 2026-01-05
• Lumbras Gigabase: 2026-03-03
• duckdb chess extension: 0.6.0

Process

Pipeline process: https://github.com/dotneB/chess_lakehouse

• Uses DVC
• Data sources:
  • Lumbras Gigabase PGNs
• Use DuckDB chess extension (source) to parse PGN into duckdb
• Export to Parquet to a Schema similar to Lichess games datasets.

Dataset Enrichment

• Eco, Opening:
  • Uses Lichess's openings dataset to resolve Eco and Opening fields.
• UTCDate:
  • Uses the first, most complete date value that exists, with the following fallback chain UTCDate -> Date -> EventDate
  • Some dates are incomplete/placeholder. Unknown month/day ?? become 01: 2000.??.?? -> 2000-01-01, 2000.06.?? -> 2000-06-01
  • Out-of-range day is clamped to month end when year/month are valid: 2015.11.31 -> 2015-11-30, 1997.02.29 -> 1997-02-28
• TimeControl:
  • A best effort lenient parsing: 40/5400+30:1800+30
  • Lenient shorthands: 3+2 -> 180+2, 75|30 -> 4500+30, 15 + 10 -> 900+10
  • Apostrophe forms: 10'+5'' -> 600+5, 3' + 2''/mv from move 1 -> 180+2
  • Free text: 3 mins + 2 seconds increment -> 180+2, 90 minutes for 40 moves + 30 minutes for the rest + 30 seconds per move from move one -> 40/5400+30:1800+30

Dataset Structure

This dataset is hive-partitioned into multiple parquet files on three keys: DataSource, year and month:

├─ data
│  └─ DataSource=LumbrasGigabase_Online
│     └─ year=2015
│         └─ month=01
│            └─ data_0.parquet

Dataset Fields

┌─────────────┬─────────────────────┐
│ column_name │     column_type     │
│   varchar   │       varchar       │
├─────────────┼─────────────────────┤
│ Event       │ VARCHAR             │
│ Site        │ VARCHAR             │
│ White       │ VARCHAR             │
│ Black       │ VARCHAR             │
│ Result      │ VARCHAR             │
│ WhiteTitle  │ VARCHAR             │
│ BlackTitle  │ VARCHAR             │
│ WhiteElo    │ UINTEGER            │
│ BlackElo    │ UINTEGER            │
│ UTCDate     │ DATE                │
│ UTCTime     │ TIME WITH TIME ZONE │
│ ECO         │ VARCHAR             │
│ Opening     │ VARCHAR             │
│ Termination │ VARCHAR             │
│ TimeControl │ VARCHAR             │
│ Source      │ VARCHAR             │
│ movetext    │ VARCHAR             │
│ DataSource  │ VARCHAR             │
│ month       │ VARCHAR             │
│ year        │ BIGINT              │
├─────────────┴─────────────────────┤
│                        20 columns │
└───────────────────────────────────┘

Credits

• Lumbras Gigabase: For collecting and curating a high quality collection of chess games PGNs