tokenizers test data Test and benchmark fixtures for huggingface/tokenizers, pulled on demand by the repo Makefiles ( make test / make bench / make fixtures via hf download ). Layout fixtures/ — multilingual + modality corpora for cross language encode benchmarks. Organized, documented, and reproducible: see fixtures/FIXTURES.md for provenance and fixtures/fixtures manifest.json for exact sources, pinned revisions, and sizes. Rebuild any file with fixtures/fetch fixtures.py . fixtures/lang/ .txt — ~5 MB of real web text per language (FineWeb 2 / FineWeb), 13 non Latin scripts + English baseline fixtures/modalities/ — source code, math/LaTeX, and coding agent trajectories Root files (legacy) — tokenizer JSONs, vocab/merges files, and text corpora referenced by their flat paths from the tokenizers repo Makefiles (Rust, Python, and Node bindings) on main and release branches. Do not move or rename them; new additions should go into organized subdirectories instead. Licensing Corpora under fixtures/ are small excerpts of public datasets, each under its own license — FineWeb/FineWeb 2 (ODC By), open web math (ODC By), SWE smith trajectories (MIT), and permissively licensed OSS repos for…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy