Pretrain shards Pre tokenized shards for pretraining. License: CC BY NC 4.0 — non commercial only. Format .bin : uint32 little endian token IDs .json : per shard sidecar with doc lengths[] and stats Layout Usage Internal research use only.
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy