Pythia 14M deduped Model Details Developed by: EleutherAI Model type: Transformer based Language Model Language: English Learn more: Pythia's GitHub repository for training procedure, config files, and details on how to use. See paper for more evals and implementation details. Library: GPT NeoX License: Apache 2.0 Contact: Join the EleutherAI Discord and post in release discussion . For general correspondence: contact@eleuther.ai Note: Pythia 14M deduped was trained after the original Pythia suite at the request of interpretability researchers who wanted a smaller model with the same training setup. It uses the same data, tokenizer, hyperparameter conventions, and checkpoint schedule as the rest of the Pythia suite. Unlike the original Pythia models (70M 12B), Pythia 14M deduped was only trained on the deduplicated Pile and does not have a separate non deduplicated variant. Model Configuration: Pythia model Non Embedding Params Layers Model Dim Heads Batch Size Learning Rate Equivalent Models 14M 1,189,888 6 128 4 2M 1.0 x 10 3 — 31M 4,739,072 6 256 8 2M 1.0 x 10 3 — 70M 18,915,328 6 512 8 2M 1.0 x 10 3 — 160M 85,056,000 12 768 12 2M 6.0 x 10 4 GPT Neo 125M, OPT 125M 410M 302,311,4…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy