Pythia 14M Model Details Developed by: EleutherAI Model type: Transformer based Language Model Language: English Learn more: Pythia's GitHub repository for training procedure, config files, and details on how to use. See paper for more evals and implementation details. Library: GPT NeoX License: Apache 2.0 Contact: Join the EleutherAI Discord and post in release discussion . For general correspondence: contact@eleuther.ai Note: Prior to Feb 27th, 2026, a model was hosted at this URL that was trained on the deduplicated Pile. That model is now at EleutherAI/pythia 14m deduped and this model was correctly trained on the standard Pile. We apologize for any confusion this has caused. Note: Pythia 14M was trained after the original Pythia suite at the request of interpretability researchers who wanted a smaller model with the same training setup. It uses the same tokenizer, hyperparameter conventions, and checkpoint schedule as the rest of the Pythia suite. A deduplicated variant is available at EleutherAI/pythia 14m deduped . Model Configuration: Pythia model Non Embedding Params Layers Model Dim Heads Batch Size Learning Rate Equivalent Models 14M 1,189,888 6 128 4 2M 1.0 x 10 3 — 31M…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy