GPT NeoX 20B is a 20 billion parameter autoregressive language model trained on the Pile using the GPT NeoX library. Its architecture intentionally resembles that of GPT 3, and is almost identical to that of GPT J 6B. Its training dataset contains a multitude of English language texts, reflecting the general purpose nature of this model. See the accompanying paper for details about model architecture (including how it differs from GPT 3), training procedure, and additional evaluations. Model details Developed by: EleutherAI Model type: Transformer based Language Model Language: English Learn more: GPT NeoX 20B: An Open Source Autoregressive Language Model. For details about the training dataset, see the Pile paper, and its data sheet. License: Apache 2.0 Contact: to ask questions about this model, join the EleutherAI Discord, and post them in release discussion . Please read the existing GPT NeoX 20B documentation before asking about the model on Discord. For general correspondence: contact@eleuther. ai. Hyperparameter Value n parameters 20554567680 n layers 44 d model 6144 n heads 64 d head 96 n vocab 50257 Sequence Length 2048 Learning Rate 0.97 x 10 5 Positional Encoding Rotary…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy