GPT J 6B Model Description GPT J 6B is a transformer model trained using Ben Wang's Mesh Transformer JAX. "GPT J" refers to the class of model, while "6B" represents the number of trainable parameters. Hyperparameter Value \\(n {parameters}\\) 6053381344 \\(n {layers}\\) 28* \\(d {model}\\) 4096 \\(d {ff}\\) 16384 \\(n {heads}\\) 16 \\(d {head}\\) 256 \\(n {ctx}\\) 2048 \\(n {vocab}\\) 50257/50400† (same tokenizer as GPT 2/3) Positional Encoding Rotary Position Embedding (RoPE) RoPE Dimensions 64 * Each layer consists of one feedforward block and one self attention block. † Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT 2 tokenizer. The model consists of 28 layers with a model dimension of 4096, and a feedforward dimension of 16384. The model dimension is split into 16 heads, each with a dimension of 256. Rotary Position Embedding (RoPE) is applied to 64 dimensions of each head. The model is trained with a tokenization vocabulary of 50257, using the same set of BPEs as GPT 2/GPT 3. Intended Use and Limitations GPT J learns an inner representation of the English language that can be used to extract features useful for…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy