GPT 2 Mini A smaller GPT 2 model with (only) 39M parameters. It was pretrained on a subset of OpenWebText, the open source version of the pretraining dataset used by OpenAI for the original GPT 2 models. Uses The purpose of this model is mainly for research and education. Its small size allows for fast experiments in resource limited settings, while still being able of generating complex and coherent text. Getting Started Use the code below to get started with the model: Output: Training Details The architecture relies on the GPT 2 model, with smaller dimensions and less layers. It uses the same tokenizer as GPT 2. We used the first 2M rows from the OpenWebText dataset, out of which we use 1k for test and validation sets. Hyperparameters Hyperparameter Value Model Parameters Vocabulary Size 50,257 Context Length 512 Number of Layers 4 Hidden Size 512 Number of Attention Heads 8 Intermediate Size 2048 Activation Function GELU Dropout No Training Parameters Learning Rate 5e 4 Batch Size 256 Optimizer AdamW beta1 0.9 beta2 0.98 Weight Decay 0.1 Training Steps 100,000 Warmup Steps 4,000 Learning Rate Scheduler Cosine Training Dataset Size 1M samples Validation Dataset Size 1k samples F…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy