Arabic GPT2 You can find more information in our paper AraGPT2 The code in this repository was used to train all GPT2 variants. The code support training and fine tuning GPT2 on GPUs and TPUs via the TPUEstimator API. GPT2 base and medium uses the code from the gpt2 folder and can trains models from the minimaxir/gpt 2 simple repository. These models were trained using the lamb optimizer and follow the same architecture as gpt2 and are fully compatible with the transformers library. GPT2 large and GPT2 mega were trained using the imcaspar/gpt2 ml library, and follow the grover architecture. You can use the pytorch classes found in grover/modeling gpt2.py as a direct replacement for classes in the transformers library (it should support version v4.x from transformers ). Both models are trained using the adafactor optimizer, since the adam and lamb optimizer use too much memory causing the model to not even fit 1 batch on a TPU core. AraGPT2 is trained on the same large Arabic Dataset as AraBERTv2. Usage Testing the model using transformers : Finetunning using transformers : Follow the guide linked here Finetuning using our code with TF 1.15.4: Create the Training TFRecords: Finetuni…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy