GPorTuguese 2: a Language Model for Portuguese text generation (and more NLP tasks...) Introduction GPorTuguese 2 (Portuguese GPT 2 small) is a state of the art language model for Portuguese based on the GPT 2 small model. It was trained on Portuguese Wikipedia using Transfer Learning and Fine tuning techniques in just over a day, on one GPU NVIDIA V100 32GB and with a little more than 1GB of training data. It is a proof of concept that it is possible to get a state of the art language model in any language with low ressources. It was fine tuned from the English pre trained GPT 2 small using the Hugging Face libraries (Transformers and Tokenizers) wrapped into the fastai v2 Deep Learning framework. All the fine tuning fastai v2 techniques were used. It is now available on Hugging Face. For further information or requests, please go to "Faster than training from scratch — Fine tuning the English GPT 2 in any language with Hugging Face and fastai v2 (practical case with Portuguese)". Model Model params Model file (pt/tf) Arch. Training /Validation data (text) gpt2 small portuguese 124M 487M / 475M GPT 2 small Portuguese Wikipedia (1.28 GB / 0.32 GB) Evaluation results In a little mor…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy