German GPT 2 model In this repository we release (yet another) GPT 2 model, that was trained on various texts for German. The model is meant to be an entry point for fine tuning on other texts, and it is definitely not as good or "dangerous" as the English GPT 3 model. We do not plan extensive PR or staged releases for this model 😉 Note : The model was initially released under an anonymous alias ( anonymous german nlp/german gpt2 ) so we now "de anonymize" it. More details about GPT 2 can be found in the great Hugging Face documentation. Changelog 16.08.2021: Public release of re trained version of our German GPT 2 model with better results. 15.11.2020: Initial release. Please use the tag v1.0 for this older version. Training corpora We use pretty much the same corpora as used for training the DBMDZ BERT model, that can be found in this repository. Thanks to the awesome Hugging Face team, it is possible to create byte level BPE with their awesome Tokenizers library. With the previously mentioned awesome Tokenizers library we created a 50K byte level BPE vocab based on the training corpora. After creating the vocab, we could train the GPT 2 for German on a v3 8 TPU over the complet…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy