GreekBERT A Greek version of BERT pre trained language model. Pre training corpora The pre training corpora of bert base greek uncased v1 include: The Greek part of Wikipedia, The Greek part of European Parliament Proceedings Parallel Corpus, and The Greek part of OSCAR, a cleansed version of Common Crawl. Future release will also include: The entire corpus of Greek legislation, as published by the National Publication Office, The entire corpus of EU legislation (Greek translation), as published in Eur Lex. Pre training details We trained BERT using the official code provided in Google BERT's GitHub repository (https://github.com/google research/bert). We then used Hugging Face's Transformers conversion script to convert the TF checkpoint and vocabulary in the desired format in order to be able to load the model in two lines of code for both PyTorch and TF2 users. We released a model similar to the English bert base uncased model (12 layer, 768 hidden, 12 heads, 110M parameters). We chose to follow the same training set up: 1 million training steps with batches of 256 sequences of length 512 with an initial learning rate 1e 4. We were able to use a single Google Cloud TPU v3 8 prov…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy