Arabic BERT Model Pretrained BERT base language model for Arabic If you use this model in your work, please cite this paper: Pretraining Corpus arabic bert base model was pretrained on ~8.2 Billion words: Arabic version of OSCAR filtered from Common Crawl Recent dump of Arabic Wikipedia and other Arabic resources which sum up to ~95GB of text. Notes on training data: Our final version of corpus contains some non Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER. Although non Arabic characters were lowered as a preprocessing step, since Arabic characters does not have upper or lower case, there is no cased and uncased version of the model. The corpus and vocabulary set are not restricted to Modern Standard Arabic, they contain some dialectical Arabic too. Pretraining details This model was trained using Google BERT's github repository on a single TPU v3 8 provided for free from TFRC. Our pretraining procedure follows training settings of bert with some changes: trained for 3M training steps with batchsize of 128, instead of 1M with batchsize of 256. Load Pretrained Model You can use this model by installing torch or tensorflow a…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy