Bangla BERT Base A long way passed. Here is our Bangla Bert ! It is now available in huggingface model hub. Bangla Bert Base is a pretrained language model of Bengali language using mask language modeling described in BERT and it's github repository Pretrain Corpus Details Corpus was downloaded from two main sources: Bengali commoncrawl corpus downloaded from OSCAR Bengali Wikipedia Dump Dataset After downloading these corpora, we preprocessed it as a Bert format. which is one sentence per line and an extra newline for new documents. Building Vocab We used BNLP package for training bengali sentencepiece model with vocab size 102025. We preprocess the output vocab file as Bert format. Our final vocab file availabe at https://github.com/sagorbrur/bangla bert and also at huggingface model hub. Training Details Bangla Bert was trained with code provided in Google BERT's github repository (https://github.com/google research/bert) Currently released model follows bert base uncased model architecture (12 layer, 768 hidden, 12 heads, 110M parameters) Total Training Steps: 1 Million The model was trained on a single Google Cloud GPU Evaluation Results LM Evaluation Results After training 1…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy