MARBERTv2 is one of three models described in our ACL 2021 paper "ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic" . We find that results with ARBERT and MARBERT on QA are not competitive, a clear discrepancy from what we have observed thus far on other tasksWe hypothesize this is because the two models are pre trained with a sequence length of only 128, which does not allow them to sufficiently capture both a question and its likely answer within the same sequence window during the pre training. To rectify this, we further pre train the stronger model, MARBERT, on the same MSA data as ARBERT in addition to AraNews dataset but with a bigger sequence length of 512 tokens for 40 epochs. We call this further pre trained model MARBERTv2 , noting it has 29B tokens . MARBERTv2 acquires best performance on all but one test set, where XLM RLarge marginally outperforms us (only in F1). For more information, please visit our own GitHub repo. BibTex If you use our models (ARBERT, MARBERT, or MARBERTv2) for your scientific publication, or if you find the resources in this repository useful, please cite our paper as follows (to be updated): Acknowledgments We gratefully acknowledge…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy