AraELECTRA ELECTRA is a method for self supervised language representation learning. It can be used to pre train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a GAN. AraELECTRA achieves state of the art results on Arabic QA dataset. For a detailed description, please refer to the AraELECTRA paper AraELECTRA: Pre Training Text Discriminators for Arabic Language Understanding. How to use the discriminator in transformers Model Model HuggingFace Model Name Size (MB/Params) : : : : AraELECTRA base generator araelectra base generator 227MB/60M AraELECTRA base discriminator araelectra base discriminator 516MB/135M Compute Model Hardware num of examples (seq len = 512) Batch Size Num of Steps Time (in days) : : : : : : : : : : AraELECTRA base TPUv3 8 256 2M 24 Dataset The pretraining data used for the new AraELECTRA model is also used for AraGPT2 and AraBERTv2 . The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars (before applying Farasa Segmentation) For the new dataset we added the un…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy