Chinese RoBERTa Miniatures Model description This is the set of 24 Chinese RoBERTa models pre trained by UER py, which is introduced in this paper. Besides, the models could also be pre trained by TencentPretrain introduced in this paper, which inherits UER py to support models with parameters above one billion, and extends it to a multimodal pre training framework. Turc et al. have shown that the standard BERT recipe is effective on a wide range of model sizes. Following their paper, we released the 24 Chinese RoBERTa models. In order to facilitate users in reproducing the results, we used a publicly available corpus and provided all training details. You can download the 24 Chinese RoBERTa miniatures either from the UER py Modelzoo page, or via HuggingFace from the links below: H=128 H=256 H=512 H=768 : : : : : : : : L=2 [ 2/128 (Tiny) ][2 128] [2/256][2 256] [2/512][2 512] [2/768][2 768] L=4 [4/128][4 128] [ 4/256 (Mini) ][4 256] [ 4/512 (Small) ][4 512] [4/768][4 768] L=6 [6/128][6 128] [6/256][6 256] [6/512][6 512] [6/768][6 768] L=8 [8/128][8 128] [8/256][8 256] [ 8/512 (Medium) ][8 512] [8/768][8 768] L=10 [10/128][10 128] [10/256][10 256] [10/512][10 512] [10/768][10 768] L…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy