WARNING : This is the official generator checkpoint as in the ELECTRA original codebase. However, this model is not scaled properly for pre training with google/electra small discriminator. The paper recommends a hyperparameter multiplier of 1/4 between the discriminator and generator for this given model to avoid training instabilities. This would not be the case when using google/electra small generator and google/electra small discriminator , which are similar in size. ELECTRA: Pre training Text Encoders as Discriminators Rather Than Generators ELECTRA is a new method for self supervised language representation learning. It can be used to pre train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a GAN. At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state of the art results on the SQuAD 2.0 dataset. For a detailed description and experimental results, please refer to our paper ELECTRA: Pre training Text Encoders as Discriminators Rather Than Generators. T…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy