CodeBERTa CodeBERTa is a RoBERTa like model trained on the CodeSearchNet dataset from GitHub. Supported languages: The tokenizer is a Byte level BPE tokenizer trained on the corpus using Hugging Face tokenizers . Because it is trained on a corpus of code (vs. natural language), it encodes the corpus efficiently (the sequences are between 33% to 50% shorter, compared to the same corpus tokenized by gpt2/roberta). The (small) model is a 6 layer, 84M parameters, RoBERTa like Transformer model – that’s the same number of layers & heads as DistilBERT – initialized from the default initialization settings and trained from scratch on the full corpus (~2M functions) for 5 epochs. Tensorboard for this training ⤵️ Quick start: masked language modeling prediction Does the model know how to complete simple PHP code? Yes! That was easy 🎉 What about some Python (warning: this is going to be meta) Results: This program can auto complete itself! 😱 Just for fun, let's try to mask natural language (not code): This (kind of) works because code contains comments (which contain natural language). Of course, the most frequent name for a Computer scientist must be undefined 🤓. Downstream task: program…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy