CodeT5+ 770M Model description CodeT5+ is a new family of open code large language models with an encoder decoder architecture that can flexibly operate in different modes (i.e. encoder only , decoder only , and encoder decoder ) to support a wide range of code understanding and generation tasks. It is introduced in the paper: CodeT5+: Open Code Large Language Models for Code Understanding and Generation by Yue Wang\ , Hung Le\ , Akhilesh Deepak Gotmare, Nghi D.Q. Bui, Junnan Li, Steven C.H. Hoi ( indicates equal contribution). Compared to the original CodeT5 family (CodeT5 base: 220M , CodeT5 large: 770M ), CodeT5+ is pretrained with a diverse set of pretraining tasks including span denoising , causal language modeling , contrastive learning , and text code matching to learn rich representations from both unimodal code data and bimodal code text data. Additionally, it employs a simple yet effective compute efficient pretraining method to initialize the model components with frozen off the shelf LLMs such as CodeGen to efficiently scale up the model (i.e. 2B , 6B , 16B ), and adopts a "shallow encoder and deep decoder" architecture. Furthermore, it is instruction tuned to align wit…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy