CodeT5+ 220M Model description CodeT5+ is a new family of open code large language models with an encoder decoder architecture that can flexibly operate in different modes (i.e. encoder only , decoder only , and encoder decoder ) to support a wide range of code understanding and generation tasks. It is introduced in the paper: CodeT5+: Open Code Large Language Models for Code Understanding and Generation by Yue Wang\ , Hung Le\ , Akhilesh Deepak Gotmare, Nghi D.Q. Bui, Junnan Li, Steven C.H. Hoi ( indicates equal contribution). Compared to the original CodeT5 family (base: 220M , large: 770M ), CodeT5+ is pretrained with a diverse set of pretraining tasks including span denoising , causal language modeling , contrastive learning , and text code matching to learn rich representations from both unimodal code data and bimodal code text data. Additionally, it employs a simple yet effective compute efficient pretraining method to initialize the model components with frozen off the shelf LLMs such as CodeGen to efficiently scale up the model (i.e. 2B , 6B , 16B ), and adopts a "shallow encoder and deep decoder" architecture. Furthermore, it is instruction tuned to align with natural lang…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy