CodeGen (CodeGen Multi 350M) Model description CodeGen is a family of autoregressive language models for program synthesis from the paper: A Conversational Paradigm for Program Synthesis by Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, Caiming Xiong. The models are originally released in this repository, under 3 pre training data variants ( NL , Multi , Mono ) and 4 model size variants ( 350M , 2B , 6B , 16B ). The checkpoint included in this repository is denoted as CodeGen Multi 350M in the paper, where "Multi" means the model is initialized with CodeGen NL 350M and further pre trained on a dataset of multiple programming languages, and "350M" refers to the number of trainable parameters. Training data This checkpoint (CodeGen Multi 350M) was firstly initialized with CodeGen NL 350M , and then pre trained on BigQuery, a large scale dataset of multiple programming languages from GitHub repositories. The data consists of 119.2B tokens and includes C, C++, Go, Java, JavaScript, and Python. Training procedure CodeGen was trained using cross entropy loss to maximize the likelihood of sequential inputs. The family of models are trained using…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy