CodeGen (CodeGen Mono 350M) Model description CodeGen is a family of autoregressive language models for program synthesis from the paper: A Conversational Paradigm for Program Synthesis by Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, Caiming Xiong. The models are originally released in this repository, under 3 pre training data variants ( NL , Multi , Mono ) and 4 model size variants ( 350M , 2B , 6B , 16B ). The checkpoint included in this repository is denoted as CodeGen Mono 350M in the paper, where "Mono" means the model is initialized with CodeGen Multi 350M and further pre trained on a Python programming language dataset, and "350M" refers to the number of trainable parameters. Training data This checkpoint (CodeGen Mono 350M) was firstly initialized with CodeGen Multi 350M , and then pre trained on BigPython dataset. The data consists of 71.7B tokens of Python programming language. See Section 2.1 of the paper for more details. Training procedure CodeGen was trained using cross entropy loss to maximize the likelihood of sequential inputs. The family of models are trained using multiple TPU v4 512 by Google, leveraging data and mod…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy