llm jp 3 150m LLM jp 3 is the series of large language models developed by the Research and Development Center for Large Language Models at the National Institute of Informatics. This repository provides llm jp 3 150m model. For an overview of the LLM jp 3 models across different parameter sizes, please refer to: LLM jp 3 Pre trained Models LLM jp 3 Fine tuned Models. Checkpoints format: Hugging Face Transformers Required Libraries and Their Versions torch =2.3.0 transformers =4.40.1 tokenizers =0.19.1 accelerate =0.29.3 flash attn =2.5.8 Usage Model Details Model type: Transformer based Language Model Total seen tokens: 2.1T Params Layers Hidden size Heads Context length Embedding parameters Non embedding parameters : : : : : : : : : : : : : : 150M 12 512 8 4096 101,874,688 50,344,448 440M 16 1024 8 4096 203,749,376 243,303,424 980M 20 1536 8 4096 305,624,064 684,258,816 1.8b 24 2048 16 4096 407,498,752 1,459,718,144 3.7b 28 3072 24 4096 611,248,128 3,171,068,928 7.2b 32 4096 32 4096 814,997,504 6,476,271,616 13b 40 5120 40 4096 1,018,746,880 12,688,184,320 172b 96 12288 96 4096 2,444,992,512 169,947,181,056 Tokenizer The tokenizer of this model is based on huggingface/tokenizers…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy