StableLM 3B 4E1T Model Description StableLM 3B 4E1T is a 3 billion parameter decoder only language model pre trained on 1 trillion tokens of diverse English and code datasets for 4 epochs. Usage Get started generating text with StableLM 3B 4E1T by using the following code snippet: Run with Flash Attention 2 ⚡️ Click to expand Model Details Developed by : Stability AI Model type : StableLM 3B 4E1T models are auto regressive language models based on the transformer decoder architecture. Language(s) : English Library : GPT NeoX License : Model checkpoints are licensed under the Creative Commons license (CC BY SA 4.0). Under this license, you must give credit to Stability AI, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the Stability AI endorses you or your use. Contact : For questions and comments about the model, please email lm@stability.ai Model Architecture The model is a decoder only transformer similar to the LLaMA (Touvron et al., 2023) architecture with the following modifications: Parameters Hidden Size Layers Heads Sequence Length 2,795,443,200 2560 32 32 4096 Position Embeddings :…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy