Introduction FLM 2 (aka Tele FLM) is our open source large language model series. The FLM 2 series demonstrate superior performances at its scale, and sometimes surpass larger models. The currently released versions include (Tele FLM)[https://huggingface.co/CofeAI/Tele FLM] and (Tele FLM 1T)[https://huggingface.co/CofeAI/Tele FLM 1T]. These models feature a stable, efficient pre training paradigm and enhanced factual judgment capabilities. This repo contains the instruction tuned 52B Tele FLM model, which we have named FLM 2 52B Instruct. Model Details FLM 2 52B Instruct utilizes the standard GPT style decoder only transformer architecture with a few adjustments: Rotary Positional Embedding (RoPE) RMSNorm for normalization SwiGLU for activation function Linear bias disabled Embedding and language model head untied Input and output multiplier Models layer number attention heads hidden size ffn hidden size vocab size params count : : : : : : : : : : : : FLM 2 52B Instruct 2407 64 64 8,192 21,824 80,000 52.85 B Training details Unlike conventional fine tuning methods, we employed an innovative and cost effective fine tuning approach. Through specialized screening techniques, we meticu…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy