Llama 3 8B Gradient Instruct 1048k Join our custom agent and long context (262k 1M+) waitlist: https://forms.gle/L6TDY7dozx8TuoUv7 Gradient incorporates your data to deploy autonomous assistants that power critical operations across your business. If you're looking to build custom AI models or agents, email us a message contact@gradient.ai. For more info see our end to end development service for custom LLMs and AI systems Join our Discord This model extends LLama 3 8B's context length from 8k to 1040K, developed by Gradient, sponsored by compute from Crusoe Energy. It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. We trained on 830M tokens for this stage, and 1.4B tokens total for all stages, which is RULER evals: Our model is behind only GPT 4 and Yi in the retrieval and Q&A tasks It’s the smallest parameter model to rank in the top 7 overall Approach: meta llama/Meta Llama 3 8B Instruct as the base NTK aware interpolation [1] to initialize an optimal schedule for RoPE theta, followed by empirical RoPE theta optimization Progressive training on increasing context lengths, similar to Large World Model […
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy