Learn how to run Grok 2 correctly Read our Guide . Unsloth Dynamic 2.0 achieves superior accuracy & outperforms other leading quants. Grok 2 Usage Guidelines Use jinja for llama.cpp . You must use PR 15539. For example use the code below: git clone https://github.com/ggml org/llama.cpp cd llama.cpp && git fetch origin pull/15539/head:MASTER && git checkout MASTER && cd .. Utilizes Alvaro's Grok 2 HF compatible tokenizer as provided here Grok 2 This repository contains the weights of Grok 2, a model trained and used at xAI in 2024. Usage: Serving with SGLang Download the weights. You can replace /local/grok 2 with any other folder name you prefer. You might encounter some errors during the download. Please retry until the download is successful. If the download succeeds, the folder should contain 42 files and be approximately 500 GB. Launch a server. Install the latest SGLang inference engine ( = v0.5.1) from https://github.com/sgl project/sglang/ Use the command below to launch an inference server. This checkpoint is TP=8, so you will need 8 GPUs (each with 40GB of memory). Send a request. This is a post trained model, so please use the correct chat template. You should be able to…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy