🐣 The Quantized LLaMA 3.3 70B Instruct Model Original Base Model: meta llama/Llama 3.3 70B Instruct . Link: https://huggingface.co/meta llama/Llama 3.3 70B Instruct 🚀 Model Inference 🔥 Real world deployment For real world deployment, please refer to the vLLM Distributed Inference and Serving and OpenAI Compatible Server. vLLM can be deployed as a server that implements the OpenAI API protocol. This allows vLLM to be used as a drop in replacement for applications using OpenAI API. By default, it starts the server at http://localhost:8000 . Please check here if you wanna change Engine Arguments . If you would like to deploy your LoRA adapter, please refer to the vLLM documentation for a detailed guide. It provides step by step instructions on how to serve LoRA adapters effectively in a vLLM environment. We have also shared our trained LoRA adapter here. Please download it manually if needed. Then, use the vLLM to serve the base model with the LoRA adapter by including the enable lora flag and specifying lora modules : Since this server is compatible with OpenAI API, you can use it as a drop in replacement for any applications using OpenAI API. For example, another way to query the…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy