This walks through taking a model from the hub to a live, OpenAI-compatible endpoint. It assumes nothing beyond an account. ## 1. Pick a model that fits Start on a model page and look at the file sizes, because that is what determines your hardware. As a rough rule, weights in VRAM are parameters multiplied by bytes-per-parameter: about 2 bytes at fp16, about 0.5 at 4-bit. A 7B model is therefore roughly 14 GB at fp16 and under 4 GB at 4-bit. That is the floor, not the requirement. You also need room for the KV cache, which grows with context length and with the number of concurrent requests. On long contexts it can rival the weights themselves. Budget for it deliberately rather than discovering it under load. ## 2. Deploy From the model page, choose Deploy, or start from the inference endpoints page. You select the model, the hardware, and whether the endpoint scales to zero when idle. Scale-to-zero is the setting most worth understanding. With it enabled you pay only while requests are being served, but the first request after an idle period pays a cold start while weights load. For an internal tool that is usually the right trade. For anything user-facing with latency expectations, it usually is not. ## 3. Call it The endpoint speaks the OpenAI chat-completions format, so existing clients work by changing the base URL and the key: ```python from openai import OpenAI client = OpenAI(base_url="https://<your-endpoint>/v1", api_key="<your-token>") resp = client.chat.completions.create( model="<your-model>", messages=[{"role": "user", "content": "Hello"}], ) ``` The same applies to the TypeScript SDK and to anything else that accepts a base URL. That compatibility is the point: you should not have to rewrite application code to change where inference runs. ## 4. Watch the first hour Two things are worth checking before you consider it done. Latency under a realistic prompt, not a one-word test. Time-to-first-token and total generation time behave differently as prompts grow, and a short test hides that. Cost per request, which is the hourly rate divided by the requests you actually serve in an hour. A GPU idling between requests costs the same as a busy one, and this is the number that tells you whether scale-to-zero or a smaller quantization is the better lever. ## When an endpoint is the wrong tool If you are running a batch job over a fixed dataset, renting a machine from the marketplace and running it directly is usually cheaper and simpler than serving an endpoint you then call in a loop. Endpoints earn their keep when something else needs to call them over the network.

Deploy your first model as an Inference Endpoint