LLM Foundations

See temperature and context length as concrete numbers rather than abstractions.

1. Open facebook/opt-125m. It is small enough that its config is readable in one screen.
2. In the Files tab, open config.json and find max_position_embeddings — that number is the context window, the hard limit on how much text the model can attend to at once.
3. Note vocab_size in the same file. That is how many distinct tokens exist; every prompt is decomposed into these before the model sees anything.
4. Now open the Playground, send the same prompt twice at temperature 0, then twice at temperature 1.

What you should see: Temperature 0 returns the same completion both times; temperature 1 diverges. Sampling is the only difference — the weights never changed.

Get JSON you can parse, without a library.

1. In the Playground, ask in plain language: "Give me the capital of France and its population."
2. Now set a system prompt: "Reply with only a JSON object with keys city and population. No prose, no code fences." Send the same question.
3. Send it twice more. You are testing consistency, not whether it worked once.
4. Try to break it: ask about a country you expect it to be unsure of, and see whether it emits prose instead of JSON.

What you should see: The constrained version parses cleanly and repeatedly. The failure mode you are hunting for is a stray sentence before the brace — that is what breaks a parser in production.

Decide honestly whether you need to fine-tune at all.

1. Write down the task and ten example inputs with the outputs you want.
2. Try to hit them with a system prompt plus two or three few-shot examples in the Playground. Count how many of the ten come out right.
3. If prompting gets you eight or nine, stop — fine-tuning a model to fix two examples costs more than it returns.
4. If it stalls below that, read the TRL docs and pick a method by what data you actually have: SFT if you have correct answers, DPO if you only have "this one is better than that one".

What you should see: A decision backed by a number. Most tasks that feel like they need fine-tuning are prompt problems, and the ten-example test is what tells them apart.

Work out what a model costs to serve before you deploy it.

1. Open a model page and look at the file sizes. Roughly, weights in VRAM ≈ parameters × bytes-per-parameter: 2 bytes at fp16, about 0.5 at 4-bit.
2. Add headroom for the KV cache, which grows with context length and concurrent requests — it is not a rounding error, and on long contexts it can rival the weights.
3. Compare that against the hardware on the Marketplace and its hourly rate.
4. Now do the division that matters: hourly rate ÷ requests you expect per hour. A GPU idling between requests costs the same as a busy one.

What you should see: A cost per request. This is usually the moment quantization stops being an optimisation and becomes the difference between viable and not.

Build a small evaluation set that catches regressions a benchmark will not.

1. Collect ten inputs from your actual use case, including the two or three that previously went wrong. Those are the valuable ones.
2. Write the output you would accept for each. Accept, not perfect — you need a pass/fail you can apply consistently.
3. Run all ten through two candidate models using Model compare, and score them yourself.
4. Check the leaderboard afterwards, not before, and see whether it agreed with your result.

What you should see: Your ten cases and the leaderboard often disagree. When they do, trust yours: the leaderboard measures general capability, you are measuring your task.