Self-hosting an LLM versus paying per token

Rohit Athaide · 2026年11月3日 公開

There are two ways to get answers from a language model. You can call a vendor’s API and pay per token. Or you can take an open-weight model, rent GPUs and serve it yourself, paying per hour. The prices look unrelated, which is why people compare them badly.

The two cost shapes

API: cost rises in a straight line with use. Zero requests cost zero, and a million requests cost a million times one. There is nothing to run.

Self-hosted: cost is mostly fixed. A GPU instance costs the same per hour whether it serves one request or a thousand. Cost per token falls as you keep the GPU busy.

So the comparison is a question about utilization: how much of the time your GPU would be doing useful work.

The break-even logic

cost per million tokens (self-hosted) =
    GPU cost per month / (million tokens served per month)

To fill in the bottom number you need the model’s throughput: how many tokens per second the GPU produces at the batch size you will use, which depends on the model, the GPU, the serving software and how many requests arrive together. A number from a benchmark for a different setup will mislead you. Measure your own model on your own hardware with your own prompts.

Then compare the result with the API’s price per million tokens for a model of similar quality. Self-hosting wins only if you can keep the GPU busy enough that its cost per token falls below the API’s.

What decides it

• Volume and steadiness. High, steady traffic favors self-hosting. Spiky or low traffic favors the API, because an idle GPU still bills.
• Model quality. The best models are often available only through an API. An open-weight model may be good enough for your task, and may not.
• GPU memory. The model must fit, with room for the working data that grows with context length. See GPU instances.
• Latency and data rules. Self-hosting keeps data in your own account and gives you control, and an API may not meet a strict residency rule.
• Discounts. API vendors offer cached input and batch rates that cut the effective price for the right workloads. See AI model token pricing.
• Commitments. GPU commitments lower the hourly price if you can commit. See cloud pricing models.

The costs people leave out of self-hosting

• Engineering time to deploy, tune and update the serving stack.
• On-call for a service that must stay up.
• Redundancy. One GPU is a single point of failure, so a real service needs spare capacity.
• Idle headroom for traffic peaks.
• Evaluation and updates whenever a better model appears.
• Networking, storage and monitoring around the GPU.

A sensible path

1. Start with an API. It needs no setup and shows what the model must do.
2. Record real usage: tokens in and out per task, requests per day, and peak load.
3. Test an open-weight model on a rented GPU for a short time, and measure throughput and quality on your own examples.
4. Compute the cost per million tokens at your real utilization, and compare it with the API’s, including the discounts you qualify for.
5. Move only if the saving is large enough to pay for the extra work and risk. Keep the API as a fallback.

What the site offers

The AI cost estimator prices tasks on API models. The GPU side is covered by the GPU accelerators catalog. The site does not yet have a self-hosting calculator, because a trustworthy one needs a sourced throughput figure for each model and GPU, and those depend on your setup.