When self-hosting LLM inference beats the API, with the math

Once an LLM feature has real traffic, someone on the team asks whether renting a GPU would be cheaper than the API bill. The question usually gets answered with a vendor benchmark and a gut feeling, and both are bad inputs. What you actually want is a single comparable number, the cost of one request that meets your quality bar and your latency target, computed the same way for every option.

There are four realistic options for a small team. A hosted model called synchronously is the default. The same hosted model called through the provider's batch or flex tier costs half as much if you can wait. A self-hosted open-weight model on a rented GPU, served by something like vLLM, trades the per-token bill for an hourly one. And a quantized model on a CPU box through llama.cpp is the cheapest hardware with by far the lowest throughput.

Which one wins depends on five inputs you should write down before anyone opens a pricing page: requests per month and at peak, average input and output tokens per request, how long a user or job can wait, the smallest model class that passes your evals, and the hours per month someone will spend running the thing. The last two decide more than people expect. If only a frontier model passes your evals, self-hosting is off the table, and if nobody owns the GPU box, its price doesn't matter.

Anything that doesn't need an answer while a user waits (nightly enrichment, document extraction behind a job queue, backfills, eval runs) should go through a provider's batch tier first, because it halves the bill without touching your infrastructure. All three big providers price batch at 50% of the standard rate.

The details differ, so check them against your workload. OpenAI's Batch API takes a JSONL file of up to 50,000 requests or 200 MB, with a completion window that can only be set to 24 hours, and it covers /v1/responses, /v1/chat/completions and /v1/embeddings among others. Anthropic's Message Batches API takes up to 100,000 requests or 256 MB, says most batches finish within an hour, expires whatever isn't done after 24 hours (expired requests aren't billed), and keeps results downloadable for 29 days. Google's Gemini Batch API targets 24 hours and expires jobs that run or wait for more than 48.

Each line of an OpenAI batch file is an ordinary request body with an ID you choose, so moving an async job over is mostly a change in how you submit and collect:

 1{"custom_id": "invoice-81723", "method": "POST", "url": "/v1/responses", "body": {"model": "gpt-6-luna", "input": "Extract the supplier VAT ID and total from: ...", "max_output_tokens": 300}}

Two cheaper variants are worth knowing. OpenAI's Flex processing (service_tier: "flex") charges Batch API rates for ordinary synchronous calls in exchange for slower responses and occasional 429 Resource Unavailable errors, which aren't charged, so it suits background workers that can retry. And Anthropic states that batch and prompt caching discounts stack, though cache hits inside a batch are best effort and its docs suggest the 1-hour cache duration when a batch shares a long prefix.

For a hosted model the cost of a request is just tokens times price. For a GPU it's the hourly rate divided by how many requests the GPU completes per hour at your latency target. Setting the two equal gives you the break-even throughput, which is the number to take into a benchmark.

 1api_cost_per_request = (input_tokens * input_price + output_tokens * output_price) / 1,000,000
 2gpu_cost_per_request = gpu_hourly_price / (requests_per_second * 3600)
 3break_even_rps       = gpu_hourly_price / (3600 * api_cost_per_request)
 4gpus_for_peak        = ceil(peak_rps / requests_per_second_per_gpu)

Here it is as a script with one worked example. The workload is assumed: 300,000 documents a month, 3,000 input tokens and 400 output tokens each. The prices are from Anthropic's pricing page and RunPod's on-demand price list as of 8 November 2026 (Claude Haiku 5.5 at $0.10 input and $0.50 output per million tokens, half that in batch; Claude Sonnet 5.5 at $2 and $10; an L4 with 24 GB on RunPod's Secure Cloud at $0.59 an hour). The 1.5 requests per second for the L4 is a placeholder, not a measurement. Replace it with your own benchmark before you believe anything below it.

 1from decimal import Decimal as D
 2from math import ceil
 3
 4MTOK = D(1_000_000)
 5HOURS_PER_MONTH = D(730)
 6
 7
 8def api_cost_per_request(in_tok, out_tok, in_price, out_price):
 9    return (in_tok * in_price + out_tok * out_price) / MTOK
10
11
12def gpu_cost_per_request(gpu_hourly, req_per_s):
13    return gpu_hourly / (req_per_s * 3600)
14
15
16def breakeven_req_per_s(gpu_hourly, api_per_request):
17    return gpu_hourly / (3600 * api_per_request)
18
19
20def gpus_needed(peak_req_per_s, req_per_s_per_gpu):
21    return ceil(peak_req_per_s / req_per_s_per_gpu)
22
23
24requests = D(300_000)
25in_tok, out_tok = D(3_000), D(400)
26
27apis = {
28    "haiku-5.5 realtime": (D("0.10"), D("0.50")),
29    "haiku-5.5 batch": (D("0.05"), D("0.25")),
30    "sonnet-5.5 realtime": (D("2.00"), D("10.00")),
31    "sonnet-5.5 batch": (D("1.00"), D("5.00")),
32}
33for name, (pin, pout) in apis.items():
34    per_req = api_cost_per_request(in_tok, out_tok, pin, pout)
35    print(f"{name:20} ${per_req * requests:>9.2f}/month  ${per_req:.6f}/request")
36
37l4_hourly = D("0.59")
38measured = D("1.5")  # req/s at your latency target, from your own benchmark
39per_req = gpu_cost_per_request(l4_hourly, measured)
40print(f"L4 per request        ${per_req:.6f}")
41print(f"L4 rented only while busy  ${per_req * requests:.2f}/month")
42print(f"L4 always on               ${l4_hourly * HOURS_PER_MONTH:.2f}/month")
43haiku_batch = api_cost_per_request(in_tok, out_tok, D("0.05"), D("0.25"))
44print(f"break-even vs haiku batch  {breakeven_req_per_s(l4_hourly, haiku_batch):.2f} req/s")
45print(f"GPUs for 4 req/s peak      {gpus_needed(D(4), measured)}")

 1haiku-5.5 realtime   $   150.00/month  $0.000500/request
 2haiku-5.5 batch      $    75.00/month  $0.000250/request
 3sonnet-5.5 realtime  $  3000.00/month  $0.010000/request
 4sonnet-5.5 batch     $  1500.00/month  $0.005000/request
 5L4 per request        $0.000109
 6L4 rented only while busy  $32.78/month
 7L4 always on               $430.70/month
 8break-even vs haiku batch  0.66 req/s
 9GPUs for 4 req/s peak      3

Read the output carefully, because it says something different depending on which line you compare. If a small hosted model passes your evals, the whole job costs $75 a month in batch, and an always-on L4 at $430.70 can't compete before anyone has spent an hour on it. If you can rent the GPU only while a nightly job runs, the raw GPU cost drops to about $33, but that assumes the L4 really sustains more than 0.66 requests per second at this prompt shape and that spinning it up and down is free in engineering time, which it rarely is. The comparison that makes self-hosting attractive is against the larger model: if an open 8B model on your own GPU matches what you'd otherwise send to Sonnet at $1,500 a month in batch, the hardware pays for itself even always on. That "if" is an eval question, and it's the one to answer first.

The throughput number in that formula is the one that most often gets copied from somewhere it doesn't apply. A single stream on a GPU decodes at some tokens per second, but a server running continuous batching interleaves many requests, so aggregate throughput is far higher than one stream, while each request's latency gets worse as the batch grows. You need the requests per second that still meet your latency target, at your input and output lengths, on the exact model and quantization you'd run.

The quantization has to come from the checkpoint. vLLM reads the quantization_config from the model's config when you don't pass --quantization, so pointing vllm serve at an original BF16 repository serves BF16 no matter what you meant, and a 4-bit setup means serving a checkpoint that was quantized to 4 bits, such as Qwen's official AWQ release of Qwen3-8B. Then benchmark with vllm bench serve, giving it your real lengths and a goodput target so that only requests finishing within your latency budget count:

 1vllm serve Qwen/Qwen3-8B-AWQ --max-model-len 8192 --max-num-seqs 32
 2
 3vllm bench serve --model Qwen/Qwen3-8B-AWQ \
 4  --dataset-name random --random-input-len 3000 --random-output-len 400 \
 5  --num-prompts 500 --max-concurrency 16 \
 6  --goodput e2el:20000

The report prints request throughput and, with --goodput, request goodput in requests per second. Goodput is the number that belongs in the formula. Run it at a few concurrency levels, because the best cost per request and an acceptable tail latency rarely happen at the same setting, and random prompts are only a first approximation of real ones, so replay a sample of production inputs once the setup looks promising.

A CPU box follows the same arithmetic with its own hourly price. llama.cpp's llama-server serves GGUF files with continuous batching on by default and a configurable number of parallel slots, for example llama-server -hf Qwen/Qwen3-8B-GGUF:Q4_K_M -c 16384 -np 4 --port 8080. Throughput on prompts of a few thousand tokens is a small fraction of what a GPU does, which is fine for a small queue that runs overnight and useless for anything a user waits on. Measure it like the GPU and let the formula decide.

An always-on GPU bills 730 hours a month whether or not traffic shows up, so bursty or low-volume workloads pay for idle time that the formula's per-request number hides. Size for peak with gpus_for_peak, multiply by the hourly price and the hours you actually keep the machines up, and compare that monthly total, not the per-request figure.

Ops time is the larger hidden cost. Somebody has to own driver and CUDA upgrades, vLLM releases, out-of-memory crashes when a long prompt lands next to a dozen short ones, autoscaling, monitoring for queue depth and time to first token, and the fallback path when the box is down. Model upgrades also become your job: a provider retires old models on a published schedule and ships better ones, while a self-hosted model stays exactly as good as the day you deployed it until you run your evals against something newer. Put a real hourly rate on those hours and add them to the monthly total.

Tail latency needs its own design on shared GPUs. Long prompts and short chat requests mixed in one batch hurt each other, so a serious setup separates them into different queues or deployments, caps input size at the gateway and sheds load rather than letting every request slow down together. If that sounds like more work than the savings justify, it usually is at small scale.

The case where self-hosting wins regardless of the math is data that can't leave your infrastructure, or a contract that forbids third-party processors. Then the formula still helps, but it compares GPU options with each other instead of with an API.

Our default is the cheapest model that passes your evals on the realtime API, then every async path moved to batch or flex, then prompt caching for long shared prefixes, and only then a self-hosted model, when a benchmark shows your real throughput clears the break-even with room to spare and someone has agreed to own it. That order puts the cheap, reversible changes first and leaves the one that adds a pager rotation for last.

Whatever you choose, keep the switch cheap. Put model calls behind one small interface in your codebase that takes your own request type and returns your own response type, records input, output and cached tokens on every call, and enforces timeouts and size limits. vLLM and llama-server both expose OpenAI-compatible endpoints, so moving a route from a hosted model to your own server can be a base URL and a model name in configuration, and moving it back when the GPU box misbehaves is the same change in reverse. With token counts logged per route, you can rerun the formula every quarter with real numbers instead of estimates, which matters because hosted prices move fast (Anthropic's pricing page currently lists Haiku 5.5 input at a tenth of Haiku 4.5's) and the break-even moves with them.