Ran Your Own LLM on Kubernetes for 30 Days — Here's the Real Cost¶

The cloud AI cost conversation is two camps shouting past each other: vendor blogs saying "our managed service is cheaper" and Reddit threads saying "self-hosting saves 90%." Neither shows real bills, so neither is useful.

This post shows the real numbers: 30 days, real production traffic, four configurations, every line item. Spoiler — self-hosting was 31% cheaper on the cloud bill and 33% more expensive once you count engineering time. Both facts are true. The details are what matter.

The Setup (Full Transparency)¶

Why we did this at all: a data residency requirement pushing us off the public API for a subset of traffic, plus a CFO who said, verbatim, "show me we'd save $200K/year or stop." This post is the document we showed him.

The model: Llama 3.1 70B Instruct — quantized to AWQ 4-bit for the cost runs, FP16 as the quality control. Our eval set confirmed 70B-class quality was sufficient for our workload (support drafting, summarization, internal copilots).

The traffic (replayed from our actual API logs from the prior 30 days):

The hardware: AWS P5 instances (H100 80GB). Two facts that changed our math versus every 2024-era blog post: P5 now comes in single-GPU sizes (p5.4xlarge), so you can scale at 1-GPU granularity instead of 8 — and on-demand pricing has dropped to $6.88/GPU-hr, with spot around $2.30/GPU-hr in us-east-1.

The stack: vLLM 0.21 on EKS (Kubernetes 1.33), KEDA for queue-depth autoscaling, Prometheus + Grafana, LiteLLM as the gateway in front of everything.

Methodology: we had the OpenAI baseline from the prior month's real bill. Weeks 1–2 ran configs B and C against mirrored production traffic; weeks 3–4 ran config D live. All figures normalized to 30 days at the traffic profile above. Yes, normalization is a judgment call — the raw data is in the appendix repo so you can re-cut it.

The Cost Breakdown: 4 Configurations, 30 Days¶

First, the unit economics everything derives from:

And the results:

You can verify the arithmetic: config B is ~20 GPUs × 720 hrs × $6.88 ≈ $98K. Config D's blended rate (0.6 × $2.30 + 0.4 × $6.88 = $4.13/GPU-hr) across ~10 average GPUs ≈ $29K. Naive self-hosting (config B) was more than double the API. Quantization closed half the gap. Spot closed the rest and went past it.

The honest conclusion up front: config D's 31% savings were real, but came with strings attached —

• 4 spot interruption events that degraded service, each handled by shedding traffic to the API
• 18 hours of engineer time tuning KEDA and rebalancing the spot/on-demand mix mid-month
• Capital allocation risk: a cluster is a commitment; the API is pay-as-you-go

And one string most posts hide entirely, which gets its own section.

The Hidden Costs Nobody Talks About¶

Eight costs that never show up in the GPU calculator:

Now redo the month-one math honestly:

Config D cloud bill:        $32,600
Setup engineering:          $15,000
Eval pipeline + obs + ops:  $15,000+
─────────────────────────────────────
Real month-one total:       ~$63,000

OpenAI, same month:          $47,200

Self-hosting was 33% MORE expensive in the first 30 days. The setup costs are one-time, though: at steady state (cloud $32.6K + ~$3K ongoing ops vs. $47.2K API), the gap is ~$11.6K/month — break-even lands around month 3 if traffic holds. That's the sentence your manager needs, and it's also the sentence the "90% cheaper!" posts and the "just use our API" posts both omit.

The Hybrid Architecture That Actually Wins¶

By day 30 we weren't running config D pure. We landed here:

                       ┌────────────────────────┐
   all traffic ───────▶│  LiteLLM gateway       │
                       │  health checks · cost- │
                       │  aware routing rules   │
                       └─────┬──────────────┬───┘
                     ~70%    │              │   ~30%
                  ┌──────────▼─────┐   ┌────▼──────────┐
                  │ TIER 1         │   │ TIER 2        │
                  │ vLLM 70B AWQ   │   │ OpenAI API    │
                  │ EKS, 60% spot, │   │ burst + spot- │
                  │ 2-replica warm │   │ recovery +    │
                  │ floor, 30s     │   │ peak overflow │
                  │ drain on       │   │               │
                  │ reclaim notice │   │               │
                  └────────────────┘   └───────────────┘

Tier 1 handles steady-state: ~70% of requests at an effective ~$0.0008/1K tokens. Spot nodes drain gracefully on the 2-minute reclaim notice (we trigger drain at 30s into it); the warm floor of 2 on-demand replicas carries the low-traffic overnight hours so we never cold-start.

Tier 2 is the API: peak-hour overflow and spot-recovery traffic. It pays for itself because it's what lets the cluster be sized for the median, not the peak — every GPU you don't buy for the 3 peak hours is pure savings.

The routing rules, in LiteLLM config:

model_list:
  - model_name: llm-prod
    litellm_params:
      model: hosted_vllm/llama-3.1-70b-awq
      api_base: http://vllm.inference.svc:8000/v1
  - model_name: llm-prod
    litellm_params:
      model: openai/gpt-4o          # same logical name → router decides
router_settings:
  routing_strategy: latency-based-routing
  health_check_interval: 5          # probe self-hosted every 5s
  fallbacks:
    - llm-prod: ["openai/gpt-4o"]
  cooldown_time: 60

Plus two custom pre-call hooks: shift 50% of traffic to the API when self-hosted p99 > 2s, error rate > 1%, or healthy replicas < 2 — and cost-aware routing that shifts to the API whenever the spot price spikes above 2× its baseline (at 2×, spot H100s cost more per token than the API does).

The month-end result of the hybrid:

$44,800 looks like a modest $2,400 savings — but that's month one, while still paying the API for 30% of traffic and amortizing nothing. At steady state the hybrid tracks toward the mid-$30Ks while keeping API-grade reliability. The hybrid isn't a compromise between the two camps. It dominates both.

Decision Framework: Should YOU Self-Host?¶

Plug in your own numbers — our 303M tokens/day is the only input that makes our answer right for us:

• Under 5M tokens/day → use the API. Don't think about it again. At this volume the engineering cost dominates everything; your "savings" are a rounding error against one senior engineer's week.
• 5–50M tokens/day, AND you have a senior engineer with K8s + GPU experience → hybrid is probably worth it. Expect 20–40% savings after break-even, which takes a quarter.
• Over 50M tokens/day → self-hosting with reserved capacity is a no-brainer on the math (3-year reserved H100s run ~$3/GPU-hr). But keep the API as fallback — that's just the hybrid again, weighted differently.
• Data residency or compliance blocks the API outright → self-host, but budget 2× your estimate. Compliance was our single largest calendar-time cost.
• No GPU/K8s experience on the team → use a managed open-model service (Together, Fireworks, Anyscale). You get open-model pricing without building the ops muscle — the legitimate middle ground.

What I'd Tell My Past Self¶

Don't trust the "90% cheaper" posts. Every one of them prices GPUs and excludes engineers. Our cloud bill dropped 31%; our month-one total rose 33%. Both numbers come from the same project.

Build the eval pipeline BEFORE you migrate. Migration took days. Knowing whether AWQ's 2–3% quality regression mattered took a week of building evals we should have had anyway. You cannot manage a quality regression you cannot measure.

Spot is great — keep at least 40% on-demand. Our 4 interruption events were absorbed only because the on-demand floor plus API fallback existed. A 100% spot cluster is a cost optimization that converts directly into an availability incident.

The hybrid is the actual answer for ~80% of teams. Self-host the predictable base, buy the unpredictable peak. Tribal "API vs. self-host" takes are arguing about which half of the architecture to delete.

Self-hosting isn't a cost optimization — it's a capability investment. The eval pipeline, the GPU autoscaling, the routing layer, the on-call literacy: you'll need all of it as AI workloads grow, whoever serves the tokens. The 31% was almost beside the point.

The Spreadsheets¶

All the raw data behind this post — billing CSVs, traffic logs, the vLLM/KEDA/LiteLLM configs, and the cost model as a spreadsheet you can plug your own traffic into — is in this repo: llm-selfhost-30days.

The numbers are reproducible. If you run a different setup and get different results, I genuinely want to hear about it — open an issue.

Summary¶

The one-line version: self-host the base, buy the burst, and count the engineers.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.