Replacing Claude Code With a Self-Hosted LLM on Kubernetes: A Production Reference¶

A single Qwen3-Coder-30B-A3B instance on Kubernetes (vLLM 0.23.0, one H100 80GB, $6.88/GPU-hour on AWS) produces code at "results comparable to Claude Sonnet" on agentic coding benchmarks, at roughly one-quarter to one-sixth the all-in cost of the hosted Anthropic API at 100-engineer scale.¹ Those numbers are real, and so are the trade-offs. The post rests on one bet: at scale the model is the cheap line on the invoice, and what decides whether you ship is everything around it. That means autoscaling, observability, the security review, the tool-calling schema, and network egress.

I wrote this as a reference, not a sales pitch. It covers what Claude Code costs in production, what the vLLM-on-Kubernetes stack looks like in mid-2026, what you give up when you cut the API cord, and the six things that break first.

The shape of the workload¶

Claude Code does four things, and you need to know all four before you decide whether self-hosting is worth it:

• Code generation and editing with multi-file context (up to ~1M tokens for the bigger models)
• Tool calling: reading files, running shell, applying patches, calling MCP servers
• Long context for agentic loops that walk a repository over many turns
• Streaming UX, where users expect the first token in under 1.5s, ideally under 1s

The token mix leans heavily toward output. A typical agentic coding session (read 5 files, propose 1, edit 3, run a test, fix the test, summarize) looks like 60% input, 40% output by raw count. But the cost mix is closer to 30/70, because output tokens cost 5× more than input on every major API.⁹

For Sonnet 4.6 (June 2026 pricing, ≤200K context): input $3/MTok, output $15/MTok. A 100-engineer team averaging 200 agentic turns per engineer per day, each turn ~6K input + 4K output tokens, lands at:

input  cost/day = 100 × 200 × 6_000  × $3e-6 = $360/day
output cost/day = 100 × 200 × 4_000  × $15e-6 = $1,200/day
total                       = $1,560/day = ~$47,000/month

That figure leaves out the prompt cache discount (which can knock 30–50% off input) and the Batch API (50% off both). Don't fixate on the exact number; the order of magnitude is what matters. At 100 engineers, Claude Code is a $30K–$70K/month line item, depending on model mix (Sonnet vs Opus) and how hard you lean on prompt caching.

What "comparable to Claude Sonnet" actually means¶

The Qwen team's claim for the Qwen3-Coder family is direct: it achieves "results comparable to Claude Sonnet" on agentic coding, agentic browser-use, and foundational coding tasks.¹ The 30B-A3B variant, the one I'll use as the reference, has 30.5B total parameters, 3.3B activated (MoE), 48 layers, 128 experts with 8 activated per token, and a 256K context window extendable to 1M with YaRN.⁷ That's a much smaller active footprint than Sonnet 4.6. Anthropic discloses neither the parameter count nor the architecture, so treat any size comparison as a guess; the real claim is that Qwen3-Coder-30B-A3B is more efficient per token.

Here's what that looks like in production. The figures below are illustrative, so measure on your own hardware before you commit to them.

• TTFT (time to first token) for 30B-A3B on a single H100 80GB at a 1K-input prompt: ~250–400ms with vLLM 0.23.0. Sonnet 4.6 typically lands at 400–700ms for similar prompts. Self-hosting wins TTFT by a small margin, mostly because there's no API hop.
• ITL (inter-token latency) at low concurrency: ~30ms/token for 30B-A3B at batch=1 versus ~25ms for Sonnet 4.6. Roughly a wash.
• Throughput at scale: vLLM 0.23.0 on one H100 sustains ~3,000 output tokens/sec aggregate for 30B-A3B at batch=16. Sonnet 4.6 has the throughput of a much larger fleet behind it, but you only feel that when you hit rate limits. At 100 engineers, you will.
• Code quality: the "comparable to Sonnet" claim rests on SWE-Bench Pro scores the Qwen team published but nobody has independently verified yet.¹ If you run a code-review organization, plan to run your own eval. I cover the recipe below.

The reference architecture¶

The stack that works in mid-2026, end to end:

                         ┌──────────────────┐
                         │   VS Code /      │
                         │   JetBrains      │
                         │   (Claude Code   │
                         │   → LiteLLM-     │
                         │   routed)        │
                         └────────┬─────────┘
                                  │ HTTPS
                                  ▼
              ┌────────────────────────────────────┐
              │ LiteLLM proxy (ClusterIP Service)  │
              │  - /v1/messages (Anthropic format) │
              │  - /v1/chat/completions (OpenAI)   │
              │  - Auth + rate limit + audit log   │
              └────────┬───────────────────────────┘
                       │ round-robin or vLLM routing
                       ▼
              ┌────────────────────────────────────┐
              │ vLLM Deployment (Karpenter-managed)│
              │  Qwen3-Coder-30B-A3B-Instruct      │
              │  vLLM 0.23.0 + tool parser         │
              │  Pod: 1× H100 80GB, 30Gi RAM       │
              │  PVC: 200Gi for model cache        │
              └────────────────────────────────────┘
                       ▲
                       │ model pull (one-time)
                       │
              ┌────────────────────────────────────┐
              │ S3/GCS bucket with HF mirror       │
              │  Qwen3-Coder-30B-A3B-Instruct/     │
              │  60GB safetensors                  │
              └────────────────────────────────────┘

Three pieces do most of the work: LiteLLM for protocol translation, vLLM 0.23.0 for inference, and Karpenter for GPU node provisioning. The Qwen3-Coder README has a section called "Prompt with Claude Code" that drives the same Claude Code CLI with a Qwen-Coder backend. For tool-calling purposes the model behaves like Claude, but you have to enable the Qwen tool parser in vLLM.¹

vLLM Deployment¶

The Deployment manifest is intentionally minimal. The model cache is on a PVC so a new pod doesn't redownload 60GB from Hugging Face on every restart.

apiVersion: apps/v1
kind: Deployment
metadata:
    name: vllm-qwen3-coder
spec:
    replicas: 1
    selector:
        matchLabels:
            app: vllm
            model: qwen3-coder-30b-a3b
    template:
        metadata:
            labels:
                app: vllm
                model: qwen3-coder-30b-a3b
        spec:
            nodeSelector:
                karpenter.sh/nodepool: gpu-h100
            containers:
                - name: vllm
                  image: vllm/vllm-openai:v0.23.0
                  args:
                      - "--model=Qwen/Qwen3-Coder-30B-A3B-Instruct"
                      - "--served-model-name=qwen3-coder-30b-a3b"
                      - "--tool-call-parser=qwen3_coder"
                      - "--enable-auto-tool-choice"
                      - "--max-model-len=131072"
                      - "--gpu-memory-utilization=0.92"
                      - "--host=0.0.0.0"
                      - "--port=8000"
                  resources:
                      limits:
                          nvidia.com/gpu: 1
                          memory: 30Gi
                      requests:
                          nvidia.com/gpu: 1
                          memory: 30Gi
                  ports:
                      - containerPort: 8000
                  volumeMounts:
                      - name: model-cache
                        mountPath: /root/.cache/huggingface
            volumes:
                - name: model-cache
                  persistentVolumeClaim:
                      claimName: vllm-model-cache
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: vllm-model-cache
spec:
    accessModes: [ReadWriteOnce]
    storageClassName: gp3
    resources:
        requests:
            storage: 200Gi

The four flags that matter:

• --tool-call-parser=qwen3_coder uses the Qwen3EngineToolParser shipped in vLLM 0.23.0, which expects the qwen_3_coder structural tag.⁸ Leave it off and the model still emits tool calls, just in its native format, which vLLM won't translate into the OpenAI tool-call schema Claude Code expects.
• --enable-auto-tool-choice lets the model decide to call a tool without a forced prompt instruction. You need it for agentic loops.
• --max-model-len=131072 caps context at 128K. The model supports 256K natively and 1M with YaRN, but a single 30B-A3B instance can't fit much KV cache beyond 128K on one H100 80GB. For 256K context you need 2× H100 with tensor parallelism, or the smaller Qwen3-Coder-Next variant.
• --gpu-memory-utilization=0.92 leaves 8% for activations and CUDA context. Go below 0.9 and you waste memory; go above 0.95 and you'll OOM under burst.

Karpenter NodePool¶

Karpenter is the right answer for GPU nodes because hand-tuning a managed node group for 1× H100 80GB on-demand turns into a quarterly cost-grief exercise. The NodePool below spins up p5.48xlarge instances on demand and consolidates them down to zero on idle.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
    name: gpu-h100
spec:
    template:
        spec:
            requirements:
                - key: karpenter.sh/capacity-type
                  operator: In
                  values: ["on-demand", "spot"]
                - key: node.kubernetes.io/instance-type
                  operator: In
                  values: ["p5.48xlarge"]
                - key: kubernetes.io/arch
                  operator: In
                  values: ["amd64"]
                - key: karpenter.k8s.aws/instance-gpu-name
                  operator: In
                  values: ["h100"]
            nodeClassRef:
                group: karpenter.k8s.aws
                kind: EC2NodeClass
                name: gpu-h100
            expireAfter: 720h
    limits:
        cpu: "1000"
        memory: 4000Gi
        nvidia.com/gpu: "8"
    disruption:
        consolidationPolicy: WhenEmptyOrUnderutilized
        consolidateAfter: 30s
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
    name: gpu-h100
spec:
    role: "KarpenterNodeRole-${CLUSTER_NAME}"
    subnetSelectorTerms:
        - tags:
            karpenter.sh/discovery: "${CLUSTER_NAME}"
    securityGroupSelectorTerms:
        - tags:
            karpenter.sh/discovery: "${CLUSTER_NAME}"
    amiSelectorTerms:
        - alias: al2023@latest
    blockDeviceMappings:
        - deviceName: /dev/xvda
          ebs:
            volumeSize: 200Gi
            volumeType: gp3
            deleteOnTermination: true

Spot instances get you p5.48xlarge at ~$25–35/hour against $55.04 on-demand¹⁰, a 40–55% saving for fault-tolerant workloads. The key requirement is karpenter.k8s.aws/instance-gpu-name=h100, which tells Karpenter's well-known label system to pick only instance types with an H100.² The 30B-A3B model fits on 1 H100, so an 8-GPU p5.48xlarge hosts 8 inference pods. Don't split a model across the whole node with tensor parallelism unless you actually need >128K context.

GPU node provisioning: NVIDIA GPU Operator¶

The moment a Karpenter-provisioned p5.48xlarge boots, it's a stock AWS Linux 2 AMI with no NVIDIA driver, no container toolkit, and no device plugin. Your pod will sit Pending forever because the node has no nvidia.com/gpu extended resource to advertise. The fix is the NVIDIA GPU Operator, a single Helm chart that runs the whole GPU stack as DaemonSets and manages the driver, container toolkit, device plugin, feature discovery, and dcgm-exporter lifecycle for you.⁴

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --set driver.enabled=true

What GPU Operator v26.3.2 (released 2026-05-29)⁴ deploys onto every GPU node:

• NVIDIA driver DaemonSet compiles and loads the matching nvidia.ko for the node's kernel
• NVIDIA Container Toolkit configures containerd to use nvidia-container-runtime as the default OCI runtime
• NVIDIA device plugin for Kubernetes v0.19.2 advertises nvidia.com/gpu on each node⁵
• GPU Feature Discovery labels nodes with nvidia.com/gpu.product, nvidia.com/gpu.count, and so on, so Karpenter's karpenter.k8s.aws/instance-gpu-name selector works
• DCGM + dcgm-exporter exposes GPU utilization, memory, temperature, and XID errors to Prometheus
• MIG manager handles H100/A100 MIG configurations (--mig-strategy=single|mixed)
• DRA driver, built in since GPU Operator v23, for the new Dynamic Resource Allocation API

Here's the part that matters for Karpenter: the GPU Operator runs on every node, including ones Karpenter just spun up. When Karpenter provisions a new p5.48xlarge, the node joins the cluster, the GPU Operator pods schedule onto it, the driver DaemonSet installs the matching NVIDIA driver, the device plugin starts, and within about 3–5 minutes the node advertises nvidia.com/gpu: 8 so your vLLM pod can schedule.

Skip the Operator only if you're on a custom AMI with the NVIDIA driver pre-baked (AWS DLAMI, GCP Deep Learning VM, Azure NV-series images) and you've installed the device plugin DaemonSet yourself. What the Operator buys you is a kernel-version-aware driver that auto-installs on new nodes, and both of those turn into manual work without it.

For the reference stack, deploy the Operator first, then Karpenter, then vLLM. Order matters here: Karpenter's karpenter.k8s.aws/instance-gpu-name label only works once GPU Feature Discovery has run on the new node.

LiteLLM proxy¶

LiteLLM is the most underrated piece. It speaks the Anthropic Messages API, the OpenAI Chat Completions API, and the Bedrock API all at once, so the Claude Code CLI works against a self-hosted Qwen3-Coder-30B with no changes. The client points at ANTHROPIC_BASE_URL=http://litellm:4000, and LiteLLM translates the requests into OpenAI chat-completions for vLLM.

apiVersion: v1
kind: ConfigMap
metadata:
    name: litellm-config
data:
    config.yaml: |
        model_list:
            - model_name: claude-sonnet-4-6
              litellm_params:
                  model: openai/qwen3-coder-30b-a3b
                  api_base: http://vllm-qwen3-coder:8000/v1
                  api_key: dummy

        router_settings:
            num_retries: 3
            timeout: 120
            allowed_fails: 5

        litellm_settings:
            drop_params: true
            set_verbose: false

        general_settings:
            master_key: os.environ/LITELLM_MASTER_KEY
            database_url: "postgresql://litellm:litellm@postgres:5432/litellm"

The router field allowed_fails: 5 is the bit that keeps a flapping vLLM pod from blackholing the whole team: after 5 consecutive errors the upstream is marked unhealthy and traffic fails over (or 503s, depending on your replica count).

The "agentic" part: tool calling, KV cache, and the loop¶

The most common production failure mode for self-hosted code assistants isn't model quality. It's the tool-calling schema. Claude Code expects a specific tool-calling format. Qwen3-Coder is trained on its own format, which vLLM's qwen3_coder parser translates into the OpenAI tool-call JSON that LiteLLM converts into the Anthropic Messages format the Claude Code CLI accepts. That's three translation layers, and any one of them can drop a tool call on the floor.

What works:

• Run --enable-auto-tool-choice with --tool-call-parser=qwen3_coder. Without auto-tool-choice, the model wraps tool calls in <tool_call>...</tool_call> XML that OpenAI clients don't parse.
• Use streaming. vLLM 0.23.0 ships streaming tool/function calling with required parameters, so the model emits the function name and partial args as tokens. A long Bash command feels responsive instead of blocking for 8 seconds while the model composes it.⁸
• Pin max_model_len to 128K for 30B-A3B on one H100. Past that, KV cache evictions start hurting multi-turn agentic loops, where the same 200K-token file context gets reread 15 times.

What breaks:

• Multiple tool calls in one turn. Qwen3-Coder is more conservative than Claude about emitting parallel tool calls. If your workflow leans on "read these 5 files in parallel," expect 2–3 sequential reads instead. Fix it by lowering temperature or adding a system prompt that pushes for parallelism.
• Long-running shell commands. Claude Code defaults to a 10-minute timeout on Bash calls. The model has to estimate runtime correctly, and Qwen3-Coder sometimes fires off a long command without run_in_background: true. Fix it with an explicit system-prompt instruction to always tag long-running commands.
• MCP servers with more than 20 tools. The model sometimes loses the right tool name in a long list. Fix it by splitting the tools into smaller, named sets in your MCP server config.

Cost model: where you actually save¶

Run the same 100 engineers at 200 turns/day, but on self-hosted Qwen3-Coder-30B-A3B across 2× H100 (one primary, one for failover plus 2× concurrency headroom):

On cost alone, self-hosting pays for itself at ~15 engineers if you stay on on-demand GPUs (compare $47K/mo hosted to $10,950/mo self-hosted); at spot prices that breakeven drops to ~10 engineers (compare $47K/mo to $7,400/mo). The operational breakeven, where the time you sink into ops, the security review, and 3 a.m. pages pays for itself, sits closer to 30 engineers. Below 10 engineers, just use the API and save yourself the complexity.

What you give up at 100 engineers matters just as much:

• Rate limit cliffs. Sonnet 4.6 enforces rate limits at the org and workspace level that a self-hosted vLLM doesn't. At 100 engineers, you'll hit the workspace limit during a coordinated push and watch latency spike.
• Constant model upgrades. Anthropic ships a new Sonnet every 4–6 months. Qwen ships a new Coder every 6–8 months. Either way, "latest" keeps moving.
• MCP ecosystem. Claude Code's MCP integrations are tested against Claude. Qwen3-Coder works with the same MCP servers through the OpenAI tool-call schema, but the edge cases (a tool with no description, a tool with 100+ params) aren't battle-tested.

The six production gotchas¶

These are what go wrong in week 1 of production, in order of frequency:

1. Model download on first pod start¶

The first vLLM pod spends 4–8 minutes downloading 60GB from Hugging Face, even on a 10Gbps link. On a Karpenter-provisioned node, that time lands inside your cold start. Always pre-pull the model to the PVC before the first Deployment rolls out. Here's the init-container pattern:

initContainers:
    - name: model-pull
      image: curlimages/curl:8.10.1
      command: ["/bin/sh", "-c"]
      args:
          - |
              cd /cache
              curl -L "https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/resolve/main/config.json" -o config.json
              # ... repeat for safetensors.index.json, tokenizer files, generation_config.json
              # Or use `huggingface-cli download` in a real image

If you're paranoid about cold-start latency, mirror the model to S3 and use vLLM's Run:ai Model Streamer to load it directly without writing to local disk. The flag is --load-format runai_streamer plus a model URL.³ The streamer pulls safetensors shards from S3, decrypts them on the fly, and writes straight into GPU memory, skipping the slow curl + disk + safetensors.load_file path. On a fresh p5.48xlarge that cuts model load time from 4–8 minutes to 1–2 minutes, and it drops the 60GB PVC the simpler initContainer approach needs.

2. KV cache pressure under burst¶

Claude Code traffic is spiky. Fifty engineers all hitting "explain this function" after a standup is a 30× burst in 60 seconds. vLLM's continuous batching helps, but --max-num-seqs=256 (the default in 0.23.0) is the real concurrency cap, and anything over it queues. Set --max-num-seqs from your KV cache headroom, not the default. A rough guide: 32 concurrent requests per H100 for 30B-A3B at 128K context, 64 at 32K, 128 at 16K.

3. Observability is on you¶

The Anthropic Console hands you a free dashboard with per-team token usage, latency, error rates, and cost. The self-hosted equivalent is Prometheus and Grafana scraping vLLM's /metrics endpoint, plus a LiteLLM-side dashboard for per-team routing. vLLM 0.23.0 exposes:

vllm:request_success_total{finished_reason="stop|length|tool_calls"}
vllm:request_decode_tokens_total
vllm:time_to_first_token_seconds
vllm:gpu_cache_usage_perc
vllm:gpu_memory_usage_bytes

Wire these to Prometheus and alert on gpu_cache_usage_perc > 0.95 for 5 minutes (you're about to OOM) and time_to_first_token_seconds{quantile="0.95"} > 2 (users are waiting). This is the operational baseline the API hands you for free.

4. Tool-call schema validation¶

The Qwen3-Coder tool parser produces OpenAI-format tool calls, which LiteLLM converts to Anthropic format. But if your tool definition has a description longer than 1024 characters or a JSON Schema with oneOf/anyOf deeper than 5 levels, the translation can drop fields silently. Validate every tool that goes into Claude Code's settings.json MCP server config with the same JSON Schema the model sees.

5. GPU memory fragmentation¶

After a few pod restarts, the H100's 80GB can fragment to the point where vLLM can't allocate a 60GB model even though nvidia-smi shows 70GB free. At the pod level, drop --gpu-memory-utilization to 0.88 if OOM-after-restart has burned you. At the cluster level, taint nvidia.com/gpu-heavy nodes so the inference workload runs alone, and keep a separate node pool for any other GPU work.

6. Security review of the prompt path¶

The Anthropic API is SOC2 Type II, ISO 27001, HIPAA-eligible (with a BAA), FedRAMP Moderate, and runs a third-party red team constantly. Your self-hosted vLLM is none of those things by default. The prompt path is your problem now: what gets sent to the model, what comes back, what gets logged, and who can read the logs. Put this on your security review checklist before you flip the switch:

• TLS termination at LiteLLM, with a cert from your internal CA
• An audit log of every request to a write-once bucket (S3 with Object Lock)
• Per-team rate limits at LiteLLM, not just at vLLM
• A red-team prompt set (50+ attack prompts) run against Qwen3-Coder before production, since the model is more permissive than Claude on jailbreak attempts
• Data classification for what code can leave the corporate VPC. If your engineers work on proprietary code, run the GPU cluster on-prem or in a single-tenant cloud account.

GPU sharing: when one-pod-per-GPU is too expensive¶

The reference stack assumes exclusive GPU access: one vLLM pod per H100, requesting nvidia.com/gpu: 1. That's the simplest model and the right default. But Claude Code traffic is spiky, and once you measure utilization you'll find most pods sitting at 20–30% outside peak hours. You have three options for sharing.

1. NVIDIA CUDA Time-Slicing (cheap, no isolation)¶

The device plugin can advertise one physical GPU as several nvidia.com/gpu resources, and vLLM processes take turns on the GPU in 1ms slices. There's no memory or compute isolation, but it works and it's free.⁵

apiVersion: v1
kind: ConfigMap
metadata:
    name: nvidia-device-plugin-config
data:
    config.yaml: |
        version: v1
        sharing:
            timeSlicing:
                renameByDefault: false
                failRequestsGreaterThanOne: false
                resources:
                    - name: nvidia.com/gpu
                      replicas: 4   # advertise each H100 as 4 GPUs

With replicas: 4 on an 8-GPU p5.48xlarge, you can pack 32 vLLM pods onto the same hardware. In practice this works for batch=1 inference, where the pods spend most of their time waiting on the model. Don't use time-slicing for high-batch sustained throughput. The context switches add 5–10% latency variance, and one pod can starve another under memory pressure.

2. NVIDIA MIG (hardware isolation, A100/H100 only)¶

Multi-Instance GPU partitions a single A100 or H100 into up to 7 hardware-isolated instances, each with dedicated SMs and memory. Every MIG instance looks like a separate GPU to a pod.

# ClusterPolicy for GPU Operator
apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
    name: gpu-cluster-policy
spec:
    migManager:
        enabled: true
    mig:
        strategy: mixed

With mig-strategy=mixed you get resources like nvidia.com/mig-1g.5gb (1/7 of an A100) that pods can request. MIG is the right answer when you need hard isolation for SLAs, so one tenant's bad prompt can't OOM-crash another tenant's pod. The catch: a MIG reconfigure takes the GPU offline for 30–60 seconds, so MIG suits stable partitioning, not dynamic multi-tenancy.

3. Dynamic Resource Allocation (DRA, the future)¶

DRA is the Kubernetes 1.30+ replacement for the device plugin model. Instead of pods requesting an opaque nvidia.com/gpu: 1, they file a ResourceClaim describing the device they need: an H100 with ≥80GB memory, a MIG 1g.5gb slice, a network-attached GPU, and so on. The DRA driver finds a matching device and binds it. GPU Operator v23+ ships a DRA driver, and the v1.36 DRA API moved to beta in April 2026.⁶

For the Claude Code reference stack, DRA is overkill. It earns its keep on a multi-tenant inference platform (different teams, different model sizes, different SLAs), where you want to say "give me a GPU with at least 60GB free" without pre-partitioning the hardware.

What to pick for Claude Code¶

For the reference stack, stick with exclusive access, one pod per H100. The 30B-A3B model at 128K context eats ~70GB of GPU memory, so there's no room to share anyway. If you need higher pod density, the next move is the smaller Qwen3-Coder-Next variant on A100s with MIG (7 instances per A100), not time-slicing a single H100.

When NOT to self-host¶

Self-hosting Qwen3-Coder-30B-A3B is a clear win at 30+ engineers, or whenever you're rate-limited on the Anthropic API. It's a clear loss in these cases:

• Under 10 engineers. The operational overhead doesn't pay back. Use the API.
• Frontier reasoning required. If your engineers tackle hard, novel algorithm work that needs Claude Opus 4.5/4.8-level reasoning, Qwen3-Coder is close on coding but Opus is still ahead on open-ended reasoning. The hosted API gives you the frontier; self-hosting gives you a 3-month-old checkpoint.
• Multi-region compliance. If you need data residency in 5 regions, the Anthropic API (or a managed Bedrock deployment) is one config knob. Self-hosting is 5 Kubernetes clusters, 5 model caches, and 5 upgrade plans.
• No GPU SRE. Self-hosting means you own the 3 a.m. page when vLLM OOMs at peak. If nobody's willing to be on call for an inference cluster, stay on the API.

Summary¶

The shape of the stack that works in mid-2026:

• vLLM 0.23.0 on a Karpenter-managed NodePool of 8× H100 80GB (p5.48xlarge), serving Qwen3-Coder-30B-A3B-Instruct with --tool-call-parser=qwen3_coder and --enable-auto-tool-choice.
• LiteLLM proxy in front, speaking Anthropic Messages API to the Claude Code CLI and OpenAI Chat Completions to vLLM.
• PVC-backed model cache, pre-populated, so cold-start is GPU load time, not download time.
• Karpenter consolidation with on-demand + spot mix, 50/50 baseline.
• Prometheus + Grafana scraping vLLM's /metrics and LiteLLM's per-team counters; alert on gpu_cache_usage_perc and time_to_first_token_seconds p95.
• Per-team rate limits at LiteLLM. Claude Code users will melt your cluster without them.
• Security review of the prompt path, the audit log, and the network egress. Standing up the model is the easy part.

TCO at 100 engineers comes in roughly 4–6× cheaper than Sonnet 4.6 hosted ($47K/mo vs $7.4–10.95K/mo), and in exchange you own the operations, the upgrades, and the security review. Below 10 engineers, the API is still the right call. Between 10 and 30, it comes down to whether you have GPU SRE capacity. Above 30, self-hosting wins on cost, on scale headroom, and on the rate-limit cliffs you'd otherwise hit on the way up.

Versions referenced¶

• vLLM 0.23.0 — verified on GitHub, released 2026-06-15. This release specifically added streaming tool/function calling with required parameters (#40700) and the Qwen3EngineToolParser that Claude Code needs.
• Qwen3-Coder-30B-A3B-Instruct — 30.5B total / 3.3B activated (MoE), 256K native context extendable to 1M with YaRN, Apache-2.0 license.
• Qwen3-Coder-Next — alternative smaller variant, based on Qwen3-Next-80B-A3B with hybrid attention, 1M+ downloads on Hugging Face.
• NVIDIA k8s-device-plugin v0.19.2 — May 2026 release, for the nvidia.com/gpu extended resource that vLLM requests. Bundled by GPU Operator.
• NVIDIA GPU Operator v26.3.2 — released 2026-05-29. Manages the full GPU stack (driver, container toolkit, device plugin, feature discovery, DCGM, MIG, DRA driver) on every GPU node.
• Karpenter v1.13.0 — released 2026-06-10. Use the karpenter.sh/v1 NodePool API and karpenter.k8s.aws/v1 EC2NodeClass. Required: expireAfter under spec.template.spec, and consolidationPolicy: WhenEmptyOrUnderutilized with consolidateAfter under spec.disruption.
• LiteLLM latest stable — Docker image ghcr.io/berriai/litellm:main-stable. Speaks Anthropic Messages, OpenAI Chat Completions, and Bedrock Converse.
• AWS p5.48xlarge — 8× H100 80GB SXM, $55.04/hr on-demand (verified Vantage 2026-06-15). Spot typically 40–55% off.
• AWS p4de.24xlarge — 8× A100 80GB, $27.45/hr on-demand (Vantage). Alternative if H100 is unavailable or expensive.
• Claude Sonnet 4.6 — $3/MTok input, $15/MTok output (≤200K context, verified Anthropic pricing page 2026-06-15).
• Claude Opus 4.5 — $5/$25 standard; $2.50/$12.50 (≤200K batch-style); $37.50/MTok for the 1M context tier (verified Anthropic pricing page).

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.