Streaming responses in 2026: SSE vs WebSockets vs gRPC for AI apps¶

Every AI app needs to stream. Most teams pick the wrong transport and spend a week debugging why the first token takes 8 seconds. The LLM is fast — the network in front of it isn't. This post compares the three transports that actually work in production (SSE, WebSockets, gRPC streaming), shows the buffering traps that kill streaming, and gives you the decision framework to pick the right one in 30 seconds.

If you've ever shipped a "ChatGPT-style" streaming endpoint and watched it work in dev, then break in staging behind nginx, this is the post for you.

The three transports that matter¶

There are only three transports you should consider for AI streaming in 2026. Everything else is a wrapper around one of them.

If you're streaming from a browser to a user, start with SSE. If you need bidirectional realtime, WebSockets. If you're piping tokens between services inside your cluster, gRPC streaming.

The rest of this post is the production gotchas for each.

1. Server-Sent Events (SSE) — the default for AI apps¶

SSE is HTTP with a Content-Type: text/event-stream response, kept open, with the server writing data: <chunk>\n\n lines as it goes. That's it. The browser's EventSource API consumes it natively.

Why SSE wins for most AI apps:

• It's just HTTP. Works with every CDN, every reverse proxy, every auth system
• Auto-reconnect is built into the browser API
• No protocol upgrade handshake (unlike WebSockets)
• Plays nicely with HTTP/2 multiplexing
• Easy to debug with curl or browser DevTools
• Works with bearer tokens, cookies, and CSRF protection the same way as any HTTP endpoint

Server side, any framework can do it. Example with Node.js (Hono 4.12.x) serving a vLLM stream:

// server.ts — Hono 4.12.x
import { Hono } from "hono"
import { stream } from "hono/streaming"

const app = new Hono()

app.post("/v1/chat", async (c) => {
  const body = await c.req.json()

  c.header("Content-Type", "text/event-stream")
  c.header("Cache-Control", "no-cache")
  return stream(c, async (s) => {
    // Pipe vLLM's OpenAI-compatible stream straight through
    const upstream = await fetch("http://vllm:8000/v1/chat/completions", {
      method: "POST",
      headers: { "Content-Type": "application/json", Authorization: `Bearer ${env.VLLM_KEY}` },
      body: JSON.stringify({ ...body, stream: true }),
    })

    const reader = upstream.body!.getReader()
    const decoder = new TextDecoder()

    while (true) {
      const { done, value } = await reader.read()
      if (done) break
      // Forward bytes as-is — they're already SSE-formatted
      await s.write(decoder.decode(value, { stream: true }))
    }
  })
})

Client side, vanilla browser:

// EventSource is GET-only — this assumes a GET /v1/chat route,
// not the POST handler above. For POST bodies, see the fetch parser below.
const evtSource = new EventSource("/v1/chat?msg=hello")

evtSource.onmessage = (e) => {
  if (e.data === "[DONE]") { evtSource.close(); return }  // not valid JSON — guard first
  const chunk = JSON.parse(e.data)
  appendToUI(chunk.choices[0]?.delta?.content || "")
}

evtSource.onerror = () => {
  // EventSource auto-reconnects; UI state is preserved
  console.log("stream interrupted, reconnecting…")
}

If you need POST (because the request body is too big for query params), EventSource won't work — use the fetch API with a manual SSE parser:

// For POST bodies, the browser fetch API is the right tool
const response = await fetch("/v1/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ messages }),
})

const reader = response.body!.getReader()
const decoder = new TextDecoder()
let buffer = ""

while (true) {
  const { done, value } = await reader.read()
  if (done) break
  buffer += decoder.decode(value, { stream: true })

  // Split on SSE event boundary (\n\n), emit complete events
  const events = buffer.split("\n\n")
  buffer = events.pop() || ""  // keep partial event in buffer

  for (const evt of events) {
    const line = evt.split("\n").find(l => l.startsWith("data: "))
    if (!line) continue
    const data = line.slice(6)
    if (data === "[DONE]") continue
    const chunk = JSON.parse(data)
    appendToUI(chunk.choices[0]?.delta?.content || "")
  }
}

SSE's one limitation: one-way only. The client can't send data back over the same connection. For AI apps that need bidirectional — multi-turn with tool calls where the client decides, realtime voice, collaborative editing — you need WebSockets.

2. WebSockets — bidirectional, realtime, but more infra¶

WebSockets give you a full-duplex connection over a single TCP socket. The client and server can both send messages at any time. After the initial HTTP upgrade, the protocol switches to binary frames.

When WebSockets are right for AI apps:

• Multi-turn conversations where the client sends tool results back mid-stream
• Realtime voice (browser MediaRecorder → STT → LLM → TTS, with overlapping streams)
• Collaborative editing (multiple users, one LLM, shared state)
• Anything where the LLM needs to push control messages ("I'm going to call this tool now") AND receive responses

Server side, ws 8.21.x on Node.js:

// server.ts — ws 8.21.x
import { WebSocketServer } from "ws"

const wss = new WebSocketServer({ port: 8080 })

wss.on("connection", async (ws) => {
  ws.on("message", async (raw) => {
    const { type, payload } = JSON.parse(raw.toString())

    if (type === "user_message") {
      // Stream from vLLM and forward each token as a WebSocket frame
      try {
        const upstream = await fetch("http://vllm:8000/v1/chat/completions", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ ...payload, stream: true }),
        })

        if (!upstream.ok || !upstream.body) {
          throw new Error(`upstream returned ${upstream.status}`)
        }

        const reader = upstream.body.getReader()
        const decoder = new TextDecoder()

        while (true) {
          const { done, value } = await reader.read()
          if (done) break
          const text = decoder.decode(value, { stream: true })
          // Parse SSE events from upstream, send each as a WS frame
          for (const line of text.split("\n")) {
            if (line.startsWith("data: ") && line !== "data: [DONE]") {
              ws.send(JSON.stringify({ type: "token", data: JSON.parse(line.slice(6)) }))
            }
          }
        }
        ws.send(JSON.stringify({ type: "done" }))
      } catch (err) {
        // Don't let upstream failures silently kill the WS — surface to client
        ws.send(JSON.stringify({ type: "error", message: (err as Error).message }))
      } finally {
        // Close so the client knows the stream is over; without this the WS
        // sits idle until ALB/nginx (60s) or Cloudflare (100s) kills it.
        ws.close()
      }
    }
  })
})

Client side:

const ws = new WebSocket("wss://api.example.com/v1/chat")

ws.onopen = () => {
  ws.send(JSON.stringify({ type: "user_message", payload: { messages } }))
}

ws.onmessage = (e) => {
  const msg = JSON.parse(e.data)
  if (msg.type === "token") appendToUI(msg.data.choices[0]?.delta?.content || "")
  if (msg.type === "done") markComplete()
}

The infra gotchas that bite WebSockets in production:

• Auth: cookies don't survive the upgrade cleanly in all browsers; bearer tokens in the URL work but end up in access logs. Use a short-lived token issued via POST, then upgrade.
• Sticky sessions: if you have multiple WebSocket servers behind a load balancer, the client must hit the same server for the lifetime of the connection. ALB supports this with sticky cookies; nginx needs ip_hash or session cookies.
• Idle timeouts: ALB's idle timeout is 60s. nginx defaults to 60s. Cloudflare is 100s. WebSocket connections that go quiet get killed. Send a ping every 30s (ws.ping()) to keep them alive.
• No HTTP/2 multiplexing: each WebSocket is its own TCP connection. 10K concurrent users = 10K sockets. SSE gives you HTTP/2 multiplexing for free.

Rule of thumb: if you need bidirectional, use WebSockets. If you only need server-to-client, use SSE — it has fewer infra traps.

3. gRPC streaming — the inside-the-cluster option¶

gRPC streaming is what you use when tokens flow between services, not from a server to a browser. Your BFF talks to your agent service over gRPC, the agent service talks to the model service over gRPC, the model service streams from vLLM.

Three flavors: - Server-streaming: client sends one request, server streams many responses - Client-streaming: client streams many requests, server sends one response - Bidirectional: both stream

For LLM token streaming, server-streaming is what you want.

Proto:

// chat.proto
syntax = "proto3";

service Chat {
  rpc Complete(CompleteRequest) returns (stream CompleteResponse);
}

message CompleteRequest {
  repeated ChatMessage messages = 1;
  string model = 2;
  float temperature = 3;
  int32 max_tokens = 4;
}

message CompleteResponse {
  string delta = 1;
  string finish_reason = 2;
  Usage usage = 3;
}

message Usage {
  int32 prompt_tokens = 1;
  int32 completion_tokens = 2;
}

message ChatMessage {
  string role = 1;
  string content = 2;
}

Server (Go):

// server.go
func (s *chatServer) Complete(req *pb.CompleteRequest, stream pb.Chat_CompleteServer) error {
    upstream, _ := vllm.StreamComplete(ctx, req.Messages, req.Model)

    for token := range upstream.Tokens {
        if err := stream.Send(&pb.CompleteResponse{Delta: token}); err != nil {
            return err
        }
    }
    return stream.Send(&pb.CompleteResponse{FinishReason: "stop"})
}

Client (Go):

stream, _ := client.Complete(ctx, &req)
for {
    resp, err := stream.Recv()
    if err == io.EOF { break }
    if err != nil { log.Fatal(err) }
    fmt.Print(resp.Delta)
}

Why gRPC wins for service-to-service:

• HTTP/2 multiplexing for free — one connection, many concurrent streams
• Schema-first (proto files) — generates typed clients across every officially supported language (Go, Java, Python, C++, C#, Node, Rust via tonic, and more)
• Built-in deadlines, cancellation, retries — the client can cancel a stream when the user closes the tab
• Backpressure — gRPC understands when the consumer is slow and applies flow control
• mTLS and auth — gRPC handles the security story well

Why gRPC loses for browser-to-server:

Browsers can't speak gRPC natively. You need grpc-web and a proxy (Envoy) to translate. It's an extra hop, an extra failure mode, and a worse debugging experience. For the browser edge, stick with SSE or WebSockets.

The right split in 2026:

Browser ←SSE / WebSocket→ BFF ←gRPC streaming→ Agent service ←gRPC streaming→ vLLM

The buffering trap — the #1 reason streaming "doesn't work"¶

Your LLM returns the first token in 80ms. The user sees it after 8 seconds. The bottleneck isn't the LLM. It's the proxies in between, each with their own buffering policy.

nginx buffering (on by default in every version — you must turn it off):¹

# CRITICAL: disable proxy buffering for SSE
location /v1/chat {
    proxy_pass http://bff;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;     # default is ON — that default is what breaks SSE
    proxy_cache off;
}

Alternatively, leave nginx alone and have the BFF emit X-Accel-Buffering: no as a response header — nginx reads it and disables buffering for that response. (It's a response header from your backend, not a proxy_set_header you send upstream.)

Cloudflare buffering (free tier buffers aggressively):²

• Free plan: buffered (streaming broken or delayed)
• Pro plan: respects X-Accel-Buffering: no
• You can also disable buffering per-response with the no-buffer transform rule

AWS ALB:³

• ALB passes streaming responses through fine, but its idle timeout (default 60s, configurable up to 4000s) kills any connection that goes quiet for that long. A slow inter-token gap or a paused agent will drop the stream.
• Fix: send a keepalive comment (: ping\n\n) during quiet stretches, raise the idle timeout, or put an NLB in front for long-lived TCP with no L7 idle timeout.

Go's http.ResponseWriter:

// Default behavior BUFFERS. Must flush after each write.
// http.ResponseController (Go 1.20+) works even when middleware wraps w,
// unlike a raw w.(http.Flusher) assertion, which silently no-ops on a wrapped writer.
rc := http.NewResponseController(w)
for token := range tokens {
    fmt.Fprintf(w, "data: %s\n\n", token)
    rc.Flush()  // <-- without this, the client waits
}

The mental model: every proxy between your LLM and the user has a "wait, maybe more is coming" buffer. Each one adds 100-500ms of perceived latency. Five proxies = 2-3 second delay before the first byte reaches the user. Disable buffering at every hop.

Time-to-first-token vs inter-token latency — measure both¶

Two metrics matter for AI apps. Track them separately.

Time to First Token (TTFT) — how long from request sent to first token received. Target: < 500ms for chat, < 2s acceptable for long-context.

Inter-Token Latency (ITL) — average time between subsequent tokens. Target: < 50ms (feels smooth), < 100ms acceptable, > 150ms feels choppy.

TTFT  ████████░░░░░░░░  ← 350ms (good)
ITL   ▎▎▎▎▎▎▎▎▎▎▎▎▎▎▎  ← 28ms avg (excellent)

A 70B model on a single H100 produces TTFT ≈ 200ms, ITL ≈ 25ms.⁴ A 400B model on 4x H100 with NVLink produces TTFT ≈ 400ms (waiting for the slowest GPU), ITL ≈ 35ms (the model itself, not the interconnect, is the bottleneck).

Common causes of bad TTFT:

Common causes of bad ITL:

Instrument both. Emit both metrics to Prometheus and Langfuse. Alert on TTFT p95 > 1s and ITL p95 > 100ms. Without these alerts, you'll find out from users, not your dashboard.

Decision framework — pick in 30 seconds¶

Start with SSE. Add WebSockets only when SSE can't do what you need. Don't reach for gRPC unless you're between services inside the cluster.

Pre-deploy checklist¶

Before you ship an AI streaming endpoint:

• Buffering disabled at every proxy — nginx, Cloudflare, ALB, your BFF framework
• TTFT and ITL instrumented — Prometheus + Langfuse, with p50/p95/p99 dashboards
• Alerts on TTFT p95 > 1s and ITL p95 > 100ms
• Client handles reconnection — EventSource for SSE, manual ping/pong for WebSockets
• Backpressure tested — slow client doesn't pin a worker; cancel upstream when client disconnects
• Auth works through the stream — bearer in header (SSE/gRPC), short-lived token for WebSocket upgrade
• First-token budget — measured end-to-end, not just LLM TTFT
• Idle keepalive — WebSocket ping every 30s, gRPC keepalive_time set
• Mobile tested — SSE reconnects gracefully on network switch (WiFi → LTE)
• Vary on Accept if you serve both SSE and JSON from the same path

Summary¶

• Start with SSE for browser-to-server streaming. It's HTTP, it just works.
• Use WebSockets when you need bidirectional (tool calls mid-stream, voice, collab).
• Use gRPC streaming for service-to-service inside the cluster.
• Disable buffering at every proxy — the LLM isn't slow, your proxies are.
• Measure TTFT and ITL separately — they're different failure modes.
• The transport choice locks you in — pick deliberately, change is expensive.

The LLM is the easy part. The transport is where AI apps live or die.

Versions referenced¶

• Hono 4.12.x — verified 4.12.25 on npm, June 2026
• ws 8.21.x — verified 8.21.0 on npm, June 2026
• Go net/http — http.ResponseController requires Go 1.20+ (Go 1.20 release notes); current stable is go1.26.4
• nginx proxy_buffering directive — default on in all versions, verified against current nginx docs
• AWS ALB — idle timeout default 60s, max 4000s, verified against AWS ALB user guide
• Cloudflare — streaming/range behavior verified against the current default-cache-behavior page; the dedicated streaming/ page has been retired. Free-plan throttling behavior is illustrative — verify on your own traffic.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.