Blog¶

Disaggregated LLM Inference on Kubernetes with LLM-D: Prefill/Decode Split, Expert Parallelism, and Real Autoscaling

LLM inference has two phases with different bottlenecks. Prefill is FLOPs-bound, decode is memory bandwidth-bound. Most production stacks shuffle both through the same GPU pool, which costs you time-to-first-token and tethers decode latency to whoever next requests a long prompt. LLM-D — the CNCF Sandbox project co-founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA — splits those phases onto separate pods, schedules expert parallelism for MoE models, and replaces CPU/GPU autoscaling with EPP-derived signals that actually predict load.

Graph Engineering: From Karpathy's Loops to Shared Knowledge Graphs

A short PDF cross-references three pieces of 2026 work — Karpathy's autoresearch loop, his AgentHub sketch, and Anthropic's Knowledge Graph Construction Cookbook — and walks through the missing layer that turns single-agent loops into collaborative swarms. It's an independent synthesis, not a Karpathy paper, but the architectural progression is precise and worth taking seriously. The pipeline has six steps, and the step that matters most is the one your loop is currently skipping.

Running Hermes Agent in Production Behind Traefik

I deployed Hermes Agent to my Hostinger VPS with the one-click Docker Compose template. The build log said Project deployed successfully. It lied — the container came up, but its HTTPS was broken and its admin dashboard was published to the public internet over plain HTTP.

This is the walkthrough of what the template actually gives you, the three defects I found on my own box, and the compose file I run now.

LLM Observability with OpenTelemetry: Instrumenting AI Workloads the Cloud-Native Way

If you already run OpenTelemetry for your services, you don't need a new telemetry stack for LLMs. You need the same tracer, the same collector, the same Grafana dashboard — pointed at a new model of work. The recent GenAI semantic conventions let spans, metrics, and logs describe what an LLM actually did: which model, which prompt, how many tokens, how long, what it cost, and which tool it called.

The Quant Developer Stack in 2026: Self-hosted Market Data, GPU Backtests with Nix and Slurm, and the Path from Notebook to Live Signal

A quant team has two completely different problems to solve. One is research: lots of analysts, lots of notebooks, lots of Parquet, slow human-in-the-loop iteration. The other is production: a small trading desk, kernel-bypass networking, microsecond budgets, almost no humans. The mistake is trying to run both on the same infrastructure. The right answer is two stacks, one boundary.

From YAML to AI: How Platform Engineers Are Embedding LLMs into Internal Developer Platforms

Platform engineering spent years turning tribal knowledge into YAML: catalog descriptors, scaffolder templates, scorecards, policies, and deployment workflows. Now LLMs are becoming the conversational layer over that machinery. The useful shift isn't replacing YAML with chat. It's giving developers a safer way to discover and invoke the platform capabilities already encoded underneath.

Scaling LLM Inference: DP, PP, and TP with vLLM

You sized the model and it doesn't fit on one GPU — or it fits but can't keep up with traffic. Both problems are solved by spreading the model across cards, but with different knobs. Pick the wrong one and you'll waste GPUs, melt your latency, or both. This post is the decision tree.

Context engineering for production AI agents

Prompt engineering is the discipline of writing the right words for a single turn. Context engineering is the discipline of curating everything that ends up in the model's context window across an entire agent run — system prompt, tool definitions, retrieved documents, conversation history, scratchpad notes, sub-agent outputs. Anthropic named this discipline in September 2025¹ because production agents kept hitting the same wall: the model was smart enough, but the context it saw was wrong.

This post covers what context engineering is, the four strategies you actually use, the failure modes that show up in production, and the tooling that does each one well.

GPUs on Kubernetes: From Bare Metal to Schedulable in One Operator

A fresh Kubernetes cluster has no idea your nodes have GPUs. kubectl describe node shows CPU, memory, and pods — nothing else. To make a pod request a GPU you need a driver, a container runtime hook, and a device plugin advertising the hardware to the scheduler, all version-matched across every GPU node. Do it by hand and you'll re-do it on every kernel bump. This post wires it up the way you actually want — one operator — and works on any Kubernetes, not a specific vendor's distro.

Stacking MIG and Time-Slicing on One GPU Operator values.yaml

MIG carves a GPU into hardware-isolated slices. Time-slicing oversubscribes each slice so more pods can share it. Stack them and one physical GPU advertises far more schedulable units than it has silicon — useful when you have more workloads than GPUs and most of them sit idle. Here's the exact values.yaml, wired into the kommander-applications GPU Operator 26.3.0 app¹, and the labels that switch a node between layouts.