The Quant Developer Stack in 2026: Self-hosted Market Data, GPU Backtests with Nix and Slurm, and the Path from Notebook to Live Signal

A quant team has two completely different problems to solve. One is research: lots of analysts, lots of notebooks, lots of Parquet, slow human-in-the-loop iteration. The other is production: a small trading desk, kernel-bypass networking, microsecond budgets, almost no humans. The mistake is trying to run both on the same infrastructure. The right answer is two stacks, one boundary.

The stack at a glance¶

This is the version-anchored map of the open-source quant developer stack as of mid-2026. Closed-source incumbents (KDB-X, kdb+, TimeBase) still dominate HFT, but they're getting squeezed where Apache 2.0 alternatives are good enough.

All versions verified on 2026-07-21 from GitHub Releases API or the project's official tags page. Two products (KDB-X, TimeBase) do not publish a public version string; treat those rows as commercial-product pointers, not numbers.

Two stacks, one boundary¶

The cleanest mental model is one platform engineering principle applied twice:

• Research stack (Kubernetes): multi-tenant, declarative, autoscaled, fault-tolerant, slow.
• Trading stack (bare metal): single-tenant, kernel-bypass, deterministic, fast, brittle.

The boundary between them is an object store — usually S3 or a compatible service — plus a versioned feature format that both sides can read without translation.

                ┌─────────── RESEARCH CLUSTER (k8s) ───────────┐
Market feed ──▶ │  Redpanda ─▶ QuestDB (hot) ─▶ ArcticDB (cold)│
                │     │             │              │           │
                │     └──── Dagster DAG ─▶ Polars ─▶ Marimo    │
                └────────────────────┬─────────────────────────┘
                                     │   S3 / Parquet / Lance v10
                                     ▼   (versioned features)
                ┌─────────── TRADING CLUSTER (bare metal) ──────┐
                │  Aeron 1.52.2 + DPDK v26.07                   │
                │  C++ strategy runtime, kernel-bypass NIC      │
                │  Risk + order router                          │
                └──────────────────────────────────────────────┘

Every component on the left has a job it does well. None of them does the other job. Trying to make Redpanda serve microsecond market-data fanout, or trying to make Aeron run in a managed-k8s service, costs latency or money — usually both.

Self-hosted market data on Kubernetes¶

For research, Kubernetes is the right answer. The parts are obvious:

# redpanda-statefulset.yaml - 3 brokers, single-binary Kafka-API
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: redpanda }
spec:
  serviceName: redpanda
  replicas: 3
  template:
    spec:
      containers:
        - name: redpanda
          image: docker.redpanda.com/redpandadata/redpanda:v26.2.1
          command: ["redpanda"]
          args: ["start", "--overprovisioned",
                  "--smp=1", "--memory=8G", "--reserve-memory=0",
                  "--check=false", "--mode=dev-container"]

QuestDB 10.0.0 serves hot tick data via SQL (still Apache 2.0, still ingesting millions of rows per second on a single node).² ArcticDB v6.21.0 from Man Group is the cold-tier DataFrame-native tick store built on LMDB; it speaks Pandas and Polars, so a researcher can read_arctic() into Polars rs-0.55.2⁴ with no schema ceremony.¹ Dagster 1.13.17 wraps the whole pipeline as software-defined assets with lineage.¹¹

What Kubernetes gives you for this:

• Multi-tenant research namespaces. One namespace per analyst team with quotas and RBAC.
• Declarative backtest jobs. Dagster asset runs as a Job; the rest is kubectl.
• KEDA scales backtest workers on queue depth from Redpanda⁸ or NATS⁹.
• Storage classes. NVMe local for scratch, S3 for the lake.

What it does not give you for trading:

• Kernel-bypass NICs. SR-IOV and DPDK need bare-metal drivers; managed EKS/AKS hide the NIC.
• Deterministic CPU pinning. A noisy neighbor on the control plane is fine for a backtest; it's fatal for an order router.
• Microsecond jitter. Trading has single-digit microsecond budgets. K8s control plane tickles add tens of microseconds.

The two-cluster pattern is the only pattern. The research cluster produces a versioned feature set; the trading cluster reads it. The handoff is the object store, not a shared message bus.

When to walk away from k8s¶

Two specific situations force a "no k8s" decision:

1. You need kernel-bypass. Solarflare/Xilinx OpenOnload or Mellanox VMA with DPDK¹⁵ v26.07 needs direct PCI passthrough, dedicated CPU cores, and isolcpus. None of that survives a managed k8s scheduler. Bare-metal with Ansible or NixOS.
2. You need sub-10µs jitter. Even self-hosted k8s with --cpuset adds scheduler latency on the order of 1–5 µs. For strategies with a budget under that, the scheduler is in the critical path.

For everything else — backtests, factor research, simulation, replay, analytics — Kubernetes is the cheaper and more reliable platform.

GPU backtesting clusters with Nix and Slurm¶

This is where 2026 has changed the practical answer. RAPIDS cuDF v26.08.00 ships against CUDA 12.x, and ABI breaks between minor CUDA releases are still the default.¹³ Trying to pin CUDA + cuDF + Python + cuML across a team with pip install is the actual hard problem. Container images alone don't fix it, because Python wheel resolution against manylinux and glibc mismatches still bites.

Nix¹⁶ 2.35.1 does fix it. The flake below gives every researcher on the cluster an identical hermetic environment with CUDA, cuDF, Polars, and uv.

# flake.nix
{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
  description = "Hermetic quant GPU env (CUDA 12.9 + cuDF 26.08 + Python 3.13)";

  outputs = { self, nixpkgs }:
    let pkgs = import nixpkgs { system = "x86_64-linux"; };
    in {
      devShells.x86_64-linux.default = pkgs.mkShell {
        buildInputs = with pkgs; [
          cudaPackages_12_9.cudatoolkit
          cudaPackages_12_9.cuda_cudart
          (python313.withPackages (p: with p; [
            polars cudf-cu12 cuml-cu12
            pytorch-bin pyarrow
          ]))
        ];
        shellHook = ''
          export LD_LIBRARY_PATH="${pkgs.cudaPackages_12_9.cudatoolkit}/lib:$LD_LIBRARY_PATH"
          export EXTRA_NIX_CFLAGS_COMPILE="-I${pkgs.cudaPackages_12_9.cudatoolkit}/include"
          # use uv for project-local resolution
          if [ -f uv.lock ]; then uv sync --frozen; fi
        '';
      };
    };
}

Three things matter in that flake:

• cudaPackages_12_9.cudatoolkit pins the CUDA minor release. cuDF 26.08 was built against CUDA 12.9. Anything else produces import errors that are not reproducible.
• (python313.withPackages ...) builds the interpreter with the right set of extensions; the resulting binary is byte-identical across team machines.
• uv sync --frozen inside the shell gives a project-local lockfile with whatever application code adds on top. Nix handles the system-level stuff; uv handles the application-level stuff.

uv 0.12.3 is fast enough (10–100× faster resolver than pip/poetry) that it doesn't matter how deep your dependency tree is, and the lockfile is portable.⁶ pixi v0.76.1 is the alternative if your team also needs non-Python C++ libraries; it adds a Conda-compatible channel on top of uv.⁷

Slurm for steady, long-running batch¶

For the backtest jobs themselves, Slurm v26.05.2 still wins when the workload is steady and long-running.¹⁴ The cluster shape:

1 login node (Dagster webhook receiver, also schedules jobs)
N GPU nodes — H100 or MI300, MIG-sliced into 7 × 10 GB instances
shared NVMe scratch on each node (~2 TB) for per-job intermediates
shared Lustre / GPFS / MinIO bucket for the dataset lake

A factor-mining job is one sbatch script.

#!/bin/bash
#SBATCH --job-name=factor-mining
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=16
#SBATCH --mem=128G
#SBATCH --time=04:00:00
#SBATCH --output=/var/log/slurm/%j.out

set -euo pipefail
source /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
nix develop --command bash -lc '
  python -m factor_mining \
    --data s3://lake/factors/us_equities/2026Q3 \
    --engine cudf \
    --mig-instance 0 \
    --output /scratch/$(hostname)/results.parquet
'

The nix develop --command wrapper ensures the job runs in the same hermetic env the developer tested on their laptop. Slurm provides GPU pinning via --gres, MIG instance selection via the env var, and a hard 4-hour wall clock to keep the cluster fair.

When k8s still wins¶

For bursty researcher-facing workloads — interactive notebooks, one-off backtests, ad-hoc sweeps — Kubernetes is the right answer:

• Dagster agent runs as a Kubernetes job; KEDA scales on queue depth.
• Marimo notebooks deploy as a Service per analyst.
• Ray / Dask / Polars jobs are short-lived and stateless.

The pattern is hybrid: Slurm for the steady GPU batch, k8s for everything interactive. Both share the same S3 lake.

From research notebook to live signal¶

The pipeline that turns a research idea into a live trading signal is the part every quant team reinvents. The simplest version that actually works in 2026 has five steps.

Step 1 - research¶

A Marimo 0.23.16 notebook against a Parquet lake. Marimo is a reactive notebook stored as a .py file, so it diffs cleanly in git and reproduces state deterministically — none of Jupyter's hidden-state problems.⁵

import marimo as mo
import polars as pl

app = mo.App()

@app.cell
def data():
    df = pl.read_parquet("s3://lake/us_equities/ohlcv/2026-07-21/*.parquet")
    return df, mo.ui.slider(df["date"].min(), df["date"].max())

@app.cell
def factor(df):
    # toy momentum factor
    return pl.col("close").pct_change(20).alias("mom20")

@app.cell
def view(df_with_factor):
    return mo.ui.table(df_with_factor.head(50))

DuckDB v1.5.5 sits next to it for ad-hoc SQL: it's an embedded OLAP engine with Parquet-native reads, faster than anything you'd point a notebook at directly.³

Step 2 - promote¶

The notebook gets promoted to a packaged Python module. uv 0.12.3 generates the lockfile from pyproject.toml; the file is committed, the env is reproducible.

# pyproject.toml
[project]
name = "factor-momentum-20"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
    "polars>=1.0",
    "pyarrow>=18",
    "duckdb>=1.5",
]

[tool.uv]
dev-dependencies = ["marimo>=0.23", "pytest>=8"]

Step 3 - schedule¶

Dagster 1.13.17 wraps the factor computation as an asset with a daily schedule.

from dagster import asset, define_asset_job, ScheduleDefinition

@asset(group_name="factors")
def momentum_20_us_equities(context):
    import duckdb
    con = duckdb.connect()
    df = con.execute("""
        SELECT symbol, date, close / NULLIF(LAG(close, 20) OVER (
          PARTITION BY symbol ORDER BY date), 0) - 1 AS mom20
        FROM read_parquet('s3://lake/us_equities/ohlcv/**/*.parquet')
    """).pl()
    df.write_parquet("s3://lake/factors/us_equities/mom20/2026-07-21.parquet")
    context.add_output_metadata({"rows": len(df)})

factor_job = define_asset_job("factor_momentum_20", selection=[momentum_20_us_equities])
daily_schedule = ScheduleDefinition(job=factor_job, cron_schedule="0 16 * * 1-5")

Step 4 - stream¶

Single-binary Kafka-API: drop-in for any client, no ZooKeeper, no KRaft control-plane games.⁸

The downstream consumer reads by symbol and date from the topic, not from S3. The S3 copy is the durable history; the topic is the live signal.

Step 5 - consume¶

The trading cluster subscribes to the topic via Aeron 1.52.2 (UDP, sub-microsecond), applies a risk check, and routes the order.¹⁰

// pseudo-strategy.cpp
auto subscriber = aeron::Aeron::connect(
    aeron::Context().aeronDir("/dev/shm/aeron-trading"));
auto fragment_handler = [&](const aeron::AtomicBuffer& buf, util::index_t offset,
                            util::index_t length, const aeron::Header& header) {
    auto signal = Signal::deserialize(buf, offset, length);
    if (risk.check(signal)) router.send(signal);
};
subscriber->addSubscription("aeron:udp?endpoint=trading-vpc:40404",
                             fragment_handler);

This is the boundary object: Redpanda on the research side, Aeron on the trading side, the topic as the contract.

Observability across the pipeline¶

Every layer ships OpenTelemetry. The minimum set:

• Marimo cell execution time via opentelemetry-instrumentation.
• Polars operation counts via a custom engine.profile() decorator.
• DuckDB query telemetry via the built-in profiler.
• Redpanda consumer lag exposed as a Prometheus metric.
• Dagster asset materialization events.
• Aeron subscriber counters via JMX.
• Nix-built binaries with a pinned --version metric for provenance.

The single rule: every metric traces back to a run ID. A failure on the trading side should be reproducible by replaying the upstream S3 versioned dataset. If it isn't, the boundary is leaking.

A practical first pass¶

If you're starting today, in order:

• Pick one research team and one trading team. Get them on ArcticDB v6.21.0 for shared feature storage.
• Stand up Redpanda v26.2.1 as the research ↔ trading message bus. Don't try to use Aeron across both — Aeron is for the trading side only.
• For researcher envs, use Marimo 0.23.16 + uv 0.12.3 + pixi v0.76.1 + a Nix 2.35.1 dev shell. Don't mix conda and uv; pick one.
• For the GPU cluster, build a Nix flake that pins CUDA 12.9 + cuDF v26.08.00. Every commit to that flake is a backtest-cluster upgrade.
• For orchestration, start with Dagster 1.13.17; switch to Prefect¹² 3.8.2 if you need dynamic workflows that Dagster's static asset graph can't model.
• For the trading runtime, don't put it on k8s. Bare metal with DPDK v26.07, Aeron 1.52.2, dedicated NICs, CPU isolation.
• Wire OpenTelemetry at every layer with a single trace context across the S3 ↔ Redpanda ↔ Aeron boundary.

When NOT to adopt this stack¶

Two categories of team should not adopt the full stack above:

1. You have less than five analysts. A single DuckDB + Polars + Marimo notebook on one analyst's laptop covers it. Don't run Slurm.
2. You are latency-bound at single-digit microseconds in production. You already know your stack: C++, kernel-bypass, FPGA/ASIC if you can afford it. The two-cluster pattern above is for teams where one side is research and one is trading, not for a single ultra-HFT shop.

Summary¶

• The 2026 quant developer stack is two clusters, one boundary: k8s for research, bare metal for trading, joined by an object store.
• The open-source core is anchored: ArcticDB v6.21.0, QuestDB 10.0.0, DuckDB v1.5.5, Polars rs-0.55.2, Redpanda v26.2.1, Dagster 1.13.17, Marimo 0.23.16, uv 0.12.3, pixi v0.76.1, Nix 2.35.1, Slurm v26.05.2, cuDF v26.08.00, Aeron 1.52.2, DPDK v26.07.
• GPU backtesting needs Nix to pin CUDA + cuDF + Python. Pip alone won't survive the ABI.
• The research-to-signal pipeline is Marimo → uv → Dagster → Redpanda → Aeron. Each step has a single owner tool and a single output format.
• K8s is right for research, wrong for trading. DPDK + Aeron + bare metal are non-negotiable for the trading side.
• Observability must cross the S3 ↔ Redpanda ↔ Aeron boundary with a single trace context.

Don't unify the two stacks. Connect them with a versioned object store and let each side do what it's best at.

Questions or discussion? Connect on LinkedIn, X, or email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.