An On-Premise RAG Reference Architecture for 100 Users: Right-Sized GPUs, HA by Design¶

You can build a production RAG platform on your own hardware for the price of two years of API bills, and it fits in half a rack. The trick is sizing the GPUs to the models instead of buying the biggest card on the truck. This is the full design for a 100-user, highly available, hybrid-plus-graph RAG stack, with every GPU choice backed by the VRAM math.

The brief: a mid-range company, ~100 knowledge workers, wants RAG over internal documents. It has to stay on-premise (data can't leave the building), survive a node failure without downtime, and not waste money on over-spec'd GPUs. Every model and card below is chosen against that brief.

I size VRAM using the method from How Much VRAM Does Your LLM Actually Need?. If a formula here looks unfamiliar, that post explains it.

The four models, and what each one does¶

A hybrid-plus-graph RAG pipeline uses four models across two paths: ingestion (run when documents change) and retrieval (run on every query).⁴

Why these models: Qwen3-Embedding and Qwen3-Reranker are the current open-source leaders for retrieval and multilingual reranking, and the Qwen3 family is Apache-2.0, which matters for on-prem.²³ The reranker is what buys you accuracy: it helps when recall is high but precision is low, so retrieve wide then rerank narrow.³

The generation model does double duty. Entity/relation extraction during ingestion is just structured generation, so it runs on the same Qwen3-32B you use for answers. Ingestion is a periodic batch job, so it borrows the retrieval GPUs during off-peak hours instead of needing its own.

Sizing the VRAM (the part that decides the bill)¶

Decoder generation is the only model that builds a growing KV cache. Embedders and rerankers do a single forward pass, so they have no KV-cache term. Their VRAM is basically weights plus a little batch activation, and that single fact is why they land on a much cheaper card.

Applying Total = (Weights + KV) / 0.9:

The generation number, worked out:

Weights   = 32B × 1 byte (FP8)                                  = 32 GB
KV/req    = 2 × 64 layers × 5120 hidden × 32768 ctx × 1 byte
            × 0.125 GQA / 10^9                                  = 2.68 GB
KV (batch 8) = 2.68 × 8                                         = 21.5 GB
Total     = (32 + 21.5) / 0.9                                   = 59.4 GB

At 59 GB with batch 8, one card with ~84 GiB usable still has room to grow to ~batch 12 before it's tight. That headroom is deliberate: it's what absorbs a traffic spike when the second replica is down.

Matching GPUs to models: the investment decision¶

Here's the whole point of sizing first. You buy two different cards, because you have two very different memory needs.

Why not the obvious alternatives:

• H100 80GB for generation? It fits (~74.5 GiB usable), but costs roughly 2× an RTX PRO 6000 for less VRAM and tighter batch headroom. HBM bandwidth doesn't help a 100-user workload that's nowhere near throughput-bound. Over-investment.
• H200 141GB or B200 for generation? You'd use 40% of the card. Over-investment by a wide margin.
• L40S for generation? 42 GiB can't hold a 32B model plus any real batch. Under-investment, it would OOM on day one.
• RTX PRO 6000 for embedding/reranking? The two small models need ~32 GB. Putting them on a 96GB card means paying for 64 GB you never touch. Over-investment, that's what the L40S is for.

The right-sized answer: RTX PRO 6000 for the generator, L40S for the encoders. Not the biggest card, not the cheapest, the one that matches the VRAM you actually computed.

The logical architecture¶

Ingestion path on the left, retrieval path on the right. The router is the 2026 production pattern: classify the query, then send simple lookups to vector search and multi-hop/relationship questions through the graph.⁴

        INGESTION (batch, off-peak)                 RETRIEVAL (every query)

   ┌─────────────┐                              ┌──────────────────────────┐
   │  Documents  │                              │   User (100 knowledge    │
   │ PDF/MD/HTML │                              │        workers)          │
   └──────┬──────┘                              └────────────┬─────────────┘
          ▼                                                  │ HTTPS
   ┌─────────────┐                                           ▼
   │  Chunker    │                              ┌──────────────────────────┐
   └──────┬──────┘                              │  RAG API + Query Router  │
          │                                     │  (Adaptive: vector vs    │
   ┌──────┴───────────┐                         │   graph vs hybrid)       │
   ▼                  ▼                          └───┬───────────────┬──────┘
┌────────────┐  ┌──────────────┐                    │               │
│ Embedding  │  │ Entity/Rel   │              dense+sparse       graph
│ Qwen3-Emb  │  │ extraction   │              search            traversal
│   -8B      │  │ Qwen3-32B    │                    │               │
└─────┬──────┘  └──────┬───────┘                    ▼               ▼
      ▼                ▼                       ┌──────────┐   ┌───────────┐
┌──────────┐    ┌────────────┐                │ Qdrant   │   │  Neo4j    │
│ Qdrant   │    │  Neo4j     │                │ (vector) │   │  (graph)  │
│ (vectors)│    │  (graph)   │                └────┬─────┘   └─────┬─────┘
└──────────┘    └────────────┘                     └───────┬───────┘
                                                           ▼ top ~50
                                                    ┌──────────────┐
                                                    │  Reranker    │
                                                    │ Qwen3-Rrk-4B │
                                                    └──────┬───────┘
                                                           ▼ top ~8
                                                    ┌──────────────┐
                                                    │  Generation  │
                                                    │  Qwen3-32B    │
                                                    └──────┬───────┘
                                                           ▼
                                                    grounded answer
                                                    + citations

Qdrant carries both dense and sparse vectors, so it handles the hybrid (semantic + keyword) search in one engine instead of bolting on a separate BM25 service. Neo4j holds the entity-relationship graph for multi-hop questions.

The physical architecture: HA by design¶

Five servers and two switches. Everything runs on a small Kubernetes cluster so failover is automatic, and every tier has at least two live replicas across at least two physical machines. No single server failure takes down a query.

        ┌───────────────┐        ┌───────────────┐
        │  ToR Switch A │◀──MLAG──▶│  ToR Switch B │   25 GbE, redundant
        └──────┬─┬──────┘        └──────┬─┬──────┘
               │ │  bonded (LACP) NICs   │ │
   ┌───────────┘ └───────────┬───────────┘ └───────────┐
   ▼                         ▼                          ▼
┌────────────────┐  ┌────────────────┐        ┌──────────────────────┐
│ GPU Node 1     │  │ GPU Node 2     │        │ Data Nodes 1 / 2 / 3 │
│ 1× RTX PRO 6000│  │ 1× RTX PRO 6000│        │ (K8s control plane + │
│ 1× L40S        │  │ 1× L40S        │        │  stateful data tier) │
│                │  │                │        │                      │
│ vLLM: gen      │  │ vLLM: gen      │        │ Qdrant cluster (RF2) │
│ TEI: embed+rrk │  │ TEI: embed+rrk │        │ Neo4j causal cluster │
│ (K8s worker)   │  │ (K8s worker)   │        │ Postgres (Patroni)   │
└────────────────┘  └────────────────┘        │ RAG API (2+ pods)    │
                                               └──────────────────────┘
        ▲                                                 ▲
        └──────────── keepalived VIP + HAProxy ───────────┘
                     (load balancer, 2 instances)

How each failure is survived:

• GPU node dies: the surviving GPU node serves all traffic at reduced batch headroom (this is why the generator runs at batch 8, not batch 12). Active-active, no failover delay.
• Data node dies: Qdrant replication factor 2, Neo4j 3-core causal cluster, and Patroni each keep quorum with 2 of 3 nodes. Reads and writes continue.
• Switch dies: bonded NICs (LACP) across two MLAG switches keep every server connected.
• Load balancer dies: keepalived floats the VIP to the second HAProxy instance.
• etcd/control plane: 3 control-plane members on the data nodes hold quorum.

The minimum for true HA is N+1 at every tier, and 3 nodes for anything that needs quorum (Kubernetes etcd, Qdrant, Neo4j). Two GPU nodes give you N+1 on inference; three data nodes give you quorum on state.

Server bill of materials¶

That's 4 GPUs total (2× RTX PRO 6000, 2× L40S), which is the entire GPU spend. No SAN: local NVMe with application-level replication keeps storage simple and avoids a six-figure array you don't need at this scale.

Capacity check for 100 users¶

Realistic concurrency for 100 users is ~10 simultaneous in-flight queries, not 100. Two generation replicas at batch 8 give 16 concurrent slots, so:

• Both nodes up: 16 concurrent, comfortable headroom.
• One node down: 8 concurrent, still above the ~10 peak with brief queuing. Degraded, not down.

Embedding and reranking are single-pass and fast, so the L40S pair clears the per-query work long before the generator finishes. The generator is always the bottleneck, which is exactly why it gets the big card.

How many documents does this support?¶

Document count isn't bound by the GPUs at all. It's bound by RAM on the three data nodes for the Qdrant index, and there's a lot of headroom for 100 users.

The math, with assumptions stated because the answer swings on them:

• ~15 chunks per document (a ~10-page doc at ~512-token chunks)
• Qdrant in-RAM cost ≈ vectors × dim × 4 bytes × 1.5 (the 1.5 covers HNSW links and payload indexes)⁷
• ~100 GB RAM per data node earmarked for Qdrant, replication factor 2, so ~150 GB effective for unique vectors
• Original vectors and chunk text live on NVMe (15 TB/node, never the constraint)

A 100-user company usually holds 10K–200K documents, so even the most expensive config gives 2–40× headroom. Two levers stretch it further:

• Store at 1024-dim. Qwen3-Embedding supports Matryoshka truncation, so drop from 4096 to 1024 unless you've measured a real recall gain. That alone 4× your capacity.
• Enable int8 quantization past ~2M chunks. It cuts vector RAM ~4× with negligible recall loss, pushing the ceiling past 6M documents on the same hardware.

Neo4j is not the constraint. It handles billions of nodes and relationships; a 100-user corpus produces a few million entities at most, which the page cache absorbs easily. When you do outgrow the vector RAM, add a data node (linear scale) or turn on quantization. You never touch the GPU tier, which is sized for query throughput, not corpus size.

Where to spend the next dollar (and where not to)¶

• More users (200–300): add a third GPU node. Generation scales linearly with replicas; you don't need bigger cards.
• Longer context / bigger answers: this grows the KV cache, not the weights. Re-run the KV formula at the new context before assuming you need new hardware.
• Faster tokens under load: swap dense Qwen3-32B for the Qwen3-30B-A3B MoE at FP8. It loads the same ~30 GB but computes only ~3B per token, so it decodes far faster at the same VRAM. Same card, more throughput.⁶
• Don't pre-buy H200/B200 "for headroom." At this scale they sit idle. Add nodes when users grow, not capacity you won't use for two years.

Summary¶

The right-sized on-prem RAG stack for 100 users:

• Models: Qwen3-Embedding-8B (embed), Qwen3-Reranker-4B (rerank), Qwen3-32B (extract + generate). All Apache-2.0, all on-prem.
• Sizing: generator ~59 GB (FP8, 32K, batch 8); encoders ~32 GB combined and no KV cache.
• GPUs: RTX PRO 6000 96GB for generation, L40S 48GB for the encoders. Right-matched, not over or under.
• Stores: Qdrant for hybrid dense+sparse vectors, Neo4j for the graph, behind an adaptive query router.
• HA: 2 GPU nodes (active-active) + 3 data nodes (quorum), redundant switches, floating VIP. N+1 everywhere, quorum-of-3 for state.
• Hardware: 5 servers, 4 GPUs, 2 switches. Half a rack.
• Scale path: add nodes for more users, switch to MoE for more speed, re-run the KV math before buying anything.

Size the model first, pick the card second, and the bill takes care of itself.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.