Personal engineering · AI infrastructure
Multi-GPU AI Infrastructure Lab
A self-operated, containerized environment for heterogeneous LLM inference: choosing and tuning the model + runtime + quantization + context strategy for each machine, then exposing those systems as usable services. Personal engineering work, independent of professional projects.
AI / Agent Clients
coding agents · research workflows · remote dev (Mac mini M4 · Tailscale)
OpenAI-compatible APIs
one surface over every engine · streaming · OpenAI / Anthropic clients
Dual RTX 5090
SGLang
Dense 27B-class · NVFP4
FP8 KV cache · FlashInfer · DSpark speculative decoding
ASUS GB10
vLLM
MoE-class · FP8
128GB unified memory · up to 131K context · prefix caching · MTP
RTX 3090
llama.cpp
GGUF quants
Secondary experiments · lower-priority workloads
Inference experimentation
quantization · speculative decoding · KV-cache · long context · throughput · memory efficiency
Tailscale mesh
encrypted zero-config remote access · per-device auth · nothing exposed to the internet
Problem / Goal
No single optimal serving configuration
Heterogeneous local inference is a systems problem: different compute platforms have different memory architectures, GPU capabilities, inference-engine compatibility, quantization support, and context constraints. Running a useful model on any one of them requires balancing model quality, memory footprint, architecture compatibility, engine support, context length, throughput / latency, and reliability — at the same time.
So there is no single optimal serving configuration, and the interesting work is choosing and tuning the appropriate model, runtime, quantization, and context strategy per machine, then exposing the result behind a usable service interface. That selection and tuning loop — grounded in measured behavior, not vendor defaults — is what this lab is about.
System Architecture
Three inference nodes, one service surface
Primary GPU system
Dual RTX 5090 · x86 · Blackwell
Two discrete 32GB GPUs on Ubuntu — the main high-throughput serving node.
Grace Blackwell system
ASUS Ascent GX10 / GB10 · 128GB unified
Unified-memory node for larger-memory and long-context experiments.
Secondary GPU system
RTX 3090 · 24GB · Ampere
Secondary inference and experimentation, kept separate from the primary nodes.
A Mac mini M4 runs the control / orchestration plane and remote development — the only supporting machine named here, because it is where routing, monitoring, and remote access actually happen. The nodes themselves are plain containerized Linux services.
Multi-GPU, precisely
The 5090 node contains two separate RTX 5090 32GB GPUs. Their memory is not a transparent pooled 64GB address space, and the configuration does not provide NVLink. Workloads therefore have to be placed or distributed with the serving engine’s multi-GPU strategy in mind. Both GPUs currently link at PCIe x8 @ 32 GT/s; for these inference workloads, PCIe topology is one part of the system design alongside GPU memory capacity, compute, and inter-GPU communication requirements.
Why the Nodes Differ
Engine, quantization, and context fit is per-machine
The three nodes differ on every axis that actually matters: discrete 32GB VRAM per card versus 128GB unified memory, three GPU generations, different engine support (e.g. 4-bit floating-point formats on Blackwell, unified-memory serving on Grace Blackwell), different quantization options, and different context headroom. A configuration that is optimal on one node is usually wrong on the next.
So the operating model is engine / workload / hardware fit, not a universal winner: a fast, structured-serving engine on the discrete-VRAM node; a long-context, unified-memory configuration on the GB10; and portable quantized models on the constrained 3090. The same model class can be served in different precision on different machines, and the right context budget differs node by node.
Engineering Decisions
Choosing model + runtime + quantization + context
Inference-engine selection
SGLang on the 5090 node: high-throughput serving with optimized attention (FlashInfer) and OpenAI-compatible interfaces, where discrete VRAM and a modern generation favor high-performance dense-model serving. vLLM on the GB10: long-context configuration, prefix caching, and speculative / MTP settings where the engine supports them — the unified-memory node is where capacity beats single-GPU bandwidth. llama.cpp on the 3090: portable GGUF quantized configurations, useful across the range of hardware and memory constraints, and as a cross-engine check. No engine is claimed to be universally faster; the fit depends on the node, the model, and the workload.
Quantization strategy
A few representative examples rather than a model catalog: NVFP4 (4-bit floating point) on Blackwell — the FP4 datapath is what makes a dense 27B-class model fit on a 32GB card with KV cache to spare; FP8 (e4m3) weights with an FP8 KV cache, which halves KV footprint versus BF16 and is the main long-context lever on the larger nodes; and GGUF Q4-class quants for the constrained node. Why it matters: quantization trades memory footprint for accuracy, which directly buys context headroom and model fit — so the format is chosen per machine and per accuracy budget, and outputs are compared when switching formats. Quantization is a matrix (engine × accelerator × accuracy), not a single dial.
Long-context and memory engineering
Memory is the hard limit: model weights + KV cache + runtime overhead must fit, and context length is what you get left over. In practice that means static memory reservation and KV-cache caps sized for worst-case context, explicit concurrency limits (concurrency vs memory), and FP8 KV cache to stretch context on 32GB cards. The advertised model context and the context a node can actually serve at a useful concurrency are different engineering questions — the configured budget is the one that matters, and it is tuned as one budget (context, batch, engine flags), not three separate dials.
Speculative decoding
A draft model proposes candidate tokens and the target model verifies them in one pass, so accepted tokens cost less per generated token and generation throughput can improve. The deciding runtime characteristic is the acceptance rate: when a workload’s acceptance is high, the draft pays for itself; when acceptance is poor, the draft overhead eats the benefit. So speculative decoding is treated as a per-workload experiment — measured, enabled where acceptance outweighs draft cost — not a global flag that speedups every workload equally.
Serving / Integration
Services behind a familiar API, not terminal demos
Each node is a persistent, restartable service: containerized (Docker) deployments on Linux with restart-on-failure behavior, not interactive terminal runs. Every backend exposes an OpenAI-compatible API — chat, completions, and tool calls — so coding agents, research workflows, and remote development sessions consume the inference the same way regardless of which engine, quantization, or hardware sits behind the wire.
Networking is the Tailscale private mesh: encrypted, zero-config remote reachability with no forwarded ports and nothing exposed to the public internet. The Mac mini M4 acts as the control / orchestration plane, routing requests to the node whose configuration fits the workload. The point is the abstraction: different inference backends, consumed behind one familiar interface even though their hardware and runtime configuration differs.
Performance / Validation
Measured behavior, not benchmark victories
Validation on the 5090 SGLang configuration is reported as measured behavior with conditions, not as a peak: greedy generation measured at roughly 49 tokens/second median, with workload-dependent variation around it. Speculative-decoding acceptance varies substantially by workload, so the same configuration shows different speedups on different workloads — a measurement result, not a tuning failure.
The discipline is the point: a configuration is a hypothesis, and labeled, condition-logged runs are how hypotheses become defensible decisions. Numbers above are personal measurements on this environment; they are not cherry-picked peaks, and they are not comparable to vendor-published benchmarks.
Failure Modes
Troubleshooting: symptom → diagnosis → adjustment
- GPU memory exhaustion (OOM). Symptom: a node dies under load. Diagnosis: context × batch KV growth, too many concurrent requests, or a restart loop after a memory spike. On the unified-memory GB10, model weights, KV cache, runtime overhead, and system memory all draw from the shared memory budget, so the same sizing discipline matters even more there. Adjustment: cap the KV cache, lower max context or concurrency, and size restart policies for worst-case context instead of hoping.
- Context-length cliffs. Symptom: a deployment fits at short context and fails at the advertised one. Diagnosis: the KV budget math — advertised model context is not practically servable context at useful concurrency. Adjustment: tune context, batch, and engine flags as one budget, and use the bigger node as the escape hatch.
- Runtime / shape mismatches. Symptom: an engine build or graph-capture path fails on certain sequence-length buckets. Diagnosis: configured max lengths and padding don’t match the shapes the runtime expects. Adjustment: align the configured limits with the workload’s actual shapes before scaling context.
- Engine / model compatibility. Symptom: a quantized model or format won’t load, or misbehaves, on a given engine / accelerator pair. Diagnosis: the quantization format is not supported by that runtime on that hardware generation. Adjustment: pick a supported quant / engine pair for the node and verify with a smoke run before promoting a configuration.
- Speculative-acceptance variability. Symptom: the same speculative setup speeds up one workload and not another. Diagnosis: acceptance rate is workload-dependent — the draft only pays off when acceptances outweigh draft cost. Adjustment: measure acceptance per workload and keep speculative decoding enabled only where it wins.
- Contained node failures. Symptom: one node’s failure must not take down the rest. Diagnosis: each node is an independently operated containerized service, so a failure is contained to that node. Adjustment: isolate the failure and restart the affected service; the other nodes keep running as their own services.
What This Demonstrates
Transferable engineering, not hardware enthusiasm
Systems thinking & memory budgets
Weights + KV cache + overhead must fit; context is what you get left over. The same budgeting that governs a production inference stack, applied per machine.
Measurement discipline
A config is a hypothesis. Labeled, condition-logged runs are how a lab stays engineering rather than a showcase — and how a speculative-decoding “win” turns out to be workload-dependent.
Runtime selection & heterogeneous ops
Engine fit over universal winners: SGLang, vLLM, and llama.cpp each win on different nodes and workloads. Because each node is an independently operated service, a failure is contained to that node instead of taking the whole environment down.
Service & API design
The OpenAI-compatible surface is the unit of reuse: agents, IDEs, and research scripts plug in without knowing which engine is behind it. Consumer-scale hardware teaches the same concepts as production infrastructure, just with smaller numbers.
Deep technical detail: current deployments
Implementation snapshot · models rotate; this is the only place they’re namedDual RTX 5090
SGLang
Dense 27B-class · NVFP4
FP8 KV cache · FlashInfer · speculative decoding
ASUS GB10
vLLM
MoE 35B-class · FP8
128GB unified · up to 131K context · prefix caching · MTP
RTX 3090
llama.cpp
GGUF models
Secondary experiments · quantization comparisons
The interesting work is the infrastructure, not which model class happens to be running today; that changes with every experiment.