by Leathery Tendons, CEO - Vintage Macro Studios
# Sovereign Stack: Local Inference Buildout v2.2
**Vintage Macro Studios — April 5, 2026**
Revised by Claude (Opus 4.6) with current source verification. v2.1 incorporated ChatGPT review on dependency ordering and release chronology. v2.2 incorporates ChatGPT’s final consistency fixes.
## Why This Revision Exists
The original buildout plan was assembled on April 5, 2026 by cross-referencing Gemini, ChatGPT, and Claude. All three models converged on a conservative sequencing: start on the Puget (x86), use Ollama as a bridge on the Sparks, treat vLLM on ARM64/sm_121 as a fragile source-build exercise, and defer multi-node clustering as an advanced topic.
That consensus was wrong — not in sequencing philosophy, but in factual premises. None of the three models searched current sources before advising. The DGX Spark software ecosystem has moved substantially since late 2025, and the “ecosystem gap” that drove the original plan’s caution has largely closed. This revision corrects the factual basis while preserving the sound engineering discipline (crawl-walk-run, validate before scaling, keep Claude Code as a parallel lane).
**Key corrections:**
**v2.1 revision (same day):** Incorporated feedback from ChatGPT (GPT-4o) review of v2. Two changes: (a) resequenced Step 2 to use NVIDIA’s official vLLM container as first proof-of-life before moving to community Docker — establishes a known-good NVIDIA baseline before depending on a community stack; (b) corrected release chronology — the unified memory reporting fix and Ubuntu 6.14 HWE kernel shipped in the **November 2025** DGX OS release, not February 2026 as v2 incorrectly stated. Verified against https://docs.nvidia.com/dgx/dgx-spark/release-notes.html.
**v2.2 revision (same day):** Second ChatGPT review pass. Two consistency fixes: (a) the “What NOT to do first” bullet about Ollama still said “go straight to community Docker,” contradicting the new Step 2 → 2b ordering — corrected to match; (b) Step 2b cited a specific stale vLLM version number (~v0.10.2) for the official container — softened to a general comparison since NVIDIA has updated their NGC tags since initial launch. No philosophy changes. Document now internally consistent.
## Hardware Reference (unchanged)
| Box | Role | Key Specs |
|-----|------|-----------|
| **Puget** | Cockpit / orchestration | x86, RTX 5090, Docker + NVIDIA Container Toolkit |
| **Spark 1** | Primary local inference | ARM64 Grace Blackwell (sm_121), 128GB unified, CUDA 13 |
| **Spark 2** | Scale / throughput | Same as Spark 1, connected via MikroTik CRS804-4DDQ 400G |
| **Yoga** | Thin client | SSH into Puget, runs Cursor via SSH Remote |
## The Revised Plan
### Step 1 — Keep Claude Code
Unchanged from v1. This entire project adds a **second lane**, not a replacement. Claude Code on a Pro or Max subscription remains the best option for Opus-level reasoning, 1M token context, and zero-setup agentic workflows. The local stack is the privacy, control, and fixed-cost lane. Hybrid is the strongest position.
### Step 2 — First proof-of-life: NVIDIA’s official vLLM container on Spark 1
**This replaces the original Step 2 (Puget-first) and Step 6 (Ollama bridge).**
The original plan routed through the Puget because vLLM on Spark allegedly required source builds, Triton patches, and ARM64 compiler gymnastics. That is no longer the case. NVIDIA publishes an official vLLM guide for DGX Spark at `build.nvidia.com/spark/vllm` with a pre-built Docker container path. Use this as your first smoke test — it’s the known-good NVIDIA baseline.
**On Spark 1 (official container):**
Follow NVIDIA’s guide at https://build.nvidia.com/spark/vllm. The official container ships a tested vLLM build for Blackwell/ARM64. Serve a model, confirm the endpoint responds, verify GPU utilization. This is your “the hardware works and NVIDIA’s own software serves a model” checkpoint.
```bash
vllm serve Qwen/Qwen3-Coder-Next \
—port 8000 \
—tensor-parallel-size 1 \
—enable-auto-tool-choice \
—tool-call-parser qwen3_coder \
—max-model-len 32768
```
**Why skip the Puget?** The Sparks are the inference hardware. The Puget’s 5090 is better reserved for DaVinci Resolve and orchestration. Using the Puget as a temporary inference box validates vLLM on the wrong architecture and delays learning on the hardware you’ll actually run in production.
**Why skip the Ollama bridge?** Ollama was positioned as a safe on-ramp before the “hard” vLLM install. With NVIDIA’s own container path now documented and working, vLLM on Spark is the direct path. Ollama won’t give you batching, PagedAttention, or OpenAI-compatible tool calling — all of which matter for Aider integration.
**References:**
### Step 2b — Move to spark-vllm-docker for newer vLLM and cluster features
Once the official container serves cleanly, switch to the community `spark-vllm-docker` project by @eugr (604 stars, 116 forks as of April 2026). This is not affiliated with NVIDIA but is the most actively maintained community tooling for Spark inference. It provides:
**On Spark 1:**
```bash
git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker
./build-and-copy.sh
```
Then serve a model (single-node, no cluster setup needed yet):
```bash
./run-recipe.sh qwen3-coder-next-int4-autoround —solo
```
Or manually:
```bash
./run-cluster-node.sh —role solo
# Inside the container:
vllm serve Qwen/Qwen3-Coder-Next \
—port 8000 \
—tensor-parallel-size 1 \
—enable-auto-tool-choice \
—tool-call-parser qwen3_coder \
—max-model-len 32768 \
—gpu-memory-utilization 0.7
```
**Why move to community after official?** The official NVIDIA container is the cleanest baseline, but the Spark-focused community stack usually moves faster, adds Spark-specific scripts and recipes, and is the more practical lane for clustering once single-node serving is confirmed. Establishing it now on a single node means the cluster transition later is just adding a `—copy-to` flag.
**References:**
### Step 3 — Start with 32K context, not 256K
Unchanged from v1. Qwen’s own model card says: reduce context to 32,768 if memory pressure occurs. Do that from the start. Validate the endpoint works before chasing max context.
**Reference:**
### Step 4 — Point Aider at the Spark 1 endpoint and test on a real repo
Same as v1 Step 5, but now pointing at Spark 1 directly instead of the Puget.
**From the Puget (or Yoga via SSH):**
```bash
cd /path/to/your/repo
aider —model openai/Qwen3-Coder-Next \
—openai-api-base http://<spark1-ip>:8000/v1 \
—openai-api-key not-needed
```
Run it against real work. Try an edit, a refactor, a test generation. Note where quality holds up and where it doesn’t compared to Claude Code. That gap is your calibration data.
Qwen3-Coder-Next is an 80B MoE with only 3B active parameters, 256K native context, Apache 2.0 licensed. It scores 70.6% on SWE-Bench Verified and 44.3% on SWE-Bench Pro. It was explicitly designed to work with Aider, Claude Code, Cline, Qwen Code, and other CLI/IDE harnesses.
**References:**
### Step 5 — Add Spark 2 to the cluster via RoCE through the MikroTik
**This replaces v1 Step 9 and upgrades it from “future experiment” to “documented, supported configuration.”**
NVIDIA officially announced 4-node DGX Spark clustering support at GTC 2026, with RoCE communication via ConnectX-7 NICs. A two-node setup through a RoCE switch is a fully supported topology — not an experiment.
The spark-vllm-docker project explicitly supports this configuration. The networking guide covers RoCE switch setups and the launch scripts auto-detect InfiniBand interfaces.
**Networking setup (one-time):**
Follow NVIDIA’s official Connect Two Sparks playbook for static IP assignment and passwordless SSH:
```bash
# On each Spark, assign static IPs on the CX-7 interfaces
sudo ip addr add 192.168.100.10/24 dev enP2p1s0f1np1 # Spark 1
sudo ip addr add 192.168.100.11/24 dev enP2p1s0f1np1 # Spark 2
# Run NVIDIA’s discover-sparks script from either node
wget https://github.com/NVIDIA/dgx-spark-playbooks/raw/refs/heads/main/nvidia/connect-two-sparks/assets/discover-sparks
chmod +x discover-sparks && ./discover-sparks
```
**Build and distribute the container to Spark 2:**
```bash
# From Spark 1 (head node):
cd spark-vllm-docker
./build-and-copy.sh —copy-to <spark2-ip>
```
**Launch the cluster:**
```bash
# Auto-discover nodes and save config
./run-recipe.sh —discover
# Run a model across both Sparks
./run-recipe.sh qwen3-coder-next-int4-autoround —setup
```
Or manually with launch-cluster.sh, which handles Docker parameters, network interface detection, NCCL environment variables, and Ray cluster initialization automatically.
**Validate RoCE bandwidth before running models:**
```bash
# Run ib_write_bw between the two Sparks
# Target: ~111 Gb/sec per logical interface
```
The spark-vllm-docker networking guide provides full instructions for this validation step.
**First goal:** Run Qwen3-Coder-Next across both Sparks for faster inference and higher concurrency. Bigger-model experiments (Qwen3-Coder-480B, Qwen3.5-397B, Llama-3.1-405B) come after the basic two-node lane is boring and reliable.
**References:**
### Step 6 — Software updates: confirm your Sparks are current
Before or during the above steps, ensure both Sparks have the latest DGX OS updates. NVIDIA has released multiple updates since launch.
**November 2025 DGX OS release:**
**January 2026 DGX OS release:**
**February 12, 2026 (software update via forum announcement):**
**March 12, 2026 (hotfix via forum announcement):**
**Update via DGX Dashboard** (recommended) or terminal:
```bash
sudo apt update && sudo apt upgrade
```
**References:**
### Step 7 — Performance tuning (once the cluster is stable)
The original plan flagged `—enforce-eager` (disabling CUDA graph capture) as an expected 20-30% throughput penalty. This may no longer be necessary — the November 2025 and subsequent updates addressed kernel and driver issues, the March 2026 hotfix resolved multi-node performance regressions, and the community Docker builds track the latest Triton (v3.5.1) and FlashInfer.
**Tuning levers to explore:**
**Performance benchmarks from the community (spark-vllm-docker):**
**References:**
### Step 8 — Optional: Agent orchestration (OpenClaw/NemoClaw)
Unchanged from v1. The original plan’s caution here was correct. OpenClaw’s own docs warn that local models for full agent use are “usually no” and that smaller models are more vulnerable to prompt injection for tool-enabled agents. NemoClaw was flagged at GTC 2026 but is not yet in the standard install plan.
Only add orchestration once the model server underneath it is stable and you’ve calibrated how well Qwen3-Coder-Next handles tool calling via Aider.
### Step 9 — Optional: Qwen Code as a second CLI front end
Unchanged from v1. Qwen Code is an open-source terminal agent optimized for Qwen models. Start with Aider first as the proven harness. Qwen Code is a nice-to-have second front end.
**Reference:**
### Step 10 — Future: Scale to four Sparks
Cecil’s long-term vision of four Sparks with 512GB unified memory is now an officially supported NVIDIA configuration, not a speculative ambition.
NVIDIA’s scaling topology:
Cecil’s MikroTik CRS804-4DDQ 400G switch already supports this topology. The QSFP cables and ConnectX-7 NICs are the correct infrastructure.
**References:**
## What NOT to do first
## Summary
| Phase | Box | Action | Outcome |
|-------|-----|--------|---------|
| 1 | Spark 1 | NVIDIA official vLLM container + Qwen3-Coder-Next | Proof-of-life on known-good baseline |
| 1b | Spark 1 | spark-vllm-docker + Aider | Local coding lane validated with current tooling |
| 2 | Spark 1 + Spark 2 | Join vLLM cluster via RoCE through MikroTik | Two-node scale and throughput |
| 3 | All | Performance tuning (NVFP4, eager mode, caching) | Optimized inference |
| 4 | Puget | OpenClaw/NemoClaw (optional) | Agent orchestration layer |
| 5 | All | Qwen Code (optional) | Second CLI front end |
| 6 | Future | Sparks 3 + 4 | 512GB unified, 700B-class models |
**Claude Code stays in the toolbox throughout.**
## Appendix: Source Index for Verification
Provide these URLs to any model asked to validate this plan:
| Source | URL |
|--------|-----|
| spark-vllm-docker (community) | https://github.com/eugr/spark-vllm-docker |
| spark-vllm-docker releases (pre-built wheels) | https://github.com/eugr/spark-vllm-docker/releases |
| spark-vllm-docker quick start | https://deepwiki.com/eugr/spark-vllm-docker/1.1-quick-start-guide |
| spark-vllm-docker NCCL/RoCE optimization | https://deepwiki.com/eugr/spark-vllm-docker/7.3-nccl-optimization-and-benchmarking |
| Scitrera pre-built containers | https://hub.docker.com/r/scitrera/dgx-spark-pytorch-runtime |
| NVIDIA DGX Spark playbooks | https://github.com/NVIDIA/dgx-spark-playbooks |
| NVIDIA Connect Two Sparks | https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/connect-two-sparks |
| NVIDIA Spark Stacking guide | https://docs.nvidia.com/dgx/dgx-spark/spark-clustering.html |
| NVIDIA Multi-Node Setups (DeepWiki) | https://deepwiki.com/NVIDIA/dgx-spark-playbooks/7-multi-node-setups |
| NVIDIA NCCL for Two Sparks | https://build.nvidia.com/spark/nccl/stacked-sparks |
| NVIDIA 4-node scaling blog | https://developer.nvidia.com/blog/scaling-autonomous-ai-agents-and-workloads-with-nvidia-dgx-spark/ |
| NVIDIA CES 2026 performance blog | https://developer.nvidia.com/blog/new-software-and-model-optimizations-supercharge-nvidia-dgx-spark/ |
| NVIDIA DGX Spark release notes | https://docs.nvidia.com/dgx/dgx-spark/release-notes.html |
| NVIDIA Feb 2026 update | https://forums.developer.nvidia.com/t/dgx-spark-software-updates-02-2026/360362 |
| NVIDIA Mar 2026 hotfix | https://forums.developer.nvidia.com/t/3-12-2026-dgx-spark-fe-update/363301 |
| NVIDIA vLLM on Spark guide | https://build.nvidia.com/spark/vllm |
| NVIDIA performance benchmarking guide | https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/connect-two-sparks/assets/performance_benchmarking_guide.md |
| 4-node cluster benchmark (community) | https://forums.developer.nvidia.com/t/multi-node-dgx-spark-cluster-4x-k3s-sglang-vllm-connectx-7-sr-iov-full-benchmark-matrix/365555 |
| 3-node mesh support (community) | https://forums.developer.nvidia.com/t/three-node-spark-clusters-without-a-switch-are-now-supported-in-spark-vllm-docker-and-sparkrun/365296 |
| Qwen3-Coder-Next model card | https://huggingface.co/Qwen/Qwen3-Coder-Next |
| Qwen3-Coder-Next technical report | https://arxiv.org/abs/2603.00729 |
| Qwen3-Coder GitHub | https://github.com/QwenLM/Qwen3-Coder |
| 4-node support (Guru3D) | https://www.guru3d.com/story/nvidia-dgx-spark-now-supports-fournode-linking-and-512gb-shared-ram/ |
| HotHardware CES 2026 | https://hothardware.com/news/nvidia-dgx-spark-performance-and-sdk-updates-ces2026 |
| StorageReview CES 2026 | https://www.storagereview.com/news/nvidia-dgx-spark-achieves-2-5x-performance-and-8x-video-speed-in-ces-2026-enterprise-update |
| Open Source For You coverage | https://www.opensourceforu.com/2026/01/nvidia-bets-on-open-source-with-local-first-dgx-spark-software-update/ |
| DGX Spark User Guide | https://docs.nvidia.com/dgx/dgx-spark/index.html |