*A field report from the Leathery Tendons Labs — running a frontier agentic MoE across two NVIDIA DGX Spark (GB10) boxes, the wedges that nearly killed it, and the exact recipe that fixed it.*
# Getting MiniMax-M2.7 Running on Two DGX Sparks: What Finally Worked (and What Didn’t)
*A field report from the Leathery Tendons lab — running a frontier agentic MoE across two NVIDIA DGX Spark (GB10) boxes, the wedges that nearly killed it, and the exact recipe that fixed it.*
## The goal
I have two DGX Sparks (GB10 Grace-Blackwell, 128 GB unified memory each, joined by a direct ConnectX-7 / QSFP56 200 Gb link running RoCE). The point of two of them was simple: run a **bigger, better open-weight model than I could fit on one box**, and use it as the brain for my self-hosted Hermes agent (”Agent 13”).
The target: **`cyankiwi/MiniMax-M2.7-AWQ-4bit`** — a ~229B-parameter Mixture-of-Experts model at 4-bit, with a native tool-call parser and a huge context window. It’s the model Techno Tim reports running on his two ASUS Ascent GX10s at ~40 tok/s, and it’s a genuinely strong agentic/ops model. The plan was tensor-parallel across both Sparks (TP=2).
It took several failed attempts and a hard lesson before it worked. Here’s the honest version.
## What didn’t work — the wedges
Every early attempt died the same way: the model would start loading, then **both Sparks would lock up** — SSH itself would time out at banner exchange, the endpoint never opened, and the only recovery was a physical power-cycle of both boxes. Classic host saturation during model load.
The natural assumption — and the one that cost the most time — was **”it’s a memory problem; lower `—gpu-memory-utilization`.”** We dropped it to 0.6, then planned 0.5. It kept wedging. That assumption was *wrong*, and here’s the tell that should have caught it sooner:
**Techno Tim runs the same model at `—gpu-memory-utilization 0.75` and 192K context and it works. We were wedging at 0.6 and 8K context.** Higher memory pressure, more context — and *his* works while *ours* died. That proves the wedge was never about the memory fraction. It was about **deviating from the known-good recipe.**
### The misses (documented honestly, because they’re instructive)
The automated agent driving the earlier attempts (”Codex”) got impressively far — it formed the Ray cluster, resolved the model path, got `world_size=2` with both ranks joined — but it deviated from the proven recipe in ways that each, on their own, can wedge a GB10:
1. **It used a self-built vLLM image instead of the proven one.** A custom `vllm-gb10:v0.24.0` build rather than the image the working recipe was validated against.
2. **It used `—no-ray`** (PyTorch-distributed) instead of the Ray backend the recipe specifies.
3. **The big one: it almost certainly never set `NCCL_NET_GDR_LEVEL=0`.** GPUDirect RDMA is *unsafe on the GB10 unified-memory SoC* — the GPU and CPU share the same DRAM, there’s no PCIe peer path for GDR, and if NCCL attempts it, it can **hard-lock the SoC** during load. That is exactly the wedge symptom. This single environment variable is the difference between a clean load and two frozen machines.
4. **It chased the memory fraction** (0.6 → 0.5) as the fix, when 0.5 is actually *too low* for this model — at ~61 GB of weights per node (TP=2), a 0.5 budget leaves no room for KV cache and fails the *other* way.
5. **Reboot persistence gaps** bit us too: the NAS mount (`/mnt/ai`) didn’t auto-remount after a Spark reboot, and the agent’s egress proxy (`tinyproxy`) wasn’t enabled on boot — both silent failures that look like model bugs but aren’t.
None of this is a knock on building your own tooling — it’s just that on brand-new hardware (GB10 / sm_121a), **the gap between “almost right” and “right” is a frozen machine.**
## What worked — Techno Tim’s recipe, verbatim
The fix was to stop improvising and run **Techno Tim’s recipe exactly as published**, changing only the model source (to a local copy on my NAS, to skip a 122 GB re-download). The pieces that mattered:
- **His image:** `ghcr.io/timothystewart6/vllm-gb10:latest` — a vLLM build pinned and compiled natively for GB10 / sm_121a, tracking upstream. (Built on eugr’s original `spark-vllm-docker` project, which deserves the foundational credit — that’s what made the two-node path possible in the first place.)
- **Ray backend + a `sleep 60` in the head command.** The head starts Ray, waits 60 seconds for the worker to join, *then* launches vLLM. Skip the wait and vLLM can silently fall back to single-GPU mode and OOM.
- **`NCCL_NET_GDR_LEVEL=0`** (with `NCCL_IB_DISABLE=0` so CPU-side RoCE stays on). This is the anti-wedge flag. The whole NCCL block — pinning the HCA, GID index 3, Ring algorithm — matters, but this is the one that stops the SoC hard-lock.
- **`—gpu-memory-utilization 0.75`** (the *floor* for a 61 GB/node model, not an aggressive setting), `—load-format fastsafetensors`, `—kv-cache-dtype fp8_e4m3`.
- **`privileged: true`, `ipc: host`, `—ulimit memlock=-1`, `/dev/infiniband`** — privileged is required on GB10 to hit full throughput.
One extra gotcha worth flagging for anyone cloning the repo: the compose file’s `command:` is a folded YAML block scalar, and on my system the extra-indented flags got parsed with **newlines instead of spaces**, so bash saw each flag as its own command and the container crash-looped. Rewriting `command:` as a single line fixed it instantly.
## The result
Clean launch. Ray cluster formed, the worker joined the TP=2 placement group (real cross-node tensor parallelism, not a single-node fallback), weights loaded over `fastsafetensors`, the endpoint came up, and memory stayed healthy — **no wedge.**
Then the test that actually matters for an agent brain — structured tool calls:
```
PLAIN CHAT -> “OK”
call 1: OK args={”city”: “Paris”}
call 2: OK args={”city”: “Paris”}
call 3: OK args={”city”: “Paris”}
call 4: OK args={”city”: “Paris”}
call 5: OK args={”city”: “Paris”}
=== TOOL-CALL SCORE: 5/5 ===
```
MiniMax-M2.7 serving across two DGX Sparks, tensor-parallel, with a perfect tool-call score. The thing that had frozen both machines repeatedly came up clean once we ran the recipe as written.
## The lessons, distilled
1. **On GB10, “almost right” is a frozen machine.** Use the known-good image and recipe before you tune.
2. **`NCCL_NET_GDR_LEVEL=0` is mandatory** on the unified-memory SoC — GPUDirect RDMA will hard-lock it.
3. **Don’t chase the memory fraction.** A wedge at *lower* utilization than a known-good config means your problem is elsewhere. (And for a big model, too-low utilization fails for lack of KV room.)
4. **`—no-ray` vs Ray, the `sleep 60` worker-join wait, single-line commands** — small things, each capable of sinking the run.
5. **Verify reboot persistence** (NAS auto-mount, proxy enabled) before blaming the model.
## Credit
- **eugr** — `spark-vllm-docker`, the original community project that made dual-Spark vLLM real.
- **Techno Tim** — the published, working `vllm-gb10` image and two-node MiniMax recipe. This worked because he did the hard part and wrote it down. The right move, when you’re on bleeding-edge hardware, is to run the proven recipe verbatim — and *then* experiment.
*— Leathery Tendons*