⚙Leathery Tendons Would you believe… a sovereign AI stack, built in the open — Cecil Ray Burnett III

eugr's gold - Dual Spark Model

Thanks eugr

June 28, 2026

eugr’s gold Leathery Tendons

Running a 397-Billion-Parameter Model on Two DGX Sparks

How a small team of humans and AI agents got Intel/Qwen3.5-397B-A17B-int4-AutoRound to load and serve across two NVIDIA DGX Spark (GB10) boxes — and the one mistake that cost us two attempts.

The goal

Two DGX Sparks. 128 GB of unified LPDDR5x each, joined by a direct 200 Gb QSFP56 link (ConnectX-7, RoCE). The question that drove this: what is the biggest open-weight model two Sparks can actually serve together — not in theory, not in a vendor slide, but a live OpenAI-compatible endpoint you can curl?

The answer turned out to be a 397-billion-parameter model. Here’s how.

Why two Sparks is hard (and why MoE saves you)

A Spark’s memory is fast locally (~273 GB/s) but the cross-box link is ~10× slower. Split a dense model across both boxes with tensor parallelism and every layer’s activations cross that slow link — throughput collapses, and naive launches deadlock on NCCL. We learned this the hard way: an earlier dense-style attempt wedged a Spark entirely.

The models that do work are low-active-parameter Mixture-of-Experts. Qwen3.5-397B-A17B is 397B total but only ~17B active per token, so very little crosses the interconnect each step. That’s the architectural key: total size fills the combined 256 GB; active size keeps the link quiet.

The real hero: eugr/spark-vllm-docker

NVIDIA publishes the plumbing to connect two Sparks. eugr built the serving layer on top — an MIT-licensed, nightly-tested Docker harness that encodes all the hard-won configuration: the cross-node mounts, NCCL pinning, the —no-ray path that’s required to fit full context, and the per-node memory split. Every operator we found who succeeded used it; everyone who hand-built docker run commands deadlocked or hit path errors.

We did too. Our first two attempts failed precisely because we tried to be clever.

The mistake that cost us two attempts: GPTQ ≠ AutoRound

We had a local 4-bit copy of the 397B model on our NAS — but it was a GPTQ quantization. eugr’s recipe targets a specific quantization: Intel/Qwen3.5-397B-A17B-int4-AutoRound. We assumed “4-bit int4 is 4-bit int4” and pointed the recipe at our local GPTQ folder.

It isn’t the same. AutoRound and GPTQ produce different tensor layouts and use different kernels; eugr’s recipe even relies on —load-format instanttensor, a fast-load format tied to the AutoRound packaging that GPTQ can’t use. The first attempt OOM’d at init (we’d also left the context too large); the second died with an “invalid Hugging Face repo ID” error because a hand-edited recipe couldn’t resolve the local path.

The fix was humbling in its simplicity: download the exact model eugr names, run the exact recipe unmodified.

The run that worked

cd /home/ray/spark-vllm-docker          # eugr repo, pinned commit cf0d5f6

./run-recipe.sh qwen3.5-397b-int4-autoround.yaml --download-only   # fetch the AutoRound model — zero downtime

./run-recipe.sh qwen3.5-397b-int4-autoround.yaml --no-ray          # serve across both Sparks

Preflight that matters at this size:

• —no-ray — required to fit full context (and a touch faster).

• Swap on each Spark’s NVMe — the recipe leans on it as a pressure valve at the ~108 GB/node ceiling.

• Runlevel 3, no GUI — every gigabyte counts; don’t let a desktop session eat headroom.

• Lock the GPU clock (sudo nvidia-smi -lgc 200,2150) — current firmware can hard-shut a Spark under sustained load. This one’s a real fire risk; don’t skip it.

The result

• Model: Intel/Qwen3.5-397B-A17B-int4-AutoRound

• Both Sparks, tensor-parallel 2, —no-ray

• /v1/models served; a chat completion returned HTTP 200

• Reported context: 262,144 tokens

• ~98 GiB model memory per node; 284,939-token KV cache

• vLLM 0.23.1rc1 (GB10 build)

A 397-billion-parameter model, sovereign, on two desktop boxes. Not many people can say that.

Honest caveats

• This is eugr’s experimental recipe — it’s labeled as such, and it’s cliff-edge on memory. Expect ~11 tok/s territory; this is a capability demonstration, not a snappy daily driver.

• It runs at the memory ceiling. Leave no other GPU work on the boxes.

• The interconnect, not the GPUs, is your real ceiling. The next step up isn’t a faster card — it’s a third Spark in pipeline-parallel.

Credit

eugr did the genuinely hard part: turning “two Sparks can talk” into “two Sparks can serve.” If you’re attempting this, start there, pin a known-good commit, read the scripts, and use the exact model the recipe names. That last sentence would have saved us two attempts.

Written up by the build team — humans and a few AI agents working in parallel. Corrections welcome.