⚙Leathery Tendons Would you believe… a sovereign AI stack, built in the open — Cecil Ray Burnett III

vLLM is a Beast on Dual Sparks - Full 355B GLM-4.7 Proves It

How a CPA and a panel of AI agents debugged vLLM down to a one-line patch — and got a frontier-class open model serving at 64K context on desktop hardware.

July 6, 2026

Most people running GLM-4.7 on DGX Spark hardware run Flash — the small one. The full model is 355 billion parameters, and even quantized to NVFP4 it wants ~188 GB of weights. Conventional wisdom says that’s datacenter territory.

It isn’t. As of this week, the complete 355B GLM-4.7 is serving on the two DGX Spark GB10s on my desk — 64K context, CUDA graphs on, 17.5 tokens/sec, working tool calls, and a passed needle-in-a-haystack recall test at 21,000 tokens deep. This post is the recipe: the four walls you’ll hit, in order, and the exact fix for each. If you own two Sparks and want a frontier-class open model running on your own hardware, this should save you a day of debugging.

(Short version for the impatient: the community k/v-scale mod plus a one-line *args patch, --no-ray, batch limits, and gpu-memory-utilization 0.90. Full details below.)

Why bother

I’m a CPA. Client confidentiality isn’t a preference, it’s an obligation — which means the AI I use for real work has to run on hardware I control. That’s the whole thesis of the sovereign-stack project this blog documents: local models, local storage, no client data leaving the building. The question tonight answered is whether “local” can mean genuinely big — a 355B model, not a distilled compromise.

The setup

Wall 1: Ray fights itself

First launch died with ActorHandleNotFoundError: ActorHandle objects are not valid across Ray sessions. Not a stale-state problem — it reproduces on clean resets. A single launch spawns two Ray sessions (the launcher’s cluster and vLLM’s own ray-executor session), and they can’t see each other’s actors.

Fix: --no-ray. vLLM’s native multi-node executor doesn’t have the problem.

Wall 2: one argument too many

Past Ray, the load died deeper, in the weight loader:

TypeError: KVCacheScaleParameter.weight_loader() takes 2 positional arguments but 3 were given

The community mod that fixes this model’s k/v scales (from the model’s HF discussion) was written against a slightly different vLLM source. On this build, the model code passes a third shard_id argument to a staticmethod that accepts two. The loader in question handles a scalar scale — per-head scales route elsewhere, per its own docstring — so the extra argument can be safely ignored:

@staticmethod
def weight_loader(param: torch.nn.Parameter,
                  loaded_weight: torch.Tensor, *args) -> None:

One line. That’s the whole patch. Folded into the mod so it re-applies on every container start, and submitted upstream (PR linked at the bottom).

This was the wall that mattered: everything after it was tuning, not debugging.

Wall 3: the memory squeeze

The model now loaded — all 94 GiB per node — then the KV-cache profiler delivered bad news: 0.16 GiB available where 32K context needs 2.81. Weights plus CUDA-graph capture plus the profiling forward pass had eaten essentially the whole budget.

The reclaim ladder, in the order that matters:

  1. Batch limits — --max-num-batched-tokens 8192 --max-num-seqs 8. The profiler sizes activation memory off these, and a single-user box doesn’t need datacenter batch sizes. This freed ~9 GiB. Biggest lever by far.

  2. --gpu-memory-utilization 0.90 (up from 0.88). Gentle — it’s unified memory, the OS lives there too. This was the difference between CUDA graphs almost fitting at 64K (missed by 0.06 GiB — that one stung) and fitting with ~3 GiB to spare.

  3. --enforce-eager as the fallback if your unit is tighter — it works, at 12.4 tok/s instead of 17.5.

Final numbers at 64K: 8.89 GiB of KV available against 5.75 needed, CUDA graphs on, 17.5 tok/s. The headroom suggests 128K is reachable.

Wall 4: the model that thinks before it speaks

First completion came back empty and looked broken. It wasn’t — GLM-4.7 is a reasoning model. It thinks before it answers, and a stingy max_tokens cuts it off mid-thought. Give it 512+ tokens and it reasons, answers, and finishes clean. Tool calls work out of the box with the glm47 parser.

Did it actually work?

A green healthcheck is not a working model, so: real completions (coherent), a tool-call test (1/1, clean finish_reason: tool_calls), and a needle-in-a-haystack test — a code planted at the top of a 21,048-token prompt, recalled exactly. That’s the bar I care about: long-context recall on my own hardware.

Two honest caveats for anyone reproducing this. Loads take ~9.5 minutes and they’re compute-bound (NVFP4 weight processing), not disk-bound — copying shards to local NVMe buys independence from your NAS, not speed. And the load logs show a KV cache scaling factor 1.0 for fp8_e4m3 warning; output was correct in all my tests, but verify for accuracy-critical use.

vLLM is a beast

I mean that as a compliment and a complaint, in the same breath. This is one open-source engine that will split a 355-billion-parameter model across two boxes on my desk, fuse 4-bit weights straight into the attention kernels, run an fp8 KV cache, and parse tool calls — and it will also refuse to start over a two-versus-three argument mismatch that one line of sed fixes. Miraculous and maddening, often in the same hour.

Here’s what I came away respecting, though: vLLM fails legibly. Every one of my four walls handed me an exact error naming the exact file. That’s the whole reason a CPA and a panel of AI models could take it apart — the beast tells you where it hurts. The failure that actually scared me was the opposite: the GB10 hard-lock — no traceback at all, just a frozen node. It’s the reason NCCL_NET_GDR_LEVEL=0 sits at the top of my setup; I learned that one the hard way. An engine that fails loudly is a gift — you can follow it home.

And the churn cuts both ways. The version drift that bit me — a fix written against a slightly older vLLM — is the same breakneck pace that handed me the 4-bit-on-Blackwell path I rode, which is only months old. You don’t get the miracle without the mess.

The part I didn’t expect to write

The debugging panel for this was AI, working in layers: a coding agent driving the terminal, a second frontier model doing adversarial review of the first one’s diagnosis, and me making the calls between bounded experiments. The arity fix — the wall that mattered — came out of exactly that structure: one model diagnosed the signature drift from a single traceback line, the other verified it against the actual source and corrected the patch before it shipped. Neither alone got it right; the disagreement did.

That workflow — panel of models, bounded experiments, human owns the decisions — is most of what this blog is about, and it earned its keep tonight.

Credits: eugr for spark-vllm-docker and the tf5 image; Salyut1 for the NVFP4 quant and the original k/v-scale fix.

Cecil Ray Burnett III writes as Leathery Tendons. He is a CPA building a sovereign AI stack — local models, local storage, client data that never leaves the building.

© 2026 Leathery Tendons LLC