There’s a particular pleasure in an AI endpoint that hums away on a shelf instead of invoicing you by the token. For the past two months I’ve been running GLM-5.3-Flash — Z.AI’s 320-billion-parameter mixture-of-experts model — locally on two MSI EdgeXperts, the lunchbox-sized GB10 machines that share their innards with NVIDIA’s DGX Spark. It has genuinely replaced a cloud endpoint: the full 1,048,576-token context window, four concurrent requests, images and video in, tool calling, structured outputs, and an OpenAI-compatible API on port 8888. Z.AI’s own model card describes GLM-5.3-Flash as “approaching Claude Opus 4.8 on coding and agentic benchmarks” at a tenth of the price; my copy of it sits on a shelf next to the printer, draws desk-lamp levels of power, and has never once asked me to accept new terms of service.

But the hardware is only half of this story, and frankly the less interesting half. What fascinates me is how the software caught up — a seven-week, 811-post group expedition on the NVIDIA developer forums that took the model from “technically fits, nobody will use it” to one of the best local inference setups I have ever used. It has a complete dramatic arc: excitement, a lull, a resurgence, a revelation, and a resolution. Here it is, with names attached, and with the engineering explained — because the engineering is the story.

The machine, and why two of them

Start with the silicon. The GB10 Grace Blackwell Superchip is a single package containing a 20-core Arm CPU — ten Cortex-X925 performance cores plus ten Cortex-A725 efficiency cores — fused to a Blackwell-architecture GPU (6,144 CUDA cores, fifth-generation Tensor Cores) over NVLink-C2C, NVIDIA’s chip-to-chip coherent interconnect. The point of NVLink-C2C is that CPU and GPU stop being two computers with a bus between them and become one pool of memory: 128 GB of LPDDR5x on a 256-bit interface, moving 273 GB/s, coherently shared. Add a ConnectX-7 network port rated at 200 gigabits, a 4TB self-encrypting Gen5 NVMe drive, and one petaFLOP of sparse FP4 AI compute, and you have the EdgeXpert MS-C931: 151×151×52 mm, 1.2 kg, a honeycomb intake grille for a personality. It is not a gaming PC and doesn’t pretend to be one; it’s a developer appliance whose entire reason to exist is AI.

The unified memory is the part that changes the economics of local inference. On a conventional PC you juggle system RAM and VRAM as two separate budgets with a PCIe tax between them; here the CPU and GPU draw from one coherent pool, so a 176 GB model checkpoint simply exists in memory the way a spreadsheet exists in RAM — no offloading choreography, no layer-by-layer streaming. One box, however, is not enough for this model once you want a million tokens of context on top of the weights. Two boxes, joined by that QSFP cable, give you 256 GB of coherent-per-node memory and a private 200-gigabit fabric between them.

And the QSFP link is more interesting than it looks. On the GB10, one QSFP port is internally wired as two separate PCIe Gen5 x4 links — the Linux system sees two network interfaces and two RDMA-capable (“RoCE”) devices per port. That means a single cable between two boxes actually provides two parallel rails for the traffic between the model’s two halves, and an all-gather — the collective operation that stitches the two halves’ work together — runs about 1.8× faster on both rails than on one. Details like this are scattered all over the recipe, and they’re the difference between a demo and an appliance.

The model, and why a million tokens is even on the menu

GLM-5.3-Flash is the first natively multimodal release in Z.AI’s GLM-5 series: 320 billion total parameters with just 18 billion active per token, trained on a 30-trillion-token multimodal corpus, released under an MIT licence, with a technical report titled — with some self-awareness — “GLM-5: from Vibe Coding to Agentic Engineering.” Z.AI’s own claim is that it approaches Claude Opus 4.8 on coding and agentic benchmarks at a tenth of the price. Two architectural decisions matter enormously for serving it at home.

First, it’s a mixture of experts. Instead of one dense 320-billion-parameter brain, the model contains a large toolbox of “expert” feed-forward networks of which only a small, learned subset fires for any given token — 18 billion parameters’ worth. That’s why a 4-bit build of the full model weighs 176 GB yet decodes at 60-plus tokens per second on hardware this size: the memory is mostly dormant experts, and only the active slice does work per token. It’s also, as we’ll see, why quantizing this particular model turned out to be uniquely tricky.

Second, it uses a hybrid attention design: some layers use sparse attention with a compressed latent key-value cache, others use linear attention with tiny recurrent states. The practical effect is that the cache for a million-token context stops being a fantasy. A naive transformer’s KV cache for a million tokens would eat hundreds of gigabytes; GLM-5.3-Flash’s compressed latent cache plus its pooled sparse-indexer keys plus a few recurrent states fit in a handful of gigabytes. Z.AI layered Manifold-Constrained Hyper-Connections (a technique for getting more out of each layer’s representations) on top, trained the whole thing on that 30-trillion-token corpus, and shipped a 1,048,576-token window as the model’s native maximum. On a datacentre GPU with HBM to spare, that’s a footnote. On two desk boxes with 256 GB between them, it’s the entire point — and it’s why the community’s obsession with KV-cache bytes was justified rather than neurotic.

There’s a third thing worth knowing, because it shaped the whole saga: the model is agentic by design. It’s built for long tool-calling workflows — the card’s own evaluation notes read like a DevOps runbook (Terminal-Bench in Claude Code, 400k-context SWE runs, six-hour timeouts). Its chat template has a thinking budget (reasoning_effort: low, high, max), and the default is max. A model tuned to think hard for hours about a repository is a different beast to serve than a chatbot, and several of the bugs the community found — we’ll get to the reasoning-decay one — only existed in that agentic regime.

Act I: excitement

When the model dropped on 26 August, the DGX Spark forum thread lit up within minutes. “Better than GLM-5.2,” announced the opener. Someone observed that the gap between frontier models and what you can run at home had collapsed from two years to under two months. The first community quantization — NVFP4, 195 GB, vision and multi-token prediction included, “first try” — appeared within the hour, and the quantizers never really stopped after that: RedHat, Intel, and the prolific local-inference-lab (“Luke”, who the thread anointed its gold-standard quantizer) all shipped versions in the first week. vLLM merged day-one support for the architecture; SGLang published a cookbook; even the tiny-model crowd got a 2.05-bit build that runs on a single Spark.

And the question everyone wanted answered was the one that mattered to me: the model was explicitly positioned for this class of hardware, but 320 billion parameters — even with 18 billion active — is a lot of weights to fit in 256 GB across two boxes alongside a million tokens of context. Could two Sparks, or two EdgeXperts, or any pair of GB10 machines, actually serve it? Day one said: yes, technically. It also said: at 23 tokens per second, with a cache that couldn’t hold four proper conversations, and a pile of new bugs. Enter the lull.

Act II: the lull

The first careful two-Spark benchmark, posted the day after release, is worth reading as a snapshot of the state of the art — because it’s also a map of every problem the next month would solve. The setup: vLLM with a native Blackwell (SM120) kernel extension, a Marlin backend for the routed experts, an FP8 KV cache, and the model’s own MTP head drafting five tokens ahead. The progression tells the story in three lines: the baseline extension managed 7.96 tok/s; the native kernels lifted it to 14.5; adding speculative decoding took it to 23.1 — with aggregate throughput across five concurrent requests reaching 60.7. The benchmark author’s conclusion was honest: “the earlier 20–25 tok/s estimate was accurate.” He also noted the cache arithmetic — roughly 849,000 logical token slots with the drafter enabled, about 4.24× a 200k context, and that the guaranteed limit was four full 200k sessions, not five — and filed a table of workarounds for bugs ranging from fluent gibberish (native MoE kernels corrupting output) to multimodal requests failing with “at most 0 images”. 86/100 on the agentic tool-calling benchmark, though. The model worked. It just wasn’t good yet.

What followed was a fortnight of the community cataloguing, one by one, every way this model could disappoint. A digest, with diagnoses:

• Quants that lied about their quality. The thread adopted a rigorous benchmark — tool-eval-bench, a 69-scenario suite (plus 28 adversarial “Hard Mode” ones) grading tool selection, parameter precision, multi-step chains, refusals, error recovery, localization, and prompt-injection defence, with reproducible seeds and configuration fingerprints so two scores are never compared unless they’re comparable. Against it, one rushed NVFP4 release — attention heads requantized to squeeze in more speed — scored a disastrous 81/100 and was described, memorably, as “braindead”. It was pulled, fixed, and re-released within a day, but not before a sharp argument about why it had been pinned in a popular recipe. The episode taught everyone to demand measured fidelity instead of vibes.
• Silent corruption. CUDA graphs combined with speculative decoding quietly wrecked output quality — the fix was to run eager. Native FP4 MoE kernels produced syntactically valid nonsense while isolated kernel tests passed. One checkpoint family leaked Mandarin and Russian into English replies. Each was hunted down with minimal reproductions and fixed upstream, but only after people burned evenings believing their hardware was haunted.
• An out-of-memory that took a week to explain. In every two-box setup, the head node died at ~88% of the weight load while the worker sailed through — drawing tens of gigabytes more memory for no obvious reason. The eventual diagnosis was beautiful: streaming the checkpoint from disk was polluting the Linux page cache, and the “missing” memory was cached weight bytes the system declined to reclaim. The fix was a pipelined instanttensor loader reading weights with direct I/O — which as a side effect cut load time to about 50 seconds for 82 GiB. The same investigation produced a warning now pinned in every recipe: never put Docker memory limits on a GB10’s unified memory — the machine wedges, hard.
• Smaller gremlins with real teeth. The stock chat template ignored the thinking toggle entirely (the model always thought, burning tokens). Long prompts stuttered to 0.08 tok/s at 40% of the window on some stacks. The best drafter, DFlash2, shipped under a non-commercial licence. A degenerate loop made the model emit “!!!” forever. And the strangest of them all: reasoning decay — past ~300,000 tokens of agentic tool use, the model silently stopped thinking and started emitting raw tool-call syntax. More on that in Act V.

For a fortnight the honest verdict was: brilliant model, unserveable hardware. Several people quietly went back to four-box clusters running other models at half precision, waiting for better quants.

Interlude: why quantizing this model was genuinely hard

It’s worth pausing on why, because it explains the entire next act — and because the community’s answer to it became the template for how this model is served today.

Quantization works by spreading error. A dense 70-billion-parameter model has 70 billion chances to distribute the small inaccuracies that 4-bit weights introduce, so no single token’s computation is wrong by much. A mixture-of-experts with 18 billion active parameters has far less redundancy: fewer weights contribute to each token, so each one matters more, and GLM-5.3-Flash’s novel hybrid attention means its caches and sparse indexer do things that generic quantization recipes were never calibrated for. The first-generation quants proved it: benchmark scores that looked fine while the underlying token distributions had drifted meaningfully from the reference.

The community’s answer was to measure fidelity the proper way. The gold standard became KL divergence — a measure of how far the quantized model’s next-token probability distribution drifts from Z.AI’s own FP8 release — computed on the exact tokens the serving stack sees, in float32, with block-bootstrap confidence intervals. Suddenly “it feels fine” had a number attached, and the number exposed two things: that the early quants left real accuracy on the floor, and that part of the error came not from the experts at all but from the dense layers, which the TensorFold engine quantizes itself at load time. There was even a measured floor — the best achievable error with FP8 experts and 4-bit dense weights — so everyone knew exactly how close each new build had come to the physical limit. When a later quant arrived claiming improvement, the thread could check its work. That discipline is the difference between a hype cycle and an engineering programme.

Act III: the performance resurgence

What happened next is why I love this community. Every single problem above got a fix, a workaround, or a patch — usually within days, and usually with the diagnosis written up like a lab notebook.

The quants grew a conscience. Mia’s AI Lab published an EXL3 4-bit build with a twist: instead of posting benchmark screenshots, they published the KL-divergence study itself. Their calibration cut the error by 11–27% over the previous best at identical size and speed, and where the FP8 reference is confident of its next token, their build disagrees 16–37% less often. On coding benchmarks both quants solved statistically the same share of problems — the measured gain was in fidelity, which is the honest thing to claim. Intel’s AutoRound W4A16 build turned out to be near-lossless — “billions of tokens, no failed tool calls” was one reviewer’s verdict — and needed only a small prefix-matching patch to unlock >92% cache hit rates. canada-quant shipped a W4A16 that leaves every accuracy-critical weight (attention, shared experts, dense layers, the MTP head) in BF16 with an Apache-licensed drafter of their own. The format war (NVFP4 versus EXL3 versus W4A16) ended not with a winner but with a menu, which is better.

The serving stacks got faster in leaps. The instanttensor loader fixed the out-of-memory and the load time in one stroke. vLLM merged day-one support for the architecture and then spent weeks fixing the corruption bugs the forum had isolated. Someone found that the GPU’s display reservation — memory a headless box can never use — could be mapped into the KV pool via a DRM dumb buffer: about 277,000 extra cache tokens per rank, free, with the caveat that the recipe checks no monitor is plugged in before it agrees. Someone else built a disk-backed KV spill tier that writes evicted conversation states to disk (with a CRC-32 every 8 MB, because a silently corrupted prompt is worse than a slow one) and resumes a 200,000-token conversation in 0.6 seconds instead of two minutes — then demonstrated a ten-million-token spill. The French-prompt bug — where 4-bit dense weights made replies never end — got a DENSE=fp8 switch and a measured explanation.

And the two-box software problem got its proper engine. The current speed king is Mia’s AI Lab’s TensorFold recipe: TensorFold — an Apache-licensed inference engine that began life as a single Zig binary serving Macs and grew a serious CUDA path — running its Python 0.6 line inside NVIDIA’s PyTorch container, with over a hundred community patches applied on top. One rank per box, weights sharded in whole blocks, a one-shot RoCE all-gather across the ConnectX-7 twin rails, and all conversations drawing from a shared FP8 KV pool of roughly 2.9 million tokens. That shared pool is the quiet genius of the design: one request can expand to the entire million-token window while three others keep their contexts resident, and when the pool runs dry, the least-recently-used conversation steps aside — spilling to disk and returning later in under a second. The engine’s exactness rules are worth quoting because they’re rare in this space: a drafted reply must equal the un-drafted one; a resumed request must equal a fresh one; a concurrent stream must equal its solo run. Benchmarks you can trust, by construction.

The clusters grew too. A three-box trick (bump the attention head count from 64 to 66 so it divides by three), four-box recipes at 90 tok/s decode with 3,500 tok/s prefill, and eventually an eight-Spark reference. A Mac Studio was bolted on via MCDMA and — to the thread’s lasting credit — documented as a reproducible integration result rather than the speed win everyone hoped for. By the time the dust settled, the pessimist’s 23 tok/s had become 60–63 tok/s single-stream and over 100 aggregate at four concurrent requests, with the context window at its native million and the pool holding 2.3–2.6 million further tokens.

Act IV: the reasoning revelation

Somewhere in the middle, the conversation shifted from how fast to how good — and this is the part I didn’t expect. The benchmarks started coming back absurd. 99% on GSM8K from a 4-bit quant — a maths score that beat the official DeepSeek release it was being compared against, on 250 problems, greedy decoding, no tricks. 96/100 on the hard-mode tool-calling suite, reproduced with zero variance across three trials. HumanEval within a single problem of the FP8 cloud endpoint — 160/164 local versus 159/164 hosted, on the strongest test-augmented variant of the benchmark. One contributor fed it 3.5 years of legacy .NET and had it port to Python overnight at a 99.7% trace match. Another quietly moved their production ERP patch workflow onto it, full trust, displacing a Claude subscription. The model the sceptics assumed would be lobotomised by 4-bit quantization turned out — with the right recipe — to reason like the frontier models it was undercutting on price by roughly 100%.

The community even turned the quality question into science. A Cyrillic long-context study swept 8k to 480k tokens across every quantization and found a precise failure signature — facts degrade first, code and needle retrieval stay intact, the onset depends on the quantization (FP8 holds to ~384k tokens, NVFP4 to ~192k, one build never degrades at all) — and, beautifully, that with thinking mode enabled the degradation vanishes entirely. “Quantization quality manifests itself precisely in thinking-off mode.” That’s the kind of finding you usually need a paper for; this one came with reproducible scripts and a benchmark repository.

It’s worth being precise about what the model is, per Z.AI’s own card: it approaches Claude Opus 4.8 on coding and agentic benchmarks. Not equals. The thread’s final quality rankings put the best two-box stacks within touching distance of the FP8 reference on coding — 91% versus 93% on LiveCodeBench v6, for instance — with the residual gap showing up in exactly the places quantization always bites: the most confident 1% of decisions, the deepest contexts, the longest agentic loops. One contributor’s blunt summary called the TensorFold EXL3 line’s reasoning “junior engineer grade”; another’s LiveCodeBench data called it “not dumbed down.” Both were right, for their workloads. What you get for the compromise is a model that runs on a shelf. Ask anyone who has paid per token for agentic workflows which side of that trade they’d rather be on.

Act V: the resolutions

Every bug that made Act II painful now has a documented resolution. The reasoning-decay bug — the model silently stops thinking after ~300k tokens of agentic tool use, collapsing into raw tool-call syntax — turned out to have a precise mechanism: turns whose conversation tail ends with a tool result rather than a user message were being served without the reasoning scaffold. It was diagnosed, reproduced deterministically, fixed by restoring the thinking scaffold after tool calls, and validated to 774k tokens across 100 rounds. The degenerate loops got a loop guard plus upstream fixes in the chat template and the sampling path. The out-of-memory got the pipelined loader, and the disk spill tier that makes long conversations instant to resume. The French-prompt end-of-turn bug got DENSE=fp8 and a measured explanation. The thinking-gate got fixed in every chat template. And the thread closed with a consolidated fourteen-configuration benchmark matrix — a map of the entire quality/speed trade-space, published for anyone to follow — plus a new deterministic agentic-coding benchmark so the next model doesn’t have to rely on vibes.

The final state of the art, as the thread wound down this week, reads like a menu rather than a winner:

Stack (2-box, tensor parallel)	Hard-mode quality	Prefill	Decode (c1)	KV pool	Long-context fidelity
Native NVFP4 stack (quality crown)	92 / 96 max	2,678 tok/s	42.7 tok/s	1.22M tokens	degrades past ~192k (thinking on: no)
TensorFold EXL3, DENSE=fp8 (speed & pool king — what I run)	91 / 93	1,601 tok/s	52.7 tok/s	2.29M tokens	degrades past ~384k (thinking on: no)
W4A16 purist build (the perfectionist’s choice)	91 / 92.3	1,510 tok/s	25.4 tok/s	1.34M tokens	zero degradation at any depth
DeepSeek-V4-Flash (reference cloud model)	89.7 / 89	1,746 tok/s	36.9 tok/s	3.62M tokens	—

There is no single “best”. There’s a menu, and that’s the point: two months after “nobody will use this”, you can pick your own point on the curve — maximum quality, maximum speed, or maximum fidelity — and all three fit on a shelf.

The cast

Threads like this one are usually remembered by their drama; this one deserves to be remembered by its roll call. A partial cast, and what each brought:

Who	What they brought	Why it mattered
Z.AI	The model: hybrid sparse/linear attention, a 1M window, MIT licence, day-one multi-framework support	Made the whole endeavour possible; the 1M window is why two desk boxes are interesting
ashhart	TensorFold — the engine the recipe runs on, with exactness guarantees (drafted replies must equal serial ones)	Two-box serving with correctness as a design goal, not an afterthought
Mia’s AI Lab	The EXL3 4bpw checkpoint, KL-divergence-calibrated; the TensorFold recipe; the disk spill tier	Turned “technically fits” into “measurably faithful”, then “fast”
SeraphimSerapis	tool-eval-bench: the 97-scenario hard-mode benchmark with configuration fingerprints	Replaced vibes with a ruler; every score in this story is comparable because of it
0rand	Day-one scepticism, then the best NVFP4 recipe, the reasoning-decay discovery, and a deterministic agentic benchmark	The thread’s conscience and its most prolific troubleshooter
ajvazan	The benchmark farm: fourteen consolidated configurations, the depth-degradation methodology	Turned anecdotes into data; the thread’s statistical backbone
technigma.ai	The quality-crown stack, the display-reservation and reasoning-decay fixes	Proof that the last 5 points of quality were findable
helge & protryt	The page-cache OOM diagnosis and the instanttensor loader	Fixed the single most common crash in every two-box setup
local-inference-lab, RedHat, Intel, canada-quant	The quantization ladder	Each rung raised quality, size efficiency, or licence freedom
wojo, florianbrede, BadAd84, lukaszraczylo, eduffd and more	The spill tier, the FP8 dense findings, long-context fixes, the Anthropic API backport, and roughly a hundred other patches	The long tail that turned a demo into infrastructure
giles8	Benchmarks of the TensorFold rebase in the thread’s final week — the numbers in the table above	That’s my run. Hi.

By the numbers

My two EdgeXperts run the TensorFold EXL3 recipe, with GPU clocks deliberately capped at 2,200 MHz — the quiet-and-cool configuration, not a benchmark-chasing one:

Metric	Result
Decode, 1 prose stream	61.8 tok/s (TTFT 162 ms)
Decode, 4 concurrent (aggregate)	105.5 tok/s
Decode, 4 concurrent (code)	128.3 tok/s
Prefill throughput	~2,000 tok/s up to 131k tokens
First token on a 131k prompt	~69 s
Tool calling (97-scenario hard mode)	91/100
Context / shared KV pool	1,048,576 tokens / ~2.9M tokens

A note on what those numbers feel like from the outside. A 131k-token prompt — a small book, or a fat stack of source code — takes about a minute to read before the first word of the answer. After that, replies stream faster than you can read them. Four people can use the box at once without noticing each other, and a conversation can sit for days and resume mid-thought in under a second. It is, in the most literal sense, a service — just one whose data centre is 1.2 kilograms and whose licence fees are zero.

Why fork it?

This is my fork of MiaAI-Lab’s original recipe — with the note that all of my commits since the fork were generated with AI assistance. It’s been a pleasant dogfooding loop: a model served from a repository it helped to write, benchmarked in the thread’s final week, and now writing blog posts about itself.

Full instructions, benchmarks and the configuration reference are in the README, and the whole saga — every dead end, drama, bug and benchmark — lives in the forum thread. The shopping list is short: two GB10 machines (DGX Sparks or EdgeXperts — the recipe doesn’t care whose logo is on the case; a QSFP cable even ships in the box on some SKUs), and about 205 GB of disk on each. What you get in return is a frontier-class assistant with a million-token memory that never phones home, and whose entire supply chain — weights, engine, patches, benchmarks, and this story — was built in the open by people arguing constructively on a forum. Long may it continue.