The engine that caught up: two desk boxes, six weeks, and a version bump survived in a day

Six weeks ago I wrote about a million tokens on my desk — the community saga that took Z.AI’s GLM-5.3-Flash from “technically fits, nobody will use it” to one of the best local inference setups I have ever used. That story ended with a sequel I didn’t plan: the serving engine itself was rewritten — its native CUDA path, no Python adaptation layer — and needed to prove itself from scratch. This is that sequel’s payoff. The engine qualified, shipped, survived a version bump and a rebuilt patch stack, and now holds every quality gate it is graded by — all in six weeks of unattended work by the loop I’ll describe below. The machine is the same two lunchbox-sized boxes next to the printer; the numbers below are where it stands now.

Two small matte-black lunchbox-sized AI computers side by side on a wooden desk in warm lamp light, joined by one cable
The setup, as an illustration: two lunchbox-sized machines, one model, one 200-gigabit cable between them.

One paragraph for the newcomer

I run a 320-billion-parameter AI model — Z.AI’s GLM-5.3-Flash, an open-weights model approaching Claude Opus-class quality — entirely on two small desktop computers on a shelf, with no cloud, no subscription and no per-token bill. They’re linked by one cable and behave as one machine with a half-million-token working memory: it can hold roughly five hundred pages of text — a codebase, a legal archive, months of project history — in mind at once, and answer questions across all of it. It answers in about fourteen milliseconds per token (fast enough to read as it writes), it reads a book-length document in seconds, it looks at images, it calls tools the way an agent does, and four people can use it simultaneously. The whole thing is open source, from the weights to the engine to the benchmark that grades it.

The short version of part one

The original article ended with a tasting menu. The dish on the table now is the native one: the TensorFold engine’s own CUDA code serving the model directly — no Python in the serving path, every exactness gate it is graded by held.

The interesting part isn’t the numbers themselves but how they were earned — because what changed was not one big breakthrough. It was eighteen small, individually-measured improvements, each adopted only after it beat the status quo in a controlled head-to-head, each reversible with one environment variable, each with its receipt filed.

Diagram: one GLM-5.3-Flash instance served by two GB10 machines — the engine and API on the head, the second rank on the worker, tensor-parallel over two 200-gigabit rails
One model, two boxes: the engine and the API live on the head; the worker holds its half of every weight matrix. The two ConnectX-7 rails carry the traffic between the halves.

The craft: eighteen improvements, each with a court case

The native engine, TensorFold, is upstream software under fast development — it moved from v1.1 to v1.2.9 during this window alone. With each release the question was the same: what does the new version change on my hardware, and what do I still need to carry myself? The answer became a patch stack — eighteen modifications layered onto the engine, each one ported from the community’s forks or written for this machine, each one A/B tested before adoption.

The flavour of them: one packs the model’s dense layers to 4-bit at load time (measured: monotonically faster at 4, 6, 8, 16 bits — 4 won). One routes the mixture-of-experts layers in a single call per prompt pass instead of two. One reads each expert’s compressed weights exactly once per decode step instead of twice. One fuses the routing glue into single kernels. One keeps the model’s cache as flexible pages from a shared pool instead of fixed windows per stream — which is what lets four half-million-token conversations share the same box. None of them is glamorous. All of them are measured.

The discipline that made it work is worth spelling out, because it is the opposite of how performance tuning usually goes. Every change ran as a controlled experiment: the same prompts, the same seeds, the same hardware, the only variable being the change itself. A reply’s content was hashed so that any patch that altered the model’s output — as opposed to its speed — was caught immediately. Three patches that the forks measured as big wins on other hardware measured as washes here, and were adopted anyway only because they were bit-safe and neutral; others would have been dropped. One patch — a warm-resume optimisation that a fork claimed as a win — survived ten windows of increasingly careful instrumentation before the verdict came back “the optimisation never actually engages on this model”, and it was retired. Negative results are kept as carefully as positive ones.

And the quality score never moved: 95 out of 100 on the community’s hard-mode benchmark, thirty-five consecutive times, across every patch, every engine version, every rebase. The bench grades a hundred-plus scenarios of tool calling, parameter precision, multi-step chains, error recovery and prompt-injection defence. The point of repeating it after every single change is that a speed patch that costs you quality doesn’t get to hide.

The bump: version 1.2.9, survived in a day

Two weeks ago upstream shipped two releases back to back. Inside them were changes that mattered to this machine — a mainline of work we’d been carrying ourselves, a rewrite of the model’s prompt-expert kernels, new defaults. The catch: our eighteen patches were built against the old version, and the new version had rewritten some of the same code from a different direction. The rebase was a genuine merge of two independent solutions to the same problems.

The loop did it in a day. The new version’s code was combined with the stack patch by patch — nine needed their conflicts resolved by hand, the rest applied cleanly — and the result was verified three ways before anything touched production: the combined tree was byte-checked against the old, proven configuration; five of the engine’s own test suites passed; and every one of the 234 GPU kernels compiled clean. Then a full battery ran: the exactness gates (does the fast path produce the same words as the slow one — always), the hard-mode benchmark (95, the thirty-fifth consecutive), the vision suite, the timed load walls. Production moved to the new version the same evening and has served it since.

The numbers, now

All measured on the shipping configuration — two GB10 boxes, the v1.2.9 engine, the eighteen-patch stack, everything receipted:

Metric Result
Time to first token, cold 83 seconds to load the whole 176 GB model across both boxes
Steady-state decode 73 tokens/second single stream — faster than you can read
Prefill (reading speed) 1,528 tokens/second — a 200-page document in under three minutes
Working memory 524,288 tokens qualified end to end, verified at 8k/64k/192k/384k depths
Quality (hard-mode bench) 95/100, thirty-five consecutive receipts
Vision 4/4 suite — images are a standing default, not an add-on
Correctness gates Drafted = undrafted; resumed = fresh; concurrent = solo. All pass, every run

What those numbers feel like from the outside: you open your laptop, the assistant is already there — it loaded in under a minute and a half and stays resident. You throw it a hundred-page contract or a month of a project’s history and it has read all of it before your coffee is poured. Replies stream faster than you can read them. Four people can use it at once without noticing each other. And it never once asked for a credit card.

The loop: how the machine maintains itself

This is the part I still find slightly absurd, in the best way. The system maintains itself through a loop of four small programs, and no human touches the stack between releases. A launcher starts a test window: it stops production cleanly, runs the experiments against the test port, and — on every possible exit path, including crashes — brings production back. A watchdog restarts production if it is ever found down. A release watcher polls upstream every thirty minutes and wakes the agent the moment a new version appears. And a wake-up caller nudges the agent session when a campaign finishes, so the results get digested and the next experiment launched without anyone typing anything.

Over six weeks this loop ran dozens of qualification windows: every patch adoption, every version bump, every benchmark repeat. It voided its own windows eleven times when instruments were faulty — and treated that as the system working, because a window that catches its own bad measurement is cheaper than a production change that doesn’t. Every result lands in a receipt file, every number in the documentation quotes one, and the whole audit trail is in the repository.

One structural choice makes the whole thing safe: the ports are keyed by intent. Production answers on 8000; anything testing answers on 8001, and the production service never touches the test port and vice versa. The two configurations even have different container names and separate logs. The discipline sounds bureaucratic until the day it saves you from benchmarking your own production endpoint by accident — which, in the project’s record, happened once and got caught by the guard, not by the data.

Where this goes next

The honest ledger: the walls that remain are the engine’s own — the model’s routed experts still dominate a decode step, and that is now upstream’s active work rather than something a home lab can out-engineer. The items parked on the bench are quality-of-life knobs from the community forks: a sharded cache for two-host serving, a couple of opt-in server behaviours. Each will get its own measured window when its value question is written, because that is how this project moves now — one brief, one experiment, one receipt at a time.

The deeper point I’d take from the last six weeks: the interesting frontier here isn’t the model — Z.AI ships those faster than I can re-base. It’s the operating discipline around one. The same two boxes that in August ran a demo now run an appliance that upgrades itself, grades itself, and refuses to ship a change it cannot prove safe. That transition didn’t cost a single new GPU.

The full receipts, the eighteen patches with their A/B records, the reproduction instructions and the benchmark cards are in the project repository. The hardware: two GB10 “desktop AI supercomputer” boxes — the same silicon as NVIDIA’s DGX Spark — joined by one 200-gigabit cable. The model: GLM-5.3-Flash under its MIT licence. The engine: TensorFold, Apache-2.0. Everything runs on-premises; nothing leaves the desk.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.