Can a CPU-Native Model Beat Ours? We Measured It on Flux

Fluxers! This one is for the engineers. We spent a fortnight measuring CPU-native ternary language models — weights that are only -1, 0 or +1 — on real Flux nodes, to find out how fast inference on this network can actually go. We came out of it with a model running at more than three times our previous generation speed, a better model for the docs assistant than the one we were running, and two fixes to file upstream.

Everything below was measured on real Flux nodes. Where a number is not ours, it is linked to where it came from.

Why ternary looked promising

Our docs assistant runs on CPU, on Flux, because that is what the network is made of. So we started by measuring where the time actually goes on a Stratus node running granite4:tiny-h at 4-bit quantisation:

• Generation: about 14 tokens per second.
• Prefill: 153 to 205 tokens per second at 8 threads — and 12 threads was 19% slower.
• Share of latency that is prefill on a 2,118-token docs-bot prompt: 89%.
• Measured memory bandwidth ceiling: about 16.6 GB/s.

Two conclusions fall straight out. Generation is memory-bandwidth-bound, so smaller weights per token is the only lever. Prefill is compute-bound and dominates, so shrinking the prompt and speeding up the matrix multiply are the levers there.

Ternary models attack both. Weights of {-1, 0, +1} store at 2 bits, so a token reads eight times fewer bytes than fp16 and about 2.3 times fewer than 4-bit; and the matrix multiply becomes additions, which is exactly what bitnet.cpp’s kernel exploits.

One thing we wrote down before starting, so the result could not flatter itself: our production model is a mixture-of-experts with about 1B active parameters, so per generated token it already reads about as many bytes as a 2.4B dense ternary model does. The bandwidth argument alone did not predict a win. Prefill was where a win would come from.

Why we measure on many nodes, not one

Before any of that, a result that changed how we benchmark on this network — and that turned into a product advantage. The same image, the same specification, three production instances, the same harness, the same evening:

• Node A: 1.3 generated tokens/s, 62 prefill, 64-second time to first token on a 3k prompt.
• Node B: 11.2 / 107 / 37 s.
• Node C: 14.8 / 137 / 29 s.

A wide spread in generation speed between nodes. A fleet of independently operated machines has genuinely varied hardware, which means any single-node benchmark tells you very little and any model comparison has to name the node it ran on. It also points straight at the design that makes the fleet an asset: our router selects on measured latency, so requests go to the fastest instance available at that moment. Measure the spread and route on it, and a diverse fleet gives you the best machine in it rather than the average one.

Round one: very fast, and a puzzle

We deployed Microsoft’s BitNet-b1.58-2B-4T on Flux using bitnet.cpp’s own I2_S kernels, behind a shim that speaks the same API as our production engine — so the gate, the docs bot and both benchmark tools ran against it completely unchanged.

On speed it delivered: 23 generated tokens/s and 204-220 prefill, roughly double the median production node.

On grounded answering it scored 2 out of 9 against the production model’s 7 out of 9, in a way that did not fit the speed at all. Given a single paragraph one sentence away that said a Nimbus node has 7 cores and 28000 MB RAM, and asked how much RAM, it answered: “70000 MB RAM and 400 GB.” It had merged two numbers that were sitting in front of it.

For a bot whose entire job is copying figures out of documentation, that needed explaining before the model could go anywhere near production. We checked the obvious confounds — chat template, repeat penalty, where in the conversation the context sat, context length — and none of them explained it.

One confound we could not close: we had built the I2_S kernel with native CPU optimisations turned off, so that a build made on a CI runner could not crash on an older node. Microsoft publishes no numbers for that build. A subtly wrong kernel looks exactly like a weak model.

Round two: the kernel was the story

So round two became a correctness check first and a speed test second. We took the same weights, converted them to TQ2_0 with upstream llama.cpp’s converter, and served them with upstream llama-server — an independent kernel, same model, same prompts, same shim.

Getting there needed two one-line patches, which are worth recording because they will bite anyone else who tries this. Upstream’s converter tensor map does not know 2B-4T’s names, and its BitNet graph hardcodes SiLU where 2B-4T uses squared ReLU. Without the second patch, the output would have been nonsense and would have looked, once again, exactly like a kernel failure.

Same question, same paragraph, the two kernels side by side:

• I2_S (our non-native build): “70000 MB RAM and 400 GB.”
• TQ2_0 (upstream): “An application on a NIMBUS node may use up to 28000 MB RAM.”

And asked for the capital of France, I2_S said “Paris. The capital of France is Paris. The capital of…” while TQ2_0 said “Paris.” TQ2_0 was also faster: 43 generated tokens/s and 267 prefill, against 23 and 210.

So the kernel was the answer. Our own non-native build had been producing degraded output the whole time, and every quality conclusion from round one went with it. This is the case for running the experiment twice on independent implementations before believing either.

What the numbers say

With a kernel we trust, the paired scores — same evening, production model on its median node, weighted out of 9:

• granite4:tiny-h: 7/9 under a strict prompt, 7/9 under a soft one. Average answer 85-123 characters.
• BitNet-2B-4T as TQ2_0: 3/9 strict, 4/9 soft. Average answer 38-48 characters.

The production model is robust to how you phrase the instruction; the 2.4B ternary model is more sensitive to it, and errs toward caution — it will say “not covered in the documentation” rather than invent something. For a documentation assistant, erring toward caution is the right direction to err in.

So: our prefill hypothesis holds (1.9-2.1x against the median node) and our generation hypothesis holds (1.6-2.1x). On grounded answering a 2.4B model does not yet match the 7B-total mixture-of-experts we run in production, which tells us where the ceiling is and what to do about it.

And note what kind of gap it is: an ordinary capacity gap between a 2.4B dense model and a 7B-total MoE, arriving with twice the speed and a runtime that has not missed a beat. That is a finding about model size, not about ternary — and it is a strong argument for the approach at a slightly larger size, which is exactly where we are pointing it next.

Two fixes to contribute upstream

Running these models in anger on a large fleet surfaced two things worth giving back to the projects we build on:

• microsoft/BitNet: the x86 I2_S path, compiled without native flags, produces wrong-but-plausible output. Either fix it or document that native flags are mandatory.
• ggml-org/llama.cpp: the bitnet architecture needs three one-line changes to load Microsoft’s own release — tensor names, squared ReLU, and the BPE vocabulary. It was built for the 2024 reproductions, and nobody had pointed it at the official weights.

Two results that change what we ship

The same harness produced two more results, and both feed straight into production.

Thread count. A sweep on production nodes confirmed generation is flat from 6 threads upward and falls past 8 — bandwidth-bound, as predicted. Prefill peaks at 8 on a fast node and at 10 on a median one, worth +10% prefill for -7% generation. Not worth a fleet-wide change: 8 threads stays.

A model sweep. Twelve models through the identical harness on one 12-core Stratus node found better answers than the thing we are running. granite4:micro-h scores 7/9 strict and 9/9 soft at usable speed. granite4.2:3b scores 8/9 strict and 9/9 soft, though it reasons out loud and produces answers four times longer. Meanwhile several much larger and much slower models scored zero, because they think and never answer inside the budget.

Bigger is not automatically better on a CPU fleet, and the only way to know which model suits your job is to run your own evaluation on your own hardware. Ours is nine questions drawn from our own documentation, and it has been worth more to us than any public leaderboard.

Where this goes

The next step is not a bigger model, it is a smaller pipeline: have a very small reader copy the relevant span out of the retrieved documentation verbatim, and let the generator only rephrase what it has been handed. That design removes the failure mode entirely, because the model never has to read a number correctly — it is given the sentence.

Alongside that, we are fine-tuning one small model on two skills, the docs bot and the Flux Cloud deploy agent, which is a separate piece of work we will write up when it has numbers.

All of the research above is in the open, in the repository — the full log, every measurement and the tools that produced them, so anyone can re-run it on their own nodes.