Flux AI: We Replaced Our Docs Assistant With One That Runs on Flux

Fluxers! The question-answering assistant on the Flux documentation site is no longer a third-party service. It is our own model, our own retrieval stack, and our own corpus — and the whole thing runs as an application on Flux, on CPU, on the same network you can deploy to.

You will find it as the small Flux mark in the corner of docs.runonflux.com, on runonflux.com, and inside FluxOS itself. Ask it something.

Why we replaced the one we had

We were paying a SaaS vendor to answer questions about our own documentation, on their infrastructure, with their model. For a project whose entire argument is that you do not need to rent someone else’s cloud, that was an awkward thing to have in the corner of the docs site.

So we built the replacement and deployed it where we tell everyone else to deploy. It is a Flux v8 application. It runs on CPU nodes. The whole thing is open source.

A corpus that cites its sources

An assistant that answers documentation questions is only as good as what it is allowed to read, so most of the work went there rather than into the model.

The corpus is built by a script that surveys the real sources — the documentation site, the articles on this blog, the deploy UI, and the public repositories — and ingests them into a single citable body of text. Along the way it does a number of unglamorous but necessary things:

• Retrieval is weighted by corpus tier, so official documentation outranks a passing mention.
• A generated fact sheet is built directly from FluxOS source, and it is the pinned one — not a hand-written sheet that drifts out of date.
• Answers cite GitHub URLs where the source is code.
• Anything that looks like a credential is redacted at ingest.
• Internal documents are refused by default when reading local repositories, and AI University course content is excluded.
• PDF text extraction is repaired before it reaches the corpus, because a mangled table is worse than no table.

The bot is also built to refuse. If the documentation does not say, it says it does not say. We scored the tail end of its answers specifically to catch unmarked reasoning passing itself off as an answer, which is the failure mode that makes these assistants untrustworthy.

Making CPU inference feel fast

Running a language model on CPU nodes rather than GPUs is the interesting engineering constraint, and it drove most of the design.

We measured where the time actually goes on a Stratus node. Generation runs at roughly 14 tokens per second and is bound by memory bandwidth. Prefill — the model reading your question plus the retrieved documentation — runs at 153 to 205 tokens per second and accounts for 89% of the latency on a typical 2,118-token docs-bot prompt. Node-to-node variance ranges from 35% to sixfold.

That measurement tells you exactly what to optimise. Send the model less, and route to the fast nodes:

• Embeddings are precomputed and shipped with the image, so a cold instance does not have to index the corpus before it can answer.
• The prompt is trimmed — pinned documents and KV prefix reuse cut a measured third off it.
• A semantic cache means a question someone already asked is answered immediately.
• Answers stream, and the wait is shown honestly rather than hidden behind a spinner.
• A load-aware router, deployed as its own Flux app, routes on measured speed rather than round-robin, and keeps a session pinned to the instance that has its context.

A good deal of plumbing went into long first turns, too. Ollama sends no response headers until prefill finishes, which on CPU can take minutes, so the gate and hub send keepalives from the moment a request is accepted and the hub heartbeats during prefill to keep the Flux Domain Manager holding the connection open.

One endpoint over many pools

Behind the widget is a hub that presents a single OpenAI-compatible endpoint over several pools of model instances, with API keys, pool profiles, sticky sessions, and token-bucket rate limits with a per-visitor cap on the public demo key. The mid-size and gpt-oss pools run at 32k context. The front page shows live status and lets you try it directly.

It is, in other words, an inference API — which is the Q4 2026 roadmap item — assembled out of ordinary Flux applications.

The widget

The widget itself is a single script tag, served with no key required, with answers cached. It carries a compact logo launcher, per-site theming so it matches the page it is embedded in, a window.ownllm API and a setTheme call for sites that switch themes at runtime, and a standalone /chat page for anyone who would rather have the whole window. Origins are checked against an allow-list in both the router and the bot.

What is next

Two things. The first is a fine-tune: one small model with two skills — the docs bot, and a deploy agent for the Flux Cloud chat row that can size, quote, deploy on your confirmation, and then manage what it deployed. The failures worth fixing there are conventions rather than intelligence (GB versus MB, quote before deploy, never invent a key, ask instead of guess), and those are exactly what supervised fine-tuning fixes. A model with the conventions baked in also needs a few hundred tokens of prompt instead of three to seven thousand of tool schemas, which on CPU is the difference between seconds and minutes to the first token.

The second is the engine underneath. We have been measuring CPU-native ternary models on Flux nodes to see whether they beat the quantised transformer we run today — there is a separate article on that, with numbers.

Go and ask it something. If it gets an answer wrong, that is a corpus bug, and corpus bugs are fixable.