How We Built and Run Our Own Docs LLM on Flux

Fluxers! Until recently, the little assistant in the corner of the Flux documentation site was Kapa — a SaaS product. You asked it a question about Flux, it sent that question to somebody else’s servers, somebody else’s model answered, and we paid a subscription for the privilege.

It has been replaced. The assistant answering questions on docs.runonflux.com, on this site and inside FluxOS is now ours: our corpus, our retrieval, our model, our router — running as an ordinary Flux application, on Flux CPU nodes, on the same network we ask everybody else to deploy to. The repository is called ownllm and it is open.

This is the long version of how it was built and how it runs. If you want to run a retrieval-augmented assistant on Flux yourself, most of this is directly reusable.

Why do this at all

There is the obvious reason and the better one.

The obvious one is consistency. A project whose whole argument is that you do not need to rent someone else’s cloud looks a bit silly renting someone else’s cloud to answer questions about itself. If decentralized infrastructure is good enough to recommend, it is good enough to run our own documentation assistant on.

The better reason is that it makes us our own customer. Every constraint a developer hits deploying an application on Flux, we hit. Every rough edge in the deployment path, the logs, the routing, the resource limits — we found by walking into it. A number of the FluxOS and Flux Domain Manager improvements that shipped over the last month exist because this project needed them.

And it means no user question about Flux leaves the Flux network.

The corpus is the product

This is the part people underestimate. An assistant that answers documentation questions is only as good as what it is allowed to read, so most of the work went into the corpus rather than the model.

The corpus is built by a script that surveys the real sources — the documentation site, the articles on this blog, the deploy interface, and the public repositories — and ingests them into one citable body of text. Along the way it does a number of unglamorous things that turn out to matter enormously:

• Retrieval is weighted by corpus tier. Official documentation outranks a passing mention in a blog post. Without this, a casual aside can outrank the reference page it is casually mentioning.
• The pinned fact sheet is generated from FluxOS source, not hand-written. A hand-written fact sheet is accurate on the day you write it and slowly becomes a liability. One generated from the code cannot drift.
• Answers cite GitHub URLs where the source is code, so you can go and read the thing the answer came from.
• Credential shapes are redacted at ingest. Anything that pattern-matches a key never enters the corpus in the first place.
• Internal documents are refused by default when reading local repositories, and AI University course content is excluded.
• PDF text extraction is repaired before it reaches the corpus. Raw pdftotext output mangles tables, and a mangled table is worse than no table — it looks like data and is not.
• Boilerplate is deduplicated, so the navigation furniture repeated on every documentation page does not drown the actual content.

The current corpus is 26,879 chunks, published openly and tracked in Git LFS alongside the vectors.

Teaching it to say “I do not know”

The failure mode that destroys trust in these assistants is not being unable to answer. It is answering confidently from nothing.

So the bot is built to refuse. If the documentation does not say, it says the documentation does not say. We also score the tail of its answers specifically, because a subtler version of the same problem was getting through: unmarked reasoning presented in the same voice as a sourced fact. The model thinking out loud and the model quoting a document should not look identical to a reader.

Our evaluation is nine questions drawn from our own documentation, run under two prompts — a strict one that tells the model never to guess, and a softer one that asks it to quote the relevant figure. Nine questions sounds small. It has been worth more to us than any public benchmark, because every one of them is a question a real person actually asks.

Making CPU inference feel fast

Here is the interesting engineering constraint. We run on CPU nodes, because that is what most of the Flux fleet is. So the first thing we did was measure where the time actually goes on a Stratus node:

• Generation: roughly 14 tokens per second, and bound by memory bandwidth — we measured the ceiling at about 16.6 GB/s.
• Prefill — the model reading your question plus the retrieved documentation — 153 to 205 tokens per second at 8 threads.
• 89% of the latency on a typical 2,118-token prompt is prefill.

That single measurement dictates the whole design. If nearly nine tenths of the wait is the model reading, then the lever is not a faster model — it is sending the model less, and sending it to the right machine.

What we did with that:

• Embeddings are precomputed and baked into the image. Originally a fresh instance indexed the corpus at boot, which took hours. Now the vectors ship with the container and a rig boots in about a minute.
• The prompt is trimmed. Pinned documents plus KV prefix reuse cut a measured third off it.
• A semantic cache answers a question somebody has already asked, immediately.
• Answers stream, and the wait is shown honestly rather than hidden behind a spinner that tells you nothing.
• Retrieval returns the top four chunks, which is what fits the context budget alongside the pinned facts.

The router

A decentralized fleet is made of independently operated machines, and their hardware genuinely varies. Measured across three production instances on the same evening with the same image, the fastest generated tokens roughly eleven times faster than the slowest.

Round-robin across that gives you the average machine. Measuring it gives you the best one. So the router — itself deployed as its own Flux application — routes on measured speed, keeps a session pinned to the instance holding its context, and falls back to discovery by node IP if the API is unreachable.

This is the design lesson we would pass on to anybody building on a decentralized fleet: measure your instances continuously and route on the measurement. Hardware diversity becomes an advantage the moment you stop pretending it does not exist.

Keeping long requests alive

A genuinely fiddly problem, and worth writing down because it will bite anyone running inference behind a proxy.

Ollama sends no response headers until prefill finishes. On CPU, with a few thousand tokens of context, that can be minutes of complete silence on the socket — and an intermediary reasonably concludes the upstream has died.

The fixes, in order of discovery: the gate and hub send keepalives from the moment a request is accepted, streaming or not. The hub heartbeats during prefill so the Flux Domain Manager holds the connection open. And we moved the gate and hub proxy onto node:http rather than fetch, because fetch abandons a response whose headers take more than 300 seconds — which a long CPU prefill legitimately does. Total timeouts sit at 30 minutes.

None of that is glamorous. All of it is the difference between an assistant that answers hard questions and one that appears to hang.

Surviving its own crashes

Some of the most valuable work was making the thing robust, and the pattern that emerged is worth stating generally: a component that dies must not take the index with it.

Each llama-server now runs under its own supervisor inside the shim, which respawns it on exit and kills-and-respawns it after 60 seconds of failed health checks. The docs bot returns await rather than a bare promise, so an upstream rejection is caught by the try/catch that was supposed to catch it rather than escaping and taking the process down. The shim aborts upstream generation when the caller disconnects. Retrieval drains streams rather than cancelling them. And the index is cleared on restart rather than appended to — an early bug had one instance make 48 passes over the same first 672 chunks.

A related one with a satisfying root cause: the embedding model has a 512-token context, and one chunk — a marketplace README table — was 823 tokens even after being cut to 2,000 characters, because table text tokenizes at about 2.4 characters per token. llama-server refuses an over-long input; ollama silently truncates it, which is why the problem never showed up under the previous engine. The indexer now halves a rejected chunk until the embedder accepts it.

The widget

The front end is a single script tag, served with no key required and with answers cached. It carries a compact logo launcher, per-site theming so it matches the page it sits on, a window.ownllm API with a setTheme call for sites that switch theme at runtime, and a standalone /chat page for anyone who wants the whole window. Origins are checked against an allow-list in both the router and the bot.

It loads on first click rather than on page load, so it costs a visitor nothing until they want it.

What is next

Two things, both underway.

The first is a fine-tune: one small model with two skills — the documentation bot, and a deploy agent for the Flux Cloud chat row that can size a deployment, quote it, deploy it once you agree, and then manage it. The failures worth fixing there are conventions rather than intelligence — GB versus MB, quote before deploy, never invent a key, ask rather than guess — and that is exactly what supervised fine-tuning is for. A model with the conventions baked in also needs a few hundred tokens of prompt instead of three to seven thousand of tool schemas, which on CPU is the difference between seconds and minutes to the first token.

The training data is generated to be correct by construction: every tool call in the deploy dialogues comes from code over a sampled scenario, and the teacher model only paraphrases the user’s side, with every number and name required to survive the paraphrase.

The second is an extract-first pipeline: a very small reader copies the relevant span out of the retrieved documentation verbatim, and the generator only rephrases what it has been handed. It is an elegant design because the model never has to read a number correctly — it is given the sentence.

Go and use it

The assistant is live on docs.runonflux.com, on this site, and in FluxOS. Ask it something specific — resource limits, what a spec field does, how to publish a second port — because specific is what it is good at.

And if you want the API rather than the widget, every model behind it is available through the Flux LLM hub at llm.runonflux.com, which is its own article.