The Flux LLM Hub: One Endpoint, Eight Models, a Hundred Instances

Fluxers! There is now a single OpenAI-compatible inference endpoint running entirely on the Flux network, and it is live at llm.runonflux.com.

One base URL. One API key. Eight models. Behind it, five pools of twenty instances each — a hundred model instances spread across independently operated Flux nodes — with health checking, latency-aware routing and session stickiness in front of them. Point any OpenAI client at it, change the base URL, and it works.

This article is how it is built, because the design is genuinely interesting and most of it is reusable for anything you want to run at scale on Flux.

What is on it today

The models currently served:

• granite4:tiny-h — the docs assistant’s production model, a 7B-total mixture-of-experts with about 1B active
• granite4.2:3b — scores highest of anything we have measured on grounded answering
• qwen3.5:2b and qwen3.5:0.8b — the small, fast end
• qwen3:8b
• gemma4:12b
• gpt-oss:20b — 21B total, 3.6B active, MXFP4
• bitnet-2b-4t — the CPU-native ternary model, served as TQ2_0 on upstream llama.cpp

The front page shows live pool status, so you can see exactly how many instances are healthy right now before you send anything. The mid-size and gpt-oss pools run at 32k context.

What a pool is

A pool is an ordinary Flux application running a gate-and-engine pair, deployed at twenty instances. Nothing exotic — it is the same kind of application anybody can register on the network.

The hub maps a model name to a pool, discovers that pool’s live instances from the Flux API, and sends each request to the best instance available. Clients see one base URL and one key; underneath, pools scale, migrate between nodes and get replaced without any client noticing. That indirection is the whole point — the thing a client depends on is the endpoint, not any particular machine.

Adding a model means deploying a pool and adding one entry to a configuration string. Retiring one is the reverse.

Stateless API keys

This is the design decision we are happiest with, and it comes straight out of a constraint the network imposes.

A Flux application runs as many independent instances with no shared database between them. So where do API keys live? The usual answer is a database, which means running one, which means it becomes the single point of failure that the twenty instances existed to avoid.

Instead, the keys carry their own proof. A key is sk-flux-<name>-<signature>, where the signature is an HMAC of the name under a master secret. Every instance can verify every key independently, with no lookup and no shared store at all. Revoke a single name through a revocation list; rotate everything by changing the secret.

It is a small idea with a big consequence: the authentication layer has no state, so it has nothing to fail, nothing to replicate, and nothing to keep consistent across a hundred instances on machines that have never heard of each other.

Sticky sessions, and why they matter more on CPU

The routing rule is: the instance this key used last, if it is healthy and idle; otherwise the least busy healthy instance, with measured latency breaking ties.

Sticky comes first, and the reason is the prompt cache. An agent harness re-sends the entire conversation on every tool call. The engine reuses its key-value cache only when the request lands on the instance that already saw that prefix. Same instance: a few hundred new tokens of prefill. Any other instance: the full twenty thousand again — which on CPU is minutes rather than seconds.

The exception is deliberate: a busy sticky instance is not waited for. If a session already has two requests in flight, it is paying for that instance’s attention, and the next request is better served elsewhere.

Busy-ness is measured as the higher of this hub instance’s own in-flight count and the gate’s reported count — because there are several hub instances, and each one only sees its own share of the traffic.

Discovery, with the network as its own fallback

The hub discovers pool instances from the Flux API every 60 seconds and health-checks them every 20.

There is a nice piece of defensive design here that only makes sense on a decentralized network. One hub instance landed on a node that could not reach the main API at all. But every Flux node answers the same query on port 16127, and every pool instance runs on a Flux node — so once a single lookup has succeeded, the pool IPs themselves become fallback API hosts. The network is its own directory service.

Instances that disappear from discovery are dropped, but only once they have no requests in flight. A discovery failure keeps the last known set rather than emptying it, because a temporarily unreachable API is not evidence that a hundred machines vanished.

Rate limiting without a shared counter

Per-key limits are token buckets: a sustained requests-per-minute rate, a burst allowance for requests that arrive together, and a cap on concurrent requests. Individual keys can have their own overrides.

The public demo key that the front page hands out gets an extra layer — a per-visitor limit on top of the per-key one, so a single script cannot consume the shared demo allowance for everybody else.

Limits and usage counters are per instance, and we say so plainly in the code: the admin endpoint shows one instance’s view, and the Flux Domain Manager spreads clients across instances, so treat those numbers as a sample rather than a ledger. That is the honest shape of counting in a system with no shared state, and pretending otherwise would be worse than saying it.

Small things that make it usable

A handful of details that only show up once real clients connect:

• Model names resolve loosely — exact match first, then case-insensitive, then treating a hyphen as a colon, because some OpenAI clients reject colons in model identifiers outright.
• Thinking is off by default for the small reasoning models, unless the client explicitly asks for it. A small reasoning model on CPU will otherwise spend its entire budget thinking and never produce an answer.
• Keepalives start the moment a request is accepted, streaming or not, because CPU prefill can be silent for a long time.
• Browser origins are checked against an allow-list.
• Both API shapes are forwarded — the OpenAI /v1 paths and ollama’s native /api paths — so existing tooling for either works unchanged.

And the whole server has no dependencies. Node’s built-in http and crypto modules, and global fetch. Nothing to audit, nothing to patch, nothing to pull at build time.

Try it

Open llm.runonflux.com. The front page shows live status, hands you a demo key and has a try-it box you can type into without configuring anything.

To use it from code, point any OpenAI-compatible client at the origin plus /v1, pass your key as a bearer token, and pick a model from /v1/models. Anything that speaks the OpenAI API — SDKs, agent frameworks, editor plugins — will work without modification.

Why this matters beyond our own bots

This is the inference API on the Flux roadmap, and it exists as an assembly of ordinary Flux applications rather than as special infrastructure. That is the part worth taking away.

A hundred model instances, health-checked, latency-routed, key-authenticated, running across independently operated machines around the world — built out of the same deployment primitives available to anybody with a Flux ID and some FLUX. The patterns in it, stateless keys, discovery with network fallback, latency-aware routing, sticky sessions for cache reuse, are the patterns for running anything serious at scale on a decentralized fleet.

It is all in the open in the ownllm repository. If you build something on top of it, we would like to hear about it.