The swarm

The swarm is where things live: agent identities and accounts, secrets, the job graph that creates and places agents, telemetry, and the UI you drive it all from. Hives are the substrate — NixOS hosts that run agent containers on the swarm's behalf. Every hive belongs to a swarm; a single host is a swarm of one.

This page is for the operator. It covers the control plane, how a hive joins the directory, and how hives and agents report upward. The steps for a fresh swarm are in setup.md; the all-local config is the README quick start.

Option reference: services.hyperhive.swarm.* (swarm-wide facts, identical on every host) and services.hyperhive.deploy.* (whether this host runs grafana, the controller, authelia, …) → options reference, or nix build .#docs-swarm / .#docs-deploy.

Where each piece lives

Swarm controller

swarm-controller is the swarm's control plane: it holds the hive directory, the agent roster and their wanted state, and the job graph. Creating an agent — from the swarm UI or swarmctl agent create --hive <h> — queues its SSO identity, forge user, config repo, store identity and matrix account, then sends the hive a deploy message. A new agent starts paused. A store identity that fails to mint logs swarm jobq: mint_agent_identity failed with the agent and the error, which the Grafana alert rule hh-agent-identity-mint-failed fires on. The agent is still created, without a store identity.

services.hyperhive.deploy.swarm-controller.enable runs it on this host; singleHostSwarm turns it on. Otherwise off: set it on the one host that runs the controller. A swarm has one controller, so enabling it states a fact about swarm topology, not about whether this host runs a hive.

What it serves, why it's a unix socket rather than a port, and the socket-directory constraint that governs where socketPath may point: swarm-controller/README.md.

Per-hive status (GET /api/hives/status)

One row per hive in swarm.hives, saying when it last reported and what it said. Hives publish upward through the swarm queue; the controller never reaches down to collect, so a hive that can't reach the swarm still knows its own state — you just can't see it from here.

A hive publishes only once it holds all three status-publish coordinates below. A hive without them reads never_reported — it's not broken, it just has nothing to say upward.

freshness what to do about it
fresh nothing — reported within staleAfterSeconds
stale the hive stopped reporting. Its last payload is still shown, so check age_seconds and the payload for what it managed to say
never_reported this hive has never reported at all — normally a deployment that hasn't happened, not an outage
unknown something is publishing under a name that's not in swarm.hives — a typo in the roster, or a hive removed from the roster but still running

Every row also carries last_seen_unix and age_seconds if you want to apply your own threshold. The timestamp is the one the queue recorded on arrival, not one the hive put in its own payload. A roster hive's row carries dashboard, the host its dashboard answers on; the swarm UI's hive list links there.

Set services.hyperhive.swarm.controller.staleAfterSeconds (default 120) above the rate hives publish at, or everything reads stale between reports. Hives publish once a minute, so the default tolerates one missed report and flags two. It takes effect on the next request; nothing has to re-publish.

Making a hive report

Two options on the hive, plus its store identity. They sit in two namespaces, because the queue's URL is a fact about this machine and the token endpoint is the swarm's single address:

option what to set it to
deploy.hive-controller.statusPublish.natsUrl tls://<swarm.nats.domain>:<swarm.nats.port>
swarm.statusPublish.tokenEndpoint the swarm IdP's /api/oidc/token

Both default to the swarm's own addresses on every hive, so there is nothing to set for them. The client secret isn't an option: hive-c0re reads it from the swarm's secret store, at swarm/hives/<hiveName>/oidc/client, under the hive's own store leaf (deploy.bao.clientCertFile). swarm-secret-publish, in authelia's container, puts it there. A hive without a store leaf doesn't publish; setting natsUrl to null turns publishing off too.

Until the secret is in the store, hive-c0re shows a swarm_queue_config warning and tries again, waiting 5 seconds after the first failure and doubling up to 5 minutes. It starts publishing as soon as a read succeeds, with no restart.

The queue URL is the same string on every hive. The queue accepts TLS only, with a certificate for swarm.nats.domain (default nats.<swarm.domain>) and no other name or address, so a URL with an IP address or nats:// fails. The queue's host resolves the name itself. A multi-host swarm needs one upstream DNS record, nats.<swarm.domain> pointing at the queue host's mesh address, the same contract as bao.<swarm.domain>. The queue's port is open on wg-hive when that host is on the mesh.

The queue's host asks the secret store for that certificate under a store leaf of its own, deploy.nats.baoClientCertFile / baoClientKeyFile. On the store's host both default to the leaf glue-bao-tls.nix mints there. A queue host without the store gets no default, even when it holds the hive's own reader deploy.bao.clientCertFile: issue a leaf whose subject is deploy.bao.natsCommonName, place it, and name it in both options. Until you do, swarm-bao-nats-tls fails naming both whenever the queue's certificate is missing or due for renewal.

The queue host's gateway also serves the queue's metrics, at https://<swarm.nats.domain>/metrics, so a swarm collector on any host scrapes them. Only the collector's authelia client gets through; any other session gets deny, and every other path on that name is 403. With the gateway's default self-signed TLS, that vhost serves the queue's own certificate, so a host whose only swarm service is the queue needs no swarm-services certificate.

The identity isn't a choice — a hive authenticates as hive-<hiveName> and publishes under hiveName, the same name that keys swarm.hives.

If a hive stops reporting, its /health/ready and the hyperhive.hive.warning metric are the place to look: a failure to publish raises a hive warning after three consecutive misses. It stays warn rather than crit on purpose — a hive that can't reach the queue isn't itself unhealthy, so it doesn't start calling itself degraded for being unable to say it's fine.

The endpoint answers 503 when this host has no swarm queue configured, or has one and can't read it — deliberately not an empty list, which would look like a silent swarm rather than a controller that can't see. The body says which. Status survives a controller restart: it's stored in the queue, not in the daemon.

Giving the agents the queue too

An agent connects to the queue as itself, with a credential swarm-controller mints for it when it creates the agent (swarm/agents/<agent>/queue in the secret store, see credentials.md). The harness reads it from the store under the agent's own certificate before each connection attempt and holds it in memory only. The one thing to configure is where the queue is:

option what to set it to
deploy.hive-controller.queue.agentNatsUrl where the queue listens, as an agent container reaches it

On every hive, agentNatsUrl defaults to the same tls://<swarm.nats.domain>:<swarm.nats.port> as the hive's own. On the queue's host the name resolves inside a container to the bridge address, where the firewall opens the port; on any other hive it resolves through the host's DNS, like the store's name. ⚠️ Never a loopback address here: an agent has its own network namespace, so 127.0.0.1 reaches the agent.

An agent can't run without the connection. Without it the swarm can neither show the agent nor send it anything, so a harness without one stops:

Either way the harness exits and comes back until the queue accepts it. Once connected, the client reconnects by itself after a dropped connection, reading the credential from the store again for each attempt.

The hive's shared queue client is still delivered into the harness unit's credentials directory, and swarm.statusPublish.tokenEndpoint still reaches the container, but only for a swarm-logs built before it read the agent's own queue secret: the harness never connects with them, and a current swarm-logs presents that secret to the log store instead.

What an agent publishes over the queue

What an agent does with that connection is publish its terminal. Every row its own web UI renders also goes to $SWARM.term.<agent>, one subject per agent, so a swarm-level terminal can follow one agent without subscribing to the swarm's whole traffic. The queue grants that subject to the agent's own credential, and the swarm controller relays it. Publishing only: an agent talks about itself here and reads nothing. The queue keeps the rows in the stream term-main-<agent> for 24 hours or 64 MiB, the same limits and the same creator as the subagent stream below, so a swarm terminal that attaches later replays the newest of them. The two streams hold disjoint subjects.

An agent's subagents publish their terminals too, from the agent's subagent daemon: each subagent's rows go to $SWARM.term.<agent>.sub.<subagent>, classified the same way. The queue grants that family to the agent's own queue credential alone, so the daemon reads it from the store under the agent's store identity, exactly as the harness does, and publishes nothing without it. The queue keeps these rows in the stream term-sub-<agent> for 24 hours or 64 MiB, whichever comes first, dropping the oldest rows once either limit kicks in so a publish never fails because of it, and the swarm lists an agent's subagents from that stream's subjects and replays a subagent's newest rows from it. The swarm controller creates that stream for every agent a hive's wanted state names, within a minute of the agent appearing there. The agent's grant is publish on its own $SWARM.term.<agent>.sub.> and no JetStream subject, so the stream's subjects and limits are the controller's and never the agent's. Subagents publish output only and read nothing.

The queue would refuse a row too large for its max_payload outright and take the connection down with it, so the harness drops such a row's body before sending and leaves a marker in its place; the summary, level and icon still arrive. The harness logs and skips a row that's too large even without its body.

The second thing an agent publishes is its turn-state header, on $SWARM.agent-state.<agent> — same shape of subject, same grant mechanics, same lack of retention. It carries what a header bar wants: what the turn loop is doing (turn_state, plus turn_state_since as an ISO 8601 UTC stamp), which model and effort (model, effort, where each comes from as model_source and effort_source, and the resolved id the last turn actually ran on), the context budget and the last turn's context and cost token blocks, agent_state, and needs_login: true while the agent's claude has no session and its turns wait for a login. The harness sets it from its own login state, so it reaches the swarm without going through the hive.

agent_state reuses the swarm's own wanted-state vocabulary (up/offline/paused/destroyed) so a reader can compare what an agent is against what the swarm declared it should be without translating between two spellings. ⚠️ From inside the container only two of those four are sayable: the harness reports up, or paused when the pause marker is present. offline and destroyed are hive-c0re's observations — a stopped agent publishes nothing and a destroyed one doesn't exist — so a view that needs the full four-state picture takes them from the agent-status bucket and uses this subject to sharpen the rest.

Headers go out on transition: the harness rebuilds the header whenever its event bus moves and publishes when the result differs from what it last sent. That's the whole point of the subject — the agent-status bucket already republishes once a minute, which is far too slow for "is this agent thinking right now." A core subject keeps nothing for a late subscriber, so the harness also re-sends the current header once a minute: a subscriber attaching mid-idle sees it within that minute. A renderer opens with the bucket's snapshot and lets this stream refine it.

Swarm-side, GET /api/agents/<name>/state/stream relays that subject as SSE, resolving the agent's hive at request time exactly as the terminal stream does. The payload passes through opaquely. Separately, the controller keeps each agent's latest model, effort and their sources for GET /api/agents/<name>/inference (inference.md), and each agent's latest needs_login for the needs_login field of GET /api/agents/status. That field is null until a header carrying the flag arrives, so for up to a minute after the controller starts, and for an agent whose harness doesn't send it.

The third thing an agent publishes is its icon, the same SVG its own GET /icon serves. It goes into the agent-icons KV bucket under the key <agent>, with no hive in it, so the swarm can show the icon of an agent that's stopped or has moved hives. The queue grants the agent's own credential $KV.agent-icons.<agent> and no other key of that bucket. The harness writes once per start, because a config change reaches an agent by restarting its container. An agent with no icon deletes its key. Swarm-side, GET /api/agents/<name>/icon serves the stored bytes, and 404 means the agent has no icon. swarm-ui's agent cards load it as an <img> and show the dimmed hyperhive mark for an agent without one, as the hive dashboard does.

The fourth is its AGENTS.md, which goes into the agents-md bucket under the key <agent>, writable by the agent's own credential alone ($KV.agents-md.<agent>). The swarm reads that copy and edits the file through the agent; An agent's AGENTS.md has the details.

The queue also limits what an agent may subscribe to, because agents are the least trusted clients on the queue. An agent's own queue credential, which its subagent daemon also presents, may subscribe to its reply inbox _INBOX.agent.<agent>.>, its operator-input subject $SWARM.input.<agent> and its command subject $SWARM.control.<agent>, and nothing else. Besides the subjects it publishes to, the credential may answer the one request it was just sent, which is how a command gets its reply. The harness and the subagent daemon connect with _INBOX.agent.<agent> as their inbox prefix, so the replies to the icon write and the schedule read arrive. An agent can't read another agent's terminal, state, input, commands or replies. The queue refuses the hive's shared agent client outright.

Hives and the swarm controller get a subscribe allow-list too, each with its own reply inbox _INBOX.client.<client-id>.>, which swarm_queue_client::connect sets as their inbox prefix. A KV watch or key listing delivers there as well. A hive may also subscribe to $SWARM.knowledge and its own $SWARM.deploy.<hive>, so it can't read another hive's deploys or any agent's terminal. The controller may also subscribe to $SWARM.term.<agent>, $SWARM.term.<agent>.sub.<subagent> and $SWARM.agent-state.<agent>, the streams it relays to swarm-ui. A hive or controller that connects with the library's default _INBOX.<nuid> prefix gets no replies, so every request it makes times out.

An agent's AGENTS.md

The file in the agent's state dir, state/AGENTS.md, is the only copy that counts, and only the agent writes it: itself, or its harness applying an edit from the swarm. No hive is in either path.

Reading. The harness publishes the file's text and its revision, the git blob id that git hash-object prints for it, at start, within ten seconds of any change, and straight after an edit it applied. An agent with no AGENTS.md publishes "". GET /api/agents/<name>/agents-md answers {content, revision} from that copy, so it works while the agent isn't running. A 404 means the agent hasn't published one: its harness hasn't run since it began to.

Initial text. POST /api/agents and swarmctl agent create --agents-md <file> take an optional agents_md, the text the file starts with, up to 64 KiB; a longer text is a 400. The controller stores it under the key <agent> of the agents-md-initial bucket, which only it writes, before it queues the creation. At start the harness reads its own key, with one direct get on $JS.API.DIRECT.GET.KV_agents-md-initial.$KV.agents-md-initial.<agent>, and writes the text to AGENTS.md only when no file exists there. It does this before the first mirror publish and the first turn. An existing file is never replaced, so a restart or a re-create keeps the agent's edits; a later change to the file never reaches the key. Destroying the agent deletes the key. When the harness can't read the key it starts without the text and logs a warning. The text holds no secret, so nothing goes through the secret store.

Editing. PUT /api/agents/<name>/agents-md takes {base_revision, diff}. The diff is a unified diff that names a/AGENTS.md and b/AGENTS.md, made against the text at base_revision. swarm-controller sends it as a request on the agent's command subject $SWARM.control.<name>, as it sends a cancel. The harness refuses it when the file isn't at base_revision. Otherwise it runs git apply --check and then git apply on a copy of the file, and writes the result over the file only when both pass:

Status Meaning
200 applied; the body is {revision}, the file's new revision
409 the file isn't at base_revision any more; nothing applied, and revision in the body is the one it's at
422 git apply refused the diff, or it touches a file other than AGENTS.md; detail carries git's message, and the file stays as it was
503 the agent isn't running or isn't connected to the queue
504 the harness didn't answer within ten seconds, so whether the edit applied is unknown
500 the harness couldn't read or write the file or run git, or the queue failed the request

The harness refuses a diff that sets any file mode other than 100644, so an edit can't turn AGENTS.md into a symlink. PUT is operator-only, behind Remote-User like cancel: whoever can call it can rewrite the agent's instructions, so don't widen who holds it without revisiting this.

An edit needs the agent running, and a read of a stopped agent shows the text it last published. A harness older than this route doesn't know the command and drops it unanswered, which reads as a 504. swarmctl agent agents-md get and apply call the two routes.

Swarm-wide forge objects

The controller also keeps the forge objects that are one per swarm, not one per hive. It ensures them at start and every five minutes after (swarm-controller/src/forge/objects.rs):

A pass that can't finish logs a warn line per object plus swarm forge objects: pass incomplete in journalctl -u swarm-controller, and retries on the next tick. While the controller is down the objects stay as they are.

The controller gives each agent read on internal/docs. A GrantDocsRead node in the agent's create job adds it as a collaborator, unless it already holds read or more. A pass over the store's hive-agent-* roster (forge/docs_grant.rs, at start and every five minutes) adds any agent that isn't one, so an agent whose create-time grant failed gets it within five minutes. The pass only adds: a roster it can't read, a forge read that fails, or an internal/docs that doesn't exist (checked once per pass, logged at warn) changes nothing. The grant doesn't depend on the hive the agent runs on, or on a forge container there.

Swarm-wide forge webhooks

At startup the controller registers its Forgejo hooks pointing at itself, all under https://<swarm.domain>/webhook/forge/:

The controller interprets a delivery and sends hives a specific message — the knowledge repo changed, deploy agent foo at rev abc123 — rather than forwarding forge payloads for each hive to re-derive. Approval happens once, at the swarm level: a hive receives a decision, not an event to adjudicate.

A config-pr delivery reporting a PR merged into main queues a deploy of its merge_commit_sha on the one hive whose wanted state places the agent; with no such hive, or several, the controller deploys nothing. See config-changes.md § Config changes. A merge whose closed delivery never arrives deploys nothing either; the operator recovers by redelivering that delivery from the forge hook page, which re-enters the same handler and re-queues the deploy.

internal/knowledge is on that path. The controller's is the only hook on it (knowledge.md covers clearing a leftover). A webhook has exactly one target URL, so a second registration would take delivery away from the first rather than add a recipient.

The agent-configs org is on it too. The controller's hook is the only one that acts on config PRs. Each hive deletes its own /webhook/config-pr org hook on startup, since a route no hive serves would otherwise sit on the org collecting failed deliveries. Removing the controller's hook stops forge-UI merges from deploying until its next start recreates it.

Nothing to configure. The controller registers the hooks only when this host also serves the swarm UI vhost — that's what publishes the endpoint, and a hook the forge can't reach would collect failed deliveries while looking healthy. The controller generates the HMAC secret on first start and keeps it (see docs/agent-lifecycle/persistence.md).

To check it's working, push to internal/knowledge and look for webhook: verified delivery in journalctl -u swarm-controller. A refused delivery logs webhook: refused delivery with the reason.

Hives: the substrate

Hive identity config

services.hyperhive = {
  swarm.domain = "example.com";     # required — the swarm's DNS domain
  hiveName = "pr1ma";               # required — this hive's label in it
  swarm.name = "constellat1on";     # shared swarm display name (optional)

  # required — the directory, identical on every host in the swarm.
  # Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
  swarm.hives = {
    pr1ma = { };                                     # this host, per hiveName
    lab   = { };                                     # a second hive
    edge  = { domain = "edge.elsewhere.example"; };  # addressed off-convention
  };
};

Swarm options are identical on every host in the swarm, so swarm.domain is required on every host that runs any hyperhive service; a host that imports the module and enables nothing evaluates without it. hiveName is required on a host that runs a hive, the secret store or the homeserver. Eval fails with a hint naming each. Neither defaults, because a guessed value here is a wrong hostname that evaluates cleanly and deploys.

The directory is one attrset, identical on every host. It describes every hive in the swarm, including this one, keyed by hiveName; what differs between hosts is only hiveName. It must contain an entry for this host's hiveName, and an empty directory fails that check too — eval names the missing hive. Peers are every entry but this host's, so a directory without this host would make every hive a peer, itself included.

# hive A                          # hive B
hiveName = "pr1ma";               hiveName = "edge";
swarm.hives = { … };              swarm.hives = { … };   # byte-identical

One entry per hive means two hosts can't hold different facts about the same third hive, such as a stale endpoint.

domain defaults to <name>.<swarm.domain>, so a conventional directory is names only. Set it only for a hive addressed by something else. This hive's own services.hyperhive.domain comes from its entry; it drives HYPERHIVE_HIVE_DOMAIN in every container so agents can form qualified labels (iris@pr1ma.example.com). Setting services.hyperhive.domain directly overrides the directory entry, with a deprecation warning: the option is local to this host while every host shares the directory, so a value written only here is invisible to the rest of the swarm.

swarm.name is display only — the dashboard chrome header and per-agent system prompts — and federated hives at different domains can share one. hiveName surfaces in the same places but is also the leftmost label of the hive's domain. swarm.name names the group; hiveName names this hive.

The env-var chain and qualify() semantics: conventions.md § Hive identity.

Trust inside a swarm comes from the secret store's pki root (ca.md): every hive's gateway leaf chains to it. Trusting a hive whose root this swarm doesn't own — another swarm's — has no mechanism.

What the directory feeds

  1. Swarm-wide hive roster — swarm-controller reads this same directory and serves it at GET /api/hives; the swarm UI's front page renders it (ui.md).

  2. Matrix federation — when this host runs the homeserver (deploy.matrix.enable), tuwunel federates with the peer's matrix server (discovered via the peer's .well-known/matrix/server delegation, which the gateway serves). Federation validates the peer's TLS certificate against the matrix container's trust bundle, independent of this directory.

    ⚠️ On a hive with self-signed gateway certificates, the container also trusts this hive's trust-bundle.pem, bound in at runtime and added to the public CAs; it holds the store's pki root, so a peer whose certificate chains to the same root validates. A peer outside this swarm's root needs a CA-issued certificate (ACME). Federation firewall and TLS requirements: integrations/matrix.md.

  3. WireGuard mesh (optional) — deploy.wireguard.enable reads each entry's wireguardPublicKey/wireguardEndpoint/wireguardAddress to configure wg-hive. See below.

WireGuard inter-hive mesh (optional)

The peer config above uses public HTTPS for all inter-hive traffic. For private deployments — or to reduce latency and TLS overhead on intra-swarm traffic — hyperhive can configure a host-to-host WireGuard mesh.

Generating keys

On each hive host:

wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key   # → share this with peer operators

Config example (two hives)

# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
  deploy.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.1/24";
    listenPort     = 51820;          # optional, default 51820
  };

  # The same `hives` attrset both hosts hold — mesh fields included,
  # since "where this hive can be dialled" is a fact about that hive.
  swarm.hives = {
    pr1ma = {
      domain             = "pr1ma.example.com";
      wireguardPublicKey = "base64keyA=";
      wireguardEndpoint  = "198.51.100.1:51820";
      wireguardAddress   = "10.100.0.1/32";
    };
    edge = {
      domain             = "edge.corp";
      wireguardPublicKey = "base64keyB=";
      wireguardEndpoint  = "203.0.113.42:51820";
      wireguardAddress   = "10.100.0.2/32";
    };
  };
};

# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
  deploy.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.2/24";
  };

  swarm.hives = { /* … identical to hive A's … */ };
};

What the mesh does

NAT / one-sided endpoints

If one host is behind NAT and can't accept incoming connections, only that host needs a null wireguardEndpoint on the peer config — the other side initiates. With keepalive on, the NAT hole stays open.

If both hosts are behind NAT, you need a STUN relay or a third host (exit node); hyperhive sets up neither.

Cross-references