Multi-hive swarms

A swarm is a collection of agents that share an identity and coordinate across one or more hives. A single hyperhive instance running on one host is already a swarm (one hive). This doc covers the additional config needed when the swarm spans multiple hosts.

Terminology

Hive identity config

services.hyperhive = {
  swarm.domain = "example.com";     # required — the swarm's DNS domain
  hiveName = "pr1ma";               # required — this hive's label in it
  swarm.name = "constellat1on";     # shared swarm display name (optional)

  # required — the directory, identical on every host in the swarm.
  # Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
  swarm.hives = {
    pr1ma = { };
    edge = { };
  };
};

swarm.domain and hiveName are required whenever hyperhive is enabled; eval fails with a hint naming each. Neither is defaulted, because a guessed value here is a wrong hostname that evaluates cleanly and deploys — an eval failure asking the operator to write the address down is the cheaper outcome. Upgrading past this release means setting both once.

domain is required too, but you no longer write it: it is read from this hive's own entry in the directory, whose domain defaults to <name>.<swarm.domain>. So a conventional swarm states no addresses at all, and a hive addressed by something else states it in the one place the other hives read — swarm.hives.edge.domain = "edge.elsewhere.example";.

Setting services.hyperhive.domain directly still works and still wins, with a deprecation warning. The reason it's deprecated is not tidiness: that option is local to one host, the directory is copied to every host, so a value written only there leaves every peer pointing somewhere else with nothing detecting the disagreement.

⚠️ Upgrading: a hive that has been running on swarm.domain + hiveName alone now needs its own directory entry — services.hyperhive.swarm.hives.<hiveName> = { };, one line, no value. Eval fails naming it if you forget.

domain drives HYPERHIVE_HIVE_DOMAIN in every container so agents can form qualified labels (iris@pr1ma.example.com).

swarm.name is purely display — it surfaces in the dashboard chrome header and per-agent system prompts, and federated hives at different domains can share one. hiveName surfaces in the same places but is not only display: it is the leftmost label of the hive's domain. That swarm.name sits under swarm and hiveName does not is the whole distinction — one names this hive, the other names the group it belongs to.

See docs/conventions.md § Hive identity for the env-var chain and qualify() / qualified_label() semantics.

Swarm CA

A hive's internal TLS chains to a swarm root CA, so a peer that trusts the root validates every hive in the swarm rather than being pinned to each one by hand. Provisioning modes, what to hand a peer (trust-bundle.pem, never ca.pem), the name constraints on a hive CA, and how an existing hive adopts the hierarchy: ca.md.

Running the swarm's shared services

One authelia, one matrix, one forge per swarm — which host runs them, and what a hive that runs none of them configures instead: services.md.

Single sign-on

Which secrets the SSO provider generates, which one has a reader in another container, and the three ways that one gets delivered: sso.md.

Secrets

Every credential the swarm holds, who mints it, where it must live, and which of the three topologies makes it the operator's job to place: secrets.md.

Swarm UI

The operator-only web surface on the swarm apex, why reaching it needs the admins group rather than just a session, and the four sites a swarm service name has to be wired into: ui.md.

The swarm's hive directory

services.hyperhive.swarm.hives = {
  pr1ma = { domain = "pr1ma.example.com"; };   # this host, per hiveName
  lab   = { domain = "lab.example.com"; };     # a second hive in the swarm
};

One attrset describing every hive in the swarm, including this one, keyed by that hive's hiveName. It is meant to be identical on every host — write it once, share it, and each host reads it correctly because services.hyperhive.hiveName says which entry is itself.

Empty (the default) means this host isn't in a swarm. Once non-empty it must contain an entry for hiveName; eval fails naming the missing hive. That assertion is load-bearing rather than pedantic — "my peers" is derived as everything that isn't me, so a directory that doesn't contain you derives every hive as a peer and you peer with yourself.

domain is required per entry and deliberately undefaulted: it is conventionally <name>.<swarm.domain>, but a wrong domain that evaluates cleanly points at a real machine that isn't the one you meant.

There is no per-hive CA field, and no per-hive cert pinning. Trust inside a swarm comes from the swarm root (ca.md): every hive chains to it, so one anchor replaces per-hive pinning entirely. What that genuinely drops is trusting a hive whose root this swarm does not own — another swarm's, or one keeping its own CA. That is a cross-swarm problem and wants a mechanism designed for it. (An earlier certFingerprint field existed for exactly that gap, pinning a peer's TLS leaf for hive-c0re's own peer HTTPS checks — removed along with the dashboard feature it existed to serve, since nothing else ever consumed it.)

What the config does at runtime

  1. Swarm-wide hive roster — swarm-controller reads this same directory and serves it at GET /api/hives; swarm-ui's overview page renders it (docs/swarm/ui.md). This is the operator-facing "what hives exist" surface — a per-hive dashboard "peer hives" display existed here once and was removed in favour of it.

  2. Matrix federation — when matrix.enable is on, tuwunel federates with the peer's matrix server (discovered via the peer's .well-known/matrix/server delegation, which the gateway serves). Federation validates the peer's TLS certificate against the matrix container's trust bundle, independent of this directory.

    ⚠️ That container currently trusts no swarm-internal CA, so a self-signed gateway certificate does not federate. The swarm root can't simply be listed there: security.pki.certificateFiles is read when the system is built, and the root is a runtime file (its key must never enter the store), so there is no build-time name for it. Bridging that needs a runtime mechanism and is tracked as its own issue. Until then, federation needs CA-issued certs (ACME). See docs/matrix.md for federation firewall + TLS requirements.

  3. WireGuard mesh (optional) — swarm.wireguard.enable reads each entry's wireguardPublicKey/wireguardEndpoint/wireguardAddress to configure wg-hive. See "WireGuard inter-hive mesh" below.

One directory, not a bilateral declaration

Both hives hold the same hives attrset; neither declares the other. What differs between the two hosts is only hiveName:

# hive A                          # hive B
hiveName = "pr1ma";               hiveName = "edge";
swarm.hives = { … };              swarm.hives = { … };   # byte-identical

That is the point of the shape, and it removes a class of bug rather than saving typing: a per-host peer list let two hosts hold different facts about the same third hive — a stale endpoint, a rotated fingerprint — with nothing to detect the disagreement. One entry per hive makes it unrepresentable.

WireGuard inter-hive mesh (optional)

The peer config above uses public HTTPS for all inter-hive traffic. For private deployments — or to reduce latency and TLS overhead on intra-swarm traffic — hive-c0re can configure a host-to-host WireGuard mesh.

Generating keys

On each hive host:

wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key   # → share this with peer operators

Config example (two hives)

# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
  swarm.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.1/24";
    listenPort     = 51820;          # optional, default 51820
  };

  # The same `hives` attrset both hosts hold — mesh fields included,
  # since "where this hive can be dialled" is a fact about that hive.
  swarm.hives = {
    pr1ma = {
      domain             = "pr1ma.example.com";
      wireguardPublicKey = "base64keyA=";
      wireguardEndpoint  = "198.51.100.1:51820";
      wireguardAddress   = "10.100.0.1/32";
    };
    edge = {
      domain             = "edge.corp";
      wireguardPublicKey = "base64keyB=";
      wireguardEndpoint  = "203.0.113.42:51820";
      wireguardAddress   = "10.100.0.2/32";
    };
  };
};

# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
  swarm.wireguard = {
    enable        = true;
    privateKeyFile = "/etc/wireguard/hive.key";
    address        = "10.100.0.2/24";
  };

  swarm.hives = { /* … identical to hive A's … */ };
};

What the mesh does

NAT / one-sided endpoints

If one host is behind NAT and can't accept incoming connections, only that host needs a null wireguardEndpoint on the peer config — the other side initiates. With keepalive on, the NAT hole stays open.

If both hosts are behind NAT, a STUN relay or a third host (exit node) is required. Out of scope for v0.

Snapshot store

One further option lives in this namespace but is documented with the service it points at: services.hyperhive.swarm.snapshotStore.{address, port} tells this hive where the swarm's btrfs receive endpoint is, so hivectl agent <name> subvol snapshot push has somewhere to stream to.

It is genuinely swarm-scoped rather than per-peer — a swarm has exactly one store, because the receiver keys destinations by agent so a migrating agent keeps one unbroken incremental chain. See snapshot-store.md.

Swarm controller

services.hyperhive.swarm.controller.enable runs the swarm-controller daemon on this host. Off by default and deliberately not derived from services.hyperhive.enable: a swarm has one controller, so enabling it is a statement about swarm topology, not about whether hyperhive is installed. Every hive runs hive-c0re (the agents on that host); one hive additionally runs this (what is true across hives).

What it serves, why it is a unix socket rather than a port, and the socket-directory constraint that governs where socketPath may point: swarm-controller/README.md.

Per-hive status (GET /api/hives/status)

One row per hive in swarm.hives, saying when it last reported and what it said. Hives publish upward through the swarm queue; the controller never reaches down to collect, so a hive that cannot reach the swarm still knows its own state — you just cannot see it from here.

A hive publishes only once it has been given the three swarm.statusPublish coordinates below. A hive that has not reads never_reported — it is not broken, it just has nothing to say upward.

freshness what to do about it
fresh nothing — reported within staleAfterSeconds
stale the hive stopped reporting. Its last payload is still shown, so check age_seconds and the payload for what it managed to say
never_reported this hive has never reported at all — normally a deployment that hasn't happened, not an outage
unknown something is publishing under a name that is not in swarm.hives — a typo in the roster, or a hive that was removed and is still running

Every row also carries last_seen_unix and age_seconds if you want to apply your own threshold. The timestamp is the one the queue recorded on arrival, not one the hive put in its own payload.

Set services.hyperhive.swarm.controller.staleAfterSeconds (default 120) above the rate hives publish at, or everything reads stale between reports. Hives publish once a minute, so the default tolerates one missed report and flags two. It takes effect on the next request; nothing has to re-publish.

Making a hive report (swarm.statusPublish)

Three options, on the hive, set together or not at all — a half-configured hive is an eval error rather than one that quietly never reports:

option what to set it to
natsUrl where the swarm queue listens, as this hive reaches it
tokenEndpoint the swarm IdP's /api/oidc/token
clientSecretFile path to this hive's client secret, plaintext

On a host that runs the queue and the IdP itself, all three default to the local ones and there is nothing to set. Any other hive needs them spelled out, and needs the secret to physically be there: the swarm does not distribute it. Copy hive-<hiveName>.secret out of the swarm host's swarm.authelia.hostClientSecretDir with whatever secret management the deployment already uses.

The identity is not a choice — a hive authenticates as hive-<hiveName> and publishes under hiveName, the same name that keys swarm.hives.

If a hive stops reporting, its own dashboard is the place to look: a failure to publish raises a warning banner there after three consecutive misses. It stays warn rather than crit on purpose — a hive that cannot reach the queue is not itself unhealthy, so it does not start calling itself degraded for being unable to say it is fine.

The endpoint answers 503 when this host has no swarm queue configured, or has one and cannot read it — deliberately not an empty list, which would look like a silent swarm rather than a controller that cannot see. The body says which. Status survives a controller restart: it is stored in the queue, not in the daemon.

Swarm-wide forge webhooks

At startup the controller registers two Forgejo hooks pointing at itself — a push hook on internal/knowledge and a pull_request hook on the agent-configs org, both under https://<swarm.ui.domain>/webhook/forge/.

Two hooks exist per swarm-wide repo: each hive's own, plus the controller's. Both are expected — do not delete either. Removing a hive's hook stops that hive acting on knowledge pushes and config PRs; removing the controller's just gets recreated on its next start.

The controller interprets a delivery and sends hives a specific message — the knowledge repo changed, deploy agent foo at rev abc123 — rather than forwarding forge payloads for each hive to re-derive. Approval happens once, at the swarm level: a hive receives a decision, not an event to adjudicate. Today the controller logs each verified delivery and sends nothing, because the swarm→hive channel does not exist yet; the hive-side hooks are what act in the meantime.

Nothing to configure. The hooks are registered only when this host also serves the swarm UI vhost — that is what publishes the endpoint, and a hook the forge cannot reach would collect failed deliveries while looking healthy. The HMAC secret is generated on first start and kept (see docs/persistence.md).

To check it is working, push to internal/knowledge and look for webhook: verified delivery in journalctl -u swarm-controller. A refused delivery logs webhook: refused delivery with the reason.

Cross-references