Multi-hive swarms
A swarm is a collection of agents that share an identity and coordinate across one or more hives. A single hyperhive instance running on one host is already a swarm (one hive). This doc covers the additional config needed when the swarm spans multiple hosts.
Terminology
- hive — a single hyperhive installation on one host. Has its
own
services.hyperhive.domainDNS name and its own set of agent containers. - swarm — one or more hives whose operators have declared them
as peers. Agents can be qualified as
agent@hive-domain. - peer hive — any hive in
services.hyperhive.swarm.hivesother than this one. Peers are derived, not declared: the directory lists every hive including yourself, andhiveNamesays which one you are.
Hive identity config
services.hyperhive = {
swarm.domain = "example.com"; # required — the swarm's DNS domain
hiveName = "pr1ma"; # required — this hive's label in it
swarm.name = "constellat1on"; # shared swarm display name (optional)
# required — the directory, identical on every host in the swarm.
# Names only: each entry's `domain` defaults to <name>.<swarm.domain>.
swarm.hives = {
pr1ma = { };
edge = { };
};
};
swarm.domain and hiveName are required whenever hyperhive is
enabled; eval fails with a hint naming each. Neither is defaulted,
because a guessed value here is a wrong hostname that evaluates cleanly
and deploys — an eval failure asking the operator to write the address
down is the cheaper outcome. Upgrading past this release means setting
both once.
domain is required too, but you no longer write it: it is read from
this hive's own entry in the directory, whose domain defaults to
<name>.<swarm.domain>. So a conventional swarm states no addresses at
all, and a hive addressed by something else states it in the one place
the other hives read — swarm.hives.edge.domain = "edge.elsewhere.example";.
Setting services.hyperhive.domain directly still works and still wins,
with a deprecation warning. The reason it's deprecated is not tidiness:
that option is local to one host, the directory is copied to every host,
so a value written only there leaves every peer pointing somewhere else
with nothing detecting the disagreement.
⚠️ Upgrading: a hive that has been running on swarm.domain +
hiveName alone now needs its own directory entry —
services.hyperhive.swarm.hives.<hiveName> = { };, one line, no value.
Eval fails naming it if you forget.
domain drives HYPERHIVE_HIVE_DOMAIN in every container so agents can
form qualified labels (iris@pr1ma.example.com).
swarm.name is purely display — it surfaces in the dashboard chrome
header and per-agent system prompts, and federated hives at different
domains can share one. hiveName surfaces in the same places but is
not only display: it is the leftmost label of the hive's domain. That
swarm.name sits under swarm and hiveName does not is the whole
distinction — one names this hive, the other names the group it belongs
to.
See docs/conventions.md § Hive identity for the env-var chain
and qualify() / qualified_label() semantics.
Swarm CA
A hive's internal TLS chains to a swarm root CA: the root signs each hive's own CA, and that hive CA signs the gateway leaf, so a peer that trusts the root once validates every hive in the swarm rather than being pinned to each one by hand.
Provisioning modes, what to hand a peer (trust-bundle.pem, never
ca.pem), the name constraints on a hive CA, and how an existing hive
adopts the hierarchy: ca.md.
Running the swarm's shared services
One authelia, one matrix, one forge per swarm — which host runs them,
and what a hive that runs none of them configures instead:
services.md.
Single sign-on
Which secrets the SSO provider generates, which one has a reader in
another container, and the three ways that one gets delivered:
sso.md.
Swarm UI
The operator-only web surface on the swarm apex, why reaching it needs
the admins group rather than just a session, and the four sites a
swarm service name has to be wired into: ui.md.
The swarm's hive directory
services.hyperhive.swarm.hives = {
pr1ma = { domain = "pr1ma.example.com"; }; # this host, per hiveName
lab = { domain = "lab.example.com"; }; # CA-trusted (Let's Encrypt etc.)
edge = { domain = "edge.corp"; certFingerprint = "sha256:…"; }; # self-signed leaf, pinned
};
One attrset describing every hive in the swarm, including this
one, keyed by that hive's hiveName. It is meant to be identical on
every host — write it once, share it, and each host reads it correctly
because services.hyperhive.hiveName says which entry is itself.
Empty (the default) means this host isn't in a swarm. Once non-empty it
must contain an entry for hiveName; eval fails naming the missing
hive. That assertion is load-bearing rather than pedantic — "my peers"
is derived as everything that isn't me, so a directory that doesn't
contain you derives every hive as a peer and you peer with yourself.
domain is required per entry and deliberately undefaulted: it is
conventionally <name>.<swarm.domain>, but a wrong domain that
evaluates cleanly points at a real machine that isn't the one you meant.
certFingerprint ("sha256:…", optional) pins that hive's TLS
leaf. Scopes only to hive-c0re's own peer HTTPS checks (the P33RS
dashboard links + agent peer discovery below); matrix federation never
consults it. Omit it for any hive under the swarm root CA or a public
CA — which is the normal case.
There is no per-hive CA field. Trust inside a swarm comes from the swarm root (
ca.md): every hive chains to it, so one anchor replaces the O(n²) pinning. What that genuinely drops is trusting a hive whose root this swarm does not own — another swarm's, or one keeping its own CA. That is a cross-swarm problem and wants a mechanism designed for it, not a field that happened to work.
Fingerprint format
The value is the string sha256: followed by exactly 64 hexadecimal
digits — the SHA-256 digest of the peer's DER-encoded TLS leaf
certificate. The hex is case-insensitive (upper or lower both parse),
carries no colon separators between bytes, and any value not matching
this shape is ignored with a warning rather than weakening trust.
sha256:b1946ac92492d2347c6235b4d2611184a3f5b6cae6c19d6e3c2f0a8e7d4c9f12
Generate it from the peer's certificate with openssl. The
-fingerprint -sha256 output is uppercase and colon-separated, so
strip the colons, lowercase, and prepend the sha256: prefix:
# from a PEM/CRT file
openssl x509 -in peer.crt -noout -fingerprint -sha256 \
| sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'
# straight from the live endpoint (port 443)
echo | openssl s_client -connect peer.example.com:443 -servername peer.example.com 2>/dev/null \
| openssl x509 -noout -fingerprint -sha256 \
| sed 's/^.*=//; s/://g' | tr 'A-Z' 'a-z' | sed 's/^/sha256:/'
Pin the leaf certificate, not an intermediate or the CA — the digest must match the exact cert the peer serves on its HTTPS endpoint. When the peer rotates its cert, update the pin to the new fingerprint (or switch the peer to a CA-trusted cert and drop the field).
The nix module serialises the attrset to a HYPERHIVE_PEERS JSON
array ([{ domain, cert_fingerprint }]) injected into the c0re
environment and forwarded to agent containers.
What the config does at runtime
-
Dashboard P33RS tab —
parse_peer_hives()indashboard.rsreadsHYPERHIVE_PEERSand includespeer_hives: Vec<{ name, url }>in/api/state. The dashboard shows a P33RS tab (hidden when the list is empty) with a card per peer linking tohttps://{domain}/. Seedocs/web-ui/dashboard.md§ P33RS tab. -
Agent identity — the same
HYPERHIVE_PEERSenv var is forwarded to agent containers bymeta.rs; agent code can callidentity::peers()to discover peer hives and address them with qualified names (agent@domain). -
Matrix federation — when
matrix.enableis on, tuwunel federates with the peer's matrix server (discovered via the peer's.well-known/matrix/serverdelegation, which the gateway serves). Federation validates the peer's TLS certificate against the matrix container's trust bundle — independently ofcertFingerprint, which it never consults.⚠️ That container currently trusts no swarm-internal CA, so a self-signed gateway certificate does not federate. The swarm root can't simply be listed there:
security.pki.certificateFilesis read when the system is built, and the root is a runtime file (its key must never enter the store), so there is no build-time name for it. Bridging that needs a runtime mechanism and is tracked as its own issue. Until then, federation needs CA-issued certs (ACME). Seedocs/matrix.mdfor federation firewall + TLS requirements.
One directory, not a bilateral declaration
Both hives hold the same hives attrset; neither declares the
other. What differs between the two hosts is only hiveName:
# hive A # hive B
hiveName = "pr1ma"; hiveName = "edge";
swarm.hives = { … }; swarm.hives = { … }; # byte-identical
That is the point of the shape, and it removes a class of bug rather than saving typing: a per-host peer list let two hosts hold different facts about the same third hive — a stale endpoint, a rotated fingerprint — with nothing to detect the disagreement. One entry per hive makes it unrepresentable.
WireGuard inter-hive mesh (optional)
The peer config above uses public HTTPS for all inter-hive traffic. For private deployments — or to reduce latency and TLS overhead on intra-swarm traffic — hive-c0re can configure a host-to-host WireGuard mesh.
Generating keys
On each hive host:
wg genkey | install -m 0400 /dev/stdin /etc/wireguard/hive.key
wg pubkey < /etc/wireguard/hive.key # → share this with peer operators
Config example (two hives)
# hive A (pr1ma.example.com, mesh IP 10.100.0.1)
services.hyperhive = {
swarm.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.1/24";
listenPort = 51820; # optional, default 51820
};
# The same `hives` attrset both hosts hold — mesh fields included,
# since "where this hive can be dialled" is a fact about that hive.
swarm.hives = {
pr1ma = {
domain = "pr1ma.example.com";
wireguardPublicKey = "base64keyA=";
wireguardEndpoint = "198.51.100.1:51820";
wireguardAddress = "10.100.0.1/32";
};
edge = {
domain = "edge.corp";
certFingerprint = "sha256:…"; # TLS trust (unchanged)
wireguardPublicKey = "base64keyB=";
wireguardEndpoint = "203.0.113.42:51820";
wireguardAddress = "10.100.0.2/32";
};
};
};
# hive B (edge.corp, mesh IP 10.100.0.2)
services.hyperhive = {
swarm.wireguard = {
enable = true;
privateKeyFile = "/etc/wireguard/hive.key";
address = "10.100.0.2/24";
};
swarm.hives = { /* … identical to hive A's … */ };
};
What the mesh does
networking.wireguard.interfaces.wg-hiveis configured on the host (not inside agent containers; containers reach peers via the host's routing table).- UDP port 51820 (or
listenPort) is opened on the host firewall. HYPERHIVE_PEERSgains awireguard_addressfield for each mesh peer so hive-c0re can reach intra-swarm services without a public DNS round-trip.persistentKeepalive = 25is set by default; override or null to disable (not needed when both sides have public IPs and no NAT).
NAT / one-sided endpoints
If one host is behind NAT and can't accept incoming connections, only
that host needs a null wireguardEndpoint on the peer config — the
other side initiates. With keepalive on, the NAT hole stays open.
If both hosts are behind NAT, a STUN relay or a third host (exit node) is required. Out of scope for v0.
Snapshot store
One further option lives in this namespace but is documented with the
service it points at: services.hyperhive.swarm.snapshotStore.{address, port} tells this hive where the swarm's btrfs receive endpoint is, so
hivectl agent <name> subvol snapshot push has somewhere to stream to.
It is genuinely swarm-scoped rather than per-peer — a swarm has exactly one store, because the receiver keys destinations by agent so a migrating agent keeps one unbroken incremental chain. See snapshot-store.md.
Swarm controller
services.hyperhive.swarm.controller.enable runs the swarm-controller
daemon on this host. Off by default and deliberately not derived from
services.hyperhive.enable: a swarm has one controller, so enabling it
is a statement about swarm topology, not about whether hyperhive is
installed. Every hive runs hive-c0re (the agents on that host); one
hive additionally runs this (what is true across hives).
What it serves, why it is a unix socket rather than a port, and the
socket-directory constraint that governs where socketPath may point:
swarm-controller/README.md.
Cross-references
docs/snapshot-store.md— the swarm'sbtrfs receiveendpoint, and theswarm.snapshotStoreoption that points a hive at itdocs/conventions.md§ Hive identity — env vars, qualified labelsdocs/matrix.md— matrix federation, TLS cert auto-generation, firewall posturedocs/web-ui/dashboard.md§ P33RS tab — dashboard surfacedocs/gateway.md— nginx vhosts and the.well-known/matrix/auto-discovery scheme