Swarm CA

A hive's gateway serves a leaf issued by the secret store's pki mount, through a role that belongs to that hive. A peer that trusts the mount's root once validates every hive's gateway and every swarm service name, present and future, instead of pinning to each one by hand.

That's the point of one issuer — it turns per-peer trust from O(n²) hand-pinning into one anchor per swarm.

The gateway leaf comes from the secret store

hive-tls-resign.service logs in to the store with this hive's own identity, calls pki/issue/<deploy.bao.hiveLeafPkiRolePrefix>-<hiveName> (swarm-hive-leaf-<hiveName> by default), and writes the result to <deploy.hive-controller.tls.stateDir>/gateway.pem and gateway-key.pem, the files the gateway copies from. The leaf's subject is the hive domain, and its SANs are the hive domain and *.<hive domain>.

Each hive's role covers that hive alone. Its allowed_domains is the hive's domain and *.<hive domain>, read from services.hyperhive.swarm.hives.<name>.domain on the store's host, with no IP SANs and no localhost. A request for any other name is a refusal from the store naming that name. The hive's policy grants update on that one issue path and nothing else. swarm-bao-hive-leaf-policy.service, on the store's host, writes the role, the policy and the cert-auth role for every hive in swarm.hives.

Every hive needs its own store identity. It's a client certificate from the store's own client CA, with the subject <deploy.bao.hiveLeafCommonNamePrefix>-<hiveName> (swarm-bao-hive-leaf-<hiveName> by default), and it's named by services.hyperhive.deploy.hive-controller.tls.baoHiveIdentityCertFile and baoHiveIdentityKeyFile:

• On the host running the store, swarm-bao-pki.service mints it at /var/lib/swarm-bao-pki/hive-leaf.pem and the two options default to it.
• On every other host, copy hive-leaf.pem and hive-leaf-key.pem from /var/lib/swarm-bao-pki/hives/<hiveName>/ on the store's host, which swarm-bao-pki.service mints for every other hive in swarm.hives, and set both options, before the host's next deploy.

Without it, hive-tls-resign.service fails with a message naming both options, and so does swarm-services-cert.service on a hive fronting no swarm name. The unit checks the identity before the leaf, so it fails even while a valid leaf keeps the gateway up. The identity is the hive's alone: it issues this hive's leaf and fetches the store's root, and it's never the identity a swarm service uses.

When it issues. The unit runs at boot, on every deploy, and from hive-tls-resign.timer once a day. It asks the store for a new leaf when the current one is missing, past half the role's 720-hour lifetime, missing a configured name, doesn't match the key beside it, or doesn't chain to the store's root (swarm-services-root.pem). The first run that reaches the store replaces a leaf that doesn't chain to that root, whoever signed it.

The gateway doesn't wait for the store. Nothing orders the gateway after hive-tls-resign.service, so a hive holding a valid leaf starts its gateway while the store can't answer — still sealed, or out of reach. A renewal that fails leaves the current leaf in place, says so in the journal, and retries every 30 seconds for a day. Only a hive with no leaf yet needs the store reachable and unsealed: its gateway's certificate import fails until the first leaf lands, and then the unit starts nginx.

nginx serves the leaf alone, without its issuer: the issuer is the self-signed root a consumer already holds in its trust bundle.

A gateway host that runs no hive (deploy.hive-controller.enable off, such as a dedicated swarm-services or secret-store host) has no hive leaf, no hive-tls-resign.service, no hive identity and no hive-domain vhost. Its gateway serves the swarm-services leaf as its default certificate too. Where it fronts other swarm names it waits for swarm-services-cert.service at start. Where the store's UI is its only swarm name it starts on a throwaway self-signed certificate and switches to the services leaf once the store issues it.

The trust bundle

hive-tls-ca.service writes <deploy.hive-controller.tls.stateDir>/trust-bundle.pem, the file every consumer anchors on. It holds one certificate, a copy of swarm-services-root.pem: the store's pki root, which the gateway leaf and the swarm-services leaf both chain to. It's the only root the swarm has.

Without the store's root, the bundle is empty and its readers go red. While <deploy.hive-controller.tls.stateDir>/swarm-services-root.pem is missing or empty, as on a first boot before the store answers, hive-tls-ca.service writes an empty bundle and logs that file's path. Every <name>-ca-bundle unit then fails with <bundle> contributed no certificate to the bundle, and the consumers that Requires= it don't start. swarm-services-cert.service restarts hive-tls-ca.service when it writes a new root.

A deploy that changes hive-tls-ca.service reloads it: the same script rewrites the bundle in place, and nothing that Requires= the unit restarts, including the store's container. A <name>-ca-bundle unit keeps the copy it made when it started.

⚠️ A consumer reads its trust when it starts. An agent or service container that last started before the store's root reached the bundle fails TLS to a gateway serving a store-issued leaf, until it restarts.

The swarm's service names come from the secret store

The swarm's service names (swarm.forge.domain, swarm.matrix.gatewayHost, swarm.authelia.domain) default to siblings of the hive domain — forge.<swarm.domain>, not forge.<hive domain>. The gateway leaf carries the hive's domain and a single-label wildcard over it, and its role allows nothing outside the hive's domain, so it can't cover them.

The secret store's pki mount issues them a leaf of their own, through its swarm-services role. swarm-services-cert.service logs in with the services-issuer identity, calls pki/issue/swarm-services, and writes the result to /var/lib/hive-tls/swarm-services.pem — the path the gateway already copies from. The store's role, not a key on the host, decides which names a gateway can serve.

Every self-signed gateway that fronts a swarm name needs the services-issuer identity, whether or not a hive runs on its host. A hive fronting none needs no copy of it: it fetches the store's root with its own store identity, as below. It's a client certificate from the store's own client CA, with the subject deploy.bao.servicesIssuerCommonName (swarm-services-issuer by default), and it's named by services.hyperhive.deploy.bao.servicesIssuerClientCertFile and servicesIssuerClientKeyFile. Its policy may issue from pki/issue/swarm-services and nothing else, and it's the same certificate on every gateway host:

• On the host running the store, swarm-bao-pki.service mints it at /var/lib/swarm-bao-pki/services-issuer.pem and the two options default to it.
• On every other gateway host fronting a swarm name, copy services-issuer.pem and services-issuer-key.pem from the store's host and set both options.

A host fronting a swarm name without both options set still evaluates and deploys, and swarm-services-cert.service fails, naming them. One that names files it can't read fails the same unit, naming the paths. The old spellings, deploy.hive-controller.tls.baoClientCertFile and baoClientKeyFile, are renames of these two: a config that still sets them evaluates with a warning naming both paths.

services.hyperhive.swarm.serviceDomains, every swarm service name and the same list on every host, populates the role's allowed_domains, so asking for a name nobody configured is a refusal from the store naming that name — not a certificate quietly issued for it. A host's own SANs are narrower still: it asks only for the names it fronts a vhost for (deploy.hive-controller.tls.localServiceDomains), so the key on one gateway can't serve a swarm service that runs behind another.

The root's public certificate is a file, on every self-signed gateway host: /var/lib/hive-tls/swarm-services-root.pem (0644), written beside the leaf it signed and folded into trust-bundle.pem. A hive fronting no swarm name asks for no services leaf and reads the root alone, from the store's pki/ca/pem, presenting its own store identity (deploy.hive-controller.tls.baoHiveIdentityCertFile / baoHiveIdentityKeyFile) for the handshake without logging in with it. Every hive must hold one: without it, or with either file absent, swarm-services-cert.service fails with a message naming both options. On the host running the store it's also at /var/lib/swarm-bao-services-pki/services-root.pem. That's the file to hand a browser, and reading it needs no store login — which matters, because every store listener but the loopback UI one demands a client certificate, and authelia gates that one to the admins group, not open to an anonymous browser fetching a trust anchor.

The granting unit generates the root once, and never again. It asks the mount whether it already has an issuer (bao list pki/issuers) before calling pki/root/generate/internal, so a rebuild or a reboot re-asserts the role and the grant without touching the anchor. A root that changed per boot would invalidate every certificate issued under it and every browser taught to trust it.

A daily timer renews the leaf. The store's role issues it for 720 hours. swarm-services-cert-renew.timer runs the same script as swarm-services-cert.service once a day, and that script asks the store for a new leaf once the current one is past half that window, is missing a configured name, or doesn't chain to the store's root (swarm-services-root.pem). A rotation re-imports the leaf into the gateway and reloads nginx. A store that can't answer — still sealed, or out of reach — fails the unit, which leaves the current leaf in place, and the next day's run tries again: about fifteen tries before the leaf lapses. systemctl list-timers swarm-services-cert-renew shows the next run.

Constraints on the material

No CA private key exists on any hive. The pki root's key stays inside the store's mount (root/generate/internal), and each hive's narrowing is its role's allowed_domains, enforced by the store at issue time: a hive's identity that leaks can get certificates for that hive's names and no others.

No host holds a CA or its key, except the store's mTLS CA (glue-bao-tls.nix), which signs only the store's server certificate and the certificates that log in to it. Every host deletes /var/lib/swarm-ca and the ca.pem, ca-previous.pem, ca-key.pem, ca.srl and .swarm-ca-adopted in <deploy.hive-controller.tls.stateDir>: systemd-tmpfiles removes them at boot, and on a deploy that changes the host's tmpfiles.d rules. The swarm distributes only certificates.

⚠️ The deletion is irreversible. Rolling back to a generation that still has swarm-ca.service finds no root under /var/lib/swarm-ca and mints a new one, unrelated to the deleted root. That's harmless: nothing issues under it.

A host whose deploy.bao.clientCaFile, deploy.bao.serverCaFile or deploy.swarm-controller.hiveClientCaFile names one of those paths fails evaluation, however it's spelled with //, ., .. or a trailing /. A symlink to one isn't resolved, so it gets past the check. Those options name the store's mTLS CA, /var/lib/swarm-bao-pki/ca.pem on the host that runs the store.

What to hand a peer

hivectl peer-config prints the services.hyperhive.swarm.hives."<name>" block a peer operator pastes into their own config. When this hive's gateway serves a self-signed leaf (detected by the presence of <deploy.hive-controller.tls.stateDir>/trust-bundle.pem), it also prints the secret store's pki root itself, read from /var/lib/hive-tls/swarm-services-root.pem. The store issues every hive's gateway leaf and the swarm's service names under that root, so it's the one anchor a peer host needs.

The operator installs the root once per swarm host, not once per peer, so a hive joining later needs no edit on the hives already running. A hive whose cert already chains to a public CA has nothing to install — peer-config says so instead.

Distributing the root

The root certificate is a runtime file: each hive fetches it from the store at boot, so nothing can name it at build time. That has a cost worth naming, because it's subtle and it bites at a distance:

Nothing whose trust store nix assembles at build time can reference the store's root. The derivation reads security.pki.certificateFiles inside itself; the root doesn't exist there.

Two consumers, both reading the root at runtime:

• Agents fetch it themselves. hive-agent-trust-bundle.service (nix/agent-modules/hive-ca.nix) reads <mount>/ca/pem from the store with the agent's own store certificate, then writes /run/hive-agent-trust-bundle/bundle.pem (system CAs plus the root) and root.pem (the root alone). The agent's units point SSL_CERT_FILE at the first (Rust, Go and OpenSSL clients) and NODE_EXTRA_CA_CERTS at the second (Node and Claude Code), and order after the unit with Requires=. The meta flake names no CA file, and a timer re-reads the root hourly. A consumer reads it at start, so a rotated root reaches an agent unit at its next start.

• The Matrix container gets a bind mount, via the shared lib/hive-ca-trust.nix helper that hive-forge and swarm-ci-runner use: the bundle is bind-mounted read-only into the container, and a oneshot concatenates it with the system CAs before tuwunel starts.

  The consumption differs per runtime and is the part worth knowing. tuwunel links no openssl, which makes SSL_CERT_FILE look inapplicable — it's not: its rustls-based TLS stack still resolves trust through the platform certificate store on Linux, and that store honors SSL_CERT_FILE, so the env var takes effect the same way it would for an OpenSSL-linked binary.

  ⚠️ Concatenate; never point SSL_CERT_FILE at the anchor alone. openssl-probe uses it instead of the default store, so naming just the bundle would drop every public CA and break federation with the wider matrix network — a much bigger outage than the one being fixed. The same caveat applies to hive-forge (Go) for the same reason.

A hive in the swarm fetches the root itself. A host outside it gets the certificate from hivectl peer-config (What to hand a peer). One anchor per host, not one per peer: a hive joining later needs no edit on the hives already running.