Swarm secrets: what exists, and where each one lives

A swarm's credentials are generated in three different places and read in a fourth, so "where does this file go" has a different answer per deployment. This page is that answer, one row per secret.

Two rules run through all of it.

Private key material and access tokens are paths, never values. Every option carrying one takes a file path (*File), because a literal written into a nix expression is rendered into the nix store — world-readable and permanent. No option in this tree accepts one inline, and adding one would be a leak rather than a convenience.

The rule is about what must stay secret, not about credentials generally. Public material is a value: a certificate, or a public nkey like swarm.nats.calloutUserPublicKey, is published to every client that connects, so the store is a perfectly good place for it.

The generator and the reader are usually in different containers. They share the host's network namespace, which makes them feel co-located, but their filesystem roots are separate. That is why delivery is a host-side copy rather than a bind mount: nixos-container refuses to start when a bind source is missing, and a secret minted on another container's first boot does not exist yet. Binding it would make one container wait on a file that waits on a container that starts after it.

The three topologies

Every row below is read against one of these.

topology what it means who places secrets
all-local one host runs the swarm's shared services and its own hive nobody — each secret is generated where it is read, or copied by a host unit
swarm-managed the swarm's services run on a host with swarmctl swarmctl writes what it owns; the rest is still generated in place
hive elsewhere a hive that federates with a swarm it does not host the operator provides the file and names it in config

Swarm-level — one of each per swarm

secret generated by lives at hive elsewhere
swarm root CA cert swarm-ca.nix first-boot unit, when autoConfigure is set /var/lib/swarm-ca/root.pem operator copies the cert in; it is public
swarm root CA key same unit /var/lib/swarm-ca/root-key.pem, 0600 stays on whichever host holds it — see the constraint below
swarm-services sub-CA (cert + key) swarm-ca.nix, signed by the root /var/lib/swarm-ca/services-ca{,-key}.pem issued where the root lives
authelia session, JWT and storage-encryption keys authelia's first-boot unit, in-container /var/lib/authelia-swarm/{session,jwt,storage-encryption}.key generated in place; nothing outside that container reads them
authelia OIDC HMAC key same unit /var/lib/authelia-swarm/oidc-hmac.key same
authelia OIDC issuer key (RSA) same unit /var/lib/authelia-swarm/oidc-issuer.key same — relying parties verify against the public half at /jwks.json
OIDC client secret, plaintext half authelia crypto hash generate --random /var/lib/authelia-swarm/oidc-clients/<id>.secret operator provides the file and names it in whichever option reads it — sso.clientSecretFile for a service, otel.clientSecretFile for the hive's telemetry collector
OIDC client secret, digest half the same mint oidc-clients/<id>.digest authelia's own half; merged at runtime via settingsFiles
authelia subject store swarmctl and swarm-authelia-bridge users.yml — one file, read and written by both swarmctl, on the host that runs authelia
wireguard private key the operator — wg genkey whatever swarm.wireguard.privateKeyFile names always operator-provided; nothing generates this for you
queue auth-callout nkeys (user seed + account seed) swarm-nats-callout-keys first-boot unit, when nats.autoGenerateCallout is set /var/lib/swarm-nats-callout/{callout-user,issuer}.seed, 0600 operator mints both with nk and names them in nats.calloutUserSeedFile / nats.calloutIssuerSeedFile

The three keys authelia mints for itself are generated in-container precisely because nothing outside that container ever reads them. That is the test worth applying to any secret added here — and the client secret's plaintext half is the one row that fails it, which is the entire reason a delivery step exists.

One reader needs no delivery step: the hive's telemetry collector, which authenticates to the swarm's collector as its own hive. It is a host unit rather than a container, so on an all-local swarm it reads authelia's file where it lies (through LoadCredential) and no second copy is made. On any other topology it is an ordinary "operator provides the file" case — see services.hyperhive.otel.clientSecretFile.

Minting the queue's callout nkeys

nats.autoGenerateCallout mints both keypairs on the host before the queue starts. It is on by default only under enableAllLocalDefaults — the one topology where the queue, its responder and the operator are the same person. On every other topology, mint them yourself:

nk -gen user    > callout-user.seed     # the responder's own identity
nk -gen account > issuer.seed           # signs the user JWTs it hands out
nk -inkey callout-user.seed -pubout     # → calloutUserPublicKey
nk -inkey issuer.seed       -pubout     # → calloutIssuerPublicKey

Keep both seeds at 0600 and name them in calloutUserSeedFile / calloutIssuerSeedFile. Possession of the issuer seed is the authority to admit anyone to the queue, so it belongs wherever the responder runs and nowhere else.

A hive that sets neither the public keys nor autoGenerateCallout fails at eval, naming the option it wants. That is deliberate: a queue that started without them would accept CONNECT {"user":"auth"} from anyone sharing the host's network namespace, and nothing would look wrong until somebody connected.

One consequence of the generated path worth knowing before you debug it: with autoGenerateCallout set, the queue's config is assembled at boot rather than at build time, so a malformed one surfaces when the container starts instead of when the system builds. The server names the offending file and refuses to run.

Hive-level — one of each per hive

secret generated by lives at
hive CA cert + key hive-tls.nix first-boot unit <tls.stateDir>/ca.pem, ca-key.pem (0600)
hive leaf certs hive-tls.nix, signed by the hive CA <tls.stateDir>/<name>.pem
matrix registration token a host activation script, on first boot /var/lib/hyperhive/matrix-register-token (0600)
the forge's copy of its OIDC secret hive-forge-oidc-secret.service copies it from authelia's tree /var/lib/forgejo-oidc/<id>.secret inside the forge container
the homeserver's copy of its OIDC secret hive-matrix-oidc-secret.service, same shape /var/lib/tuwunel-oidc/<id>.secret, handed to tuwunel through LoadCredential

Both delivery units wait for authelia's first boot to mint the secret — a bounded wait, 120s — and then fail loudly rather than skipping. A silent skip produces a service whose login button always fails, which is a symptom several layers from its cause.

The constraint that decides where the root lives

A hive CA carries nameConstraints=permitted;DNS:<hive domain>, and a swarm service name is a sibling of the hive domain rather than a child — forge.<swarm> next to <hive>.<swarm>. So a hive CA cannot issue a certificate for a swarm service. Not by policy: by construction, and openssl enforces it.

Whatever holds the swarm root is therefore what makes swarm-service certificates possible at all. Two things follow:

Adding a secret

State three things, in the row you add above: who mints it, which container reads it, and what happens when they differ. If they differ, it needs a delivery unit, and the unit copies — it does not bind.