Swarm secrets: what exists, and where each one lives

A swarm generates its credentials in three different places and reads them in a fourth, so "where does this file go" has a different answer per deployment. This page is that answer, one row per secret.

Two rules run through all of it.

Private key material and access tokens are paths, never values. Every option carrying one takes a file path (*File), because a literal written into a nix expression ends up in the nix store — world-readable and permanent. No option in this tree accepts one inline, and adding one would be a leak rather than a convenience.

The rule is about what must stay secret, not about credentials generally. Public material is a value: a certificate, or a public nkey like the queue callout's user key, reaches every client that connects, so the store is a perfectly good place for it.

The generator and the reader typically live in different containers. They share the host's network namespace, which makes them feel co-located, but their filesystem roots are separate. That's why delivery is a host-side copy rather than a bind mount: nixos-container refuses to start when a bind source is missing, and a secret minted on another container's first boot doesn't exist yet. Binding it would make one container wait on a file that waits on a container that starts after it.

Topologies, by who places secrets

Read every row below against one of these. This is a different cut from the deployment shapes --- those say where services run, these say who is responsible for a secret file being there --- the two lists don't line up one-to-one, and neither is a renaming of the other.

Swarm-level — one of each per swarm

Authelia mints the three keys for itself, in-container, precisely because nothing outside that container ever reads them. That's the test worth applying to any secret added here — and the client secret's plaintext half is the one row that fails it, which is the entire reason a delivery step exists.

Three telemetry collectors exist, and each one fails that test: each reads its own secret out of the store, under its own identity.

The hive's collector authenticates to the swarm's collector as its own hive, so it reads its secret as the hive: swarm-bao-hive-otel-oidc.service fetches swarm/hives/<hive>/oidc/client under the hive's own leaf (deploy.bao.clientCertFile), on every topology. A hive with no leaf has no default, and the build stops at the collector's identity assertion unless services.hyperhive.otel.clientSecretFile names a file delivered some other way.

The swarm's collector reads its own under a leaf of its own, the fourth store reader — the same shape as Grafana's, described in full below. In short: swarm-bao-otel-oidc.service reads swarm/services/<client-id>/oidc/client out of the store on every host that runs the collector and holds a store identity, whether authelia is a network away or in the container next door. The same call glue-grafana-oidc-client.nix makes for Grafana's registers the client it reads for unconditionally: authelia refuses a bearer-authz client with no audience, and every host publishes at least forgejo's /metrics as a scrape target, so the audience is never empty.

⚠️ A collector with no store identity isn't refused. Unlike Grafana, where an absent OIDC block is a container with no way in at all, a collector with nothing to authenticate with still receives every hive's telemetry — its pushes to the stores go out unauthenticated and get refused there, and its scrape jobs for the published targets read a credential file nothing loads, so they can't get a token. swarm-bao-otel-oidc.service therefore doesn't render at all without deploy.bao.otelOidcClientCertFile / otelOidcClientKeyFile, the same choice glue-queue-agent-credential.nix makes for its own reader, rather than the hard assertion swarm-grafana.nix uses for its non-optional one. The operator can still deliver the secret by hand, at services.hyperhive.deploy.swarm-otel.clientSecretFile, on a collector this unit never reaches.

The secret store's own collector — the forwarder inside the swarm-bao container — needs a delivery step too, and it takes the same route with one principal of its own: swarm-bao-forwarder-oidc.service reads swarm/services/<client-id>/oidc/client under a leaf of its own and writes it to /run/swarm-bao-forwarder-oidc on the host's tmpfs, which the container bind-mounts read-only and where LoadCredential hands it to the collector. The client id is its own (services.hyperhive.swarm.bao.otel.clientId), registered by glue-swarm-bao-otel-oidc-client.nix wherever authelia runs, and the swarm collector admits it on a receiver of its own — one identity per principal, so this forwarder borrows neither the swarm collector's client nor swarm-controller's.

⚠️ This one has no degraded mode, and the difference is structural. The route it exports over accepts an authenticated request and nothing else, so a forwarder without the secret has nothing to fall back to: the unit fails and retries, and the collector refuses to start until the file arrives, after every boot, rather than exporting into a 401 while reporting healthy. Its ordering differs from the unit above it for the same structural reason — it reads the store inside the container it delivers into, so it runs after that container rather than before it. The fetch creates the file, so when the store stays sealed or unreachable after a boot the forwarder stays down, then resumes from its saved journal cursor once the store answers.

The queue's callout nkeys

The responder holds two nkeys besides the callout xkey below. It signs every user JWT it issues with the issuer key, an account key, so possession of the issuer seed is the authority to admit anyone to the queue. It connects to the queue as the user key, the one identity the server exempts from callout. You configure nothing about either: both live in the swarm secret store beside the xkey, and come into being on the queue's first boot.

• Issuer seed at secret/swarm/services/swarm-nats/callout/issuer-seed, public half at secret/swarm/public/swarm-nats/callout-issuer.
• User seed at secret/swarm/services/swarm-nats/callout/user-seed, public half at secret/swarm/public/swarm-nats/callout-user.

The unit that creates the xkey creates these too, under the same rules: only the swarm-nats-auth role reads or creates them, nothing replaces a stored seed, and the policy of swarm-secret-publisher denies it both seed paths. The server renders both public halves into its auth_callout block as it starts, and doesn't start without them.

A seed file at /var/lib/swarm-nats-callout/issuer.seed or /var/lib/swarm-nats-callout/callout-user.seed on the queue's host is what swarm-nats-auth ensure-callout-keys stores for that key when the store holds none, so the key keeps its public half. Once a read of the store returns a seed, the verb removes the file. When the store already holds a different seed, the verb keeps the store's seed, logs a warning naming the file, then removes it.

A hive that sets autoGenerateCallout, calloutUserPublicKey, calloutIssuerPublicKey, calloutUserSeedFile or calloutIssuerSeedFile, under deploy.nats or swarm.nats, fails at eval with a message naming the option and where the keys live now.

To replace either key, delete both of its paths the way the xkey's section below shows for the xkey, then restart swarm-bao-nats-callout-xkey.

One consequence worth knowing before you debug it: the queue assembles its config each time the server starts rather than at build time, in every mode, because the callout keys exist only at runtime. A malformed config surfaces when the server starts instead of when the system builds. The server names the offending file and refuses to run.

The queue's callout xkey

The server seals every auth-callout request to a curve key, the callout xkey, and the responder seals its reply back, so nothing else in the callout account or on the wire can read the request or the user JWT minted for it. You configure nothing about the key: it lives in the swarm secret store and comes into being on the queue's first boot.

• Seed at secret/swarm/services/swarm-nats/callout/xkey-seed.
• Public half at secret/swarm/public/swarm-nats/callout-xkey.

Only the swarm-nats-auth role reads either path, and it may create each but never replace it. The policy of swarm-secret-publisher, which writes the rest of swarm/services/, denies it the seed's path.

On the queue's host, swarm-bao-nats-callout-xkey runs before the swarm-nats container, under the responder's store leaf:

1. swarm-nats-auth ensure-callout-keys creates the pair when the store holds no seed, and otherwise only reads it. It needs no queue, so on a first boot the key exists before the server that needs it.
2. It reads the public half back from the store into /var/lib/nats-callout/callout-xkey.pub inside the container.
3. Each time nats-server starts, it renders that key into auth_callout.xkey. With no file, the server doesn't start.
4. The responder starts after the server and reads the seed from the store. A responder that can't read it doesn't start, rather than answering requests it can't open.

The store has to be reachable each time the responder starts. When it's down, the fetch unit fails and retries for a day. The server starts with the key fetched last time, or not at all on a first boot. A responder that starts while the store is down exits and retries every 5 seconds, and until one start reads the seed the queue refuses every client, OIDC ones included. The retry that lands the file starts or restarts the server, and the responder comes up with it.

The responder's store leaf is deploy.nats.authBaoClientCertFile / authBaoClientKeyFile. On the store's host both default to the leaf glue-bao-tls.nix mints there. A queue host without the store gets no default, even when it holds the hive's own reader deploy.bao.clientCertFile: issue a leaf whose subject is deploy.bao.natsAuthCommonName, place it, and name it in both options. Until you do, evaluation fails naming both.

Where the queue and the store run on different hosts, deploy the store's host first. swarm-bao-nats-auth-policy there writes the role's grants on the keys' paths, and swarm-bao-secret-publisher-policy the publisher's denial of the seeds' paths; where the queue runs on that host too, the fetch waits for both. A queue host deployed before it can't read or create the keys, so its fetch fails, and the queue refuses every client until you deploy the store's host: the server starts only on public keys fetched before, and the responder can't load its seeds. Seed files the import is waiting for stay where they are. If that takes longer than the fetch's day of retries, run systemctl restart swarm-bao-nats-callout-xkey on the queue's host.

Two starts that create the key at once can't both win: each write is a create that fails once the path holds anything, and the start that lost reads the winner's key back.

To replace the key, remove both paths with their metadata and every version, then restart the fetch unit on the queue's host. Its restart of nats.service restarts the responder too, which then loads the new seed:

bao kv metadata delete secret/swarm/services/swarm-nats/callout/xkey-seed
bao kv metadata delete secret/swarm/public/swarm-nats/callout-xkey
systemctl restart swarm-bao-nats-callout-xkey

bao kv delete isn't enough. It deletes the latest version but keeps the path's metadata, so the store still counts the path as existing and refuses the create that makes a new key. No principal hyperhive configures may delete either path's metadata, so run the two deletes with a token that may, such as the root token.

Hive-level — one of each per hive

The forge and the homeserver read their copies out of the store, the way Grafana does below: the forge under the forge publisher's leaf, the homeserver under matrix-ctl's. Each fails loudly and retries until the publisher in authelia's container has written the secret, rather than skipping. A silent skip produces a service whose login button always fails, which is a symptom many layers from its cause.

The store's first reader is the agent containers' queue credential: glue-queue-agent-credential.nix lands it as two files, the client secret and the client id it authenticates, because that's how a queue client reads them — the secret by path, the id by value. No local fallback exists, and none is possible, so absent files mean an agent built before swarm-logs read its own queue secret has no working swarm-logs, which is what a swarm looks like before the publisher in authelia's container has run. The reader runs before hive-c0re.service, which wants it (doesn't require it), so an agent container never renders ahead of the credential; an unreachable store delays the daemon's start rather than failing it.

That credential still has one hop left, because the reader of it lives inside an agent container. It crosses as a systemd credential, not as a bind mount, and the mode decides that: the secret is root:0600 and a harness runs as its own unprivileged agent user, so a bind would deliver a file that user can't open. hive-c0re stats the two files — the directory is 0755, so it needs no read access to either — and hands them to systemd-nspawn --load-credential through hive-priv, which runs as root. Inside, hive-agent.service names the same two ids in LoadCredential=, which lands them in its own $CREDENTIALS_DIRECTORY, owned by the agent user and by nobody else, where a swarm-logs built before it read the agent's own queue secret reads them. The harness itself doesn't: it connects to the queue with the agent's own credential. hive-c0re never reads the bytes at any point: it runs as hive-core, which is the reason a copy wasn't an option either. When the files aren't there the daemon forwards nothing and says so in its journal, and the container starts without them.

The second reader is Grafana's OIDC client secret, and it's the first one that's a swarm service's own credential rather than a hive's. It has exactly one delivery route: swarm-bao-grafana-oidc.service reads swarm/services/<client-id>/oidc/client out of the store, on every host that runs Grafana. Where authelia is doesn't enter into it — the publisher in authelia's container writes that path whether the reader is a network away or in the container next door.

That's a deliberate refusal of the shortcut. Copying the plaintext straight out of authelia's tree when the two happen to share a host is one round trip cheaper and costs a second delivery unit, a second way for the file to be wrong, and a gate deciding between them — and the gate is the expensive part, because whatever it's wrong about is an outage nobody can read. The store exists so the only secrets a host holds out of band are certificates, and a host reads everything else with them.

Two things follow, and swarm-grafana.nix asserts both rather than degrading: running Grafana requires swarm.authelia.url (this module disables its local login form unconditionally, so SSO isn't a feature of some topologies — it's the only way into the UI), and it requires this host's deploy.bao.grafanaOidcClientCertFile / grafanaOidcClientKeyFile. Each refusal names the option to set. Both used to be silent: a null URL dropped the OIDC block, a missing leaf produced a warning, and either one left a Grafana with no SSO and no password box — nothing failed, so nothing said anything. Registration is separate and stays where authelia is (glue-grafana-oidc-client.nix): a client is a row in authelia's own config, so it's declared on the host running authelia, which isn't necessarily the host running Grafana. The swarm controller, the queue, the homeserver and the forge register the same way (glue-controller-oidc-client.nix, glue-nats-oidc-client.nix, glue-matrix-oidc-client.nix, glue-forge-oidc-client.nix), and the queue's responder, the homeserver and the forge read their secrets from the store too: the responder itself under its own leaf, swarm-bao-matrix-oidc.service under matrix-ctl's, swarm-bao-forge-oidc.service under the forge publisher's. Each fails, naming the option pair, when that leaf is unset — neither the responder nor the homeserver starts without its secret, and the forge registers no login source without its.

Grafana's admin password rides the same leaf and the same store. Where Grafana and the store run on different hosts, deploy the store's host first: swarm-bao-grafana-oidc-policy there writes the grant that lets Grafana create the password, and until it has, the store refuses swarm-bao-grafana-admin on the Grafana host, which retries, while Grafana stays down because its config reads that file.

The third reader is the swarm collector's own OIDC client secret, the same shape one level down: swarm-bao-otel-oidc.service reads swarm/services/<client-id>/oidc/client out of the store on every host that runs the collector, and registration lives separately in glue-swarm-otel-oidc-client.nix for the same reason Grafana's does. Where it differs is what an absent credential means: the collector still receives telemetry with none, so swarm-otel.nix doesn't assert deploy.bao.otelOidcClientCertFile / otelOidcClientKeyFile the way Grafana does — it doesn't render the reading unit without them, the shape glue-queue-agent-credential.nix uses for its own read.

The fourth reader is the secret store forwarder's OIDC client secret: swarm-bao-forwarder-oidc.service, on the store's own host, registered by glue-swarm-bao-otel-oidc-client.nix on authelia's. It differs from the third in both directions. It renders unconditionally, because the export it authenticates has no unauthenticated mode to degrade into — and it orders itself after container@swarm-bao, because the store it reads runs in the container it delivers into. Nothing circular sits behind that: the identity it logs in with is the static forwarder-oidc.pem leaf swarm-bao-pki signs, not anything the store mints. With no leaf it has no fallback, so swarm-bao.nix refuses the build and names deploy.bao.forwarderOidcClientCertFile / forwarderOidcClientKeyFile.

A service's secret is one value for the whole swarm rather than one per hive, so it lives under the services prefix, and a hive's read policy grants that prefix whole. That's because nothing in a swarm's configuration records which hive runs a given service — placement is a deploy.* fact, per-host by definition — so there is nothing to scope the grant to. A hive's own leaf can therefore read every swarm service's client secret, and that's stated in swarm-secret-client's policy module beside the grant itself.

⚠️ The readers no longer present it. Five units used to log in with deploy.bao.clientCertFile, which is the hive's own leaf, and bao identifies a principal by the subject of the certificate it presents — so five readers behind one leaf were one principal holding the union of their needs. Each now holds a leaf of its own, and a policy naming only the path that unit reads. See per-principal identities below.

Absence behaves as it does for the other readers: a store that says "not here" leaves the file alone and says so, since there is no such thing as a locally valid OIDC client secret to stand in.

⚠️ Service↔store mTLS is its own trust domain. You can't fetch a credential you must already hold to authenticate from the thing it authenticates you to, so the store's identity can't come from an authority the store distributes — which excludes the store's pki root, and has nothing to do with the gateway's HTTPS certificates either way. glue-bao-tls.nix mints a CA that signs a fixed, short list of leaves — the store's server certificate, the hive's own reader certificate, and one per service principal — and distributes nothing. A deployment with a real internal CA deletes that file and names its own paths in deploy.bao.serverCertFile / clientCaFile; the store itself has no opinion. A hive that reads from a store on another machine names the reader's half — clientCertFile, clientKeyFile, serverCaFile — and places that leaf by hand, plus one leaf per principal it runs (the table below names the options). Those are the credentials that can't come out of the store, being what opens it; everything else a hive needs does.

Agents are the one principal whose leaf the store issues. Its pki-agents mount (deploy.bao.agentPkiMountPath) holds an agent CA generated inside the store, and swarm-controller, already logged in under its own host leaf, asks that mount's one role for a hive-agent-<agent> client certificate at agent creation. Host roles never pin that CA and agent roles pin only it, so an agent's certificate opens that agent's own swarm/agents/<agent>/* and nothing else: read on the values there and list on their names. swarm-controller writes that policy at agent creation and rewrites every agent's at its own start, so a change to it reaches existing agents with the next controller restart.

Per-principal identities

Bao matches a cert-auth role on the certificate's subject, so a certificate is an identity and sharing one merges the identities. Each of these ten leaves has a role and a policy of its own, and reaches one path, except the review classifier's, the CI runner's, the builder's, the CI run poller's, the claude login minter's, the forge host's and Grafana's, which reach more than one:

The first exists per hive, because the path it reads carries a hive name and every hive runs its own reader. Its subject is <deploy.bao.queueAgentCommonNamePrefix>-<hive>; swarm.nix reserves that composed spelling as a hive name, so nobody can name a hive into another hive's role. The other nine reach a path that names a swarm service rather than a hive, so one role each is enough and their subjects are the flat deploy.bao.grafanaOidcCommonName, deploy.bao.otelOidcCommonName, deploy.bao.forwarderOidcCommonName, deploy.bao.ciRunnerCommonName, deploy.bao.reviewClassifierCommonName, deploy.bao.builderCommonName, deploy.bao.ciMetricsCommonName, deploy.bao.claudeLoginCommonName and deploy.bao.forgePublisherCommonName. The last two are the writers, and the last is the one leaf two units share: the forge's host holds one store identity for the token it publishes and the OIDC client secret it reads.

On a host that mints its own PKI, glue-bao-tls.nix signs all nine and defaults all eighteen options, and there is nothing to do. Elsewhere you copy each leaf from the store's host and name it here: a service principal's from the top of /var/lib/swarm-bao-pki/, another hive's queue-agent pair from /var/lib/swarm-bao-pki/hives/<hive>/ (bao.md § TLS lists that directory). That's one file per principal rather than one file shared by nine, which is the whole of what this buys. swarm-ci-runner.nix refuses the build wherever the runner runs without its pair, as swarm-review-classifier.nix does for the classifier, swarm-ci-metrics.nix for the CI run poller and swarm-grafana.nix for Grafana. The forge's pair refuses nothing: without it forgejo-swarm-controller-account and swarm-bao-forge-oidc render, fail at start naming both options, and the controller runs without forge access and the forge without a login source until an operator places the leaf.

On a host without the store, where the forwarder never runs, forgetting one of the other nine isn't a quiet degrade. A hand-configured remote-store hive that names the hive's own clientCertFile and misses a principal's pair has the build refused, by the module of the reader it missed, naming that pair — see what a missing leaf costs below.

The roles of swarm-nats-auth, swarm-logs-auth and swarm-metrics-auth issue batch tokens with a five-minute TTL (token_type=batch, token_ttl=5m). The responders log in on every check, and the store keeps no record of a batch token, so unauthenticated requests don't grow its token storage. Nothing can renew or revoke a batch token. The other roles keep service tokens, because some of their readers hold a handle for longer than five minutes.

What a missing leaf costs

Each of the three modules asserts its own pair, on the same condition: this host already reads the store — it holds deploy.bao.clientCertFile and clientKeyFile — and the consumer is on, and this principal's pair is null. The refusal names the two options, so an operator acts on the message without opening the nix.

That condition fires on exactly one shape, the hand-configured remote-store hive that named the hive's own pair and missed a principal's. It stays silent everywhere else, and each half of that matters on its own:

• A host with no store identity at all holds no clientCertFile either, so the first clause is false. That's the supported no-store deployment, and also the one an operator passes through while bringing a hive up — neither should fail to evaluate.
• A host that runs the store gets every pair from glue-bao-tls.nix as defaults, so the third clause is false. Nothing to set, nothing to refuse.
• A consumer that's off — no collector, no Grafana — has no unit to skip, so the second clause is false. Only the queue-credential reader has no toggle of its own to check: every hive runs one for its own agents, so reading the store at all is what asks for it.

The collector keeps a separate degrade underneath this, unchanged: a host with no store identity runs a collector with no OIDC client secret and still receives telemetry. The assertion doesn't touch that shape, because that host has no clientCertFile either.

How a reader reaches the store

Every reader dials the same URL — https://bao.<swarm domain>:<port> — and on the host that runs the store that name resolves twice. /etc/hosts answers a host-side unit with loopback, where openbao binds; the hive's dnsmasq answers a container with the bridge IP, where nginx holds the same port. One BAO_ADDR, and the network namespace that asked decides which half of it answers.

The container's half goes through nginx and still doesn't terminate TLS. It is an nginx stream server with ssl_preread on: it reads the SNI off the ClientHello and splices the rest of the connection through byte for byte, so openbao completes the handshake itself. A vhost would decrypt here instead, and the client certificate — the store's whole authentication — would stop at nginx, leaving openbao seeing one client for every hive in the swarm. That's why the store has no vhost and gets a passthrough: not an exception to the no-vhost rule, the one shape that keeps it.

The stream listener binds the bridge IP rather than every address, because openbao already holds loopback on that port in the same network namespace and a wildcard bind would fail with EADDRINUSE — taking nginx, and with it every hive domain behind the gateway, down. network.exposeHostPorts opens the port on the bridge firewall and nowhere else.

Reaching the port grants nothing by itself: openbao answers nothing without a client certificate signed by deploy.bao.clientCaFile or by the store's own agent CA. The listener reads both from listener-client-ca.pem, which no cert-auth role pins. The passthrough carries whichever certificate the reader presents, unchanged.

The browser UI

OpenBao's built-in web UI is at https://bao-ui.<swarm domain>/ (swarm.bao.ui.domain), for members of authelia's admins group only. Log in with the OIDC tab, or with a bao token under Other. The gateway vhost checks the session with authelia and proxies to an nginx inside the store's container on 127.0.0.1:<deploy.bao.uiProxyPort>. That nginx forwards /ui/ and /v1/ to a second openbao listener on 127.0.0.1:<deploy.bao.uiPort>, answers 403 on the unseal, seal, step-down, rekey and generate-root endpoints, and 404 on everything else. ⚠️ That listener asks for no client certificate, because a browser has none to present, so on this one door a bao token is the whole credential. Anything on the store's host that can dial loopback, and any admins session through the vhost, needs only a token to use the API. Unseal from the host's bao CLI; the UI's unseal form gets a 403. On a self-signed gateway where the UI is the only swarm name its host fronts, that host's nginx doesn't wait for the store: it serves the hive leaf, or on a host that runs no hive a throwaway self-signed certificate, on the UI's name (a browser warning) until the unsealed store issues the services leaf, so the stream passthrough readers use on that host comes up with the store sealed.

OIDC login

The OIDC tab logs in through authelia as the client swarm-bao-ui (swarm.bao.ui.oidc.clientId). An admins member gets a token under swarm-operator-viewer: list and read on secret/metadata/*, and create and update on secret/data/swarm/services/review-classifier/anthropic-api-key and secret/data/swarm/services/review-classifier/acp-provider. That shows the key tree and each key's versions and timestamps, and no value: the store refuses every secret/data/ read. The two writes are the review classifier's keys, which the operator fills in by hand. Anything more needs a token.

The granter holds no sys/auth, so enabling the mount stays a bootstrap-token step. A store set up before the mount existed needs the one-time granter step again: it re-writes bao-bootstrap, which covers sys/auth/oidc, and the granter's own policy, which covers auth/oidc/. Until then swarm-bao-operator-viewer-policy writes the viewer policy, logs the step, and exits 0, and the UI offers token login only. swarm-bao-granter-role re-runs it each time the step succeeds, so OIDC login needs no further restart.

The constraint that decides where the root lives

The store issues a hive's gateway leaf through a role whose allowed_domains is that hive's domain and *. that domain, and a swarm service name is a sibling of the hive domain rather than a child — forge.<swarm> next to <hive>.<swarm>. That role can't issue a certificate for a swarm service; the store refuses it.

The store's swarm-services role issues them instead, from the root inside its pki mount. That root's private key never leaves the mount, and its certificate is the only anchor the swarm has. Each hive fetches the certificate from the store at boot, so nothing build-time can name it — the system reads security.pki.certificateFiles' value at build time, which makes it the wrong tool here. Host containers get a bind-mounted bundle assembled at boot; an agent fetches the root itself with its own store certificate.

Adding a secret

State three things, in the row you add above: who mints it, which container reads it, and what happens when they differ. If they differ, it needs a delivery unit, and the unit copies — it doesn't bind.