swarm-ci-runner: Forgejo Actions Runner

The swarm-ci-runner module runs the swarm's Forgejo Actions runner in a swarm-ci-runner nixos-container, executing CI jobs from .forgejo/workflows/ci.yml on every PR. It's a swarm service: it runs on whichever host sets it, and needs neither the forge nor a hive on that host.

For operators

Enabling it's one line: services.hyperhive.deploy.ci-runner.enable = true in the NixOS config of the host that should run it. No manual token provisioning: swarm-controller publishes the forge's runner registration token to the secret store, and the runner reads it with a store identity of its own. On the store's host glue-bao-tls.nix mints that identity; on any other host, copy ci-runner.pem and ci-runner-key.pem from the store host's /var/lib/swarm-bao-pki/ and name them in services.hyperhive.deploy.bao.ciRunnerClientCertFile and ciRunnerClientKeyFile.

Two things worth knowing before you rely on it:

Everything below this point is implementation detail: registration internals, container design, and the full security writeup.

CI checks

Several jobs run on every PR (and on workflow_dispatch for manual re-triggers), defined in .forgejo/workflows/ci.yml. The first four below are required checks (forge branch protection) — a hit on any of them blocks merge. The two vale jobs split what used to be a single check: the error-only one exists so it can become a required check without also gating on the full job's standing warning/suggestion backlog, but flipping that switch on the forge is a separate operator action, not something this table can assert ahead of it happening.

Job What it runs Required?
nix flake check treefmt formatting, cargo clippy -D warnings, cargo test, module evaluation yes
tracker-tag lint flags #NNN issue tags in source and comments (scripts/check-issue-refs.sh) yes
comment-block lint flags contiguous comment blocks over 30 lines (scripts/check-comment-blocks.sh) yes
doc-pointer lint flags docs/…md paths and relative markdown links that no longer resolve (scripts/check-doc-refs.sh) yes
attribution-trailer lint flags Co-Authored-By trailers with Claude/Anthropic attribution in PR commits (scripts/check-attribution-trailers.sh) not yet — intended to be
prose lint (vale, errors) vale, docs/ tree, error-severity rule hits only not yet — intended to be
prose lint (vale) vale, docs/ tree, full severity range (warnings + suggestions too) no — advisory, standing backlog

hive-forge ci-rerun --pr N dispatches a workflow_dispatch retrigger without an empty commit.

ci-rerun --pr verifies the code but doesn't reliably move the PR's own status checks. Measured directly (raw GET /repos/.../actions/tasks JSON): a genuine PR-triggered run carries event: pull_request with the #<n> pseudo-ref as its head_branch; a ci-rerun --pr-dispatched run carries event: workflow_dispatch with the real branch name instead. Forgejo's PR commit-status tracking appears scoped to pull_request-event runs specifically, so a dispatched run — even a genuinely successful one — never writes to that status slot. Reproduced with three dispatches on one PR over 20+ minutes, all genuinely green, the PR's own status unmoved throughout. Always re-check pr status after a dispatch and believe what it says — don't push a commit just to unwedge it, that's the empty-commit anti-pattern this verb exists to avoid. If hive-forge ci-log --run <n> --job 0 shows the code green (verdict is the last line) but the status is stuck, ask the operator to select "rerun" in the forge web UI — it's CSRF-gated, so only they can do it.

Running checks locally

Don't run nix flake check directly — it dispatches to the shared build farm and wastes a remote-builder slot. Use the devshell equivalents instead:

nix develop -c cargo clippy --all-targets -- -D warnings
nix develop -c cargo test
nix fmt                         # authoritative; the formatter set is nix/treefmt.nix
sh scripts/check-issue-refs.sh  # tracker-tag lint
sh scripts/check-comment-blocks.sh  # comment-block lint

scripts/pre-push provides a git pre-push hook that automates the two lint checks. Install it once per clone:

ln -sf ../../scripts/pre-push .git/hooks/pre-push

After that, any git push automatically runs both lints and aborts with a diagnostic if either fails — catching the issue locally before CI sees it. Note that the hook does not run cargo clippy or cargo test (those are slow); run those manually before pushing Rust changes.

Configuration reference

services.hyperhive.deploy.ci-runner.enable = true on the host that runs it (see For operators above). The runner reaches the forge at https://<swarm.forge.domain>, so that name has to resolve from the host: the gateway's dnsmasq answers it locally where the forge runs, and forwards it to the operator's DNS everywhere else.

Optional tuning under services.hyperhive.deploy.ci-runner: name (runner name in the forge admin panel), concurrency (parallel job capacity), labels (workflow targeting; the default hive-ci:host is what this repo's workflows name), jobTimeout (per-job wall-clock cap, default "1h", Go duration string for example "3h" — a job that exceeds it's killed so a hung or runaway build can't hold the runner's single slot indefinitely).

The old services.hyperhive.deploy.forgejo.ci.{enable,name,concurrency,labels,package,jobTimeout} and services.hyperhive.swarm.forge.ci.{enable,name,concurrency,labels,package} are still accepted: each forwards to the same-named option under deploy.ci-runner, and evaluation warns naming both paths. Set the new name to silence the warning. The two name defaults differ (hive-ci before, swarm-ci-runner now), so a host that never set the old name shows up under the new default.

The forwarded enable carries the new module's requirements, not the old module's: evaluation refuses a host that sets only the old enable unless services.hyperhive.deploy.bao.ciRunnerClientCertFile and ciRunnerClientKeyFile are both set. They default to the identity glue-bao-tls.nix mints when services.hyperhive.deploy.bao.enable is true; elsewhere place them as described under For operators. The runner no longer needs deploy.forgejo.enable or a hive on its host.

Container design

Registration

  1. swarm-controller (swarm-controller/src/forge/runner_token.rs) reads the forge's instance-wide token from GET /api/v1/admin/runners/registration-token at start and every five minutes, and writes it to the secret store at swarm/services/ci-runner/registration-token when it differs from the value it last wrote. Its store grant on that path is write-only.

  2. swarm-bao-ci-runner.service on the runner's host logs in to the store with the ci-runner leaf, whose swarm-ci-runner role reads that one path and nothing else, retrying every 30 seconds until the token is there. It writes the token to a root-only file inside the container, and restarts the runner when the value changes. It runs at boot, on deploy, and every ten minutes on a timer.

  3. swarm-ci-runner-register.service in the container is the only unit that loads that file, with LoadCredential=. systemd hands a unit's credentials to all of its processes, and for a host label act_runner runs every job step as a process of the daemon with the daemon's whole environment, so the daemon unit carries the token in neither. The register unit is a oneshot under the daemon's dynamic user and state directory. It reads the token from $CREDENTIALS_DIRECTORY and runs act_runner register only when .runner is missing, so the .runner it writes is the daemon's own. The runner unit (gitea-runner-swarm) starts it in a pre-start step and runs the daemon once it succeeds. A root step of the register unit, ahead of registering, removes .runner in three cases:

    • The forge has no record of the runner. Forgejo answers every call from an unknown runner with unregistered runner, and the daemon exits 1 when its startup Declare gets that answer. On the next start the step finds the message in the daemon's journal since its last start, and registers again. A daemon that's already running when an admin deletes its runner keeps polling and logging the error without exiting; the next restart of the unit or container registers it again.
    • .runner names another forge address than the configured one.
    • .runner names other labels than the configured ones, compared in the form the daemon writes back: <name>:<scheme>, with host for a label that names no scheme.

    A changed token alone doesn't register again: the daemon authenticates with the per-runner token in .runner. Without .runner and with no token yet the register unit fails and the runner unit's pre-start step with it; Restart=on-failure on the runner unit starts both again every 15 seconds, with no start limit.

    Host jobs run as the runner's own user, so they can still read .runner and act as this runner; see the security section below.

A new registration is a new runner on the forge: the previous entry stays in the forge admin panel as offline until an admin deletes it. The runner the old hive-ci module registered is one such entry, since this runner registers under its own name and token. Delete it under Site administration, Actions, Runners.

Actions mirrors

With services.hyperhive.deploy.forgejo.actionsFromSelf enabled, the swarm controller seeds actions/checkout and actions/upload-artifact pull-mirrors (from github.com/actions/*) on the forge and Forgejo's DEFAULT_ACTIONS_URL points at the forge itself. CI uses: actions/<name>@vN steps then resolve on the forge, with no external DNS on the CI critical path. A workflow that names any other bare actions/* action fails at job setup with "repository not found" until that action joins the list in nix/host-modules/hive-forge/default.nix.

A forge host that runs the swarm controller therefore resolves those actions from itself, with or without a runner on that host. Set actionsFromSelf = false there to resolve actions from data.forgejo.org and seed no mirror.

The option defaults to whether the forge's host also runs the swarm controller (services.hyperhive.deploy.swarm-controller.enable; set by glue-forge-controller.nix): on when it does, off otherwise. Setting it true on a forge host without the controller leaves the actions unresolved, because nothing seeds the mirrors; evaluation warns about it.

The swarm controller seeds the mirror in its swarm-wide forge-objects pass (swarm-controller/src/forge/objects.rs), beside the orgs it ensures. glue-forge-controller.nix forwards the effective mirror list as SWARM_CONTROLLER_FORGE_MIRRORS in the swarm-controller service environment (JSON-encoded [{upstream, dest}] list). It reads the list of the host the controller runs on: mirrors declared on any other host aren't seeded, and evaluation warns about them.

General-purpose mirrors: you can pre-seed any external repo as a pull-mirror via services.hyperhive.deploy.forgejo.mirrors:

services.hyperhive.deploy.forgejo.mirrors = [
  { upstream = "https://github.com/actions/checkout"; dest = "actions/checkout"; }
  { upstream = "https://github.com/example/tool";    dest = "mirrors/tool"; }
];

The controller creates each entry as a real Forgejo pull-mirror — not a one-off clone. Forgejo re-syncs the mirror on every pull (git-upload-pack request), so a DNS blip during that sync will propagate back to the runner as a hard git clone failure. The controller autocreates the <owner> org in dest. Keep mirror dests out of the hive-c0re-managed namespaces (config/, shared/, agents/, core/) to avoid provisioning collisions.

Security: unsandboxed builds and trusted contributors

The runner should only run CI for trusted contributors. The security boundary is weaker than it looks:

What unsandboxed builds mean

nspawn containers can't create user-namespaces, so the module sets nix.settings.sandbox-fallback = true in the container. This means every nix build (and nix flake check) runs without a build sandbox — the build process has full access to the container filesystem, network, and any bind-mounts during the build phase.

A malicious default.nix or build script in a PR can therefore:

No forge admin token enters the container: swarm-controller holds it and publishes only the registration token, which only swarm-ci-runner-register.service loads, so a job step can't read it. A build process can still reach the forge over the network, and it runs as the runner's own user, so it can read .runner and act as this runner; it can't issue admin API calls.

Note: nix flake check --no-build (eval-only) reduces the attack surface but doesn't eliminate it — builtins.fetchGit, builtins.fetchurl, and import-from-derivation can reach the network and filesystem during evaluation. The default CI workflow runs full nix flake check (builds derivations), which is the higher-risk path.

Mitigation

For a hive used by a single operator or a small trusted team, the risk is low — all contributors are already trusted with forge access anyway.

For repos with external contributors or fork PRs:

The current design is appropriate for a trusted-team hive where all contributors have implicit forge access.

The CI runner builds derivations through the host nix-daemon — the swarm-ci-runner container shares the host store and has no daemon of its own. Build outputs accumulate in /nix/store with no automatic collection, and a busy CI day can fill the disk until every job fails fast with ENOSPC.

Store GC is a host-level concern, so it belongs in the host's own NixOS configuration, not in the hyperhive service modules — a single service should not reach out and change the host's global nix-daemon options. Add the following to your host config:

{
  # Daily GC: delete store paths not referenced by a live root and older
  # than a day. Keeps the store bounded between builds.
  #
  # `randomizedDelaySec` because a fixed daily time lands on the same wall
  # clock as whatever else runs nightly. A collection that overlaps a CI
  # job doesn't fail it — the runner builds through the host nix-daemon,
  # so an in-flight build's temp roots are protected — but the first build
  # after a collection rebuilds what the collection deleted, so a run that
  # lands just behind one pays for it in wall time.
  nix.gc = {
    automatic = true;
    dates = "daily";
    randomizedDelaySec = "45min";
    options = "--delete-older-than 1d";
  };

  # Disk-pressure GC: when free space drops below min-free mid-build, the
  # daemon collects garbage up to max-free before continuing. This is the
  # real-time net the daily timer can't provide — a same-day build burst is
  # what fills the disk. Tune to your disk size.
  nix.settings.min-free = 20 * 1024 * 1024 * 1024;  # 20 GiB
  nix.settings.max-free = 50 * 1024 * 1024 * 1024;  # 50 GiB
}

Measure a collection on your own hosts before relying on that stagger. randomizedDelaySec spreads the start, not the run, and the sweep over unused .links entries dominates the run — which scales with the size of the store, not with what this collection deletes. On a large store it can take most of an hour, so two hosts both set to daily may overlap however much jitter you give them. journalctl -u nix-gc.service reports each run's wall clock.

Remote builders: if CI dispatches builds to a remote builder (for example via nix.buildMachines / ssh-ng://), the build outputs land in that host's store, so apply the same GC config wherever the builder runs — GC on the coordinator host won't reclaim space on the builder.

⚠️ Two stores also means the temp-root protection above doesn't span them. Each daemon holds roots only for the builds it runs, so a collection on the builder knows nothing about a coordinator build waiting on a path there. Stagger the two hosts' GC windows rather than giving them the same dates, and prefer min-free/max-free on the builder — disk-pressure collection at least runs while nothing is mid-copy.

References