CI metrics
swarm-ci-metrics puts the forge's Actions run times into the swarm's metrics
store, one sample per finished run, so Grafana's hyperhive · CI runs board
can show run time and queue wait per repo, workflow and PR, over any range back
to a repo's first run.
Where it runs
A oneshot swarm-ci-metrics.service on a 5-minute timer, on the host that sets
deploy.ci-metrics.enable. allSwarmServices defaults it on wherever the host
has the poller's store identity, which the host that runs the store has. Run it
on one host per swarm: each poller keeps its own cursor, so two of them push
every run twice.
It reaches the forge, authelia, the store and the collector by name, so it needs
none of them on its host. On a host without the store, its leaf
(deploy.bao.ciMetricsClientCertFile and ciMetricsClientKeyFile) is the one
file to copy, as for the CI runner.
With a self-signed gateway, those names serve leaves the swarm's own CA issued.
The poller reads them through swarm-ci-metrics-ca-bundle.service, which joins
the system CAs and the host's trust-bundle.pem into
/run/swarm-ci-metrics-ca/trust-bundle.pem and sets that file as the poller's
SSL_CERT_FILE (CA). Without it, every pass fails
with invalid peer certificate: UnknownIssuer.
What it reads, and as whom
| what | where | identity |
|---|---|---|
| its forge token | swarm/services/ci-metrics/forge/token in the secret store, once per pass |
its store leaf, subject deploy.bao.ciMetricsCommonName |
| its OIDC client secret | swarm/services/swarm-ci-metrics/oidc/client in the secret store, once per pass |
the same leaf |
| the repos | the forge's repo search: every repo with Actions on, neither archived nor a mirror | the forge account ci-metrics, with that token |
| each repo's runs | the forge's runs list, newest first, down to the repo's cursor, at most a page a second | the same account |
swarm-controller creates the ci-metrics account and mints its token with the
scope read:repository, the way it does for the builder. It also keeps a
ci-metrics team in every organisation, with read on the Actions unit of every
repo in it, and the account as its one member. The poller therefore covers:
- every public repo, which needs no grant;
- every private repo an organisation owns, through that team;
- no private repo a user owns. The forge grants a user's repo only to a collaborator, and a collaborator reads every unit, code included.
What it pushes
Two gauges, pushed over OTLP to https://<swarm.otel.domain>/swarm-ci-metrics
as its own OIDC client (swarm.ci-metrics.clientId). The swarm collector
forwards them to the metrics store. Each sample carries the run's stopped
time as its timestamp, which the store keeps, so a run from last month lands
last month.
| metric | value, in seconds |
|---|---|
hyperhive.ci.run.duration |
stopped − started |
hyperhive.ci.run.queue_wait |
started − created |
Both carry the labels repo (owner/name), run (the run number), workflow
(the workflow file), pr (the PR number, or the ref for a run outside a PR),
sha (12 characters), outcome (success or failure) and event, plus
service.name="swarm-ci-metrics" and the collector's swarm. With run among
them, every run is a series of its own, and the board reads each with
last_over_time.
A run becomes no sample when its status is cancelled or skipped, it has no
start (or the epoch as its start, which the forge writes for a run cancelled
before it started), or it stops no later than it starts. A run still going a day
after its created time counts as abandoned. The journal line for each repo counts
these by reason.
The cursor and the backfill
The poller keeps one file, cursor.json, in its state directory
(/var/lib/swarm-ci-metrics). Per repo, it holds a run number below which the
poller has pushed or skipped every run, and the run numbers above it already
handled. A run still going holds that number, so a run that finishes after a
newer one still gets pushed.
With no file, the cursor of each repo is at run 0, so the first pass is the backfill: it walks every run back to the first, a page of 50 a second. For a repo with 5000 runs that takes a few minutes.
The poller writes the file only after the collector accepted a push, never before. A crash between the two pushes those runs again on the next pass. The store then holds the same point twice in the same series, which changes no panel. Deleting the file re-runs the backfill the same way.
A failed repo leaves its cursor where it was, the pass goes on with the next
repo, and the unit fails, so systemctl status swarm-ci-metrics shows it. The
timer runs it again 5 minutes later.
The board
nix/host-modules/swarm-grafana/dashboards/ci.json, with a repo and a workflow
variable:
- mean and 90th percentile run time of successful runs, over time;
- queue wait over time, mean and longest;
- finished runs per day, by outcome;
- a table per PR over the time range: mean and longest successful run time, runs, failed runs.