Review classifier

⚠️ PLANNED, NOT IMPLEMENTED YET. This page describes the full target shape of the review classifier at component level, and none of it exists in the tree today. The forge epic "pr review classifier pipeline" tracks the work. Details land here as each part ships, and this banner goes away when the epic closes.

The review classifier checks a pull request against short, named yes/no rules that the repo itself defines, and posts the result as a forge review. A small model answers one rule at a time, so mundane rule checks come out the same way on every PR.

It doesn't replace a full review by a human or agent reviewer. A swarm can request that review only once the classifier has passed the PR, and use a larger model there. The classifier takes the cheap, repetitive checks off that reviewer.

Repo configuration

A repo opts in with a rules file at its root, in YAML as .hyperhive/review-classifier.yml or in JSON as .hyperhive/review-classifier.json. It holds one of the two; with both, the rules don't load. swarm-controller always reads the file from the repo's default branch, never from the PR, so a PR can't edit the rules it's judged by. It reads it with its own forge token on every load, with no cache, so a rule merged to the default branch counts from the next load on.

The file holds one key, classifications: a list of one to ten entries, each with exactly these four keys:

Any other key, at the top or in an entry, is an error: a misspelt key fails the load instead of vanishing. An empty list, more than ten entries, an empty name or description, and two entries with one name are errors too. Nothing falls back to a pass: a repo whose file doesn't load has no rules.

An example:

classifications:
  - name: datetime
    description: All datetimes carry time zone information.
    polarity: positive
    threshold: 80
  - name: no-ads
    description: >-
      A commit message contains a co-authored-by line or other
      advertising for an AI model.
    polarity: negative
    threshold: 90
  - name: bao-first
    description: Any added secret reaches its reader through the bao store.
    polarity: positive
    threshold: 70
  - name: docs-updated
    description: >-
      A change to what an operator or user sees, such as a command, flag,
      option, endpoint, service or default, updates the matching page under
      docs/ or the crate's README in the same diff. A diff that changes
      nothing an operator or user sees makes the question not applicable.
    polarity: positive
    threshold: 65

Trigger and lifecycle

The classifier has a dedicated forge service account, review-classifier (see The service account). Requesting that account as a reviewer on a PR starts a run:

  1. The forge sends the pull_request_review_request delivery to swarm-controller, the swarm's one webhook owner, on its instance-wide hook /webhook/forge/review-request. A delivery for any other reviewer, or a removed request, does nothing.
  2. The controller reads the PR and the repo's rules from the default branch, and hands the run to the classifier service as one queue request on $SWARM.review-classifier.try-out, the subject a try-out uses.
  3. The classifier runs every classification against the PR's commits and diff.
  4. The controller posts the result on the PR as the service account's review, on the commit it read.

When a post can't reach the secret store or the forge, because the connection fails or a try gets no answer within two minutes, the controller tries it up to six times over about five minutes, enough to outlast a swarm-bao restart, with the same verdict. It doesn't retry any other failure, such as the forge refusing the review. A post that still fails ends the run as post-failed, and nothing appears on the PR. The account's review request is still pending then, and the forge sends no event for a request that's already standing: remove the service account's review request, then request it again to start a fresh run.

A verdict belongs to the commit it ran on. A push starts no run. Once the PR's head moves, requesting the service account as a reviewer again starts a fresh run on the new head.

Requesting the account again on a head it already reviewed depends on that head's newest recorded review, as the run store holds it. When that review approved or requested changes, nothing runs: the account posts the same verdict and result table again, opening with "Already reviewed <sha>, verdict unchanged." When it had no clean verdict, or its post failed, the request starts a fresh run. A head with no recorded review, such as one whose runs have dropped out of the newest 100, or a run store the controller can't read, gets a fresh run too.

The re-request also retires the account's earlier review. The forge clears the official flag on a reviewer's earlier reviews when it's requested again, and only official reviews count toward the required approvals or block a merge. Removing the request before the new review lands restores the earlier one. A repo whose branch protection also dismisses stale approvals (see PR review gate) stops counting an approval on an older commit even without a re-request.

How many requested reviews run at once is a swarm-wide setting, two by default (see Swarm-level settings). Up to eight more wait in the controller's memory, a bound that isn't configurable, so a controller restart drops them, and nothing gets posted for them. Waiting reviews start in no guaranteed order. A request for a PR and head that's already running, waiting or re-posting starts nothing, and the head stays taken until the controller writes its record. A request with every running place and all eight waiting places taken starts no run: the account posts a **No verdict:** comment saying so.

Components

The classifier service

A swarm-level service in its own container, on whichever host the operator puts it. It talks to swarm-controller through async messages only, and gets its secrets from the swarm's secret store, bao (bao.md, credentials.md).

It looks like an agent from the inside, with three things missing: it has no full agent identity, no state directory and no tools. Every model session is ephemeral, and nothing carries over from one run to the next.

On the API key runtime, each classification is one claude --print the service spawns through the hive-claude crate directly, with no session to resume, no tools (--tools ""), no MCP server (--strict-mcp-config and no config) and an empty directory of its own. The prompt reaches it on stdin as one user message in --input-format stream-json, so that it can carry content blocks (see Shared context).

The claude is the one its host's services.hyperhive.deploy.hive-controller.claudeCodePackage pins for agents, or its container's own nixpkgs' where the host pins none.

On the ACP runtime, each classification is one prompt on a new opencode acp session. Its config and opencode's state live in a directory of its own, removed after it. Its working directory is one empty directory that every classification shares: opencode writes that path into the system message, so one path keeps the system message the same from one classification to the next, and the provider's prompt cache can hold. The service drives it through hive-runtime's AcpRuntime, as the subagent daemon does a one-run subagent, and stops the agent when the classification ends. The agent has no tool. opencode runs some tools, read among them, without asking the client, so the service sets opencode's own permission to deny every tool (OPENCODE_PERMISSION={"*":"deny"}), which leaves the model with no tool offered at all, and refuses any permission request on top. The config it writes for opencode names one provider and model, from the settings, and turns off opencode's title call, which would send the prompt a second time. It also defines the agent the session runs as, whose prompt is the classifier's system prompt, so opencode sends it as the system message, with only its environment block (model, directory, platform, date) after it. The first user message holds the PR alone. That agent has no permission of its own: opencode merges an agent's permission over OPENCODE_PERMISSION, so one there would offer the model tools again.

The service is swarm-review-classifier, and its container is the one services.md describes. It holds no forge credential: swarm-controller reads the PR and sends it along.

Swarm-level settings

The runtime, the model, the effort and how many requested reviews run at once are one document for the whole swarm. swarm-controller keeps it in its review-classifier-settings KV bucket under the key all, and no other service reads it. With no document stored, the classifier runs on the API key runtime on haiku at claude's default effort, two reviews at a time. The controller ignores a stored field it doesn't know, and a document it still can't decode counts as none, with a warning in its log, until the next PUT replaces it.

GET /api/review-classifier/settings answers the settings in force:

{
  "model": "haiku",
  "effort": null,
  "max_concurrent_reviews": 2,
  "runtime": "api-key",
  "acp": null
}

PUT /api/review-classifier/settings takes any of these fields, changes only those, and answers the whole document it stored. A field left out keeps its stored value; null sets effort or acp to none. A form that sends the model, effort and limit alone therefore leaves the runtime as it was. No secret goes in the document: an api_key field is an unknown field.

The controller reads the settings for every run and sends the runtime, model and effort in the run request, so a change applies from the next run with no restart. A try-out reads them when it's posted, a review when it gets its place. Lowering the limit lets the runs already going finish. A run request with no model runs on haiku, and one with no runtime on the API key.

The PUT needs the Remote-User header the gateway sets, as the try-out POST does. Every error is problem+json with the reason in detail:

When the controller can't read the settings, a try-out answers the same 503 or 500, and a review posts a no-verdict comment. The operator writes each runtime's key by hand in the secret store (services.md).

The per-agent equivalent is inference.md, where the controller writes model and effort to a queue KV bucket (swarm-wire/src/agent_inference.rs) and each agent reads its own key.

The service account

A dedicated forge user, separate from swarm-controller's site-admin account and from every agent's account. It's the reviewer someone requests, it authors every classifier review, and it pushes the branches the try-out section creates.

swarm-controller creates it as review-classifier, with the email review-classifier@services.hyperhive.local and no repo creation, and mints its one token, swarm-review-classifier with scope write:repository, into the store at swarm/services/review-classifier/forge/token. A pass at start and every five minutes re-mints when the token is missing or no longer matches the forge. The controller refuses an existing account by that name with any other email. The controller posts the reviews and pushes the try-out branches with that token, and the classifier service never sees it.

Giving a repo access is a repo admin's step, by hand. Nothing grants it automatically. In the repo's settings, add review-classifier as a collaborator:

The try-out's open a PR pushes a branch to the repo itself, so it needs write whatever the branch protection says.

The repo also needs a rules file on its default branch (see Repo configuration). Without one, a request posts a comment saying so.

The classification run

Shared context

Every classification in a run sees the same context, in the same order:

  1. a short system prompt
  2. the PR's commits, each with its full message
  3. the full diff

Only the last part differs per classification: that classification's description and the task to classify the PR against it. The identical prefix lets the runtime's prompt cache reuse it across questions.

The controller builds the diff itself, from each changed file at the PR's merge base and at its head, with 25 lines of context around each change rather than the 3 of the forge's .diff. An enclosing gate, such as a config = lib.mkIf … or an option's default, is in view when it's within 25 lines of the change, so a move that restates it doesn't read as adding one. A gate further away is still out of view. At review time the controller takes the merge base from the PR's commits: the one parent that isn't a PR commit. A rebased or force-pushed PR therefore gets the diff against the base it sits on now, not the base it started from. The controller sends the forge's .diff instead when the PR changes more than 100 files, when the commits don't name a single merge base (the head merges the base branch in), when reading a changed file fails, or when the wide diff is over 256 KiB. The review then says so, with the reason.

On the API key runtime the commits and diff are one content block of the user message and the question another, with a cache breakpoint at the end of the first. The cache writes an entry only where a breakpoint is, and claude sets one only after the last block, so that breakpoint is what lets the next question read the commits and diff from the cache. The PR stays in the user message rather than the system prompt, which tells the model to judge it as data. A shared context shorter than the model's minimum cacheable length is not cached at all.

No object hash is in that prefix: the header names the repo and the PR number, not the head commit, each commit's heading gives its position instead of its sha, and the diff loses each file's index <hash>..<hash> line, keeping its diff --git paths and every hunk. A rebase rewrites all of those hashes, so with them in, a rebased PR with the same messages and hunks would miss the cache from the first line. The head commit still goes to the controller, which posts the review against it.

One question, one answer

The model answers each classification in JSON. The answer holds:

An answer:

{
  "verdict": "no-match",
  "certainty": 92,
  "reason": "Two new timestamps use naive datetimes without a zone."
}

The verdict field holds match, no-match or not-applicable, and the certainty field a whole number. The classifier reads the object from the reply, or from the first code fence in it. When an answer isn't that object, the classifier asks that classification once more, with a reminder to reply with the JSON object alone; the other classifications aren't asked again. The second ask runs on what the first left of the classification's three minutes, so a retry never lengthens the run, and it doesn't happen when the first ask leaves less than 20 seconds. The classifier also doesn't ask again after a runtime error or a timeout. A runtime error, a timeout, or an answer that still isn't that JSON object after the second ask leaves the classification without a verdict: it's listed with its error, and doesn't count. A run with no key for its runtime, or in which no classification got an answer, fails as a whole, and shows its error instead of a verdict.

Aggregation into the verdict

The classifier collects the answers into one result, keyed by classification name, and logs it. A classification counts toward the overall verdict only if it's applicable and its certainty reaches the threshold. The table still lists every classification below its threshold or not applicable, marked as excluded.

A counted classification fails when its verdict lands on the wrong side of its polarity: no match on a positive one, match on a negative one. Any failing classification makes the verdict request changes. Otherwise the verdict is approve, including when the run counted no classification at all.

The review

The service account posts the verdict as a forge review, approve or request changes, and the table is the review's body. An illustrative table:

result check certainty reason
❌ no match datetime 92 / 80 Two new timestamps use naive datetimes.
✅ no match no-ads 99 / 90 No commit message carries an AI attribution.
❔ unsure bao-first 95 / 70 The PR adds no secret.
❔ match docs-updated 41 / 65 Below threshold.

Overall: ❌ request changes

The result cell is the symbol for what the row did to the review, then the model's verdict. ✅ is a counted row that passed its polarity and ❌ a counted row that failed it, so match is ✅ on a positive rule and ❌ on a negative one. ❔ is a row that doesn't count: below its threshold, not applicable (shown as unsure), or without an answer. The certainty cell reads certainty / threshold.

The approval is a real one: it counts toward a repo's required approvals like any other (pr-review-gate.md). The Repos that rely on it typically raise their required approval count to account for it.

A run with no clean verdict never approves. When the PR or the rules don't load, the run fails, the classifier answers a different number of classifications than the controller sent it, or any classification has no answer, the account posts a comment instead, starting with **No verdict:** and the reason, followed by the table when there is one. A comment neither approves nor blocks the merge: nine timeouts and one pass post a comment, not an approval.

The controller flattens every cell to one line and backslash-escapes the characters that could end the cell or start markup, math or raw HTML: |, \, backticks, [, ], <, >, &, *, ~, $, and _ except between two letters or digits (ci_metrics.rs stays as is). Ordinary punctuation such as (, ) and . passes through unescaped. Each @, #, !, :// and www. gets a zero-width space after it, with no backslash, so a model's reason can't break the table, embed markup, mention anyone, reference an issue or PR, or carry a link. The prompt asks the model for plain text in reason.

Runtimes

The classifier runs on one of two runtimes, the settings' runtime:

Model names don't carry across runtimes. A claude alias means nothing to an ACP provider, so each runtime has a model field of its own.

To switch to ACP, an admin:

  1. writes the provider key in the secret store's UI under the OIDC login, at secret/swarm/services/review-classifier/acp-provider, field api_key

  2. PUTs the runtime and the provider:

    {
  "runtime": "acp",
  "acp": {
    "base_url": "https://llm.example/v1",
    "model": "Qwen/Qwen3-35B",
    "prices": { "input": 0.5, "output": 2 }
  }
}

The next run goes to that provider. {"runtime": "api-key"} switches back and keeps the provider for next time.

A split deploy runs on the API key: a classifier older than the ACP runtime ignores runtime and runs every classification on claude, and a controller older than it ignores the stored runtime and acp and sends neither.

A third runtime, claude subscription (the claude CLI on a Claude subscription login), comes later.

Open point. The subscription runtime needs a login, which is the one piece of durable state the service would keep. A swarm-level claude subscription login may have to exist before this runtime works.

Swarm UI

Two pages in the swarm UI, admins only like the rest of it.

Recent runs

One card per run: the repo, the PR, and how many classifications ran. Selecting a card expands it to show each classification's result, the same rows as the review table.

Try-out

A page for writing classification prompts without touching the repo. Pick a repo and a PR, write or edit a classification, and run it against that PR; change the prompt and run it again until it answers the way you want. Nothing reaches the forge while you iterate.

The page posts the run to swarm-controller (POST /api/review-classifier/try-out), which reads the PR with its own forge token and sends it to the classifier as one queue request on $SWARM.review-classifier.try-out. The page then reads the result back (GET /api/review-classifier/try-out/{id}): a run can take longer than a request stays open. The controller keeps the result in memory for an hour, and the finished run goes into the run store.

GET /api/review-classifier/rules?repo=owner/name returns the repo's own rules, for the page to load into its editor; a run posts them like typed ones. The route answers:

The page's repo and PR fields offer values to pick from, and stay free text, since a run also works on a merged or closed PR. Two routes feed them:

Every error is problem+json with the reason in detail.

When it's ready, open a PR (POST /api/review-classifier/open-pr, with the repo and the classifications) has the service account push a branch that adds the classifications to the repo's rules, with you, the swarm UI user, as commit author. The page then links you to the forge's compare page for that branch, where you log in as usual and open the PR yourself.

The route answers 201 with branch, path (the file written) and compare_url, or:

Run store and retention

swarm-controller records every run when it ends: a try-out when the classifier's reply arrives, a requested review once the controller has posted its review or the post has failed, and a request turned away because the queue was full once its comment is on the forge or the post has failed. The controller records a re-posted verdict the same way, with the outcome and answer of the run it repeats and that run's id in its detail. A review request for a head that's already running, waiting or re-posting starts nothing and isn't recorded. A run still going isn't in the list, and a controller restart drops a run in progress.

The newest 100 runs are one JSON array under the key recent in the review-classifier-runs KV bucket on the swarm queue, which only the controller reads or writes. The controller creates the bucket on first use and adds each run with a compare-and-set write, so the list survives a controller restart or a move to another host. Recording the 101st run drops the oldest; there's no time-based retention. The controller leaves a run it can't decode, such as one a newer build recorded, out of the list with a warning in its log. It keeps the run in the store as it was and counts it toward the 100. A document that isn't a list reads as no runs. The controller logs a failed write, and it changes nothing on the forge.

GET /api/review-classifier/runs returns the list, newest first. Like the try-out GET, the gateway's admins rule gates it, and the controller checks no header. Each run has:

A run holds no diff, no commit text and no token count or cost. The route answers 503 when the controller has no swarm queue or can't reach the bucket, and 500 when it can't read the list, both as problem+json.

Each recorded run is also one info event from swarm-controller with the message review classifier run finished. The event carries the same fields, with rows as JSON. It reaches VictoriaLogs through the host journal, where each field arrives as F_ and the name in capitals (F_OUTCOME, F_REPO), so a run whose write failed is still there:

$ swarm-logs query '_time:7d _SYSTEMD_UNIT:="swarm-controller.service" _msg:="review classifier run finished" | fields _time, F_TRIGGER, F_REPO, F_PR, F_OUTCOME, F_DETAIL, F_REVIEW_URL' --format json

Add a filter such as F_OUTCOME:="post-failed" before the | to narrow it.

Observability and cost

The swarm UI shows no token counts, cache counts or cost, on any page. On the claude runtimes, every classifier run starts its claude process with OTEL_RESOURCE_ATTRIBUTES naming the repo and the PR, so the cost reads from the metrics store by repo and PR. The process exports to the hive-tier collector on the classifier's host, where that host runs one. On a subscription, cost is notional.

On the ACP runtime, the service computes each classification's cost itself, from the settings' acp.prices and the token counts the agent reports for the prompt, and adds it to the counter hyperhive.review_classifier.cost.usage, labelled repo, pr, model (the provider's) and currency (USD). It exports to the same collector claude does. A classification stopped at its deadline, or whose agent reports no token counts, adds nothing. opencode reports no cost of its own for a provider without prices in its config, which is why the prices are a setting.

Each such classification also adds the tokens it used to the counter hyperhive.review_classifier.token.usage, with the same labels bar currency and a type of input, output, cacheRead or cacheCreation. Those are the types of claude_code.token.usage, which claude exports on the claude runtimes, so the cache hit rate reads the same way on every runtime: cacheRead over the sum of input, cacheRead and cacheCreation.

The classifier logs each run's result to its journal, which ships to VictoriaLogs like every swarm service container's (services.md).

Still open

These get settled, and filled in here, as the work reaches them: