Persistence + retention

Where state lives, what survives what, and how it's bounded.

For operators

The short answer to "will I lose anything": destroying an agent keeps its state, purging it doesn't.

Beyond that:

Everything below this point is implementation detail: exact table schemas, file layouts, and internal migration mechanics.

Sqlite databases

/var/lib/hyperhive/db/broker.sqlite (host)

Three tables, all in one file — the message queue, a small key/value table, and the per-agent power-intent registry:

On a host whose hive-c0re once kept scheduled prompts, the file also holds scheduled_prompts and scheduled_prompt_targets. Nothing opens them; scheduled prompts live at swarm level (swarm/schedules.md). They're safe to drop by hand.

Retention:

/harness/hyperhive-events.sqlite (per agent)

Lives inside each container's bind-mounted /harness/ dir (host path: /var/lib/hyperhive/agents/<name>/harness/hyperhive-events.sqlite). One table:

The harness both writes and vacuums it. Cleanup runs in-container because hive-c0re runs as the unprivileged hive-core user under privsep and can't delete agent-owned files (host-side deletes hit PermissionDenied on the bash-task trio and a readonly database error here). hive-agent's vacuum::run (hive-agent/src/vacuum.rs) sweeps hourly. Retention is type-scoped: it deletes only the verbose stream rows (the raw claude stream-json deltas — one per text chunk / tool use, the bulk of the file's size) older than 14 days, and keeps every other kind (turn_start, turn_end, note, status_changed, model_changed, token_usage_changed, turn_state_changed) indefinitely — those are small and carry the semantic per-turn history the operator scrolls back through when debugging a regression. Age-only within the stream kind — no row cap — so a chatty turn doesn't lose its stream history sooner than a quiet one. The trade-off (accepted): a misbehaving harness can skip its own cleanup — but a compromised harness is already inside the container trust boundary (docs/trust-boundary/security.md), and these are ephemeral local artifacts, so cleaning them up where they live is the honest fix.

Path overridable via HYPERHIVE_EVENTS_DB (for dev / no-/harness setups). On open failure the Bus falls back to no-store mode rather than crashing the harness — events still broadcast over SSE, just nothing persisted.

/harness/hyperhive-turn-stats.sqlite (per agent)

Per-turn analytics sink. One row per claude turn captures identity (model, wake_from, result_kind), timing (started_at, ended_at, duration_ms), cost (input / output / cache_read / cache_creation token counts), behaviour (tool_call_count + tool_call_breakdown_json), and post-turn snapshot metrics (open_threads_count, open_reminders_count — fetched via the same socket the harness already uses for GetOpenThreads + CountPendingReminders). Bin-loop helpers build_row + record land each row at turn_end; writes are best-effort, a sqlite hiccup logs + lets the turn loop continue.

The hive-bash-daemon (not the harness) writes a sibling bash_commands(ts INTEGER, head TEXT) table in the same file: one row per executed bash task recording the normalised command head - the basename of the first real command, looking past cd repo && prefixes, env-assignments, and prefix-runners like sudo/env. It backs the "favorite tools" view on the /stats page (aggregated host-side). Best-effort and created on first write (CREATE TABLE IF NOT EXISTS), so it's absent until a bash task runs.

turn-stats.sqlite has no vacuum — it's one small row per turn (~hundreds of KB even over months), read directly by the /stats page and the hive-wide stats view, so pruning it would only lose trend history for no space gain.

/state/hyperhive-harness.json (per agent)

Consolidated harness state file written atomically (.tmp + rename) by Bus::emit_status whenever rate-limited or login-failed flags change. Shape:

{ "rate_limited": false, "needs_login": false, "active_model": "…" }

The turn loop is the only writer today, but it still goes read-modify-write under a shared in-process lock and merges into the existing object rather than reconstructing it — so a second writer would preserve fields it doesn't own, and the lock closes the lost-update window between a writer's read and its rename. The lock is in-process only, so it wouldn't serialise a writer running as a separate process; none of today's writers are.

hive-c0re reads this file on each build_all sweep (~10s) via container_view::read_harness_flags. Falls back to the individual sentinel files (hyperhive-rate-limited, hyperhive-needs-login) when the JSON is absent.

/var/lib/hyperhive/db/build_logs.sqlite (host)

Full stdout + stderr capture for every nixos-container / nix build invocation the lifecycle layer fires. One row per invocation; the row accumulates lines as the child runs.

Capturing the full stream (rather than a short tail buffer) matters because real eval errors routinely run long — "tried alternatives" blocks alone are often 30+ lines — so a truncated tail would cut off the actual failure and leave only the host journal holding the complete output. With this table the dashboard can surface the entire log.

Three indices:

Writes are best-effort: append_stdout / append_stderr / finish log a warning on sqlite error and let the build continue. A failed log row never blocks a rebuild.

A row is open (finished_at IS NULL) only while its child runs. At startup Coordinator::open closes every row still open as fail and appends a line saying hive-c0re stopped: the previous process died mid-build, and the retention sweep never deletes open rows.

/harness/hyperhive-model (per agent)

Single-line text file holding a claude model name. The harness only reads it, at boot in Bus::new and again when the swarm clears the model it set. It holds the model the agent runs when neither the swarm nor HIVE_DEFAULT_MODEL (agent.nix) names one. Nothing in the harness writes it. Path overridable via HYPERHIVE_MODEL_FILE. Survives destroy/recreate, gone on --purge.

/harness/paused (per agent)

Empty marker file. Its presence parks the agent's turn loop: the harness keeps serving its web UI and MCP daemons but drives no turns, and inbox messages queue unacked until it's removed (see turn loop).

Unusually, it's read and written from both sides of the harness bind-mount, and that's the whole design: the harness stats it in-container via hive-agent's paths::paused_marker, while hive-c0re stats it on the host (Coordinator::is_paused) to populate the paused field on the agent card, and creates/removes it (Coordinator::set_paused) for hivectl agent <name> pause|resume and the dashboard toggle. Because the file itself is the only shared state there's no protocol between them, no round-trip into the container, and pause keeps working when the harness is wedged or the container is stopped.

It lives in /harness/ rather than /state/ deliberately: /state/ is the agent's own space to fill, and this is harness control state. Survives destroy/recreate, gone on --purge — so a paused agent comes back paused after a restart, which is the intended behaviour rather than an accident of storage.

State dirs (per agent)

Under /var/lib/hyperhive/agents/<name>/:

Cross-agent access to state

An agent holding the ManageRootAgent capability gets every other agent's state dir bind-mounted read-write and its config dir read-only (bind_child_agent_dirs in lifecycle/host_config.rs). The RW on state is deliberate, not an oversight: the holder recovers other agents, which includes writing into their state (for example seeding notes, clearing a stuck sentinel) as well as reading it.

This is the only cross-agent mount. An agent holding no capability sees its own dirs and nothing else.

harness isn't mounted at all. It holds that agent's own runtime material — bash-tasks/, the turn-stats and event sqlite dbs — and nothing argues for anyone else reading it, let alone writing it. hive-c0re reads a harness dir directly on the host when it wants those stats, which needs no mount into another container.

config is read-only, including for the holder. A config change is a PR on that agent's config repo, made from a clone and merged after review — so the bind-mounted config dir is a copy to read, never a tree anyone edits in place. Mounting it writable would leave a second path to the same file that skips the review entirely, which makes the boundary a convention rather than a permission.

⚠️ Don't confuse it with the config-repo seeding hive-c0re does at spawn (lifecycle::setup_proposed): that writes the agent's initial config repo as hive-c0re, against the host path, and read_only on a bind constrains writers inside a container only. The two are unrelated — conflating them can lead you to reason your way into thinking this mount should be writable when it shouldn't.

Isolation still holds by default: a container has its own dirs and, unless it holds the capability, nothing else.

Under /var/lib/hyperhive/applied/<name>/ — the hive-c0re-only applied repo. Tracks flake.nix (module-only boilerplate; never edited after first spawn) + agent.nix (the actual config; edits land here from merged config PRs) + any other files those PRs commit. .git/ carries the proposal / approved / building / deployed / failed / denied tag history.

Under /var/lib/hyperhive/meta/ — the swarm-wide deploy flake plus system-level config files. Single git repo for the whole host; hive-c0re commits every mutation that should survive a restart here. Contents:

The root agent has the meta dir RO-mounted at /meta/.

hive-c0re renders containers onto meta#<n> at creation.

Destroy vs purge

See For operators above for what each action does to an agent's state. The mechanics, for completeness:

actions::destroy implements the root/bootstrap agent's specialness as a soft policy guard that refuses to destroy it, backstopped by auto_update::ensure_root_agent, which recreates it on the next hive-c0re startup if it's ever absent — so even without the guard, destroying it would only be transient.

btrfs subvolumes for /var/lib/hyperhive/agents/<name>

On a btrfs host, lifecycle::ensure_agent_state_subvolume creates a brand-new agent's state root as a btrfs subvolume instead of a plain directory (progressive enhancement). This is a no-op fallback on non-btrfs hosts and for any agent whose root already exists, so hive-c0re automigrates nothing: existing agents keep their plain dirs until an explicit opt-in upgrade.

Per-subvolume disk-usage accounting and optional quotas have since landed as the qgroup work: hivectl quota-enable turns on btrfs qgroup accounting hive-wide (opt-in, no-op on non-btrfs hosts), and hivectl agent <name> quota show|set reads/limits one agent's subvolume usage through the same hive-priv-mediated path as subvolume creation/deletion above.

This is the same subvolume hivectl agent <name> subvol snapshot push sends to the swarm's snapshot store — see docs/networking/snapshot-store.md for what a pushed snapshot contains and how the store authenticates a sender.

The subvolume is the whole agent state root, so work/ is part of every snapshot and migration of it and can make them much larger.

/var/lib/swarm-controller/ (swarm-controller host only)

Only present on the one host running services.hyperhive.deploy.swarm-controller.enable. systemd StateDirectory=, so it survives restarts and redeploys.

If the file is unreadable at startup the daemon still starts and logs webhook secret unavailable; the webhook endpoint then answers 503 rather than accepting deliveries it can't verify. Everything else the controller serves is unaffected.

Run-time dirs

/run/hyperhive/ is tmpfs-backed (systemd RuntimeDirectory=) but preserved across hive-c0re restarts via RuntimeDirectoryPreserve=yes. Without that, every restart wipes bind sources and existing containers can't start.

On startup, Coordinator::register_agent drops any prior socket task before rebinding — idempotent so a hive-c0re restart followed by rebuild alice recreates the agent's socket without a clean reinstall.

First-boot agent-user migration

The harness runs as a per-agent unix user inside the container (services.hyperhive.agent.user.name, defaults to the agent's logical label so each container has a uniquely named user). The activation script below moves root-owned state to the agent user so the claude session survives.

system.activationScripts.hive-agent-user-migrate (in nix/agent-modules/user.nix) runs on every activation, marker-guarded so the substantive moves only happen once per container lifetime:

  1. ${homeDir} exists with the right ownership — covers the the first boot before useradd's createHome has had a chance to chown. Also re-applies on every rebuild in case the meta-flake's per-agent name evolves (rare).
  2. Migrate any leftover /root/.claude content into ${homeDir}/.claude. Marker (/var/lib/hive-agent-user-migrated) guards single-shot; it's only written once there's nothing left to migrate, so a cp failure leaves it absent and the next boot retries. cp -an (no-clobber) so any pre-existing files at the new location win — never blow over data already there, and so a retry after a partial copy is as safe as the first attempt.
  3. Chown the bind-mounted state dir (/agents/*/state) recursively so the agent user can read/write it. Wildcard matches the single agent that container sees; -h skips symlinks the agent might have planted.
  4. Chown the ~/.claude/ bind-mount recursively. The harness reads ~/.claude/ as the agent user to decide Online vs NeedsLogin in login::has_session. Without the chown, credentials owned by another uid get silently treated as "no session" and the operator re-prompts every boot.
  5. Hand the socket dir /run/hive-agent/<name> to the agent user, 0751, not recursive. hive-priv creates it 0751 root on the host before every start; the harness binds its sockets there as the agent user. Why the mode matters: boundary.md.

Matrix per-agent daemon + token-arrival trigger

hive-matrix-daemon is a long-running matrix-sdk Client + sync process per agent. Serves its MCP tools directly over streamable-http (services.hyperhive.agent.mcp.matrixHttpPort, no stdio bridge — same shape as hive-bash-daemon), emits hyperhive wake signals on incoming room events via /run/hive/mcp.sock. Conditional on the agent having at least one services.hyperhive.agent.matrixAccounts entry — there is no separate enable switch, and the same condition gates the daemon, its path watcher AND the autoinjected extraMcpServers.matrix entry. The module declares the hive-internal main entry whenever services.hyperhive.agent.matrix.url is non-null, so on a real hive (where hive-c0re renders that URL per agent) every agent has one; a null URL with no operator-declared account is the "this agent has no matrix" state.

First-boot ordering: a token can arrive after the container comes up — a declared token file, or a token the swarm mints into the store. Without the path-trigger sibling (systemd.paths.hive-matrix-daemon, PathExistsGlob = <this agent's state dir>/matrix-token* — the trailing * also catches a declared secondary token file like matrix-token-ccc), the daemon would exit 0 quietly the first time it ran and the MCP would have no daemon until the next restart. The .path unit makes the appearance of the token re-fire the service so the daemon comes alive in the same boot cycle as provisioning. A token in the store changes no file, so a store-backed agent also gets a timer that restarts the daemon five minutes after it last exited. A running daemon re-lists its linked accounts in the store every two minutes and, when the set changed, exits with status 75, which the unit's RestartForceExitStatus restarts. On the same two-minute tick it tries again each secondary account that didn't come up — a declared account with no token yet, a token the homeserver rejected, or a homeserver that was down — waiting two minutes after its first failure and twice as long after each further one, up to sixteen minutes. One that comes up joins the running daemon; the primary and the other accounts aren't restarted. The same token watcher also drives avatar setting: on a restart the daemon re-runs each account's bring-up, which sets the avatar (see below).

matrix avatar (set by the daemon over the live Client)

hive-matrix-daemon itself publishes the agent icon (services.hyperhive.agent.icon, an SVG) as each matrix account's profile avatar (hive-matrix-mcp::client::sync_avatar), not a separate oneshot. After the daemon builds + restores an account's Client (authenticated, pointed at that account's resolved homeserver), it calls matrix-sdk's account().upload_avatar() — one call that uploads the media and sets avatar_url. Because it reuses the live Client, there is no hardcoded homeserver URL, no token re-read, and no token-file globbing: the daemon already iterates every configured + dashboard-discovered account in its bring-up loop, so it sets the avatar for every account.

Nix rasterizes the SVG to a 512x512 PNG at build time (iconPng, via librsvg) and forwards its store path as HIVE_ICON_PNG on the daemon unit, gated on services.hyperhive.agent.icon != null. No icon configured → the env is unset → sync_avatar returns early and sets no avatar.

Idempotency is per-account: an avatar-icon-hash file in each account's matrix-sdk state_dir. The daemon hashes the PNG bytes and skips the upload when unchanged, because every upload mints a fresh mxc:// URI that emits a profile state event in every joined room — re-uploading identical bytes is timeline spam. A swarm-UI-linked account gets its avatar when the daemon restarts on a changed account listing (exit 75, which re-runs the per-account bring-up), so no separate avatar trigger is needed. The daemon swallows avatar failures (logged, non-fatal) so they never break account bring-up or sync.