Persistence + retention

Where state lives, what survives what, and how it's bounded.

For operators

The short answer to "will I lose anything": destroying an agent keeps its state, purging it doesn't.

• DESTR0Y (the default action) stops and removes the container but keeps everything on disk — config history, claude login, /state/ notes, harness data. The agent shows up as a tombstone (K3PT ST4T3 on the C0R3 page) with a ⊕ R3V1V3 button that recreates it from the kept state, no re-login needed.
• PURG3 (opt-in, from the dashboard or hivectl agent <name> destroy --purge) is DESTR0Y plus wiping all of it — config history, claude credentials, /state/ notes, everything. No undo. Only reach for this when you actually want the agent gone for good.

Beyond that:

• Approvals and questions are kept forever — they're audit trails, not caches. Nothing about them ever ages out.
• Broker messages: acked ones vacuum after 30 days; anything undelivered or delivered-but-not-yet-acked is always kept, however old.
• An agent's own /state/ notes and claude login survive every restart and rebuild — only an explicit purge (or a hive --purge-style host operation, or the agent's own choices) touches them.
• The root/bootstrap agent is special: it isn't really destroyable in practice — hive-c0re recreates it automatically on its next startup if it's ever gone.

Everything below this point is implementation detail: exact table schemas, file layouts, and internal migration mechanics.

Sqlite databases

/var/lib/hyperhive/db/broker.sqlite (host)

Seven tables, all in one file — four queues, the schedule header/targets split, and the per-agent power-intent registry:

• messages — every inter-agent / operator-bound message. sender / recipient / body / sent_at / delivered_at / acked_at / in_reply_to. in_reply_to links a reply to its parent row id; the dashboard and per-agent inbox render these as threaded rows.
• reminders — mcp__hyperhive__remind queue. agent / message / file_path / due_at / created_at / sent_at / attempt_count / last_error. file_path set when a body exceeded the inline soft-cap and got auto-spilled to a file under the agent's state dir; the worker delivers a short pointer instead. attempt_count / last_error accumulate on delivery-failed retries.
• approvals — the queue. agent / kind (merge_config_pr | spawn | init_config | update_meta_inputs | schedule_prompt) / commit_ref / requested_at / status / resolved_at / note.
• operator_questions — ask / answer queue (despite the table name, stores both operator-targeted + agent-to-agent questions since the ask rename). asker / question / options_json / multi / asked_at / deadline_at (ttl) / answered_at / answer / target. target IS NULL = operator path (dashboard); target = '<agent>' = peer Q&A (HelperEvent::QuestionAsked pushed into target's inbox, answered via Answer request). Migrated via ALTER TABLE ADD COLUMN against pragma_table_info.
• scheduled_prompts — recurring + one-shot prompt queue. owner / body / interval_seconds (NULL = one-shot) / next_fire_at_unix / created_at_unix / source ("operator" or "approval:<id>") / cancelled_at_unix / description. owner drives cancel-permission checks (operator vs the submitting agent). Cancelled rows are tombstoned and reaped by the worker on its next pass.
• scheduled_prompt_targets — per-target state for each schedule. schedule_id / target / cancelled_at_unix / last_fired_at_unix / last_result. ON DELETE CASCADE from scheduled_prompts(id) — requires PRAGMA foreign_keys = ON per connection (set at open).
• agent_power — one tiny row per agent: agent PK / wanted (up | offline) / updated_at — the durable power intent behind the job queue's desired-state reconciliation (docs/coordinator.md::Job queue; owner: hive-c0re/src/stores/power.rs). Written synchronously by every operator/agent power action (dashboard start/stop, hivectl stop, the MCP kill/start tools, spawn approval); read by Reconcile nodes and the boot reconcile. Intent survives hive-c0re restarts — in-flight queue work deliberately does not. Agents without a row are seeded from observed state on first touch (running ⇒ up); destroy deletes the row.

Retention:

• Broker::vacuum_delivered runs hourly via a tokio task in hive-c0re::main. Drops acked message rows older than 30 days (acked_at IS NOT NULL). Undelivered + delivered-but-not-acked rows are always kept — the harness ack_turns only after a successful turn, so an unacked row can still be requeued via requeue_inflight on a crash.
• Approvals and questions are kept indefinitely — both are audit trails. actions::destroy and answered questions stay visible to anything that queries by id.
• Reminder rows are kept after sent_at is set (audit trail); no automatic vacuum today.
• Scheduled prompts: one-shot rows are deleted on fire by the worker; recurring rows live until the operator cancels them (cancel_schedule MCP / dashboard ✗) which tombstones via cancelled_at_unix, then reap_cancelled drops the row on the next worker pass.
• agent_power rows live until the agent is destroyed (one row per agent — nothing to vacuum).

/harness/hyperhive-events.sqlite (per agent)

Lives inside each container's bind-mounted /harness/ dir (host path: /var/lib/hyperhive/agents/<name>/harness/hyperhive-events.sqlite). One table:

• events(id, ts, kind, payload_json) — every LiveEvent the harness emits during turn loop execution.

The harness writes; the host vacuums. hive-c0re::events_vacuum runs hourly and sweeps every existing agent harness dir. Retention is type-scoped: it deletes only the verbose stream rows (the raw claude stream-json deltas — one per text chunk / tool use, the bulk of the file's size) older than 14 days, and keeps every other kind (turn_start, turn_end, note, status_changed, model_changed, token_usage_changed, turn_state_changed) indefinitely — those are small and carry the semantic per-turn history the operator scrolls back through when debugging a regression. Age-only within the stream kind — no row cap — so a chatty turn doesn't lose its stream history sooner than a quiet one. Centralising retention on the host means a misbehaving harness can't disable its own vacuum and agents don't need any cleanup wiring of their own.

Path overridable via HYPERHIVE_EVENTS_DB (for dev / no-/harness setups). On open failure the Bus falls back to no-store mode rather than crashing the harness — events still broadcast over SSE, just nothing persisted.

/harness/hyperhive-turn-stats.sqlite (per agent)

Per-turn analytics sink. One row per claude turn captures identity (model, wake_from, result_kind), timing (started_at, ended_at, duration_ms), cost (input / output / cache_read / cache_creation token counts), behaviour (tool_call_count + tool_call_breakdown_json), and post-turn snapshot metrics (open_threads_count, open_reminders_count — fetched via the same socket the harness already uses for GetOpenThreads + CountPendingReminders). Bin-loop helpers build_row + record land each row at turn_end; writes are best-effort, a sqlite hiccup logs + lets the turn loop continue.

A sibling bash_commands(ts INTEGER, head TEXT) table in the same file is written by the hive-bash-daemon (not the harness): one row per executed bash task recording the normalised command head - the basename of the first real command, looking past cd repo && prefixes, env-assignments, and prefix-runners like sudo/env. It backs the "favorite tools" view on the /stats page (aggregated host-side). Best-effort and created on first write (CREATE TABLE IF NOT EXISTS), so it's simply absent until a bash task runs.

turn-stats.sqlite has no vacuum — it's one tiny row per turn (~hundreds of KB even over months), read directly by the /stats page and the hive-wide stats view, so pruning it would only lose trend history for no space gain.

/state/hyperhive-harness.json (per agent)

Consolidated harness state file written atomically (.tmp + rename) by Bus::emit_status whenever rate-limited or login-failed flags change. Shape:

{ "rate_limited": false, "needs_login": false, "active_model": "…" }

• rate_limited — set when the harness detects a 429 from the Claude API; cleared by any subsequent status emit. Drives ContainerView.rate_limited on the dashboard.
• needs_login — set when a turn hits 401 (expired OAuth credentials); cleared by "online" status (re-auth completed). Drives the needs_login flag alongside the claude_has_session check.
• active_model — the resolved Claude model for the dashboard badge.

The turn loop is the only writer today, but it still goes read-modify-write under a shared in-process lock and merges into the existing object rather than reconstructing it — so a second writer preserves fields it doesn't own, and the lock closes the lost-update window between a writer's read and its rename. (The forge notification poller used to be that second writer, for a delivery-dedupe cursor. It persists nothing now — forge's own read-state is the durable record of what has been delivered — and it is a separate process, which an in-process lock could not have serialised anyway.)

hive-c0re reads this file on each build_all sweep (~10s) via container_view::read_harness_flags. Falls back to the legacy individual sentinel files (hyperhive-rate-limited, hyperhive-needs-login) if the JSON is absent, so existing containers keep working through the transition window before their next rebuild.

/var/lib/hyperhive/db/build_logs.sqlite (host)

Full stdout + stderr capture for every nixos-container / nix build invocation the lifecycle layer fires. One row per invocation; the row accumulates lines as the child runs.

Replaces the legacy 32-line stderr ring buffer that lifecycle::run kept. The ring tail routinely truncated real eval errors ("tried alternatives" blocks alone are often 30+ lines), so failures bailed with an arbitrary tail whose full stream only lived in the host journal. With this table the dashboard can surface the entire log.

Two indices:

• (agent, started_at) — backs the per-agent latest-N lookup used by the agent card chip.
• (status, finished_at) — backs the retention sweep that runs as part of the existing hourly vacuum.

Writes are best-effort: append_stdout / append_stderr / finish log a warning on sqlite error and let the build continue. A failed log row never blocks a rebuild.

/harness/hyperhive-model (per agent)

Single-line text file holding the claude model name currently selected for this agent (default haiku when absent). Written by Bus::set_model whenever the operator flips it via /model <name> in the web terminal. Read once at harness boot in Bus::new. Path overridable via HYPERHIVE_MODEL_FILE. Survives destroy/recreate, gone on --purge.

/harness/paused (per agent)

Empty marker file. Its presence parks the agent's turn loop: the harness keeps serving its web UI and MCP daemons but drives no turns, and inbox messages queue unacked until it's removed (see turn loop).

Unusually, it's read and written from both sides of the harness bind-mount, and that's the whole design: the harness stats it in-container via hive_agent::paths::paused_marker, while hive-c0re stats it on the host (Coordinator::is_paused) to populate the paused field on the agent card, and creates/removes it (Coordinator::set_paused) for hivectl agent <name> pause|resume and the dashboard toggle. Because the file itself is the only shared state there's no protocol between them, no round-trip into the container, and pause keeps working when the harness is wedged or the container is stopped.

It lives in /harness/ rather than /state/ deliberately: /state/ is the agent's own space to fill, and this is harness control state. Survives destroy/recreate, gone on --purge — so a paused agent comes back paused after a restart, which is the intended behaviour rather than an accident of storage.

State dirs (per agent)

Under /var/lib/hyperhive/agents/<name>/:

• config/ — the proposed nix repo (root-agent-editable). Bind-mounted read-only to /agents/<name>/config inside the sub-agent's own container so the agent can inspect what defines it and request precise changes from the root agent; RW into the root agent via the /agents tree bind.
• claude/ — claude OAuth credentials, bind-mounted RW to /home/<name>/.claude inside the container.
• state/ — durable notes and hyperhive-harness.json. Bind-mounted to /agents/<name>/state inside the container (uniform for all agents). The $HYPERHIVE_STATE_DIR env var exposes the same path to in-container scripts. Notable files written here by the harness:
  • hyperhive-status — single-line free-text status string written by set_status; cleared on explicit set_status(""). Read by hive-c0re and the per-agent /api/dashboard-state endpoint to surface the status chip on the dashboard. Absent when no status is set.
  • hyperhive-harness.json — rate-limited / needs-login flags read by the dashboard's async container-state fetch. See docs/web-ui/dashboard.md::Container row.
• harness/ — harness-internal ephemeral state; not intended for agent consumption. Bind-mounted to /agents/<name>/harness inside the container ($HYPERHIVE_HARNESS_DIR). Contents:

  • bash-tasks/ — task JSON + stdout/stderr files for background mcp__bash__run jobs. JSON files are <id>.json (status + tails), <id>.out / <id>.err (full captured output). hive-c0re::bash_tasks_vacuum runs hourly and deletes terminal task trios older than 48 hours; non-terminal (still-running) tasks are never deleted by vacuum.

  • hyperhive-todos.sqlite — loose-ends-v2 todo store. In-container daemons (hive-bash-daemon, hive-matrix-daemon, hive-forge-notify) upsert keyed todos here over the harness's in-agent socket (HIVE_AGENT_SOCKET); the harness merges them into get_loose_ends output and clears a row on mark_todo_done. Replaced the old file-based mcp-loose-ends/ scanner.

    The harness itself is also a producer, not just the socket server: boot wiring's spawn_todo_socket starts todo_server::run (the socket the out-of-process daemons above dial) alongside disk_watch::run — an in-process todo producer that shares the store + wake Notify directly rather than dialling its own socket. disk_watch raises a keyed disk todo when the filesystem backing this agent's state gets tight, naming the agent's own biggest directories; the summary is bucketed and carries no raw byte counts, so an unchanged situation re-upserts as changed == false and never re-wakes.

  • hyperhive-events.sqlite — turn-loop event log.

  • hyperhive-turn-stats.sqlite — per-turn timing stats.

  • hyperhive-model — single-line model name override file.

Parent access to child state

A parent agent gets each direct child's state dir bind-mounted read-write and its config dir read-only (bind_child_agent_dirs in lifecycle/host_config.rs). The RW on state is deliberate, not an oversight: a parent manages its children, which includes writing into a child's state for recovery (e.g. seeding notes, clearing a stuck sentinel) as well as reading it.

harness is not mounted at all. It holds the child's own runtime material — bash-tasks/, the turn-stats and event sqlite dbs — and nothing argues for a parent reading it, let alone writing it. It used to be mounted RW for "the same management reasons" as state, which was never an argument so much as the side-effect of one loop treating all three dirs alike. hive-c0re reads a child's harness dir directly on the host when it wants those stats, which needs no mount into the parent.

config is read-only, including for the parent. A config change is a PR on the child's config repo, made from a clone and merged after review — so the bind-mounted config dir is a copy to read, never a tree anyone edits in place. Mounting it writable would leave a second path to the same file that skips the review entirely, which makes the boundary a convention rather than a permission.

⚠️ Not to be confused with the seeding done when an InitConfig approval resolves: that writes the child's initial config repo as hive-c0re, against the host path, and read_only on a bind constrains writers inside a container only. The two are unrelated, and reading them as the same thing is what kept this mount writable.

Per-child isolation still holds: a container only ever has its own dirs plus its direct children's bind-mounted, never a sibling's.

Under /var/lib/hyperhive/applied/<name>/ — the hive-c0re-only applied repo. Tracks flake.nix (module-only boilerplate; never edited after first spawn) + agent.nix (the actual config; the root agent's edits land here via the approval flow) + any other files committed via the approval flow. .git/ carries the proposal / approved / building / deployed / failed / denied tag history.

Under /var/lib/hyperhive/meta/ — the swarm-wide deploy flake plus system-level config files. Single git repo for the whole host; every hive-c0re mutation that should survive a restart is committed here. Contents:

• flake.nix — declares one nixpkgs input per agent + one nixosConfigurations.<n> output per agent. flake.lock is the canonical "what's deployed where." The git log is the deploy audit trail (one commit per successful deploy or hyperhive bump).
• topology.json — parent/child agent graph ({ "alice": "root", "bob": "alice", "root": null }). Written by topology::set_parent; read by the dashboard, the renderer, and <parent> / <children> recipient resolution.
• tool-groups.json — per-agent MCP tool group grants ({ "alice": ["messaging", "inbox", "execution"] }). Written by tool_groups::set_groups; injected as HIVE_TOOL_GROUPS env var into each agent's container.
• capabilities.json — per-agent capability grants ({ "atlas": ["read_host_journal"] }). Written by capabilities::set_caps; injected as HIVE_CAPABILITIES env var. Absent agents have no extra capabilities.
• resource-limits.json — per-agent container resource overrides ({ "sock": { "cpu_quota": "400%", "memory_max": "8G" } }). Written by resource_limits::set_limits; read where the systemd drop-in is generated (lifecycle::write_dropins), not injected into the container — these are host-side caps on the container, so the capped party never sees or sets them. Fallback is per field: an absent file, absent agent, or absent field falls back to the hive-wide services.hyperhive.agentCpuQuota / agentMemoryMax, so an agent can override only its memory and still track the hive default for CPU. The CPUWeight= / IOWeight= shares in the same drop-in have no per-agent override — they are hive-wide only and come straight off HiveEnv, so this file has no field for them.

The root agent has the meta dir RO-mounted at /meta/.

There is no longer a .meta-migration-done marker: the one-shot container repoint it guarded has been removed, since containers are rendered onto meta#<n> at creation. A stale marker file left over from an older hive is inert and can be deleted.

Destroy vs purge

• DESTR0Y (default) — stops + removes the nspawn container, drops the systemd drop-in, fails any pending approvals. State dirs stay put; the agent appears in the dashboard's K3PT ST4T3 section as a tombstone with ⊕ R3V1V3 and PURG3 actions. R3V1V3 queues a Spawn approval that reuses the kept state on approve (no re-login).
• PURG3 (opt-in via the dashboard button or hivectl agent <name> destroy --purge) — DESTR0Y plus wipes /var/lib/hyperhive/{agents,applied}/<name>/. Config history, claude creds, /state/ notes, and the harness dir are all gone. No undo.

The root/bootstrap container is imperative infrastructure — managed end-to-end by hive-c0re, not declared in the host's NixOS config. auto_update::ensure_root_agent recreates it on the next hive-c0re startup if it's absent (bypassing the approval queue, as required infrastructure). A soft policy guard in actions::destroy currently refuses to destroy it; even without that guard, destroying it would only be transient — hive-c0re brings it back on the next startup.

btrfs subvolumes for /var/lib/hyperhive/agents/<name>

On a btrfs host, a brand-new agent's state root is created as a btrfs subvolume instead of a plain directory (progressive enhancement — see the #1762 lane). This is a no-op fallback on non-btrfs hosts and for any agent whose root already exists, so nothing is auto-migrated: existing agents keep their plain dirs until an explicit opt-in upgrade.

• Creation: lifecycle::ensure_agent_state_subvolume runs before the per-agent subdirs are created (spawn / rebuild / InitConfig). It skips the work when the root already exists; otherwise it asks hive-priv (EnsureAgentSubvolume) to btrfs subvolume create the root when the FS is btrfs (statfs magic gate) and chown it to the hive-core user so the normal state/ claude/ harness/ mkdirs succeed inside it.
• DESTR0Y keeps the subvolume exactly like a plain dir — revival reuses it untouched.
• PURG3 deletes it correctly: a subvolume root can't be removed with rmdir/remove_dir_all, so purge first calls hive-priv (DeleteAgentSubvolume) which btrfs subvolume deletes it iff it's actually a subvolume, then the normal remove_dir_all sweep covers plain-dir agents + the applied dir.

Per-subvolume disk-usage accounting and optional quotas are a follow-up (the qgroup work), not part of the base migration.

Run-time dirs

/run/hyperhive/ is tmpfs-backed (systemd RuntimeDirectory=) but preserved across hive-c0re restarts via RuntimeDirectoryPreserve=yes. Without that, every restart wipes bind sources and existing containers can't be started.

• /run/hyperhive/host.sock — admin socket (host-side CLI).
• /run/hyperhive/agents/<name>/mcp.sock — per-agent socket (bind-mounted into the container as /run/hive/mcp.sock).

On startup, Coordinator::register_agent drops any prior socket task before rebinding — idempotent so a hive-c0re restart followed by rebuild alice recreates the agent's socket without a clean reinstall.

First-boot agent-user migration

The harness runs as a per-agent unix user inside the container (hyperhive.user.name, defaults to the agent's logical label so each container has a uniquely-named user). Operators with legacy root-owned state dirs need a one-time data shuffle so they don't lose their claude session.

system.activationScripts.hive-agent-user-migrate (in nix/agent-modules/user.nix) runs on every activation, marker-guarded so the substantive moves only happen once per container lifetime:

1. ${homeDir} exists with the right ownership — covers the very first boot before useradd's createHome has had a chance to chown. Also re-applies on every rebuild in case the meta-flake's per-agent name evolves (rare).
2. Migrate any leftover /root/.claude content into ${homeDir}/.claude — legacy claude wrote to root's empty home; the bind mount didn't exist yet. Marker (/var/lib/hive-agent-user-migrated) guards single-shot. cp -an (no-clobber) so any pre-existing files at the new location win — never blow over data already there.
3. Chown the bind-mounted state dir (/agents/*/state) recursively so the agent user can read/write it. Wildcard matches the single agent that container sees; -h skips symlinks the agent might have planted.
4. Chown the ~/.claude/ bind-mount recursively. Legacy claude wrote .credentials.json 0600 root:root; the current harness reads ~/.claude/ as the agent user to decide Online vs NeedsLogin in login::has_session. Without the chown the existing credentials get silently treated as "no session" and the operator re-prompts every boot.

The activation script will eventually become unnecessary once no operators have legacy root-owned state dirs left to migrate; drop the body + marker check at that point.

Matrix per-agent daemon + token-arrival trigger

hive-matrix-daemon is a long-running matrix-sdk Client + sync process per agent. Serves its MCP tools directly over streamable-http (hyperhive.mcp.matrixHttpPort, no stdio bridge — same shape as hive-bash-daemon), emits hyperhive wake signals on incoming room events via /run/hive/mcp.sock. Conditional on hyperhive.matrix.enable (which both the daemon AND the auto-injected extraMcpServers.matrix entry read).

First-boot ordering: hive-c0re provisions the matrix token AFTER agent containers come up. Without the path-trigger sibling (systemd.paths.hive-matrix-daemon, PathExistsGlob = /agents/*/state/matrix-token), the daemon would exit 0 quietly the first time it ran and the MCP would have no backend until the next restart. The .path unit makes the appearance of the token re-fire the service so the daemon comes alive in the same boot cycle as provisioning. The same token watcher also drives avatar setting: on a restart the daemon re-runs each account's bring-up, which sets the avatar (see below).

matrix avatar (set by the daemon over the live Client)

The agent icon (hyperhive.icon, an SVG) is published as each matrix account's profile avatar by hive-matrix-daemon itself (hive-matrix-mcp::client::sync_avatar), not a separate oneshot. After the daemon builds + restores an account's Client (authenticated, pointed at that account's resolved homeserver), it calls matrix-sdk's account().upload_avatar() — one call that uploads the media and sets avatar_url. Because it reuses the live Client, there is no hardcoded homeserver URL, no token re-read, and no token-file globbing: the daemon already iterates every configured + dashboard-discovered account in its bring-up loop, so the avatar is set for every account.

Nix rasterizes the SVG to a 512x512 PNG at build time (iconPng, via librsvg) and forwards its store path as HIVE_ICON_PNG on the daemon unit, gated on hyperhive.icon != null. No icon configured → the env is unset → sync_avatar returns early and no avatar is set.

Idempotency is per-account: an avatar-icon-hash file in each account's matrix-sdk state_dir. The daemon hashes the PNG bytes and skips the upload when unchanged, because every upload mints a fresh mxc:// URI that emits a profile state event in every joined room — re-uploading identical bytes is timeline spam. A dashboard-provisioned account gets its avatar when the systemd.paths.hive-matrix-daemon token watcher restarts the daemon (which re-runs the per-account bring-up), so no separate avatar trigger is needed. Avatar failures are swallowed (logged, non-fatal) so they never break account bring-up or sync.