Graphden documentation

Operations runbook (day-2)

For a self-hoster or cloud operator running Graphden in production. Graphden is entirely Postgres-backed — the graph (code), versions, tenants, execution records, service desired-state, and the fleet placement map all live in one database. So your day-2 story is largely a Postgres story: get backups, recovery, and (for a fleet) HA right, and the rest follows.

See also DEPLOYMENT.md (install/config) and SCALING.md / FLEET_DEPLOY.md (fleet).

What holds state

State Where
Graph (fns/slots/bindings), branches, versions Postgres
Tenants, users, grants, plans Postgres
Execution records, usage metering Postgres
Service desired-state, fleet placement map Postgres
Secret values Vault / OpenBao (Postgres holds only the paths)
Compiled registry, per-branch caches in-memory (rebuilt from Postgres on boot / NOTIFY)

Everything durable is in Postgres + Vault. Back up both. The in-memory state is derived and self-rebuilds — it needs no backup.

Backups

Choose one (managed Postgres usually gives you both):

Back up the Vault/OpenBao store on its own schedule (its own backup mechanism) — a graph that reads a secret is useless if the secret is lost.

Test the restore. An untested backup is not a backup — periodically restore into a scratch instance and boot Graphden against it (bb verify <url> after).

Restore

  1. Restore Postgres (from dump, or PITR to the target time).
  2. Ensure Vault/OpenBao holds the referenced secret paths.
  3. Boot the executor against the restored DB. Schema init is idempotent and fleet-safe (a transaction-scoped advisory lock serializes it — see Migrations & upgrades below), so a normal boot reconciles cleanly.
  4. Verify with bb verify <base-url> (per-section build-hash match) and the /health endpoint.

The in-memory compiled registry rebuilds from the restored rows on boot — no separate cache restore.

Postgres HA & failover

Graphden does not manage Postgres HA — use a managed Postgres, or replication with a failover mechanism (e.g. Patroni, or your cloud’s managed failover). Two things to know about how Graphden interacts with a failover:

Recommendation: point the app’s DB connection at a stable failover endpoint (a managed endpoint, or PgBouncer / a virtual IP in front of the primary) so a failover is a transparent reconnect rather than a config change. This also matters for CRaC restore (its DB endpoint is baked at checkpoint time — see FLEET_RFC.md §5.1).

Migrations & upgrades

Health & readiness

Cold-boot cost (canonical figure). A cold start — package load, the type-check sweep, then eager-compile of every fn-def — takes ~30–40 s to serving (measured, after the compile/sweep memoization that cut compile-all from ~60 s to ~1.2 s; the sweep now dominates). Size the Dockerfile healthcheck start-period and any k8s startup probe against this. Other docs reference THIS number rather than restating it. A runtime full recompile (no sweep — just compile-all) is now a few seconds, not the tens of seconds it once was.

Editor bricked by an asset override

The Operate → Assets panel edits the editor’s own JS/CSS (versioned :resource-override rows). A syntax-broken JS override is refused client-side, but a broken override applied by another path — or a bad CSS/structure change — can leave the editor unusable, and the panel lives inside that editor. Recovery:

  1. Restart the pod(s) with GRAPHDEN_DISABLE_ASSET_OVERRIDES=1 — every frontend asset then serves its shipped classpath baseline (bytes and the ?v= hash both ignore the override rows), so the editor loads clean.
  2. Open Operate → Assets and revert the offending override.
  3. Unset GRAPHDEN_DISABLE_ASSET_OVERRIDES and restart.

The hatch is read at serve time, so it takes effect on restart and needs no data change. It is a platform/self-host knob — in the cloud the entity is tenant-forbidden, so only an operator can create or revert overrides.

Suspending an abusive org

To freeze a misbehaving tenant near-instantly, set its plan to suspended — no effects, all row ceilings 0, so it can neither run nor create anything (it can still delete its own data). The route is platform-only (authenticate as the platform/operator, not as the tenant):

# freeze
curl -sS -X POST "$BASE/api/orgs/plan" -H "Authorization: Bearer $OPERATOR_TOKEN" \
  --data-urlencode name=<org-slug> --data-urlencode plan=suspended

# restore (back to its real tier)
curl -sS -X POST "$BASE/api/orgs/plan" -H "Authorization: Bearer $OPERATOR_TOKEN" \
  --data-urlencode name=<org-slug> --data-urlencode plan=free

plan must be one of free / network / dedicated / suspended; an unknown slug is rejected rather than silently applied. The change is effective on the org’s next request (no memo). The same route upgrades / downgrades a paying org. See PLANS.md § Suspending an org.

Inactive free organizations

The tenancy addon’s reaper (:tenancy/demo-gc, the same sweep that purges expired demo orgs, hourly) also runs the free-org lifecycle when the accounts tables exist:

  1. Mark — a plan = free org with no activity for GRAPHDEN_FREE_ORG_INACTIVE_DAYS (default 60; <= 0 disables the lifecycle) gets expires_at = now + GRAPHDEN_FREE_ORG_GRACE_DAYS (default 14, floored at 1). Activity is any of: a session of the owner or a member created OR used (session.last_used_at, refreshed at most hourly by every authenticated request — a browser cookie or an API token in daily use), an execution in the org, a graph edit in the org (an fn or binding version), or the owner account being younger than the window.
  2. Warn — in the same pass, the owner and every user-member with a verified primary email receive “Your Graphden organization X will be deleted on " (the accounts Mailer; pending email invitees are not mailed), and the org row's `warned_at` is set. Any marked org without `warned_at` — one marked before the warning shipped — is caught up on the next sweep. `GRAPHDEN_FREE_ORG_REMIND_DAYS` (default 2) before the deletion a "Last reminder" goes out once (`reminded_at`). Both columns clear with the mark on rescue. Without `:mailer` / `:link-origin` on `:tenancy/demo-gc` the mark is only logged (`demo-gc: marked orgs NOT warned`).
  3. Rescue — any sign-in, token use, run or edit inside the grace clears expires_at (and the mail ledger) on the next sweep, before that sweep purges anything.
  4. Purge — past expires_at the org, its graph rows and its grants are hard-deleted. Every account is kept — the owner’s too: a real person may own or belong to other orgs. (Only a demo org’s purge also takes the ephemeral anonymous account that owns it, with its sessions and grants.)

Paid (network / dedicated) and suspended orgs are never selected. Logs: demo-gc: marked inactive free orgs …, demo-gc: warned <org> → n of m recipients, demo-gc: un-marked returned free orgs …, demo-gc: purged expired org ….

Your responsibility vs Graphden’s

Graphden gives you: idempotent fleet-safe schema migration, self-healing in-memory state, health/version/verify surfaces. You own: Postgres backups, HA, and failover topology; Vault/OpenBao backups; the network perimeter and TLS; and OS/container patching.