For a self-hoster or cloud operator running Graphden in production. Graphden is entirely Postgres-backed — the graph (code), versions, tenants, execution records, service desired-state, and the fleet placement map all live in one database. So your day-2 story is largely a Postgres story: get backups, recovery, and (for a fleet) HA right, and the rest follows.
See also DEPLOYMENT.md (install/config) and SCALING.md / FLEET_DEPLOY.md (fleet).
| State | Where |
|---|---|
| Graph (fns/slots/bindings), branches, versions | Postgres |
| Tenants, users, grants, plans | Postgres |
| Execution records, usage metering | Postgres |
| Service desired-state, fleet placement map | Postgres |
| Secret values | Vault / OpenBao (Postgres holds only the paths) |
| Compiled registry, per-branch caches | in-memory (rebuilt from Postgres on boot / NOTIFY) |
Everything durable is in Postgres + Vault. Back up both. The in-memory state is derived and self-rebuilds — it needs no backup.
Choose one (managed Postgres usually gives you both):
pg_dump of the graphden database. Simple; restore
granularity is the dump interval.Back up the Vault/OpenBao store on its own schedule (its own backup mechanism) — a graph that reads a secret is useless if the secret is lost.
Test the restore. An untested backup is not a backup — periodically restore
into a scratch instance and boot Graphden against it (bb verify <url> after).
bb verify <base-url> (per-section build-hash match) and the
/health endpoint.The in-memory compiled registry rebuilds from the restored rows on boot — no separate cache restore.
Graphden does not manage Postgres HA — use a managed Postgres, or replication with a failover mechanism (e.g. Patroni, or your cloud’s managed failover). Two things to know about how Graphden interacts with a failover:
LISTEN/NOTIFY are session state and are NOT
replicated. On failover the singleton/leader advisory locks and every
NOTIFY subscription are lost with the old primary’s connections.services/reconciler, the fleet controller), NOTIFY listeners re-subscribe,
and cache invalidation self-heals via the graph-epoch check + the periodic
reconcile tick. Nothing needs manual intervention, but expect a short window
of degraded invalidation/singleton coverage around the failover.Recommendation: point the app’s DB connection at a stable failover endpoint (a managed endpoint, or PgBouncer / a virtual IP in front of the primary) so a failover is a transparent reconnect rather than a config change. This also matters for CRaC restore (its DB endpoint is baked at checkpoint time — see FLEET_RFC.md §5.1).
ALTERs) do not race DDL — each pod
blocks, then sees the migrated schema and no-ops. A rolling upgrade is safe.bb verify <base-url> to confirm the
frontend / packages / backend sections all match the shipped build.uuid-v5(ns, name)
id, _anon-* shapes included) is purged only when nothing the
current graph still follows references it — the newest version of each
row on a live branch, never a create-time identity row or a superseded
version (identity-repair/inbound-refs-many :live-only?). A retired fn
a user fn still references stays, with one aggregated boot warning, and
shows in the editor’s ⚐ lint filter as a duplicate. Editor-created rows
(random ids) are never touched. Preview before an upgrade from a REPL:
(packages.sync/reconcile-moved-identities! storage packages {} {:preexisting-fn-ids …
:dry-run? true}) returns {:moves :purgeable :kept} and writes nothing.Cold-boot cost (canonical figure). A cold start — package load, the type-check sweep, then eager-compile of every fn-def — takes ~30–40 s to serving (measured, after the compile/sweep memoization that cut
compile-allfrom ~60 s to ~1.2 s; the sweep now dominates). Size the Dockerfile healthcheckstart-periodand any k8s startup probe against this. Other docs reference THIS number rather than restating it. A runtime full recompile (no sweep — justcompile-all) is now a few seconds, not the tens of seconds it once was.
GET /health — readiness signal (and startup). It is a graph fn behind
the compiled registry, so it stays 200 only while the registry is warm: a pod
mid-cold-boot reads not-ready and is correctly de-rotated. Give the startup
probe a start-period generous enough for the boot compile (see the Dockerfile
healthcheck start-period). A runtime full clear (a nil-seed write, a
migration) no longer de-rotates the pod: it is stale-while-revalidate —
the ctx keeps serving its existing (stale) registry while a background
rebuild-optimistic! refreshes it, so /health stays 200 and no request
blocks behind the recompile (a few seconds since the compile-all
memoization — see the cold-boot note above). Staleness is bounded by one rebuild; the
window is served, never hung. (Only a genuinely cold holder — boot, or a
divergent cold branch’s first access — still compiles on the request path,
and that is per-ctx, so it never hangs the whole pod.)GET /livez — liveness signal. A static, registry-independent 200 (it
short-circuits in branch-router/dispatch before any registry access). Point
the k8s livenessProbe / any restart-on-unhealthy check HERE, never at
/health: a liveness probe on /health kills a busy-but-alive pod during a
runtime recompile → discards the in-flight compile → a cold-boot cascade
(the full ~30–40 s cold start above, paid over and over). /livez keeps the
pod alive; /health (readiness) de-rotates it
until the compile lands. (Helm chart: startup + readiness → /health,
liveness → /livez.)GET /version — the three build-section hashes (bb verify consumes these).GET /metrics — a JVM + structural-counter snapshot (JSON). For a Prometheus
scrape target, see the observability notes in SCALING.md.The Operate → Assets panel edits the editor’s own JS/CSS (versioned
:resource-override rows). A syntax-broken JS override is refused
client-side, but a broken override applied by another path — or a bad
CSS/structure change — can leave the editor unusable, and the panel
lives inside that editor. Recovery:
GRAPHDEN_DISABLE_ASSET_OVERRIDES=1 — every
frontend asset then serves its shipped classpath baseline (bytes and
the ?v= hash both ignore the override rows), so the editor loads
clean.GRAPHDEN_DISABLE_ASSET_OVERRIDES and restart.The hatch is read at serve time, so it takes effect on restart and needs no data change. It is a platform/self-host knob — in the cloud the entity is tenant-forbidden, so only an operator can create or revert overrides.
To freeze a misbehaving tenant near-instantly, set its plan to suspended — no
effects, all row ceilings 0, so it can neither run nor create anything (it can
still delete its own data). The route is platform-only (authenticate as the
platform/operator, not as the tenant):
# freeze
curl -sS -X POST "$BASE/api/orgs/plan" -H "Authorization: Bearer $OPERATOR_TOKEN" \
--data-urlencode name=<org-slug> --data-urlencode plan=suspended
# restore (back to its real tier)
curl -sS -X POST "$BASE/api/orgs/plan" -H "Authorization: Bearer $OPERATOR_TOKEN" \
--data-urlencode name=<org-slug> --data-urlencode plan=free
plan must be one of free / network / dedicated / suspended; an unknown
slug is rejected rather than silently applied. The change is effective on the
org’s next request (no memo). The same route upgrades / downgrades a paying org.
See PLANS.md § Suspending an org.
The tenancy addon’s reaper (:tenancy/demo-gc, the same sweep that purges
expired demo orgs, hourly) also runs the free-org lifecycle when the accounts
tables exist:
plan = free org with no activity for
GRAPHDEN_FREE_ORG_INACTIVE_DAYS (default 60; <= 0 disables the
lifecycle) gets expires_at = now + GRAPHDEN_FREE_ORG_GRACE_DAYS
(default 14, floored at 1). Activity is any of: a session of the owner
or a member created OR used (session.last_used_at, refreshed at most
hourly by every authenticated request — a browser cookie or an API
token in daily use), an execution in the org, a graph edit in the org
(an fn or binding version), or the owner account being younger than
the window.expires_at (and the mail ledger) on the next sweep, before that
sweep purges anything.expires_at the org, its graph rows and its grants are
hard-deleted. Every account is kept — the owner’s too: a real person may
own or belong to other orgs. (Only a demo org’s purge also takes the
ephemeral anonymous account that owns it, with its sessions and grants.)Paid (network / dedicated) and suspended orgs are never selected.
Logs: demo-gc: marked inactive free orgs …, demo-gc: warned <org> → n of m
recipients, demo-gc: un-marked returned free orgs …, demo-gc: purged
expired org ….
Graphden gives you: idempotent fleet-safe schema migration, self-healing in-memory state, health/version/verify surfaces. You own: Postgres backups, HA, and failover topology; Vault/OpenBao backups; the network perimeter and TLS; and OS/container patching.