# Operations runbook (day-2)

For a self-hoster or cloud operator running Graphden in production. Graphden is
**entirely Postgres-backed** — the graph (code), versions, tenants, execution
records, service desired-state, and the fleet placement map all live in one
database. So your day-2 story is largely a **Postgres** story: get backups,
recovery, and (for a fleet) HA right, and the rest follows.

See also [DEPLOYMENT.md](DEPLOYMENT.md) (install/config) and
[SCALING.md](SCALING.md) / [FLEET_DEPLOY.md](FLEET_DEPLOY.md) (fleet).

## What holds state

| State | Where |
|---|---|
| Graph (fns/slots/bindings), branches, versions | Postgres |
| Tenants, users, grants, plans | Postgres |
| Execution records, usage metering | Postgres |
| Service desired-state, fleet placement map | Postgres |
| Secret **values** | Vault / OpenBao (Postgres holds only the paths) |
| Compiled registry, per-branch caches | in-memory (rebuilt from Postgres on boot / NOTIFY) |

Everything durable is in Postgres + Vault. **Back up both.** The in-memory
state is derived and self-rebuilds — it needs no backup.

## Backups

Choose one (managed Postgres usually gives you both):

- **Logical** — periodic `pg_dump` of the graphden database. Simple; restore
  granularity is the dump interval.
- **Physical + WAL archiving (PITR)** — continuous WAL archiving for
  point-in-time recovery. Recommended for production: recover to any moment,
  not just the last dump.

Back up the **Vault/OpenBao** store on its own schedule (its own backup
mechanism) — a graph that reads a secret is useless if the secret is lost.

**Test the restore.** An untested backup is not a backup — periodically restore
into a scratch instance and boot Graphden against it (`bb verify <url>` after).

## Restore

1. Restore Postgres (from dump, or PITR to the target time).
2. Ensure Vault/OpenBao holds the referenced secret paths.
3. Boot the executor against the restored DB. Schema init is idempotent and
   **fleet-safe** (a transaction-scoped advisory lock serializes it — see
   *Migrations & upgrades* below), so a normal boot reconciles cleanly.
4. Verify with `bb verify <base-url>` (per-section build-hash match) and the
   `/health` endpoint.

The in-memory compiled registry rebuilds from the restored rows on boot — no
separate cache restore.

## Postgres HA & failover

Graphden does not manage Postgres HA — use a managed Postgres, or replication
with a failover mechanism (e.g. Patroni, or your cloud's managed failover).
Two things to know about how Graphden interacts with a failover:

- **Advisory locks and `LISTEN/NOTIFY` are session state and are NOT
  replicated.** On failover the singleton/leader advisory locks and every
  NOTIFY subscription are lost with the old primary's connections.
- **Recovery is automatic, with a brief window.** After the connections
  reconnect to the new primary: singleton/leader locks are re-contended
  (`services/reconciler`, the fleet controller), NOTIFY listeners re-subscribe,
  and cache invalidation self-heals via the graph-epoch check + the periodic
  reconcile tick. Nothing needs manual intervention, but expect a short window
  of degraded invalidation/singleton coverage around the failover.

**Recommendation:** point the app's DB connection at a **stable failover
endpoint** (a managed endpoint, or PgBouncer / a virtual IP in front of the
primary) so a failover is a transparent reconnect rather than a config change.
This also matters for CRaC restore (its DB endpoint is baked at checkpoint
time — see [FLEET_RFC.md](FLEET_RFC.md) §5.1).

## Migrations & upgrades

- Schema init/migration runs **automatically on boot** and is **idempotent**.
- It is **fleet-safe**: the whole read → plan → DDL section is serialized across
  pods by a transaction-scoped advisory lock, so N pods booting concurrently
  (fresh DB or a rolling upgrade shipping `ALTER`s) do not race DDL — each pod
  blocks, then sees the migrated schema and no-ops. A rolling upgrade is safe.
- After deploying a new image, run `bb verify <base-url>` to confirm the
  frontend / packages / backend sections all match the shipped build.
- The package sync on boot is **declarative for package identities**: a fn
  a bundled package no longer declares (deterministic `uuid-v5(ns, name)`
  id, `_anon-*` shapes included) is purged **only** when nothing the
  current graph still follows references it — the newest version of each
  row on a live branch, never a create-time identity row or a superseded
  version (`identity-repair/inbound-refs-many` `:live-only?`). A retired fn
  a user fn still references stays, with one aggregated boot warning, and
  shows in the editor's ⚐ lint filter as a duplicate. Editor-created rows
  (random ids) are never touched. Preview before an upgrade from a REPL:
  `(packages.sync/reconcile-moved-identities! storage packages {} {:preexisting-fn-ids …
  :dry-run? true})` returns `{:moves :purgeable :kept}` and writes nothing.

## Health & readiness

> **Cold-boot cost (canonical figure).** A cold start — package load, the
> type-check sweep, then eager-compile of every fn-def — takes **~30–40 s to
> serving** (measured, after the compile/sweep memoization that cut
> `compile-all` from ~60 s to ~1.2 s; the sweep now dominates). Size the
> Dockerfile healthcheck `start-period` and any k8s startup probe against this.
> Other docs reference THIS number rather than restating it. A *runtime*
> full recompile (no sweep — just `compile-all`) is now a few seconds, not the
> tens of seconds it once was.

- `GET /health` — **readiness** signal (and startup). It is a graph fn behind
  the compiled registry, so it stays 200 only while the registry is warm: a pod
  mid-cold-boot reads not-ready and is correctly de-rotated. Give the startup
  probe a start-period generous enough for the boot compile (see the Dockerfile
  healthcheck `start-period`). A *runtime* full clear (a nil-seed write, a
  migration) no longer de-rotates the pod: it is **stale-while-revalidate** —
  the ctx keeps serving its existing (stale) registry while a background
  `rebuild-optimistic!` refreshes it, so `/health` stays 200 and no request
  blocks behind the recompile (a few seconds since the `compile-all`
  memoization — see the cold-boot note above). Staleness is bounded by one rebuild; the
  window is served, never hung. (Only a genuinely cold holder — boot, or a
  divergent cold branch's first access — still compiles on the request path,
  and that is per-ctx, so it never hangs the whole pod.)
- `GET /livez` — **liveness** signal. A static, registry-independent 200 (it
  short-circuits in `branch-router/dispatch` before any registry access). Point
  the k8s livenessProbe / any restart-on-unhealthy check HERE, never at
  `/health`: a liveness probe on `/health` kills a busy-but-alive pod during a
  runtime recompile → discards the in-flight compile → a cold-boot cascade
  (the full ~30–40 s cold start above, paid over and over). `/livez` keeps the
  pod alive; `/health` (readiness) de-rotates it
  until the compile lands. (Helm chart: `startup + readiness → /health`,
  `liveness → /livez`.)
- `GET /version` — the three build-section hashes (`bb verify` consumes these).
- `GET /metrics` — a JVM + structural-counter snapshot (JSON). For a Prometheus
  scrape target, see the observability notes in [SCALING.md](SCALING.md).

## Editor bricked by an asset override

The Operate → Assets panel edits the editor's own JS/CSS (versioned
`:resource-override` rows). A syntax-broken **JS** override is refused
client-side, but a broken override applied by another path — or a bad
CSS/structure change — can leave the editor unusable, and the panel
lives *inside* that editor. Recovery:

1. Restart the pod(s) with `GRAPHDEN_DISABLE_ASSET_OVERRIDES=1` — every
   frontend asset then serves its shipped classpath baseline (bytes and
   the `?v=` hash both ignore the override rows), so the editor loads
   clean.
2. Open Operate → Assets and **revert** the offending override.
3. Unset `GRAPHDEN_DISABLE_ASSET_OVERRIDES` and restart.

The hatch is read at serve time, so it takes effect on restart and needs
no data change. It is a platform/self-host knob — in the cloud the
entity is tenant-forbidden, so only an operator can create or revert
overrides.

## Suspending an abusive org

To freeze a misbehaving tenant near-instantly, set its plan to `suspended` — no
effects, all row ceilings 0, so it can neither run nor create anything (it can
still delete its own data). The route is platform-only (authenticate as the
platform/operator, not as the tenant):

```bash
# freeze
curl -sS -X POST "$BASE/api/orgs/plan" -H "Authorization: Bearer $OPERATOR_TOKEN" \
  --data-urlencode name=<org-slug> --data-urlencode plan=suspended

# restore (back to its real tier)
curl -sS -X POST "$BASE/api/orgs/plan" -H "Authorization: Bearer $OPERATOR_TOKEN" \
  --data-urlencode name=<org-slug> --data-urlencode plan=free
```

`plan` must be one of `free` / `network` / `dedicated` / `suspended`; an unknown
slug is rejected rather than silently applied. The change is effective on the
org's next request (no memo). The same route upgrades / downgrades a paying org.
See [PLANS.md § Suspending an org](PLANS.md).

## Inactive free organizations

The tenancy addon's reaper (`:tenancy/demo-gc`, the same sweep that purges
expired demo orgs, hourly) also runs the free-org lifecycle when the accounts
tables exist:

1. **Mark** — a `plan = free` org with no activity for
   `GRAPHDEN_FREE_ORG_INACTIVE_DAYS` (default 60; `<= 0` disables the
   lifecycle) gets `expires_at = now + GRAPHDEN_FREE_ORG_GRACE_DAYS`
   (default 14, floored at 1). Activity is any of: a session of the owner
   or a member created OR used (`session.last_used_at`, refreshed at most
   hourly by every authenticated request — a browser cookie or an API
   token in daily use), an execution in the org, a graph edit in the org
   (an fn or binding version), or the owner account being younger than
   the window.
2. **Warn** — in the same pass, the owner and every user-member with a
   verified primary email receive "Your Graphden organization X will be
   deleted on <date>" (the accounts Mailer; pending email invitees are not
   mailed), and the org row's `warned_at` is set. Any marked org without
   `warned_at` — one marked before the warning shipped — is caught up on
   the next sweep. `GRAPHDEN_FREE_ORG_REMIND_DAYS` (default 2) before the
   deletion a "Last reminder" goes out once (`reminded_at`). Both columns
   clear with the mark on rescue. Without `:mailer` / `:link-origin` on
   `:tenancy/demo-gc` the mark is only logged (`demo-gc: marked orgs NOT
   warned`).
3. **Rescue** — any sign-in, token use, run or edit inside the grace
   clears `expires_at` (and the mail ledger) on the next sweep, before that
   sweep purges anything.
4. **Purge** — past `expires_at` the org, its graph rows and its grants are
   hard-deleted. Every account is kept — the owner's too: a real person may
   own or belong to other orgs. (Only a demo org's purge also takes the
   ephemeral anonymous account that owns it, with its sessions and grants.)

Paid (`network` / `dedicated`) and `suspended` orgs are never selected.
Logs: `demo-gc: marked inactive free orgs …`, `demo-gc: warned <org> → n of m
recipients`, `demo-gc: un-marked returned free orgs …`, `demo-gc: purged
expired org …`.

## Your responsibility vs Graphden's

Graphden gives you: idempotent fleet-safe schema migration, self-healing
in-memory state, health/version/verify surfaces. **You** own: Postgres
backups, HA, and failover topology; Vault/OpenBao backups; the network
perimeter and TLS; and OS/container patching.
