Graphden documentation

RFC: Dynamic fleet — load-based placement & rebalancing

Status — maturity, honestly:

Layer State Verified where
Static fleet (org sharding, 421 backstop, per-org quota, advisory-lock singletons, SSE invalidation, BYO) implemented single-host multi-process + testcontainers
Dynamic fleet (placement, rebalancing, control loop, discovery) — Phases 0-3 core implemented single-host kind cluster, in-cluster Postgres (§8, §12)
T4.5 overlap-accounting implemented, opt-in (GRAPHDEN_FLEET_OVERLAP_WEIGHT, default off) —
CRaC checkpoint / T5.3 scale-to-zero demonstrated PoC KEDA+CRaC on kind; CI bake is manual/off-PR

Not yet exercised by anything above: real cross-machine network latency / partitions, a real load balancer, and — critically — advisory-lock singleton / leader behaviour across a real Postgres failover. Read every “SHIPPED” below as code merged and single-host-verified, not production-hardened across machines. Ops guide: FLEET_DEPLOY.md.

T4.5 overlap-accounting is SHIPPED opt-in (GRAPHDEN_FLEET_OVERLAP_WEIGHT, default off); still open: the per-route split (evidence-gated) and the CRaC-gated pieces (T5.3 scale-to-zero is demonstrated on KEDA+CRaC — the CRaC checkpoint CI bake shipped too). The as-built STATIC fleet (org sharding, 421 backstop, per-org quota, advisory-lock singletons, SSE invalidation, BYO) is the base layer in SCALING.md; this document designed — and the branch then built — the dynamic step on top: automatic placement and rebalancing of tenant graphs across a fleet of executors by load and size, and the runtime substrate to run it on.

Read SCALING.md first — this RFC extends it and reuses most of its machinery. The problem statements and phased checklist below retain the original design context. For current installation commands and the shipped L1 dedicated-shard assembly, use FLEET_DEPLOY.md; the CRaC/scale-to-zero track remains a demonstrated PoC, as distinguished in the maturity table.

1. Problem & goals

Today assignment is static and operator-configured: each pod is told its org set via GRAPHDEN_EXECUTOR_ORGS, loads that shard at startup, and 421s anything else. There is no brain that reacts to load, and the shard predicate is fixed at context-build time.

Goals:

Non-goals (explicit, to bound scope):

2. The core tension

Static org sharding is deliberately brainless: the org is in the request’s hostname, so routing needs no coordinator and no per-call lookup (SCALING.md § “Sharding by org”). Load-based rebalancing breaks that assumption and forces three new pieces into existence:

  1. A live routing map unit → executor. Today: none (static env var). If assignments change at runtime, both the router and the pods must agree on the current owner.
  2. Dynamic membership. An executor must load and evict a unit at runtime. Today it only loads its shard at startup. (See §6.2 — most of this machinery already exists.)
  3. The placement controller (the brain). Computes each unit’s weight from fn-count (from the graph) and load (from live metrics), packs units onto executors, and moves units when the fleet drifts out of balance — with hysteresis so it doesn’t thrash, and churn-minimisation so a rebalance moves the fewest units.

None of the three exist yet; all three are prerequisites for rebalancing.

3. Unit of placement: the cell, not the org

The as-built shard key is the whole org. This RFC generalises it: the unit of placement is a cell = a root fn plus its transitive forward ref-closure. An org has one or more cells; a pod holds a mix of cells from many orgs (a sleeping admin endpoint of org A packed next to a hot checkout endpoint of org B). “Whole org = one cell” is the trivial case we start from.

Indivisibility (the §1 non-goal made concrete). A cell is atomic: its whole closure must live on the pod that runs it, because a single call can’t span pods. A cell can only be split off at an independent entry point — a route, endpoint, or service whose closure doesn’t call into another cell’s private fns. A cell’s closure never leaves org ∪ public — already guaranteed by reject-cross-org-refs!.

3.1 What is already a cell (verified against the code)

3.2 Overlap reality → default granularity

Independent route-cells heavily overlap: within an org they all converge on the same shared library (web/reitit, web/html, web/response, web/crud, :pg-query, the whole _app-ring-response wrap chain). Splitting one app across pods duplicates that shared closure on each pod. So per-route splitting pays off only under genuine hot/cold asymmetry with a large per-cell unique closure relative to the shared base.

Therefore the default granularity for Phase 0/1:

Per-route splitting is a Phase 2+ refinement, gated on evidence that route-level asymmetry justifies the overlap cost. The cell abstraction is designed to allow it; we don’t build it early. (This matches the “combine sleeping + hot pieces” intent — but the pieces should be genuinely independent, like separate services or apps, not routes of one small app.)

3.3 Routing consequence

Addressing becomes (org, entry), not just org — needs either L7 (path-aware) edge routing, or an internal forward hop between sibling pods (§6.1). This is the price of fine granularity, called out as a first-class trade-off.

4. Two orthogonal axes (correcting a false coupling)

Pod footprint and placement model are independent — do both:

Axis Options Note
Footprint / start fat JVM (~655 MB, slow start) ↔ faster-start options (§5.1) Worth doing under any model; shrinks the base tax and the cold-start that makes evict/rebalance/scale-to-zero cheap.
Placement model own control plane (this RFC) ↔ delegate to Knative-per-service Independent of footprint.

Even a much smaller base doesn’t remove the packing problem: at 100–200 orgs it is the same order of magnitude, and we deliberately pack pieces of orgs together for load levelling. The footprint work is a cost/latency optimisation, not a substitute for the control plane.

5. Chosen direction

Own control plane (grouped cells) + a footprint/start track in parallel.

5.1 Footprint / start track — GraalVM is OUT, CRaC is the candidate

GraalVM native-image is NOT feasible without a package-layer rearchitecture (verified). The blocker is fundamental, not incidental: base-fn implementations are shipped as Clojure source (resources/packages/**/impls.clj, 32 files) and eval‘d at boot with macroexpansion (packages/loader.clj load-impls-via-eval) — runtime Clojure compilation, which native-image forbids, on the production startup path. Going native would mean converting all 32 impls to AOT namespaces, killing eval-load, force-AOTing ~20 requiring-resolve targets, dropping the runtime-require addon manifest, and authoring reflection config for 5 native-unfriendly deps (postgres, next.jdbc, HikariCP, http-kit, hiccup). That is a rewrite of the package layer — not a spike. Shelved.

Candidates that are compatible with the eval-load model, in payoff order:

Environment feasibility CONFIRMED in this sandbox (2026-07). Both pieces are obtainable and functional here:

So the gate is no longer “is the environment capable” — it is the CRaC integration itself (§8 track).

Integration PoC run (development/crac/, 2026-07): a warm graphden JVM that has done the expensive load-packages (eval of 32 impls.clj → 251 base-fns, 4484 ms) checkpointed to a 201 MB image and restored in ~41 ms —

100× vs the load, >800× vs a cold boot. That is the value, demonstrated. The PoC also surfaced the two blockers a production path must solve:

  1. Native-library temp-mmap — FIXED. brotli4j (deps.edn, the :brotli-bytes base-fn) extracted libbrotli.so to a random ${tmpdir}/com_aayushatharva_brotli4j_<nanoTime>/ dir marked deleteOnExit and mmapped it; CRIU restore then failed once that file was gone (Cannot open mapped file …/libbrotli.so) — so only the very first restore worked. Resolved in web/http/impls.clj: extract the .so once to a deterministic path (-Dgraphden.native-lib.dir, default ${tmpdir}/graphden-native) and point brotli4j at it via its supported brotli4j.library.path property → System.load of a stable, image-resident file, no temp dir, no deleteOnExit. Restore is now reproducible (3/3 restores OK, ~30–38 ms). General rule for any future extract-and-mmap native lib: pin it to a stable path the checkpoint image ships.
  2. Live external connections. A full-system checkpoint needs CRaC Resource handlers to close-before / reopen-and-rewire-after every live resource (Hikari→Postgres pool, http-kit listener, vault/openbao client, notify + advisory-lock connections, SSE). CRIU won’t snapshot live sockets without app cooperation; the executor context holds a reference to the pool-wrapping storage, so reopening means re-threading it.

So CRaC is feasible and high-value, but the integration is a real feature, not a config flip — see development/crac/README.md.

Corollary: since the ~655 MB base can’t shrink cheaply, that reinforces grouping (Path A) — the base is paid once per pod and shared by all cells the pod holds. The base duplication across pods (cells overlap heavily on web/*, §3.2) is exactly why packing many cells per pod wins.

6. Architecture

                     ┌────────────────────────────────────────────┐
   request           │  Edge router (L7)                           │
  (org, path) ─────► │  reads the live routing map (org,entry)→pod │
                     └───────────────┬────────────────────────────┘
                        route to owner│         (miss → 421 backstop,
                                       ▼          or internal forward)
              ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
              │  executor 1 │  │  executor 2 │  │  executor 3 │
              │ cells: a1,  │  │ cells: b1,  │  │ cells: a2,  │
              │  c3, (base) │  │  a3, (base) │  │  b2, (base) │
              └─────────────┘  └─────────────┘  └─────────────┘
                     ▲ load/evict cell (delta-compile / evict)
                     │ heartbeat: held cells + live weight
              ┌──────┴───────────────────────────────────────────┐
              │  Placement controller (the brain)                 │
              │  weight(cell)=f(fn-count, load); pack; rebalance  │
              │  writes the routing map; drives load/evict        │
              └───────────────────────────────────────────────────┘

6.1 Live routing map

6.2 Dynamic membership — most of the machinery already exists

Verified against the executor:

6.3 Placement controller

7. Runtime substrate

7.1 Tenant service isolation — effect gate + dedicated shard + CRaC

A tenant :service is a persistent process running the tenant’s own graph (cron/:schedule, long-poll bots, always-on listeners). The free tier is a full FaaS without services precisely because a persistent tenant process raises a concern the per-request FaaS path never does: resource fairness in a shared JVM. This section is the isolation model that makes tenant services safe, and the concrete path to enabling them (task #6 parts 2/4).

Two boundaries, not one. The effect gate and a resource boundary are orthogonal, and a safe tenant service needs both:

Boundary Bounds Status Mechanism
Capability (authz) What the service may do — :network/:env/:io/… SHIPPED (task #6 parts 1+3) cr/*allowed-effects* per the org’s plan, conveyed into the service’s worker thread (conveyed-dynamic-vars) via cr/run-service-scoped
Resource (fairness) How much CPU / heap / threads / wall-clock it consumes NOT shipped — the launch gap OS/cgroup limit on a runtime the tenant does not share with others

The effect gate is an authz boundary; it does not cap CPU, heap, or thread count. A (loop [] (recur)) service pins a core; an unbounded accumulator OOMs the shared heap; and — critically — the JVM has no hard per-thread heap cap, so a malicious tenant can crash a shared pod no matter how the in-process code is sandboxed. The honest resource boundary therefore lives below the JVM, at the OS/container level. And the fleet packer (§6.3) deliberately spreads cells for load, not for isolation (fleet/packer.clj: “spread for load, not for isolation”) — a shared pod holds a mix of many orgs’ cells — so the default fleet does not provide this boundary. It must be added as a placement policy, not new isolation tech.

Three levels, in order of strength (and cost):

Enabling task #6 (safe tenant services) via L1 — concrete checklist:

  1. Plan flag :dedicated-executor? (or :tier :dedicated) in tenancy/plan.clj → only such an org may create a :service. Free/shared-tier orgs stay service-less (the FaaS boundary is unchanged).
  2. Provision the dedicated shard: a StatefulSet with :executor-orgs #{org} + tight Helm resources.limits. This is config assembly of shipped primitives (the predicate + the 421 path already run everywhere) plus the ops runbook — no new executor code.
  3. Task #6 parts 2/4 (Option B, below) so the tenant can create/list services, capped by plan. :service stays in tenant-forbidden-entities (the platform reconciler must read every org’s services, and moving :service into default-scoped-entities hides tenant rows from the platform at the DB layer — Postgres FORCE ROW LEVEL SECURITY under a non-BYPASSRLS app role, so the base read never sets the org GUC and the RLS policy filters it to public). Instead: add an :org-id field to :service; a tenant creates and lists services through dedicated endpoints that write/read via the base (unrestricted) storage, stamping / filtering by (tc/current-org) — exactly how tenancy/plan.clj reads the tenant-forbidden :org — and enforcing a :max-services cap. The shipped cr/run-service-scoped seam then sandboxes each service by its :org-id.

The result: a paid org with :dedicated-executor? runs its services on its own cgroup-limited pod, where the shipped effect gate and the k8s limit together bound both capability and resources.

8. Phased plan (incremental — each phase ships value alone)

9. Relationship to graph hot-reload (already shipped)

A likely question: “when the DB graph changes (package install, branch merge, user edit), every executor using that graph must refresh, and requests must only reach up-to-date executors — isn’t that this task?” Mostly it is already built, and it is the foundation this RFC reuses — not new work.

10. What we reuse from today

11. Open questions / risks

12. Decision log