Status — maturity, honestly:
| Layer | State | Verified where |
|---|---|---|
Static fleet (org sharding, 421 backstop, per-org quota, advisory-lock singletons, SSE invalidation, BYO) |
implemented | single-host multi-process + testcontainers |
| Dynamic fleet (placement, rebalancing, control loop, discovery) — Phases 0-3 core | implemented | single-host kind cluster, in-cluster Postgres (§8, §12) |
| T4.5 overlap-accounting | implemented, opt-in (GRAPHDEN_FLEET_OVERLAP_WEIGHT, default off) |
— |
| CRaC checkpoint / T5.3 scale-to-zero | demonstrated PoC | KEDA+CRaC on kind; CI bake is manual/off-PR |
Not yet exercised by anything above: real cross-machine network latency / partitions, a real load balancer, and — critically — advisory-lock singleton / leader behaviour across a real Postgres failover. Read every “SHIPPED” below as code merged and single-host-verified, not production-hardened across machines. Ops guide: FLEET_DEPLOY.md.
T4.5 overlap-accounting
is SHIPPED opt-in (GRAPHDEN_FLEET_OVERLAP_WEIGHT, default off); still open: the
per-route split (evidence-gated) and the CRaC-gated pieces (T5.3 scale-to-zero is
demonstrated on KEDA+CRaC — the CRaC checkpoint CI bake shipped too). The as-built STATIC fleet (org sharding,
421 backstop, per-org quota, advisory-lock singletons, SSE invalidation, BYO)
is the base layer in SCALING.md; this document designed — and the
branch then built — the dynamic step on top: automatic placement and
rebalancing of tenant graphs across a fleet of executors by load and size, and
the runtime substrate to run it on.
Read SCALING.md first — this RFC extends it and reuses most of its machinery. The problem statements and phased checklist below retain the original design context. For current installation commands and the shipped L1 dedicated-shard assembly, use FLEET_DEPLOY.md; the CRaC/scale-to-zero track remains a demonstrated PoC, as distinguished in the maturity table.
Today assignment is static and operator-configured: each pod is told its
org set via GRAPHDEN_EXECUTOR_ORGS, loads that shard at startup, and 421s
anything else. There is no brain that reacts to load, and the shard predicate is
fixed at context-build time.
Goals:
421 a rare backstop, not the mechanism.Non-goals (explicit, to bound scope):
Static org sharding is deliberately brainless: the org is in the request’s hostname, so routing needs no coordinator and no per-call lookup (SCALING.md § “Sharding by org”). Load-based rebalancing breaks that assumption and forces three new pieces into existence:
unit → executor. Today: none (static env var). If
assignments change at runtime, both the router and the pods must agree on the
current owner.fn-count (from the graph) and load (from live metrics), packs units onto
executors, and moves units when the fleet drifts out of balance — with
hysteresis so it doesn’t thrash, and churn-minimisation so a rebalance moves
the fewest units.None of the three exist yet; all three are prerequisites for rebalancing.
The as-built shard key is the whole org. This RFC generalises it: the unit of placement is a cell = a root fn plus its transitive forward ref-closure. An org has one or more cells; a pod holds a mix of cells from many orgs (a sleeping admin endpoint of org A packed next to a hot checkout endpoint of org B). “Whole org = one cell” is the trivial case we start from.
Indivisibility (the §1 non-goal made concrete). A cell is atomic: its whole
closure must live on the pod that runs it, because a single call can’t span pods.
A cell can only be split off at an independent entry point — a route,
endpoint, or service whose closure doesn’t call into another cell’s private fns.
A cell’s closure never leaves org ∪ public — already guaranteed by
reject-cross-org-refs!.
:service row names a :fn-id that MUST have
zero free arguments (enforced at create-time — schema/services/schema.clj),
and the reconciler runs it standalone via cr/execute ctx fn-id {}. A
zero-free-arg root the executor compiles independently is a cell. :web-server
ships as the baseline service. So every service row already names a cell root
— these are the first placeable units, no new work.:app-route today. The fleet reads each
:app-route row’s handler-fn-id (the graph :domain-router resolves
Host→route and executes it with {:request request}). Routes WITHIN a handler
exist as data (each is a fn-def parented from :get-route, gathered into one
list fed to one compiled reitit router), but they are not independently
addressable — every request to that app enters the one top handler.:router-or-nil already exists for fall-through composition), and point a
per-cell handler-fn-id at each. Deferred — see §3.2.Independent route-cells heavily overlap: within an org they all converge on
the same shared library (web/reitit, web/html, web/response, web/crud,
:pg-query, the whole _app-ring-response wrap chain). Splitting one app across
pods duplicates that shared closure on each pod. So per-route splitting pays off
only under genuine hot/cold asymmetry with a large per-cell unique closure
relative to the shared base.
Therefore the default granularity for Phase 0/1:
Per-route splitting is a Phase 2+ refinement, gated on evidence that route-level asymmetry justifies the overlap cost. The cell abstraction is designed to allow it; we don’t build it early. (This matches the “combine sleeping + hot pieces” intent — but the pieces should be genuinely independent, like separate services or apps, not routes of one small app.)
Addressing becomes (org, entry), not just org — needs either L7 (path-aware)
edge routing, or an internal forward hop between sibling pods (§6.1). This is the
price of fine granularity, called out as a first-class trade-off.
Pod footprint and placement model are independent — do both:
| Axis | Options | Note |
|---|---|---|
| Footprint / start | fat JVM (~655 MB, slow start) ↔ faster-start options (§5.1) | Worth doing under any model; shrinks the base tax and the cold-start that makes evict/rebalance/scale-to-zero cheap. |
| Placement model | own control plane (this RFC) ↔ delegate to Knative-per-service | Independent of footprint. |
Even a much smaller base doesn’t remove the packing problem: at 100–200 orgs it is the same order of magnitude, and we deliberately pack pieces of orgs together for load levelling. The footprint work is a cost/latency optimisation, not a substitute for the control plane.
Own control plane (grouped cells) + a footprint/start track in parallel.
GraalVM native-image is NOT feasible without a package-layer rearchitecture
(verified). The blocker is fundamental, not incidental: base-fn implementations
are shipped as Clojure source (resources/packages/**/impls.clj, 32 files)
and eval‘d at boot with macroexpansion (packages/loader.clj
load-impls-via-eval) — runtime Clojure compilation, which native-image forbids,
on the production startup path. Going native would mean converting all 32 impls to
AOT namespaces, killing eval-load, force-AOTing ~20 requiring-resolve targets,
dropping the runtime-require addon manifest, and authoring reflection config for
5 native-unfriendly deps (postgres, next.jdbc, HikariCP, http-kit, hiccup). That
is a rewrite of the package layer — not a spike. Shelved.
Candidates that are compatible with the eval-load model, in payoff order:
Resource handlers to close/reopen the DB pool
and sockets across the checkpoint, and privileged CRIU at restore.
Highest-payoff realistic path; architecturally a fit.Environment feasibility CONFIRMED in this sandbox (2026-07). Both pieces are obtainable and functional here:
apt-get install criu (v3.16.1), criu check → “Looks good”.
Kernel 5.15 has CAP_CHECKPOINT_RESTORE, and the process already holds
CAP_SYS_ADMIN / CAP_SYS_PTRACE / CAP_CHECKPOINT_RESTORE. In prod a pod
needs those caps (--privileged, or --cap-add=CHECKPOINT_RESTORE,SYS_PTRACE,SYS_ADMIN
--security-opt seccomp=unconfined).zulu21.50.19-ca-crac-jdk21.0.11 runs and exposes
CRaCCheckpointTo / CRaCRestoreFrom. A checkpoint of a dirty ~300 MB heap
succeeded → a ~330 MB image; restore is functional (criu “Restore successful!”).So the gate is no longer “is the environment capable” — it is the CRaC integration itself (§8 track).
Integration PoC run (development/crac/, 2026-07): a warm graphden JVM that
has done the expensive load-packages (eval of 32 impls.clj → 251 base-fns,
4484 ms) checkpointed to a 201 MB image and restored in ~41 ms —
100× vs the load, >800× vs a cold boot. That is the value, demonstrated. The PoC also surfaced the two blockers a production path must solve:
brotli4j (deps.edn, the
:brotli-bytes base-fn) extracted libbrotli.so to a random
${tmpdir}/com_aayushatharva_brotli4j_<nanoTime>/ dir marked deleteOnExit
and mmapped it; CRIU restore then failed once that file was gone
(Cannot open mapped file …/libbrotli.so) — so only the very first restore
worked. Resolved in web/http/impls.clj: extract the .so once to a
deterministic path (-Dgraphden.native-lib.dir, default
${tmpdir}/graphden-native) and point brotli4j at it via its supported
brotli4j.library.path property → System.load of a stable, image-resident
file, no temp dir, no deleteOnExit. Restore is now reproducible
(3/3 restores OK, ~30–38 ms). General rule for any future extract-and-mmap
native lib: pin it to a stable path the checkpoint image ships.Resource
handlers to close-before / reopen-and-rewire-after every live resource
(Hikari→Postgres pool, http-kit listener, vault/openbao client, notify +
advisory-lock connections, SSE). CRIU won’t snapshot live sockets without app
cooperation; the executor context holds a reference to the pool-wrapping
storage, so reopening means re-threading it.So CRaC is feasible and high-value, but the integration is a real feature, not
a config flip — see development/crac/README.md.
Corollary: since the ~655 MB base can’t shrink cheaply, that reinforces
grouping (Path A) — the base is paid once per pod and shared by all cells the pod
holds. The base duplication across pods (cells overlap heavily on web/*,
§3.2) is exactly why packing many cells per pod wins.
┌────────────────────────────────────────────┐
request │ Edge router (L7) │
(org, path) ─────► │ reads the live routing map (org,entry)→pod │
└───────────────┬────────────────────────────┘
route to owner│ (miss → 421 backstop,
▼ or internal forward)
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ executor 1 │ │ executor 2 │ │ executor 3 │
│ cells: a1, │ │ cells: b1, │ │ cells: a2, │
│ c3, (base) │ │ a3, (base) │ │ b2, (base) │
└─────────────┘ └─────────────┘ └─────────────┘
▲ load/evict cell (delta-compile / evict)
│ heartbeat: held cells + live weight
┌──────┴───────────────────────────────────────────┐
│ Placement controller (the brain) │
│ weight(cell)=f(fn-count, load); pack; rebalance │
│ writes the routing map; drives load/evict │
└───────────────────────────────────────────────────┘
:placement entity in the shared Postgres —
(org, entry-fn-id) → executor-id, epoch-versioned so a move is atomic to
readers.graphden_events / SSE).
Two shapes:
(Host, path) → upstream.421. Simpler edge,
extra hop. Start here, add L7 later.Verified against the executor:
:compiled-registry
= {fn-id → closure} (executor/context.clj). delta-recompile!
(compile_runtime.clj) already does a surgical swap!: it dissocs pruned
fn-ids and re-inserts only a recompiled subset via ce/compile-subset on top of
the existing map. So per-cell add/evict is a swap! on an atom that already
supports it — not a wholesale rebuild.compile-subset over the FORWARD closure of its root. The
:forward-deps index is already built (compile/deps.clj build-deps-state);
compile-subset already compiles an arbitrary dependency-ordered subset. The
one missing named piece is a forward-closure walk (transitive-blast
already does this walk, but over :reverse-deps for invalidation; we need the
same walk over :forward-deps).forward-closure(root) \ ⋃ forward-closure(other
loaded roots) — the :forward-deps index gives the per-fn sets to compute the
difference.:invalidation-lock ReentrantLock +
CAS swap! mean per-cell add/evict under the lock is safe with no new
synchronization.:executor-orgs already accepts
a fn (not just a set); pass a fn closing over a mutable atom. The real work is
not the predicate read — it’s keeping the compiled registry coherent as the set
changes, which is the load/evict problem above.fn-count per cell from the graph (static-ish). load from the
per-org pending-execution counter we already keep, request rate, and pod CPU.
weight = w1·fn-count + w2·load.docker-compose container — fine for one
node, not a fleet.A tenant :service is a persistent process running the tenant’s own graph
(cron/:schedule, long-poll bots, always-on listeners). The free tier is a full
FaaS without services precisely because a persistent tenant process raises a
concern the per-request FaaS path never does: resource fairness in a shared
JVM. This section is the isolation model that makes tenant services safe, and
the concrete path to enabling them (task #6 parts 2/4).
Two boundaries, not one. The effect gate and a resource boundary are orthogonal, and a safe tenant service needs both:
| Boundary | Bounds | Status | Mechanism |
|---|---|---|---|
| Capability (authz) | What the service may do — :network/:env/:io/… |
SHIPPED (task #6 parts 1+3) | cr/*allowed-effects* per the org’s plan, conveyed into the service’s worker thread (conveyed-dynamic-vars) via cr/run-service-scoped |
| Resource (fairness) | How much CPU / heap / threads / wall-clock it consumes | NOT shipped — the launch gap | OS/cgroup limit on a runtime the tenant does not share with others |
The effect gate is an authz boundary; it does not cap CPU, heap, or thread
count. A (loop [] (recur)) service pins a core; an unbounded accumulator OOMs
the shared heap; and — critically — the JVM has no hard per-thread heap cap,
so a malicious tenant can crash a shared pod no matter how the in-process code is
sandboxed. The honest resource boundary therefore lives below the JVM, at the
OS/container level. And the fleet packer (§6.3) deliberately spreads cells for
load, not for isolation (fleet/packer.clj: “spread for load, not for
isolation”) — a shared pod holds a mix of many orgs’ cells — so the default fleet
does not provide this boundary. It must be added as a placement policy, not
new isolation tech.
Three levels, in order of strength (and cost):
L0 — in-JVM governance (REJECTED for untrusted tenants). Per-service thread-pool cap + a wall-clock/CPU watchdog that interrupts a runaway service + per-org soft heap accounting. This is best-effort fairness, not isolation: with no hard per-thread heap cap, a determined tenant still OOMs the pool. Acceptable only for a trusted tenant set, never for open signup.
:executor-orgs (docs/SCALING.md §Sharding) — a membership predicate on the
ExecutionContext. A pod compiles + serves only its shard’s orgs; the
reconciler on that pod starts only that shard’s services; an off-shard
request 421s. Set :executor-orgs #{org} and the pod is that org’s
dedicated executor.resources.limits (deploy/helm/graphden/values.yaml) — the
StatefulSet already carries k8s CPU/memory requests + limits (cgroups).Together: run a small StatefulSet whose shard predicate is a single paid org, under tight k8s limits. That org’s service cells run only there; a runaway or OOM crashes only that tenant’s own pod, never the shared fleet. Effect gate (shipped) bounds what, the cgroup bounds how much — that is the safe tenant service. The cost is a warm pod (or small pool) per paid tenant — real infra spend, which is exactly why services are the honest paid line.
Enabling task #6 (safe tenant services) via L1 — concrete checklist:
:dedicated-executor? (or :tier :dedicated) in tenancy/plan.clj
→ only such an org may create a :service. Free/shared-tier orgs stay
service-less (the FaaS boundary is unchanged).:executor-orgs #{org} +
tight Helm resources.limits. This is config assembly of shipped
primitives (the predicate + the 421 path already run everywhere) plus the
ops runbook — no new executor code.:service stays in tenant-forbidden-entities
(the platform reconciler must read every org’s services, and moving :service
into default-scoped-entities hides tenant rows from the platform at the DB
layer — Postgres FORCE ROW LEVEL SECURITY under a non-BYPASSRLS app role, so
the base read never sets the org GUC and the RLS policy filters it to
public). Instead: add an :org-id field to :service; a tenant creates and
lists services through dedicated endpoints that write/read via the base
(unrestricted) storage, stamping / filtering by (tc/current-org) — exactly
how tenancy/plan.clj reads the tenant-forbidden :org — and enforcing a
:max-services cap. The shipped cr/run-service-scoped seam then sandboxes
each service by its :org-id.The result: a paid org with :dedicated-executor? runs its services on its own
cgroup-limited pod, where the shipped effect gate and the k8s limit together
bound both capability and resources.
421 becomes rare; assignments change
with no restart. Concrete tasks:
forward-closure(root) — transitive walk over :forward-deps
(compile/deps.clj). Defines a cell’s contents. Small; building blocks exist.load-cell!(ctx, root) — forward-closure → compile-subset on top of
the existing registry → swap! under :invalidation-lock. Reuses the
delta-recompile! plumbing.loaded-roots bookkeeping + refcount + evict-cell! — the one new
abstraction (§6.2).org-in-shard? → cell-held?. (Decision: keep read-graph org-coarse at
first, or move it to “load closures of held roots + public” — smaller step
first.):placement routing table (org, entry) → executor, epoch-versioned;
populated manually.421).fleet.metrics — cell-fn-count (forward-closure size),
pending-loads (the quota’s own :pending :fn-execution rows, one
query per tick, counted per org),
cell-weight folding the two with overridable weights.fleet.controller/move-cell! — the load→flip-epoch→evict
orchestration with its invariants (load-before-flip, abort-before-flip,
evict-after-flip), cross-pod effects behind load-on / evict-on seams,
serialized under a move-monitor. Fully in-JVM tested.fleet.command — the directed, ACK-gated cross-pod transport
that fills the seams: POST /internal/fleet/cell/{load|evict}/{root} on
each pod (runs load-cell! / evict-cell!), gated by a shared internal
token (GRAPHDEN_INTERNAL_TOKEN), NOT the tenant auth-provider (a move is
cross-org platform authority). execute-move! is the ops/REPL entry.move-monitor orders a single controller’s moves; the leader-lock makes
ONE controller authoritative fleet-wide.fleet.packer — LPT bin-pack (heaviest cell → least-loaded pod),
the desired-placement baseline; budget-free (capacity is a separate read).fleet.rebalance — churn-minimising steepest-descent (move the
fewest cells that flatten load) + magnitude hysteresis (:min-improvement);
surfaces :unplaced cells separately for initial placement.fleet.control-loop — the pure plan-tick (initial placement
every tick, rebalance only on SUSTAINED imbalance via :sustain-ticks) plus
live-fleet discovery (:org app cells ∪ :placement, :service excluded)
and run-tick! executing through a move-fn seam.:exec/fleet-controller — the leader-locked periodic component
(own advisory-lock conn, re-asserted per tick) that drives run-tick! via
the directed cell-command transport. One controller fleet-wide.packer/best-target scores a pod as
load − w-overlap·|cell-closure ∩ pod-fns|, so cells sharing a forward-closure
co-locate (shared fns compiled once → less memory, cheaper moves). Threaded
through pack + control-loop/plan-initial-placements + discover-cells
(attaches :closure only when enabled). Off by default (w-overlap 0 =
pure LPT, the verified behaviour); a deployment opts in via
GRAPHDEN_FLEET_OVERLAP_WEIGHT — honouring the evidence-gate (turn it on when
multi-cell-per-org proves common). Per-route cell-splitting (§3.2) still waits
on that same evidence. Overlap biases BOTH initial placement (best-target)
AND rebalance: rebalance/best-move scores an imbalance-reducing move
imbalance-after − w-overlap·overlap(cell, dest-fns), so among moves that
equally flatten load the co-locating one wins (load stays the hard constraint).fleet.discovery) — headless-
Service SRV → live executor set, tracking StatefulSet/HPA scaling. Pure DNS,
no k8s API / RBAC.deploy/helm/graphden) — StatefulSet + headless +
front-door Service + Secret/ConfigMap + opt-in HPA. Each pod’s
GRAPHDEN_EXECUTOR_ID is its FQDN; the controller runs in-app (no separate
operator/CRD). See FLEET_DEPLOY.md.privileged and cannot
capabilities.add, while CRIU restore needs CAP_SYS_ADMIN /
CAP_CHECKPOINT_RESTORE in-container. So Knative’s fast-restore is blocked
by design, not by plumbing./health 200 in ~3.1–3.5 s from 0 pods (raw CRIU
restore in-pod ~1.2 s; the rest is KEDA detect + schedule + 1 s probe). vs
Knative’s 115 s → ~33×. This is the practical scale-to-zero path; Knative
stays the reference for the caps limitation.graphden.crac
quiesce/resume blockers fixed — see development/crac/README.md). Measured on
the real image: checkpoint 219 MB, restore→serving ~178 ms vs ~141 s cold
boot (~780×). Decision: adopt CRaC as the footprint track — it directly
unblocks scale-to-zero (T5.3). AppCDS is DE-PRIORITISED: the boot is
compute-bound (type-check + eager-compile), which AppCDS doesn’t touch.
The CI bake shipped: .github/workflows/crac-bake.yml (manual, off the PR
path — CRIU needs a privileged/CRIU-capable runner) builds the jar, checkpoints
in the azul base against a Postgres service, builds Dockerfile.crac, and
smoke-tests restore→/health. NOTE the restore image only restores if the
checkpoint was path-aligned (all mmap paths under /app + the base’s JDK) —
the host build-checkpoint.sh is NOT restorable by the container image; use
build-checkpoint-in-container.sh (found 2026-07-12).
NOT GraalVM.
resume! does
not re-read JDBC_URL; it resumes the SAME HikariDataSource from the
checkpoint, re-dialing the frozen jdbcUrl + creds. So a CRaC image is pinned
to the DB URL it was baked against — you cannot bake against a throwaway PG in
CI and set JDBC_URL=…rds at restore. Prod shape: front the cloud PG with a
STABLE logical endpoint (k8s ExternalName, or a PgBouncer proxy holding the
real secret) identical at bake (CI) and restore (prod); the demo’s crac-pg
Service name is that stand-in. Env-portable alternative (NOT shipped): have
resume! REBUILD :db/postgres from env on restore instead of resuming the
frozen pool — a real refactor (the system map holds direct references to the
frozen HikariDataSource; needs a DataSource indirection + consumer re-init),
worth it only if the endpoint-indirection proves operationally awkward. Full
write-up: deploy/kind/keda/README.md § “Production: cloud PG”.A likely question: “when the DB graph changes (package install, branch merge, user edit), every executor using that graph must refresh, and requests must only reach up-to-date executors — isn’t that this task?” Mostly it is already built, and it is the foundation this RFC reuses — not new work.
fn:invalidate:<fn-id>|<branch>|<org> on graphden_events; every local pod’s
LISTEN applies the delta (recompiles just the affected fns via
delta-recompile!), and remote/BYO executors get the same over the SSE relay
(SCALING.md § SSE). Package install/update/fork/materialize and branch-merge were
all made delta-invalidating. So “the graph changed → holders refresh” already
works, fleet-wide.load-cell! (T2.2) is the same
compile-subset/swap! delta path pointed at a forward closure instead of a
reverse (invalidation) one. Hot-reload and cell-load are two directions of one
mechanism.delta-recompile!, compile-subset, :forward-deps)
→ cell load, and the whole hot-reload story (§9).421 misdirected → the backstop when the map is stale.reject-cross-org-refs! → guarantees a cell’s closure ⊆ org ∪ public.graphden_events → routing-map + registry invalidation transport.:executor-orgs predicate → generalises to the live cell set (it already
accepts a fn).:router-or-nil + route fn-defs → per-route cell split with no new base-fn.compile-all ~60 s → ~1.2 s
(memoize fn-typed-fn-ids + env-bindings) and the type-check sweep
~141 s → ~24 s (memoize effective-ref-return across passes) — so the earlier
clean measurement no
longer holds; the sweep now dominates what little remains. It is still
compute-bound, not dependency-reconnect or class-loading, so AppCDS barely
helps; CRaC (which captures the whole compiled registry) is still the lever for
eliminating the residual boot cost, though the memoization sharply narrowed the
gap it closes.compile-all’s cache is keyed on the whole
graph shape; per-cell partial loads bypass it. Not blocking, but the cache is at
the wrong granularity for per-cell.Resource-handler integration + measuring the
real ~655 MB image.graphden.crac quiesce/resume blockers were found + fixed (lazy class-init on
the checkpoint thread, Hikari pool not draining to zero, http-kit
listener/selector). Cold boot re-measured clean = compute-bound (type-check +
eager-compile), so AppCDS is de-prioritised and CRaC is the footprint
lever. Remaining is operational (CI checkpoint bake); this unblocks
scale-to-zero (T5.3).fleet.{placement,router,command,metrics,controller,packer,rebalance,
control-loop,discovery} + :exec/fleet-controller + the Helm chart). The
fleet auto-places new tenant cells and rebalances sustained load imbalance
under a leader-locked controller, over a k8s StatefulSet whose SRV membership
tracks HPA scaling.packer/best-target + discover-cells closures) behind
GRAPHDEN_FLEET_OVERLAP_WEIGHT (default 0 = pure LPT). Remaining: per-route
split (evidence-gated), scale-to-zero
(demonstrated on KEDA+CRaC, T5.3).