Graphden documentation

Scaling the Executor Fleet

How graphden runs on more than one executor process, and what is still missing. Companion to SERVICES.md (what a service is), VERSIONING.md (per-branch contexts) and TENANCY_SEAM.md (orgs, RLS, the effect gate).

This document is the STATIC layer — fixed :executor-orgs shards + the 421 backstop. The DYNAMIC layer built on top of it (automatic load-based placement, cell load/evict, a leader-locked rebalancing controller, and forward-hop routing to a cell’s holder instead of 421) is FLEET_RFC.md, with the ops guide in FLEET_DEPLOY.md.

Where the tenancy code lives. This repo (the open core) ships only the core-side seam tenancy/context.clj. The policy pieces this doc names — tenancy/addon.clj, tenancy/app_router.clj, tenancy/storage.clj (OrgScopedStorage, visible?), reject-cross-org-refs!, and the 421 misdirected-request enforcement — live in the private graphden-tenancy repo and plug into core via config (see TENANCY_SEAM.md). Without the addon the system is single-tenant and none of them are present.

The one idea

Four separate-looking problems — “don’t compile every tenant’s fns”, “let a customer run the executor on their own hardware”, “sell a faster tier”, “route load across pods” — are the same problem: which orgs does this executor serve?

Organization is the only closed dependency boundary in the graph. A tenant’s reads are filtered to {own-org, public} (tenancy/storage.clj visible?), so no edge a tenant can write leaves that set — and since 2026-07 reject-cross-org-refs! enforces it instead of merely implying it. Therefore a pod that holds own-orgs ∪ public can resolve every ref inside the fns it compiled, and nothing else needs to be present.

That is why the sharding key is the org, not fn popularity. ROADMAP’s “popularity-based fn-set distribution” (a routing layer that dispatches by fn-id heat) stays Not planned — it needs a router in front of every call and pays back only at millions of fns. Org sharding needs no router at all: a request already names its org.

What multi-pod means today

Run N containers against one Postgres, put a load balancer in front.

Concern Mechanism Where
Each pod compiles its own registry rebuild! at boot executor/compile_runtime.clj
A pod never runs two full compiles at once (each holds the whole graph + lookups + a fresh registry live — two working sets OOM’d prod on 2026-08-05: cold-branch build racing the epoch heal) process-wide full-compile-semaphore, 1 permit (GRAPHDEN_MAX_CONCURRENT_COMPILES widens); queued compiles count :registry/compile-queued. Held only across the pure read+compile section — never across a lock (call-with-compile-permit’s deadlock-freedom contract) executor/compile_runtime.clj
A write on pod A reaches pod B NOTIFY graphden_events → delta invalidate (LATENCY optimization) storage/postgres/notify.clj, system/init/services.clj on-notify
A skipped/lost invalidation self-heals (client abort mid-request, a write path with no NOTIFY, a dropped NOTIFY) graph EPOCH: a PG sequence bumped BEFORE every graph-shaped write; the branch-router validates its cached ctxs against it on fetch (TTL ≈1s). A heal rebuilds the BASE ctx in place (stale-while-revalidate) and DROPS every other cached branch ctx — rebuilt on its next request — so a heal costs one compile, not one per cached branch (merged source branches are permanent, so a workspace holds many); a branch with a RUNNING service is pinned (the reconciler registers set-pinned-branches-fn!) and rebuilds in place too, so the service keeps the ctx the router serves. Every heal logs its reason (graph-epoch heal reason — the aborted epochs with the entity that bumped them, or the foreign ones). An un-noted local bump counts as aborted only after 45 s, above the 30 s abort-shield budget, so a merely slow write never triggers a heal; the boot package sync notes its own bumps (note-all-applied!) so a fresh boot does not heal once for nothing, a never-called sequence reads as epoch 0, and dispatch binds the request bump log itself when its caller did not. Ledger entries survive the watermark for five minutes (*ledger-retention-ms*) so a second epoch STATE that is behind (the parallel test plugin isolates one per NS thread) still classifies them. Eager invalidate + NOTIFY stay as the fast path; correctness no longer depends on them. Under the tenancy addon too: every ledger call resolves the pool-bearing handle beneath the org-scoped decorator (epoch-handle), and the heal’s deleted-branch check reads the raw handle at platform context storage/postgres/graph_epoch.clj, system/branch_router/epoch.clj validate-graph-epoch!
Only one pod runs a cron :cardinality :singleton + pg_try_advisory_lock services/reconciler.clj
Every pod runs the HTTP listener :cardinality :per-pod resources/packages/app/package.edn
A pod dies mid-service its PG session ends → advisory lock auto-releases → a sibling re-takes it on the periodic reconcile tick (~15s; the crash emits no NOTIFY, so the level-triggered pass — not an event — heals it) storage/postgres/advisory_lock.clj, services/reconciler.clj
Cancel reaches the pod actually running the execution execution:cancel:<id> NOTIFY fan-out crud/fn_execution.clj, persist/cancel-local!

Cardinality is the thing people get wrong

A :service is “keep this fn running”. Two service shapes want opposite placement, and the difference is not inferable from the fn:

Hence the explicit :cardinality field. nil reads as :singleton so rows written before the field keep their old behaviour; the package seeder backfills it from package.edn on boot, which is what flips a pre-existing :web-server row to :per-pod.

Invalidation is per-branch, not per-pod

Each branch has its own compiled registry. A branch with no version rows of its own gets the base registry’s {fn-id → closure} map by reference (instantiate-from-templates!), so branches are nearly free in memory — but that shared map is a value, held in the branch ctx’s own atom. Recompiling main therefore does not update it.

So a write invalidates every cached branch ctx that inherits from the written branch — written-branch ∈ (collect-branch-chain … C) — and nothing else. Locally that is branch-router/invalidate-affected-ctxs!; across pods the branch-id rides in the NOTIFY payload (fn:invalidate:<fn-id>|<branch-id>) and the receiving pod answers the same question about its own cache. It never builds a branch ctx it hasn’t served: an unbuilt branch has no stale state, and compiling it on a NOTIFY would make a pod pay for a branch nobody asked it about.

Sharding by org

:executor-orgs on the ExecutionContext is a membership predicate over org-ids. read-graph filters the graph to org-id ∈ predicate ∪ NULL before compiling. nil (the default, and the only value a self-hosted deployment uses) compiles everything, exactly as before.

# a pod serving two tenants plus the platform packages
GRAPHDEN_EXECUTOR_ORGS=public,acme,beta

A set is already a predicate in Clojure, so a hash-sharded fleet can pass a function instead and never enumerate its tenants:

{:executor-orgs (fn [org] (= my-ordinal (mod (hash org) pod-count)))}

Rows with a NULL :org-id are un-owned — the platform graph from resources/packages/ — and are in every shard. Name the public org explicitly: once the tenancy decorator writes the platform packages, they carry org-id = "public", not NULL.

What this buys:

Reaching the wrong pod: 421

A load balancer that doesn’t know about shards will eventually send a request for org acme to a pod that doesn’t hold acme. That pod answers 421 Misdirected Request — the status HTTP defines for exactly this: the server cannot produce an authoritative response for this authority, and the client should retry on another connection.

Both entry points check, because both execute graph fns:

Path Where Response
Editor / API (/api/*) tenancy/addon.clj :tenancy/request-scope 421 {"error":"misdirected-request"}
Tenant app (FaaS subdomain) tenancy/app_router.clj 421 text/plain

Two ordering decisions worth keeping:

421 is not a security response. A tenant that reaches the wrong pod did nothing wrong; the balancer sent it there. Routing by subdomain at the LB (which the addon’s Host-based org resolution already makes natural) avoids the round trip; the 421 is the backstop that keeps a misrouted request honest.

Routing by subdomain at the load balancer

To keep the 421 a backstop rather than a hot path, route each tenant’s subdomain to the pool of pods that hold its shard. Since an org’s slug IS its subdomain (<org>.<base-domain>), the map is direct. An nginx sketch, one upstream per shard:

# pods serving orgs acme + beta (GRAPHDEN_EXECUTOR_ORGS=public,acme,beta)
upstream shard_a { server pod-a1:8080; server pod-a2:8080; }
# pods serving the rest (GRAPHDEN_EXECUTOR_ORGS=public,gamma,delta)
upstream shard_b { server pod-b1:8080; }

map $host $shard {
    ~^(acme|beta)\.        shard_a;
    ~^(gamma|delta)\.      shard_b;
    default                shard_a;   # apex / editor → any hosted pod
}

server {
    server_name ~^.+\.example\.com$;
    location / { proxy_pass http://$shard; }
}

A BYO org points its subdomain (or custom domain) straight at the customer’s own executor instead — that pod runs with GRAPHDEN_EXECUTOR_ORGS=<their-org> + GRAPHDEN_BYO_EXECUTOR=true, and hosted pods 421 it anyway if the LB ever misroutes. The map is the only thing that changes as shards are rebalanced; the pods need no LB awareness.

Fleet-wide per-org quota

*max-concurrent-executions-per-org* (default 32, GRAPHDEN_MAX_CONCURRENT_EXECUTIONS_PER_ORG) has two layers in acquire-execution-slot!. Every org has a per-pod live counter, covering inline pure runs as well as persisted/async work. A tenant additionally checks shared non-terminal (:pending) :fn-execution rows for the fleet-wide persisted-run budget. Pure inline runs that write no row remain bounded on each pod; they are not counted globally by that query. The pending-row budget self-heals through the zombie/TTL sweeper after a crash.

The global cap (*max-concurrent-executions*, default 128) stays per-pod: it sizes the worker count of each JVM’s bounded execution pool (make-execution-pool — ThreadPoolExecutor + bounded queue), which is a per-process thread-exhaustion safety property. It no longer rejects at admission: overflow PARKS in the pool queue, and only a full worker set AND full queue turns a submit into 503 + Retry-After (:queue-full). The per-org fleet count has a bounded TOCTOU slack (two pods can both admit before either row exists), which is fine for a fairness limit — the global per-pod bound is the hard safety net.

The public org (platform / single-tenant) keeps the per-pod atom for its per-org cap: it isn’t a metered tenant, and its executions are the hot editor path we don’t add a query to. So single-tenant self-hosted is unchanged.

Tenant isolation: shared vs dedicated (resource-isolation caveat)

On the shared tier many orgs are co-located on one JVM/pod (the packer spreads for load, not isolation). Those tenants are isolated by capability and bounded for fairness:

What the shared tier does not give is a hard resource boundary: the co-located tenants share the pod’s CPU, heap, and threads, so a heavy or misbehaving neighbour can degrade others (a classic noisy neighbour), and a per-tenant out-of-memory can restart the pod and drop co-tenants’ in-flight executions. The wall-clock execution deadline and the per-org quotas blunt this, but they do not partition CPU or heap.

A tenant that needs a hard resource boundary uses the dedicated tier: its own :executor-orgs shard with cgroup CPU/memory limits, so its load can never touch another tenant — see FLEET_DEPLOY.md § Dedicated tenant shard. This is why safe always-on tenant services are the dedicated (paid) line, not the shared one.

External / BYO executor

A customer runs the executor on their OWN hardware, but the graph stays in our Postgres. Built in these pieces:

Running one (graphden.byo)

The pieces are assembled by graphden.byo/start-byo!, and -main reads the config from the environment — so a customer runs the same jar with:

GRAPHDEN_HUB_URL=https://hub.example.com \
GRAPHDEN_SSE_URL=https://hub.example.com:8081 \
GRAPHDEN_EXECUTOR_TOKEN=<bearer> \
GRAPHDEN_EXECUTOR_ORG=acme \
GRAPHDEN_EXECUTOR_BRANCH=main \
GRAPHDEN_APP_HANDLER_FN=<org-app-handler> \
GRAPHDEN_PORT=8080 \
  clojure -M -m graphden.byo

It loads the packages LOCALLY for their base-fn impls (no DB sync — the graph lives on the hub), reads the graph over HTTP into a RemoteStorage, compiles it, serves the org’s handler fn over HTTP directly (not via the PG-backed service registry), and refreshes on each SSE push. GRAPHDEN_HUB_URL is the hub’s APP url (/api/export/graph-rows); GRAPHDEN_SSE_URL is the relay’s separate port. Without a relay, set GRAPHDEN_REFRESH_POLL_MS to poll the hub for changes on a fixed cadence instead; with NEITHER set the graph freezes at the bootstrap snapshot until restart and start-byo! warns loudly.

Provisioning a BYO customer, operator-side (platform-only, :org is tenant-forbidden): create the org, mint its executor token, point the org at its handler, then flip it with POST /api/orgs/execution-mode {name, execution-mode=byo} (tenancy-admin/registration), which drops the byo memo so hosted pods start 421’ing it at once.

Boundaries (what a BYO executor does NOT do)

The read-only, one-org shape is deliberate:

Advisory-lock reconnect

The lock connection lives behind a reconnecting holder; each reconcile pass calls advisory-lock/ensure-live!, and a reconnect (DB restart / network blip) triggers reassert-lock-ownership! — the pod re-takes every :singleton it was running and stops any a sibling grabbed during the outage. See SERVICES.md § Roadmap.

Service-to-service

A service that wants another service’s address names it — the service fn is the value (:service-endpoint :service :orders-service, a :fn-ref slot), and the reconciler has recorded where that fn answers (SERVICES.md § Endpoints). What the address IS depends on the deployment, and that is the whole distribution story for services:

Deployment Producer Address the consumer gets
Single pod :http-server service 127.0.0.1:<port>
Multi-pod fleet :singleton / :pool holder <executor-id>:<port> — the holder’s pod-FQDN, the same name the forward-hop dials
Multi-pod fleet :per-pod listener the last pod to record it (any pod serves)
Cloud tenant an :app-route https://<label>.<apps-domain> via the addon resolver seam — the public origin, so the call is an ordinary egress-guarded outbound request and no internal address is ever handed out

Cross-pod calls are explicit HTTP in the graph, never a transparent RPC inside a lazy call chain: the request body is a typed slot, the cost is visible, and the closure-capture / cycle rules stay exactly what they are (a :fn-ref edge is an identity, not a dependency — a pair of services may name each other). “Which executor runs what” stays the org/cell model above; this section only answers “and how does one reach the other”.

Still open

Nothing about CORRECTNESS. (The 2026-08 audit found — and fixed — one real correctness hole here: the byo refusal covered ALL of /api/*, which made the byo flip brick the org because the BYO pod’s own bootstrap read was refused too. It is execute-scoped now, with tests on both repos’ gates.) The multi-node software topology — separate JVMs, real TCP between them, a BYO executor that reaches the graph over HTTP (RemoteStorage) instead of touching Postgres directly — needs no special hardware: it’s separate processes + a config boundary, reproducible with containers, VMs, or even two ports on one host. That loop is proven three ways: storage.remote.e2e-test runs the BYO half over REAL http-kit servers (bootstrap, SSE live-refresh, reconnect); fleet.two-container-e2e-test boots the executor image TWICE as a real two-pod fleet over a shared Postgres (testcontainers) and drives the token-gated /internal/fleet/* control plane over real HTTP (bb test-fleet-e2e); and the fleet was also run on a real two-pod kind cluster (forward-hop, leader election, cross-pod transport, DNS-SRV — 2026-07).

What remains is purely OPERATIONAL and not a test: a live demonstration of a customer running a BYO executor on their own infrastructure (their hardware / another cloud / another region). That’s a deployment showcase — reproducible with a second cloud VM or container, never literally a second physical machine — not a correctness gap. The boundaries above are intentional scope.

Rolling upgrade note

An old pod holds the :web-server advisory lock until it exits; a new pod reads :cardinality :per-pod and ignores the lock. So a rolling restart works, and the first new pod serves traffic immediately. The seeder backfills :cardinality on boot, before the reconciler’s first pass.