Declarative long-running services backed by the :service entity.
Packages declare baseline services in package.edn; an admin can
also write :service rows directly through /api/entities/service.
The reconciler turns enabled rows into running futures, supervises
startup failures, and stops them on shutdown.
For a public URL that invokes a handler once per request, see Temporary HTTP handlers. That finite host uses the existing HTTP origin or isolated apps domain and does not start a persistent service.
Reconcile fires on integrant init, on every CRUD mutation, on a
service:* NOTIFY from a sibling pod, and on a periodic tick
(:exec/service-reconciler, ~15s). The tick makes the reconciler
level-triggered rather than purely edge-driven: it re-takes a
:singleton’s advisory lock after the holder pod crashes (the crash
emits no NOTIFY), picks up an out-of-band DB edit, and reconverges a
transient start failure — all without a restart.
Multi-pod coordination is built. Every pod runs its own
reconciler; a service’s :cardinality decides how many pods run it
(:singleton → advisory-lock-gated, :per-pod → everywhere). Cron
and fixed-period schedules ship as graph compositions (§ Service shapes).
Before this feature, exactly one long-running fn was supported via a
package’s :startup-fn field — typically :web-server baked into
app/package.edn. You couldn’t:
:web-server from the editor used to
crash with Address already in use)The :service registry fixes all three. Packages now contribute
baseline services through a :services [...] declaration in their
package.edn — the reconciler seeds them idempotently at boot, the
admin retains the option to disable them.
A :service row is just a marker on a fn that says “keep this alive”.
The fn it points at must have no start-blocking free arguments —
every slot the fn needs to compute/configure itself at start must be
bound via fn-defs / bindings. The reconciler invokes the fn with empty
args. (A listener’s handler args — supplied by the deferred invoker per
request/tick — are NOT start-blocking; see § Guard below.)
To run the same impl with different parameters, create a derived fn-def that binds the slot differently, then declare a service for the derived fn:
{:name :web-server-9001 :parent :http-server
:args {:handler :_app-ring-response :port 9001}}
Then POST /api/entities/service {fn-id: <web-server-9001-id>, enabled?: true, restart-policy: always}.
This is intentional: it keeps service config visible in the fn-graph (versioned, type-checked, composable) and avoids duplicating the binding mechanism. Service rows stay tiny and declarative.
The redundant alternative would be a separate :service-arg table
that overlays slot values — we evaluated it and rejected it: it
duplicates :binding shape, hides config from the graph, and adds an
entity type that the project’s “minimal entities” principle pushes
against.
Env-based config falls out of this naturally — :port {:ref :_port-from-env}
where :_port-from-env is itself a fn-def calling :env "PORT" and
parsing the result. Visible in the graph, type-checked, versioned.
The service registry doesn’t care WHAT shape of long-running work the
fn spawns — only that the fn’s return is a 0-arg stopper-thunk and
that the fn declares the :process effect. These patterns ship with
graphden today:
| Pattern | Parent | Shape | Example |
|---|---|---|---|
| Long-lived listener | :http-server (Ring/http-kit) |
bind a port + a handler fn → returns a stopper that releases the port | :web-server in packages/app/server |
| Cron-driven loop | :schedule (composed over :future + :loop-until-interrupted + :sleep-until-ms + :cron-next-after + :call-noargs) |
bind :cron (Quartz cron-6 string) + :fn (any 0-arg callable) → returns a stopper that interrupts the daemon thread |
:ex-cron-heartbeat in packages/examples/schedule-cron |
| Fixed-period loop | :interval (composed over :future + :loop-until-interrupted + :sleep + :call-noargs); :interval-now fires once at start as well |
bind :every-ms (period) + :fn (any 0-arg callable) → same stopper shape as :schedule |
{:name :poll :parent :interval :args {:every-ms 300000 :fn :poll-inbox}} |
| Many triggers, one service | :start-all (base-fn) |
bind :triggers to a LIST of :schedule / :interval derivatives → every trigger starts in its own daemon thread; the ONE returned stopper stops them all, liveness is the children’s combined (alive while any runs, :failed once any threw) |
{:name :jobs :parent :start-all :args {:triggers [:_nightly :_poll]}} |
:schedule is itself a pure fn-def composition — no monolithic
base-fn. The two captured args :cron and :fn propagate through
three HOF boundaries via closure-capture (docs/CLOSURE_CAPTURE.md).
This is the canonical template for adding more long-running patterns
(queue consumer, websocket listener, file watcher): compose over the
existing concurrency primitives, return a stopper, declare :process.
1-to-many triggers are a graph list, not a child entity. The
roadmap once sketched a :service-schedule row (N cron/interval rows
per service). It ships instead as :start-all over a list — the same
reasoning as :migrate’s :migrations and :do’s :steps: the list
is versioned, type-checked ([:list [:fn {} :null]]), diffable, and
editing it restarts the service through the ordinary fn-graph path,
with no schema, no CRUD surface and no second reconciler loop. A
trigger whose target should show up on the Runs tab wraps it in
:traced-call (core/system): every fire then lands as an
:fn-execution row of the target with a fresh trace id, exactly like a
queue consumer’s :call-traced hop. Listing the same trigger twice is
refused (:start-all/duplicate-trigger): the executor runs a list item
once per start, so the second entry would be the same loop, not a
second one — derive a second fn-def to run a job twice. A trigger whose
thread dies makes the whole set :failed; the restart policy then
restarts every trigger, because the service handle is one thing (per-
trigger supervision would need the executor to hand :start-all the
triggers as re-invocable callables, which a list of refs is not).
A service fn is any no-arg fn, so “do X before the listener comes up”
is plain sequencing — :do — not a field on the :service row. The
storage/pg package ships the Flyway shape as two fn-def templates,
:migration and :migrate (composition only, no new base-fn — the
same role :schedule plays for cron):
{:name :m-001 :parent :migration
:args {:id "001-users"
:ddl {:value {:create-table [:users :if-not-exists]
:with-columns [[:id :bigserial [:primary-key]]
[:email :text]]}}}}
{:name :m-002 :parent :migration
:args {:id "002-users-created-at"
:ddl {:value {:alter-table :users :add-column [:created_at :timestamptz]}}}}
{:name :app-migrate :parent :migrate :args {:migrations [:m-001 :m-002]}}
;; The service fn: migrate, then start the listener whose stopper
;; `:do` returns (it also inherits the listener's `:process` effect).
{:name :app-service :parent :do :args {:steps [:app-migrate :web-server]}}
| One migration | :migration — inside a transaction, when the schema_migrations journal has no marker for :id, run :ddl (a HoneySQL map) and insert the marker (returns 1); otherwise a no-op (nil). |
| The run | :migrate — ONE transaction under pg_advisory_xact_lock: ensures the journal table, then runs :migrations in order. N pods starting at once serialise and the late ones find nothing pending; a throwing migration rolls the whole run back, markers included. |
| Adding one | Append to the list — {:migrations {:append [:m-003]}} on a derived fn-def, or edit the vector. The write hook restarts every service whose closure contains the list (invalidation.clj), the :do runs the migrator again, only the new id is pending. |
| Merging | :merge-post-commit! restarts the same set on the target branch, so migrations added on a feature branch apply on main’s next start. |
| Ids | Stable, append-only, sortable strings ("001-users"). A duplicate id in one run fails the journal insert’s primary key and rolls the run back — loudly, by design. |
Each migration is a graph node, not a row of data: the editor shows the
journal as the :migrations chain, and the type-checker sees every
:ddl. graphden.packages.storage.migrations-test pins the contract.
Branches version the graph, not Postgres. :migration runs
against graphden’s OWN datasource, so a service on a feature branch
migrates the same physical database main’s service uses, and the
journal table is shared. Keep platform-table migrations on the branch
that owns the deployment; for a user database of your own, put the
DSN in a branch-local? :env fn-def and compose over web/sql
(:sql-exec / :sql-query) instead — then each branch can point at
its own database. The two differ in what they record: the platform-DB
templates (:pg-query / :pg-execute / :migration) carry the
:raw-sql effect, which the cloud effect gate forbids to tenant graphs
(SECURITY_MODEL.md) — on the cloud they are a
platform / self-host affordance — while web/sql against a database of
your own is :db + :network, an ordinary tenant effect.
One non-versioned entity + two enums (:restart-policy, :cardinality;
admin mutates in place; the audit trail for what actually ran lives in
:fn-execution rows the services spawn).
:service| Field | Type | Notes |
|---|---|---|
:id |
:uuid |
Returned to clients as service-id. |
:fn-id |
:ref :fn |
Logical fn id — service tracks the current graph; editing the fn picks up at next restart. (Compare :fn-execution.fn-version-id which is a frozen snapshot.) Must point at a fn with no start-blocking free args (§ Guard). |
:enabled? |
:bool |
Reconciler only starts enabled rows. Toggle this + reconcile to stop a service without deleting it. |
:restart-policy |
:restart-policy |
:always / :on-failure / :never — see § Supervisor below. |
:cardinality |
:cardinality |
:singleton / :per-pod / :pool — how many pods run it at once; see § Cardinality. Nullable; nil ≡ :singleton (rows that pre-date the field). |
:pool-size |
:int |
Pod count for :cardinality :pool (ignored otherwise). Nullable — a :pool row with nil/non-positive size degrades to a singleton. |
:branch-id |
:ref :branch |
Per-branch scope. Reconciler routes the start through branch-router/ctx-for branch-id, so the same :fn-id can run with branch-specific bindings on dev + prod simultaneously. Nullable — nil is normalized to the router’s default branch at reconcile time (effective-branch-id), so a legacy nil-branch row behaves exactly like an explicit default-branch row, including the post-merge restart-services-depending-on! pass. Without a router (tests) nil falls back to the reconciler’s base ctx. The editor’s ⚙ popover picker defaults to the editor’s current branch on create. |
:org-id |
:text (null) |
Tenant owner; NULL ≡ platform. Load-bearing for fleet sharding: the reconciler drops services whose org this pod doesn’t serve (service-in-shard?), so a dedicated tenant’s services run only on its own pod. Stamped by the tenant service-create endpoint (NOT OrgScopedStorage — :service is tenant-forbidden write-through). |
:service-instanceOne row per RUNNING copy (§ Endpoints, § Liveness) — the desired-state
row says keep it running, an instance row says this pod runs it,
here, and was alive at seen-at. Written by the pod that starts the
copy, deleted on every stop path, heartbeat every reconcile tick.
| Field | Type | Notes |
|---|---|---|
:service-id |
:ref :service |
The desired-state row this copy belongs to. |
:executor-id |
:text |
The pod — its fleet :executor-id, "local" on a single pod. |
:host |
:text |
Where the copy answers — the executor-id, loopback on a single pod. |
:port |
:int (null) |
The port the listener bound (from the handle’s :endpoint metadata); nil ≡ not a listener (a cron loop still has a row — it heartbeats). |
:started-at |
:timestamptz |
|
:seen-at |
:timestamptz |
Heartbeat. Older than default-stale-after-ms (45 s, three ticks) ⇒ presumed dead: consumers ignore it, and after ten windows any pod deletes it. |
:org-id |
:text (null) |
The service’s tenant, copied from its :service row when the copy starts (the reconciler writes with the platform handle). Org-scoped on the cloud: a tenant reads ITS running copies — the ⚙ popover’s Running copies — while the :service row stays platform-only. |
:restart-policy enum:always, :on-failure, :never.
:cardinality enumHow many pods run the service simultaneously:
| Value | Lock? | Use it for | If you get it wrong |
|---|---|---|---|
:singleton |
pg_try_advisory_lock(service-id, slot 0) on the pod’s dedicated lock connection; losers idle with a ::not-our-lock placeholder |
cron / :schedule loops, one-shot migrations, anything whose side-effects must happen once per tick |
a :per-pod cron fires N times per tick, once per pod |
:per-pod |
none — every pod starts its own copy | listeners (:http-server), anything a load balancer fans traffic into |
a :singleton listener means only ONE pod ever binds a port; every other pod fails its healthcheck and the LB sees a single backend no matter how many pods you run |
:pool (:pool-size N) |
races for the first free of N advisory-lock slots (keys service-id+0 … +N-1); holds it |
a background worker that should be redundant / parallel across a bounded number of pods — not one, not all | too small an N under-provisions; too large just caps at the pod count |
:pool generalises :singleton (a pool of 1, slot 0). With ≥ N pods
exactly N run; with fewer pods, one copy each. The pool SIZE is fixed —
load-driven autoscaling of N is intentionally out of scope (it would need
a per-service load signal + a scaling controller; the request path scales
via cells + HPA instead — see docs/SCALING.md).
app/package.edn declares the seeded :default → :web-server service
as :per-pod for exactly this reason.
The running-atom entry records :locked? — whether this pod holds the
advisory lock — so stopping a service releases only locks it actually
took. A cardinality flip is picked up by the reconciler’s config-drift
detector and stop+restarts the service (dropping the lock on the way to
:per-pod, racing for it on the way back).
Upgrade note: the seeder backfills :cardinality onto an existing row
whose value is nil, taking it from the package declaration. That is what
moves a pre-existing :default row off the singleton default. It only
touches nil, so a deliberately-set value survives, same as :enabled?.
reconcile-once! groups enabled rows by :branch-id, asks the
branch-router for each branch’s ExecutionContext (built lazily
via build-actual-entry!), and starts the service against that
ctx. A typical workflow:
;; dev: port 9001 (sticky-local, doesn't merge to main)
{:name :dev-server :parent :http-server
:args {:handler :app-handler :port 9001}}
;; main: port 8080 (also sticky-local)
{:name :prod-server :parent :http-server
:args {:handler :app-handler :port 8080}}
Two :service rows — {:fn-id :dev-server :branch-id dev} and
{:fn-id :prod-server :branch-id main} — both run.
The merge’s :merge-post-commit! step calls
recon/restart-services-depending-on! on the target, seeded by the
merged fn-ids — only the services whose closure the merge touched
restart — so cron loops pick up new fn-versions (HTTP servers re-
read their registry lazily, cron closes over the fn-graph at
spawn time). delete-branch! cascade-soft-disables services
scoped to the deleted branch before the row goes away.
Port / resource conflicts surface as OS-level failures —
Address already in use records :start-failed-at and shows up
in the running-atom for the UI badge.
What the editor shows comes from :enrich-running (GET
/api/services): the raw entry fields plus the reconciler’s verdict
(recon/running-state → :state / :next-attempt-at), so every
placeholder has a name — running, start-failed, exited (policy
leaves it down), backoff (restart delayed; the next attempt time
rides along), not-our-lock (another pod holds the slot; the badge
says “elsewhere”), pending (nothing recorded yet). Before the
verdict every placeholder rendered as “pending”.
Lives in graphden.services.reconciler. Diff-driven, idempotent. A
running copy’s instance row + stop path live in
graphden.services.instances; exit detection, the restart policy at
runtime and the exit backoff in graphden.services.liveness.
:exec/service-reconciler integrant key):
recon/running.:services into the :service table
(idempotent — see § Packages-based seeding below).:service rows from DB.reconcile-once! starts each.POST /api/services/reconcile or programmatic
reconcile-once!), one pass under reconcile-monitor:
begin-pass! — heal the lock connection (re-assert ownership),
drop the transient ::not-our-lock / ::start-failed placeholders
and elapsed ::backoffs, run the liveness + heartbeat pass, reap
stale instance rows.plan-pass (no I/O) — diff desired set (DB) vs running set
(in-process atom), plus config drift (entry-drifted?).start-one! each missing one with empty args
(the fn is fully bound by definition) — outcome :started /
:start-failed / :not-our-lock / :branch-ctx-failed.halt-key!): drain recon/running via stop-all!.{service-uuid → {:fn-id :uuid
:restart-policy :always | :on-failure | :never
:cardinality :singleton | :per-pod | :pool
:pool-size Int | nil (slot count; 1 for singleton)
:locked? Bool
:pool-slot Int | nil (which advisory-lock slot we hold)
:branch-id :uuid (effective branch: the row's, or the
router's default for nil-branch rows;
absent only when no router is active)
:stopper (fn []) | nil
:instance-id :uuid (this copy's `:service-instance` row —
heartbeat each tick, deleted on stop)
:started-at Instant
:start-attempts Int
:start-failed-at Instant (set only when retries exhausted)}}
:stopper is whatever the fn returned (web-server-shape: a thunk
that stops the listener; other shapes are logged on stop but otherwise
ignored). nil means the start failed and retries were exhausted.
:locked? says whether THIS pod holds one of the service’s advisory-lock
slots, and :pool-slot records WHICH slot (0 for a singleton), so stop
releases exactly the key it took and a post-reconnect re-assert re-takes
the right one. Only lock-gated (:singleton / :pool) services take a
slot; :cardinality + :pool-size are mirrored onto the entry so the
config-drift detector notices an admin flipping either.
A pod that couldn’t get a slot stores the sentinel ::not-our-lock
instead of a map. That placeholder is transient: the top of every
reconcile pass drops it, so the service is re-attempted each pass. This is
what makes the periodic tick heal a crashed owner — its slot auto-releases
(no NOTIFY), and a sibling re-acquires it on the next tick rather than
idling until a :service edit happens by.
start-service! wraps start-service-once! with a bounded
exponential-backoff retry loop per :restart-policy:
| Policy | Behaviour |
|---|---|
:always |
Retry on start exception up to max-retries (default 3) with backoff 1s → 2s → 4s; at runtime, restart after ANY exit — a clean stop included (restart-after-exit?, the liveness path). The first restart is immediate; further exits of a copy that lived under 60 s back off 1s → 2s → 4s → … → 60s (parked as ::backoff, restarted on the first pass after the delay), and a run of 60 s or more resets the counter — so a one-shot fn under :always (use :interval for that) is not a hot loop. |
:on-failure |
The same start-time retries; at runtime, restart only after a :failed exit — a clean stop stays down. |
:never |
Single attempt. On failure, record :start-failed-at, leave :stopper nil. |
After the retries within a single pass are exhausted, the reconciler
does not hold the entry as “running” forever. It records the
transient ::start-failed sentinel and RELEASES the advisory slot,
so (a) a healthy sibling can immediately fail over and take the service,
and (b) the top of the next reconcile pass drops the sentinel and
RE-ATTEMPTS the start (one cheap retry-free attempt per tick). So a
transient cause — a port briefly taken, a file not yet present — self-heals
on the next pass with no operator action. Admin can still poll for
:start-failed-at to see which services are currently failing to start;
a persistent failure (bad :fn-id, permanently-taken port) is re-probed
each pass until the underlying issue is fixed.
Every automatic trigger is retry-free ({:max-retries 0 :backoff-ms 0}):
the periodic tick, the :service NOTIFY, and the edge-triggered restarts
(restart-services-depending-on! / restart-services-on-branch!, which
fire on every fn / binding write and run no pass at all when no running
service matches) — a start retried inline would sleep 1+2+4 s under
reconcile-monitor and block every other trigger. The defaults apply to
the explicit POST /api/services/reconcile, boot and CRaC resume. Tests pass the same
start-opts to reconcile-once! for zero-time runs.
Endpoints use the normal authentication seam: a session or bearer when authentication is enabled. A bare self-host with authentication disabled retains its open mode. Tenant service management uses the addon’s own dedicated-tier endpoints rather than generic service CRUD.
POST /api/entities/service (generic CRUD)Form-encoded body. Create a service row.
curl -X POST -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "fn-id=$FN_UUID&enabled?=true&restart-policy=always" \
http://server/api/entities/service
PUT /api/entities/service/:id updates the row in place;
DELETE /api/entities/service/:id removes it.
Rejection: start-blocking free args on target fn. If fn-id points
at a fn with a free arg it needs to START (a direct unbound operand, or
one lifted through a data slot), the create is rejected with:
<p class="error">Cannot make a :service for a fn that has free args:
[:nums]. Create a derived fn-def that binds them, then
declare a :service for the derived fn.</p>
The guard uses :service-blocking-free-args, the service-ability
projection of :free-arg-slot-map: it drops the fn’s own top-level
CALLBACK-slot subtrees (an :http-server handler, a :schedule body),
because those free args are the callback’s per-invocation concern —
supplied by the deferred invoker (per request / per tick), not needed
to start the listener/loop. So a whole-app listener like web-server
(whose ~45 free args all live below its :handler HOF slot) is
service-able, while a genuinely unstartable fn is still rejected. The
full free-arg surface stays visible to /api/execute’s arg form.
POST /api/services/reconcileTrigger reconciliation on demand — the admin’s “apply changes” button. (A periodic tick also reconciles every ~15s, so an out-of-band change is picked up either way; this endpoint just makes it immediate.)
{"ok": true,
"reconcile": {"started": ["uuid", …],
"stopped": ["uuid", …]}}
Caveat: never call this endpoint to displace the very web-server that serves it. http-kit’s stop interrupts in-flight request threads, including the one running the reconcile call — the managed service start fails mid-query and the port ends up unbound. Workarounds:
For services that DON’T displace the API-serving web-server (a metrics server on a different port, say), the endpoint works fine synchronously.
When validate-execute resolves a Run request to a fn-id that the
service registry already considers alive, the request is rejected
upfront with :status :rejected:
{"ok": false, "status": "rejected",
"error": "Function is already running as a managed service. …",
"error-data": {"reason": "already-running-as-service",
"source": "service",
"service-id": "uuid"}}
Prevents the foot-gun where clicking ▶ on :web-server in the editor
tries to re-bind its port.
A service that listens somewhere answers somewhere, and the other services want that address. The service fn IS the value:
:http-server returns its stopper with {:endpoint {:port p}}
metadata — the port actually bound (so :port 0 reports the
OS-picked one). The handle’s contract (a 0-arg stopper) is unchanged.:service-instance row — its own dialable host (the fleet
:executor-id, a pod-FQDN in k8s; loopback on a single pod), the
port, and a heartbeat it refreshes every tick — and deletes it on
every stop path (stop-and-forget!, the lost-lock stop, stop-all!
with a ctx at halt / CRaC). A :pool or :per-pod service has one
row per copy. A crashed pod never deletes its row: the heartbeat goes
stale, consumers stop picking it after one window (45 s), and any pod
reaps it after ten — a scan each reconciler runs at most once per
window, not on every graph write (§ Liveness).:fn-ref slot —
:service-endpoint :service :orders-service. :fn-ref is the
identity primitive (TYPES.md):
the impl receives the fn’s id, the fn is never evaluated (binding a
listener does not start a second one), its free args don’t surface,
and the edge is not a dependency — so two services may name each
other (CONSTRAINTS.md).
web.service/:service-endpoint picks a LIVE listener
instance of the fn’s enabled row on the caller’s branch (a nil-branch
row — package-seeded — serves every branch), this pod’s own copy
first (loopback beats a hop), then the freshest heartbeat; else it
checks a finite HTTP publication and then the addon-installed resolver seam (on the cloud
an :app-route’s public origin — service-to-service traffic goes
through the public domain like any outbound call, SSRF-guarded, no
internal address to exempt), else throws :service/not-running.web/service composes the call: :service-url → :service-get /
:service-post (:http-get / :http-post with :url = origin +
:path) → :service-get-json / :service-post-json (body parsed).
Free args: :service, :path, plus http-client’s :headers /
:auth-value / :timeout-ms (and :body on POST). The caller’s
trace header rides along, so the producer records the request as an
execution linked to the caller
(EXECUTION.md § Tracing across services).The contract between two services lives in the graph too: a shared
namespace (svc-a.api) holds the path constants and the request /
response type-rows that BOTH sides reference — the producer’s
:get-route + handler return type, the consumer’s :service-get +
decoded result narrowed to the same shape. One edit moves both sides;
the type-checker catches a drift at write time. Tutorial:
lesson 38.
Start-time failures were always caught (port in use, a throw in the
constructor). A copy that started and THEN died — the listener object
stopped, the daemon thread ended — used to sit in running as alive
forever. Now a handle may carry two more metadata keys, and the
reconciler’s per-tick pass reads them:
| Key | Set by | Meaning |
|---|---|---|
:alive? |
:http-server (http-kit’s listener status), :future (thread state) |
0-arg probe: is the copy still running? |
:exit |
:future |
atom — nil while running, :done after a clean return / interrupt, :failed after an uncaught throw |
Each reconcile-once! pass, before the diff, check-liveness! walks
this pod’s running copies: a live one gets its heartbeat; a dead one is
stopped (lock released, instance row deleted) and then the row’s
:restart-policy finally means what it says —
| Policy | On a copy that died in place |
|---|---|
:always |
restarted in the same pass |
:on-failure |
restarted only when :exit is :failed; a clean exit is parked |
:never |
parked |
Parked = an ::exited placeholder in running that keeps the diff
quiet; toggling the row (enabled? off → on) runs it again. A handle
without :alive? (a nil stopper, a fire-and-forget) is trusted as
running, as before. Stale instance rows of a pod that CRASHED (no pass
ever ran there) are ignored by consumers after one window and deleted
by whichever pod ticks next after ten (each reconciler scans for
them at most once per window — reconcile also runs on every graph
write, and that path must not pay for the scan).
HTTP (§ Endpoints) is synchronous: the consumer waits, and a slow or
absent producer is the consumer’s problem. A queue decouples them. It
is Postgres-backed — one table, the standard shape every PG job queue
uses (Oban, graphile-worker, River, pgmq): FOR UPDATE SKIP LOCKED
for the claim, a visibility timeout while a consumer holds a message,
bounded retries with a delay, a dead-letter state, and a NOTIFY
(queue:publish:<name>) that wakes a waiting consumer so it never
polls hot. No extension, no second service to run; RLS / org-scoping
apply as to any row. A broker (Kafka, NATS) stays possible as an
external package: the graph contract below doesn’t name the backend.
:queue-messageNon-versioned work rows (schema graphden.schema.queue.schema):
| Field | Type | Notes |
|---|---|---|
:queue |
:text (indexed) |
The channel name. |
:payload |
:jsonb |
Whatever was published — narrow it to the contract’s type-row on both sides. |
:org-id |
:text (null) |
Tenant owner (stamped by the org-scoped decorator on the cloud). |
:state |
:text |
pending (takeable) or dead (retries exhausted, kept for inspection). A successful ack DELETES the row — the table holds work, not history. |
:attempts |
:int |
Claims so far. |
:available-at |
:timestamptz |
Due time (publish delay / retry delay). |
:locked-until |
:timestamptz (null) |
The visibility lock of the current claim. |
:error |
:text (null) |
The last handler’s message, kept on retry and on dead. |
:trace-id, :parent-execution-id |
:uuid (null) |
The publisher’s trace and execution (cr/*execution* at publish time); the consumer’s handler runs as a child of that execution (§ Tracing across services in EXECUTION.md). Published outside a persisted run, the message opens a trace of its own (:trace-id fresh, no parent) — the handling is persisted either way. |
storage/queue)| Base-fn | Does |
|---|---|
:queue-publish queue payload delay-ms |
insert + NOTIFY; returns the id |
:queue-take queue batch visibility-ms wait-ms |
claim up to batch due messages (SKIP LOCKED, attempts+1, lock for visibility-ms); when none is due, wait up to wait-ms for a publish and try once more |
:queue-ack message-id |
delete |
:queue-nack message-id error retry-ms max-attempts |
release for retry after retry-ms, or dead once attempts ≥ max-attempts |
:queue-extend message-id visibility-ms |
renew the claim’s lock — the lease heartbeat of a handler that outlives its claim (true while the row is still pending) |
:queue-requeue message-id |
a dead letter back to pending (attempts 0, error cleared, consumers woken) — what Operate → Queues’ Requeue calls |
:queue-stats |
one row per queue — {:queue :pending :in-flight :dead} — from ONE aggregate query, org-scoped like the entity read |
:queue-dead-letters limit |
the newest limit dead letters, org-scoped |
:queue-consumer is a service like any other (:future → a loop →
take a batch → run :handler on each → ack on return, nack on a
throw). While the handler runs, :extend is called every
:lease-every-ms under :with-heartbeat, so a handler slower than
the visibility timeout keeps its claim instead of being re-delivered
mid-flight. The handler itself runs through :call-traced: with the
trace ids the publisher stamped on the message, its execution is
persisted as a child of the publisher’s — the Run pane’s Downstream
calls continue across the queue exactly as across HTTP. Its :take /
:ack / :nack / :extend are fn-typed slots (:lease-every-ms
a knob), so the backend is a binding: :pg-queue-consumer binds the
Postgres primitives above (batches of 10, 30 s visibility renewed every
10 s, 5 s wait; retry after 5 s, dead after 5 attempts); a broker
package binds its own :take / :ack / :nack to the same template —
extending :queue-consumer-leaseless (the no-op :extend,
:_queue-no-extend) when it has no leases; the template itself leaves
:extend free, because a slot a parent pins is final for every
descendant (tutorial lesson 07). A consumer is two bindings away:
{:name :orders-worker :parent :pg-queue-consumer
:args {:queue "orders" :handler :handle-order}}
:handle-order sees the message under message
({id, queue, payload, attempts}). To change the knobs, derive your
own take / nack ({:parent :queue-take :args {…}}) and bind them to
:take / :nack on the consumer. Make the worker a :service
(:singleton for a strict in-order-ish drain, :pool for parallel
workers — a claim is exclusive either way). The contract is the
message’s type-row, referenced by the publisher’s payload and the
handler’s message narrowing, exactly like the HTTP contract.
Operate → Queues (GET /partials/queues-panel, editor-queues.js)
lists every queue with its pending / in-flight / dead counts
(:queue-stats — one aggregate query) and the newest 200 dead letters
(:queue-dead-letters) with Requeue (:queue-requeue) and Delete —
org-scoped on the cloud, so a tenant sees its own queues, and the same
cost on a queue of a million rows as on ten. Tutorial:
lesson 39.
A run is a bounded future: it answers within its inline budget or returns
a pending execution ID. The wall-clock watchdog defaults to five minutes
(GRAPHDEN_MAX_EXECUTION_WALL_MS); cancellation is cooperative. The hourly
zombie sweep is a separate cleanup backstop (EXECUTION.md).
There is deliberately no
suspended execution that sleeps for days waiting for a callback — a
paused thread is state the reconciler cannot restart, move between
pods or merge across branches. The graph-native shape of “wait for
X” is two runs and a row:
:pg-execute, or
a table of your own through web/sql) with what it is waiting for —
an id, a token in the callback URL, a due time;:post-route), the approval (a form on a page), or a
:schedule tick reads the row back and runs the second half.Everything in between is a table row, so it survives restarts, shows in
SQL, and is branch-scoped like any other write. :queue-publish with
:delay-ms covers the “wait, then continue” case without a callback.
Packages contribute baseline :service rows through a
:services [...] field in their package.edn. Each entry is a map:
{:name :default ; required — seed name, used for id
:fn-name :web-server ; required — fn-name to resolve
:enabled? true ; optional, default true
:restart-policy :always ; optional, default :always
:description "..."} ; optional, comment-only
The reconciler invokes seed-package-services! at integrant init.
For each entry:
ids/seeded-service-id package-name name so re-runs land on the
same row.:fn-name against the :fn table; if not found, logs
and skips (admin can re-create the fn and the next boot picks it
up).:enabled? toggle survives restarts.The seeded rows go through the regular reconcile path — no separate fallback code, no separate stopper handle. Each seed becomes a fully-supervised service.
app/package.edn ships one seed by default — :default → :web-server.
Other packages can add their own; the loader aggregates seeds across
all loaded packages.
| Step | What |
|---|---|
| Done | :service schema, reconciler, integrant, generic CRUD via /api/entities/service, supervisor for startup failures, packages-based seeding, already-running rejection, validation that target fn has zero free args |
| Done | Multi-pod: per-pod reconcilers, PG advisory-lock ownership for :singleton services, :cardinality so :per-pod listeners run everywhere, service:* NOTIFY so siblings reconcile within ~1s, lock auto-release on pod crash. No :owner-pod-id column — ownership is implicit in who holds the lock. |
| Done | Periodic reconcile tick (:exec/service-reconciler, ~15s) — level-triggered convergence: re-takes a :singleton lock after the holder crashes (no NOTIFY is emitted), picks up out-of-band DB edits, reconverges transient start failures. Retry-free under reconcile-monitor so a failing start never blocks the listener. The tick actually heals a crash because the reconciler now drops ::not-our-lock placeholders at the top of every pass (they were sticky before, so an idle pod never re-attempted the freed lock). |
| Done | :pool cardinality (:pool-size N) — a service runs on up to N pods, coordinated by N advisory-lock slots (generalises :singleton = slot 0). Fixed :pool-size only; load-driven autoscaling of N is out of scope (the request path scales via cells + HPA). |
| Done | Startup steps — :migration / :migrate templates in storage/pg (§ Startup steps): journaled, advisory-locked, one-transaction schema migrations sequenced ahead of the listener with :do. |
| Done | Many triggers per service — :start-all over a list of :schedule / :interval derivatives (§ Service shapes); :interval for fixed periods; :traced-call to persist each fire as a run. Replaces the planned :service-schedule child entity. UI: row-actions “Make service” + sidebar “Only services” filter. |
| Done | Advisory-lock connection-drop reconnect + re-acquire. The lock connection is held behind a reconnecting holder; every reconcile pass runs advisory-lock/ensure-live!, and on a reconnect reassert-lock-ownership! re-takes each :singleton this pod was running (stopping any a sibling stole during the outage). Closes the “two pods double-run one service until the next reconcile” window. |
| Done | Cross-pod cancel routing for :fn-execution — execution:cancel:<id> NOTIFY fan-out; see EXECUTION.md. |
| Done | Endpoints — one :service-instance row per running copy (host + bound port from the handle’s :endpoint metadata, heartbeat each tick), deleted on stop; :service-endpoint (web/service) resolves a service fn named through a :fn-ref slot to a LIVE copy, with the addon resolver seam for cloud app-routes (§ Endpoints). |
| Done | Liveness — handles carry :alive? / :exit; the per-tick pass restarts a copy that died in place per :restart-policy (:always any exit, :on-failure a throw, :never parks), heartbeats live copies and reaps the rows a crashed pod left (§ Liveness). |
| Done | Queues — :queue-message (Postgres, SKIP LOCKED + NOTIFY wake), the four primitives, and the backend-swappable :queue-consumer / :pg-queue-consumer templates (§ Queues). |
| Future | Pluggable supervisor strategies |
src/graphden/schema/services/schema.cljsrc/graphden/services/reconciler.clj (the pass),
src/graphden/services/instances.clj (instance rows, stop, stale reap),
src/graphden/services/liveness.clj (exit detection, exit backoff)src/graphden/services/endpoint.clj (native row and heartbeat reads) and resources/packages/web/service/ (graph source priority and call templates)src/graphden/schema/queue/schema.clj, resources/packages/storage/queue/ (primitives + the consumer templates)src/graphden/system/core.clj → :exec/service-reconcilerresources/packages/app/execution/fns.ednresources/packages/web/crud/fns.edn → :_create-service-free-args-rej (fn has start-blocking free args) and :_create-service-no-process-rej (fn doesn’t declare the :process effect)resources/packages/app/execution/fns.ednresources/packages/app/execution/{fns.edn,impls.clj} → :_reconcile-servicesresources/packages/app/routes/fns.edn → :api-services-reconciletest/graphden/services/reconciler_test.clj,
test/graphden/services/service_endpoint_e2e_test.clj (two services over a real socket),
test/graphden/packages/web/service_test.clj, test/graphden/executor/fn_ref_test.clj,
test/graphden/packages/storage/queue_test.clj, test/graphden/services/queue_consumer_e2e_test.clj,
test/graphden/packages/app/execution_routes_test.clj,
test/graphden/crud/fn_execution_test.clj (already-running cases),
test/graphden/crud/entities_test.clj (parser cases)