Graphden documentation

Function Execution

Run any fn-graph from the editor (or HTTP), get the result inline when it’s fast, or hand off to async polling when it isn’t. Persisted runs feed the per-fn history sidebar.

This is the execution-as-event model — every (future …) we submit is a moment-in-time fact that can be replayed but not rewritten.

Architecture overview

   POST /api/execute  (the :execute route — a graph fn-def)
        │
        ▼
   parse-execute-request   (json → {fn-id args timeout-ms persist?})
        │
        ▼
   graph validation :cond  (fn exists, args fit free-args, caps —
        │                    `:_execute-validation` in app/execution/fns.edn;
        │                    no Clojure mirror, the graph is the sole path)
        ▼
   apply-execute  ──► (future …)  ──► executor.compile-runtime/execute
        │                                  │
        ├──◄ deref f timeout ──► inline:   │
        │   {status: succeeded, result}    │ ── future resolves
        │                                  │
        └──► timeout fires:                ▼
             persist :pending,  ◄── record-completion! tail-future
             return :execution-id          writes :status :succeeded
                                           | :failed | :cancelled
                                           + :result/:error/:finished-at

Every run is submitted to a bounded ThreadPoolExecutor (make-execution-pool in crud/fn_execution/persist.clj): core == max workers (GRAPHDEN_MAX_CONCURRENT_EXECUTIONS, default 128) plus a bounded LinkedBlockingQueue. Runs beyond the worker count park in the queue; only when the queue is ALSO full does submit throw RejectedExecutionException, which the handler turns into 503 + Retry-After (:error-data {:reason :queue-full}) — never an unbounded queue, never a silent drop. A watchdog (*max-execution-wall-ms*) hard-kills a runaway. The HTTP handler waits on the submitted future with a timeout; long fns gracefully degrade to polling.

Storage schema

Three non-versioned entities + one enum live in src/graphden/schema/executions/schema.clj:

:fn-execution

Field Type Notes
:id :uuid Returned to the client as execution-id.
:fn-version-id :ref :fn-version Frozen snapshot of which version ran. Base :fn-id derives via JOIN.
:started-at :timestamptz When the future was submitted.
:finished-at :timestamptz (null until resolved) Duration computed at read-time (finished - started). No denorm.
:status :execution-status enum :pending :succeeded :failed :cancelled.
:trace-id :uuid (null, indexed) Cross-service tracing: the top-level execution’s id, shared by every hop (§ Tracing across services). nil ≡ untraced / a top-level run.
:parent-execution-id :uuid (null, indexed) The execution that called INTO this one over the wire (a :service-get from another service).
:graph-hash :text (null, indexed) SHA-256 of the RESOLVED graph the run executed — every reachable fn / slot / binding / list item as the branch saw them at submit time (lookup/graph-hash). The content anchor: a binding edit below the root changes it while :fn-version-id does not (VERSIONING.md § Execution anchoring). nil on pre-field rows.
:result :jsonb (null) Capped at 5 MB. Oversize → :result nil + :result-truncated? true.
:result-truncated? :bool (null) Set when the 5 MB cap fires.
:error :text (null) Exception message, truncated to 4 KB.
:error-data :jsonb (null) ex-data, truncated to 64 KB (or {:type … :truncated true}).
:declared-effects :jsonb (null) Snapshot of :effects from rich-types registry at submit time.
:runtime-effects :jsonb (null) Observed effects captured via cr/record-effect! inside instrumented impls. Diffed against :declared-effects to surface drift in the editor’s result pane.
:user-id :uuid (null) Reserved for future user/session auth.
:cancel-requested? :bool (null) Set by /cancel; observed by *cancel-check*.

:fn-execution-arg

Mirrors :binding shape — one row per free-arg supplied:

Field Type Notes
:execution-id :ref :fn-execution  
:slot-id :ref :slot Which free-arg.
:value :jsonb (null) XOR with :ref-fn-version-id.
:ref-fn-version-id :ref :fn-version (null) Frozen snapshot of ref-targets (HOF args).

Unique on (execution-id, slot-id).

:fn-execution-arg-item

Sequence content under list-typed args (mirrors :binding-list-item):

Field Type
:execution-arg-id :ref :fn-execution-arg
:position :int
:value :jsonb (null)
:ref-fn-version-id :ref :fn-version (null)

Unique on (execution-arg-id, position).

Why non-versioned

Executions are events (immutable historical facts). Mutations are limited to :status flipping :pending → terminal. Versioning machinery would record meaningless one-step trails — not in versioning.storage.resolution/entity-config.

HTTP API

When authentication is enabled, these endpoints require an authenticated session or bearer token (auth-required-middleware).

POST /api/execute

The selected function receives its own execution authorization check before submission. Permission to run the HTTP handler does not authorize that function. An API token without the execute scope receives HTTP 403 with error: "token-scope"; a namespace permission refusal returns HTTP 403 with error: "forbidden". Neither refusal creates an execution row or future. The worker also checks the selected root using the submitting principal.

Request:

{
  "fn-id":      "uuid",       // XOR with fn-name
  "fn-name":    "add",
  "args":       {"a": 1, "b": {"ref": "uuid"}, "c": [1, 2, 3]},
  "timeout-ms": 10000,         // default 10000, cap 60000
  "persist?":   false,         // default false
  "trace?":     false,         // default false — Debug P1 path capture
  "capture-values?": false     // default false — Debug P3 value capture
                               //   (implies trace?; the editor sets it only
                               //   behind an explicit confirm dialog)
}

Arg value shape (per slot):

Form Stored as
1, "hi", true, null :value (literal jsonb)
{"ref": "uuid"} :ref-fn-version-id (resolves logical fn → version)
[1, 2, {"ref": "..."}] Per-item :fn-execution-arg-item rows

Response — 4 shapes:

// a) finished within timeout, NOT persisted (default for pure fast fns):
{"status": "succeeded", "result": …, "declared-effects": null}

// b) finished within timeout, persisted (?persist=true OR effects ≠ #{}):
{"status": "succeeded", "result": …, "declared-effects": ["env"],
 "execution-id": "uuid"}

// c) timeout fired — polling mode:
{"status": "pending", "execution-id": "uuid"}

// d) pre-flight validation failed (400):
{"status": "rejected", "error": "...", "error-data": {"reason": "..."}}

Auto-persist matrix:

Condition Persist?
client persist?=true yes
declared-effects ≠ #{} (audit trail) yes
timeout fired (need polling target) yes
pure fn finished inline AND ¬persist? NO

GET /api/execute/:id

Returns the full row + nested :args array (with per-item :items sub-array for list args). 404 when row not found (after TTL cleanup or wrong id).

POST /api/execute/:id/cancel

{"ok": true, "cancel-requested": true}

Sets :cancel-requested? true + future-cancel. Best-effort:

GET /api/executions?fn-id=X (or ?fn-version-id=V)

{"ok": true, "executions": [/* most-recent rows, latest first */]}

Pass ?fn-id=X to list runs of the version that resolves on the current branch (the one ▶ would execute now — not every historical version), or ?fn-version-id=V to list runs of one specific version (the ⌛ history panel’s per-version expand uses this; it wins when both params are present). Ordered :started-at desc. ?limit is clamped to 1–100, defaulting to 20 when absent or non-numeric. Summary shape (no nested args). Editor’s History panel calls this. Two optional narrowings ride on the same query, applied in the SQL (so the page is the last N matching runs, not the matches among the last N): ?secrets=1 keeps only audit-trail rows (touched-secret?), and ?graph-hash=H keeps only the runs of one exact resolved graph (the row’s :graph-hash anchor).

Concurrency caps

Two caps gate how many executions run at once, with different scopes and different overflow behaviour (see crud.fn-execution.persist/acquire-execution-slot! and make-execution-pool):

Cap Env Default Scope Overflow
Global GRAPHDEN_MAX_CONCURRENT_EXECUTIONS 128 Per-POD. Sizes the bounded execution pool’s worker count. Parks in the pool queue (bounded LinkedBlockingQueue); it no longer rejects at admission. Only when workers AND queue are both full does submit throw → 503 + Retry-After {:reason :queue-full}.
Per-org GRAPHDEN_MAX_CONCURRENT_EXECUTIONS_PER_ORG 32 Per-pod live counter for every org, including pure inline runs; tenants additionally check pending :fn-execution rows for a fleet-wide persisted-run budget. Rejects at admission, no row or future created: {:ok false :status :rejected :error-data {:reason :over-capacity}}.

The per-org cap is the admission gate. The global cap moved to the bounded pool: it queues work (park-then-503) rather than dropping it, and the per-pod bound remains the exact thread-exhaustion safety net. See SCALING.md § Fleet-wide per-org quota.

A second submit-time refusal precedes even the cap check (error-tolerance Phase 4): an fn with RECORDED type diagnostics on the current branch is rejected with {:ok false :status :rejected :http-status 400 :error-data {:reason :unresolved-type-errors :diagnostics […]}} — the message names the fn + its first error. See ERROR_CODES.md § execute rejection.

Wire caps

Field Cap Behaviour on overflow
:result 5 MB Store nil + :result-truncated? true
:args (sum, serialised) 256 KB Pre-flight reject with 413 / :args-too-large
:error 4 KB Truncate with …
:error-data 64 KB Truncate JSON path, keep :type if possible
:timeout-ms 60 s Reject :timeout-out-of-range

TTL cleanup

Integrant component :exec/cleanup-scheduler runs a single ScheduledExecutorService thread hourly (override via CLEANUP_PERIOD_MS env var).

Status Retention Action when past
:succeeded 7 days DELETE
:failed 30 days DELETE
:cancelled 7 days DELETE
:pending > 1 h (zombie sweep) Flip to :cancelled (NOT delete); next sweep applies the 7-day TTL

Implementation: sweep-executions! pool now (in graphden.crud.fn-execution.retention; graphden.system.init.cleanup owns only the cadence). The now argument is injectable so tests can verify TTL behaviour without sleeping real time.

The sweep is four set-based statements — zombie UPDATE, TTL DELETE, then two anti-join DELETEs that reclaim :fn-execution-arg / :fn-execution-arg-item rows whose parent is gone. Those child tables carry no foreign key (:ref fields become plain indexed UUID columns), so nothing else removes them; the anti-join also clears rows detached before this sweep existed.

Usage rollups (:usage-stat)

Every terminal transition (both inline arms in apply-execute and the async arms in record-completion!) also increments a pre-aggregated rollup row: one per (UTC-hour bucket, org, fn, status) with a run count and summed wall-clock duration — via an atomic INSERT … ON CONFLICT DO UPDATE (graphden.crud.fn-execution.stats/bump!, best-effort: a failed bump logs and never fails the execution). The table stores counts and durations only — never args, results, or error text — so it is privacy-safe to aggregate and grows with distinct keys, not traffic.

Reads: the :usage-fn-stats graph fn-def — a :quot-derived average over the :fn-stats-raw read base-fn ({:runs :failed :cancelled :avg-ms} for one fn over a trailing window, scoped to the CURRENT org) feeds the “7d: N runs · M failed · avg K ms” strip at the top of the editor’s execute-history panel; org-stats (src-level) lists an org’s busiest fns for operator tooling. Retention: the cleanup scheduler sweeps buckets older than 90 days (sweep-stats!) — trends outlive the raw :fn-execution TTLs, which is the point.

Unresolved failures (GET /api/failures, the ✕ failed chip)

The editor’s ✕ failed filter chip (Explorer) focuses on, and the Inspector’s Runs tab lists, the current branch view’s unresolved recent failures (newest first) straight off the :fn-execution audit rows — no new storage beyond two columns (:branch-id stamped at create, :acknowledged-at set by dismiss). GET /api/failures returns the per-fn counts the chip, the namespace rows and the fn cards read (editor-problems.js); the Runs tab’s block renders the rows. Privacy holds because the write path already sanitised each row: redact-outcome hides secret-tainted bodies, scrub-outcome replaces internal error types with an opaque ref: on the cloud. A row carries a ✕ Dismiss button, the finish time, the error text and a collapsible ex-data block; the chip’s chip row carries Dismiss all.

A failure counts as unresolved only while ALL of these hold (the panel is a worklist — every red counter has an action that clears it):

Backed by the :recent-failures / :failure-ack / :failure-ack-all base-fns (graphden.crud.fn-execution.errors) — raw SQL with an explicit org filter, 7-day window, 50-row cap; the still-current-version half of the predicate goes through the versioned resolver. The audit rows themselves keep their TTLs (failed rows sweep after 30 days) regardless of dismissal.

Editor UI

The ▶ button on a fn-card root row (auth-required, visible only to signed-in users) opens the Run pane — it selects the fn and lands the right inspector on its Runs tab, so the canvas stays fully visible (and pannable) next to the form, the result, and the fn’s run history below. In the history, a row whose run consumed a secret and produced an effect carries a 🔒 (SECRETS.md § Audit trail), and the Secret flows chip narrows the list to those rows — in the API’s SQL (GET /api/executions?fn-id=X&secrets=1), so the page is the last N secret flows, not the secret flows among the last N. Each row also carries a ⌗ chip with the first 8 hex of its :graph-hash; clicking it narrows the list to runs of that exact graph (&graph-hash=H), and the active chip in the strip clears the filter:

Source: resources/packages/app/editor/editor-execute*.js.

Typed value representations

The result pane’s first non-trivial dispatch tier is the value-repr registry — the read-side sibling of the value-form system: :_value-form-registry answers “how do I EDIT a value of this type”, :_value-repr-registry (app/reprs/fns.edn) answers “how do I SHOW one”.

Component preview is the neighbouring tier, not a registry row: when the fn’s declared return type is semantically :hiccup-node (mutual-subtype? equality — one-way ⊆ would swallow every scalar, :hiccup-node being a union), the result renders as markup in a fully sandboxed iframe (sandbox="" — no scripts, no same-origin; unlike the HTML-response pane’s allow-scripts). This is the static half of a devcards-style component preview. The narrowed :script-tag / :style-tag returns deliberately fail the mutual-equality test (refinement ⊂ base, never ⊇) — a page asset is not a visual component, and its repr above wins the dispatch anyway.

Interactive component preview — GET /preview

The dynamic half: the component-preview caption carries an “Open interactive preview ↗” link to GET /preview?fn-id=<uuid> (&branch= preserved), which renders the component as a REAL page — components stylesheet, htmx, and the runtime dispatcher scripts all live — in a new tab. A tab keeps the editor DOM out of reach, avoids the global frame-ancestors 'none' / X-Frame-Options: DENY headers entirely (no framing), and authenticates like any page navigation (the session cookie under accounts; an auth-off self-host is open by the same :get-auth-required seam).

Components gallery — GET /preview with NO fn-id (devcards L3): one card per named fn whose return type is semantically :hiccup-node, sorted by namespace-path + name, capped at 60 with a visible overflow count. A PURE component with zero free args renders LIVE inside its card (per-card :_execute-apply; inline pure runs write no rows); effectful/unknown or parameterised components show their effect chips / a hint and defer to their single-fn page (where the confirm gate applies). Same self-host-only 403 gate. Missing vs malformed fn-id is decided on the RAW query param — absent serves the gallery, present-but-unparseable is a 400. The :filter / :sort-by callbacks stay pure per the HOF contracts; the effectful lookups live in :map callbacks (:_pvg-*).

Tests: graphden.packages.app.value-repr-test, graphden.web.hiccup-sanitize-test, graphden.packages.app.preview-page-test.

Path trace — the call tree

A trace? submission records one entry for the run’s own fn — the outermost frame (compile-eager/traced-root-call, applied by run-future, the request trap and the cross-service hop alike) — plus one per :ref invocation and one per call of a callable handed to a higher-order fn (compile-eager/tagged-callable, behind hof-wrap, the env-binding HOF case and make-single-arg-callable: :map over three items gives three frames of the mapped fn, nested under the mapping fn’s), into the row’s :path-trace jsonb (capture-values? additionally stores each non-hidden frame’s return, 4 KB/entry). Every entry carries :seq (entry-order frame number) and :parent-seq (the frame that forced it), so the completion-ordered flat vector reassembles into the call tree, the root at its top.

Secret handling is capture-time AND read-time:

Concealment (multi-tenant) is read-time too: a trace shows only what the viewer could have drawn from the graph they see. Every surface that serves a trace — the inline POST /api/execute answer, GET /api/execute/:id, GET /api/executions, the execute-trace partial, MCP execute-fn with trace: true — passes it through the view-impl seam (fn-execution.conceal/conceal-path-trace, via fn-execution/viewer-path-trace). A frame of a fn whose composition the viewer may not see (another org’s shared fn, no :view-impl grant) stays as a leaf marked :concealed? true (its own duration / value stay); every frame beneath it is dropped, and the kept frames are renumbered so :seq gaps do not count them. A frame whose ancestry is lost (truncated parent, pre-tree entry) is dropped for a tenant, since it cannot be placed. The row itself keeps the raw trace: visibility is per viewer, and grants change.

Debug: catch next request

One-shot, TTL-bounded trap on a branch’s web handler (graphden.crud.debug-capture; hook in branch-router/ring-callable-for-ctx, unarmed cost = one atom deref). While armed, the next matching HTTP request runs with the path trace bound (optionally value capture) and persists a standard :fn-execution row: the sanitized request as the handler’s arg (credential headers — authorization, cookie, x-api-key, … — are stripped BEFORE anything is stored; body capped at 64 KB), the Set-Cookie-stripped Ring response as the result, the call tree as :path-trace. The captured run then replays through the same history/trace UI as any traced run.

Scoping: one trap per (org, branch), armed and fired under the SAME org (a trap can never fire on another org’s request). The registry is runtime-only in-process state (the *traced-fn-ids* doctrine — no stored field; restart disarms). Org-keyed sharding routes an org’s requests to its own pod, so arming editor and captured request meet on one process by construction.

Editor surface: the Runs tab of a fn that is a service on the branch (the ⚙ services chip) opens with the trap block (/partials/debug-catch composed into the history partial + editor-debug.js) — arm form (path prefix, capture-values behind the explicit confirm), armed status with Cancel, «open last captured trace»; when the capture lands the tab re-mounts, so the request is a run row with its tree. (Until 2026-09-04 this was the diagnostics drawer’s Debug tab; the drawer under the canvas is gone.)

Execution support and boundary validation

The callable type’s fifth element records supported execution profiles, separate from its fourth-element effect ceiling. #{:browser-main} requires a browser implementation; #{:jvm-server} requires a JVM implementation; the two-profile set proves both. :unknown grants no profile proof and #{} proves none. Legacy callable declarations retain unknown support. Scoped execution variables are solved per call, including nested map/list callback contracts.

The trusted primitive capability registry is shared by checker derivation and the browser exporter. Purity alone does not establish browser support: an unsupported native primitive, unsupported capture/lambda projection or callable construction stays unproven. Authored type overrides can require support but cannot manufacture it. Invalid saved drafts remain diagnostic and do not gain execution or export permission. These rules do not replace ACL checks or runtime effect authorization.

Use explicit core.logic/ensure-type at an ingress, RPC or persistence boundary to validate a supported data definition after parsing. Success returns the original value; safe path/code diagnostics report failure without its contents. It preserves secrecy and does not validate every internal graph edge. A web.html/UiView is a presentation result, not the field’s saved value; intentional serialization uses web.html/encode-view, which retains secrecy labels. See the type contracts and validator subset.

Runtime effect tracing

Each apply-execute future binds a fresh *effect-trace* atom-set (see graphden.executor.compile-runtime). Effectful base-fn impls call (cr/record-effect! :env) etc. to declare what they’re about to do; the reaper snapshots the set onto the row’s :runtime-effects field at terminal status.

Categories: same vocabulary as rich-type-of :effects — :db, :env, :io, :network, :time, :random, :process, :state, :raw-sql.

Currently instrumented (in resources/packages/core/system/impls.clj):

(:jvm-version and :heap-memory are fn-def compositions in fns.edn, not impls.clj defbases — their :io propagates from the JVM/heap primitives they compose.)

Adding instrumentation is a one-liner inside the defbase body:

(defbase my-effectful-impl [arg]
  (cr/record-effect! :network)
  (http-get arg))

(cr/record-effect! cat) is a no-op outside an execution trace context, so direct unit-tests of the impl don’t need special setup.

The editor’s Run pane surfaces the runtime set as a “ran:” strip below the result. Two drift visuals:

The unobserved chips render only when a runtime trace WAS recorded for this row — runs from before instrumentation existed (or for fns that never call record-effect!) leave both fields empty and the strip is suppressed entirely.

Cancellation contract

The *cancel-check* dynamic var (graphden.executor.compile-runtime) is checked at the top of every execute invocation. Default unbinding is nil (no-op). The fn-execution future wrapper binds it to a closure that reads the per-execution cancel-flag atom and throws InterruptedException when flipped.

For long-running impl bodies (e.g., compute loops, scan jobs) that want to participate cooperatively:

(defbase my-slow-loop [coll]
  (reduce (fn [acc x]
            (cr/check-cancel!)         ; cooperatively yields
            (process x))
          [] coll))

(cr/check-cancel!) is a no-op outside the execution context, so impls can call it unconditionally.

Tracing across services

The path trace above stops at the socket: a :service-get to another service is one frame here and, on the other side, a request nobody records. Tracing continues the tree across the wire:

  1. A persisted run binds cr/*execution* ({:id :trace-id}) for its thread (persist/run-future); a top-level run is its own trace root. Effectful fns are always persisted, so a consumer that calls a service always has an identity.
  2. :service-get / :service-post merge :trace-headers into the request — X-Graphden-Trace: <trace-id>;<execution-id> — {} when the run is not persisted.
  3. :http-server receives its handler as a callable tagged with the fn it wraps (hof-wrap metadata); a request carrying the header runs through fn-execution.trace/run-traced!: a fresh execution id is bound as *execution* (so the handler’s own outbound calls link to it), the handler runs under a path trace, and the outcome is persisted as an execution of the handler fn with the caller’s :trace-id and :parent-execution-id (debug-capture/ persist-captured!, the same writer the Debug trap uses — request sanitised, response minus Set-Cookie). A request without the header is not persisted; ordinary traffic pays a header lookup. Neither is one whose header names no execution the listener’s org can see — the parent hop’s row, or the trace root’s when the parent is itself a traced listener that writes its row only when it finishes. The header reaches a PUBLIC listener, so an outside caller must not be able to turn every request into a persisted, fully traced run.
  4. GET /api/execute/:id returns :children — the runs this one called into, each with its fn name and status — and the Run pane’s result partial renders them as Downstream calls under the result. Open a child and the tree continues on the other service’s side.

  5. The queue is a hop too. :queue-publish stamps the publisher’s :trace-id / :parent-execution-id on the message row (outside a persisted run it opens a fresh trace — the handling is persisted, parentless, findable by trace id); the consumer template runs its handler through :call-traced (core/ system — :call as a trace hop: with a trace id it persists the call as an execution of the callee, linked to the parent, with *execution* bound for the handler’s own calls and publishes; nil → a plain call). Both HTTP and the queue go through one seam, fn-execution.trace/run-traced-with!; run-traced! is its HTTP case. The callable’s hof-wrap metadata carries the callee’s identity AND its lambda-param names, so the persisted argument sits under the handler’s own name (message).

Rows of a whole trace: GET /api/entities/fn-execution?trace-id=… (or query the column). Tutorial: lesson 38 and lesson 39.

Tests

File Coverage
test/graphden/crud/fn_execution_test.clj parse / validate (rejection reasons) / apply (inline + persisted + args rows) / get / cancel (flag + real interrupt) / list (branch-scoped + per-version + ?limit clamp) / bounded-pool queue-full → 503 / per-org over-capacity / TTL sweep (incl. as-instant Date regression) / failed-path / args-too-large / result-truncation / exec-stats rollups
tools/browser-test/edit-execute.test.js E2E: ▶ Run pane, fill args, Run, inline result, history row reveal
test/graphden/crud/fn_execution/trace_test.clj, test/graphden/services/service_endpoint_e2e_test.clj Trace header wire format; a traced hop persisted on the listener side and listed as a child

Total: ~59 deftests / ~267 assertions backend, 8 assertions browser.