🔧 Internal engineering record — not user documentation. A contributor-facing investigation note, indexed from CLAUDE.md and outside the reader path (see docs/README.md).
Not a roadmap. Production endpoints comfortably fit their request budgets today. This file holds the diagnosis + attempted-fix lessons from the 2026-05 perf investigation, preserved for the next time we need to push further on the executor hot path.
| path | wall | size | notes |
|---|---|---|---|
/api/graph/entities (base-fn impl) |
~95 ms | 4.5 MB | direct Clojure call, no executor — baseline. Full scope (CLI / e2e helper). |
/api/graph/entities?scope=tree |
~15 ms | O(namespaces), ~few KB | namespaces + per-namespace named-fn counts, NO fn rows. The editor’s initGraph / post-mutation refresh use this — the O(all-fns) scope=index pull is gone from the hot path. |
/api/graph/entities?scope=namespace&namespace-id=X |
~5-15 ms | O(ns fns) | one namespace’s light fn rows (id/name/role/counts). Fetched lazily when the sidebar expands that namespace. |
/api/graph/entities?scope=search&q=… |
~5-20 ms | O(matches), capped 200 | name-substring matches (light rows). Backs the sidebar filter box, the fn/namespace/reparent pickers, and name→id resolution. |
/api/graph/entities?scope=index |
~30 ms | 1.6 MB | fns + namespaces only. Still O(all-fns); retained for CLI / backward-compat — the editor no longer uses it (superseded by scope=tree). |
/api/graph/entities?scope=subtree&root-id=X |
~5-40 ms | 1.5 KB - 4.2 MB | BFS closure from root-id. Leaf fn = 1.5 KB; app-root = 4.2 MB. Editor uses this per selectFn (commits bec65163 + 55bee689). |
/api/branches (graph composition via :resolve-fn-rows) |
~20 ms | 498 B | small dataset |
resolve-versioned-rows-matches-clojure-end-to-end (test) |
~16 s | n/a | bootstrap-included; 4× executes + 4× Clojure SOT compares; per-execute slice estimated ~0.5–1 s |
The graph-executor path is now within an order of magnitude of direct Clojure (vs the original 900× pre-eager-compile). Wins that brought it here:
CREATE DATABASE … TEMPLATEThe 4-step real fix sketch (below) is preserved but the
priority dropped sharply. Before allocating multi-commit
work, re-benchmark with clj-async-profiler on the current
executor pipeline — the 2026-05 flame-graph predates the
eager-compile refactor and may show a different hot-frame
profile now.
Re-benched the current pipeline with the harness now committed at
development/bench/graphden/exec_profile.clj (bootstrap [core],
map a 2-node composed callback over (range 2000) ≈ 4–6k
node-executes, criterium). Result: ~3.8 ms → ~1 µs/node — the
executor is now near the sketch’s <500 ns/node target; the
eager-compile refactor already captured the big win.
Sketch status: #3 (HashMap call-cache, not Atom) is DONE —
::call-cache is a java.util.HashMap today. #1 (transients)
is PARTLY DONE — the arg-map builder loop already uses
transient/persistent!. Landed a variant of #4: skip the
per-call fa-ref volatile when a fn has no env-bindings (commit
9d79b29d). That point-fix measured ~5% — inside the run’s
std-dev, i.e. noise, re-confirming the 2026-05 lesson: single-site
fixes don’t move a smeared profile. Kept only because it’s
provably-equivalent + cleaner, not for the number.
Do NOT expect executor micro-opts to speed up bb test — the
slow tests (entities-graph, branches-lifecycle,
compile-packages) are CRUD / compile / HTTP-bound, not
execute-bound. The banked test-suite wins (2026-07-09, ~36 min →
~23 min) came from delta-invalidation + minimal package sets, not
execute-path tuning.
A test-suite audit reached “registry-test costs 91 s because each
install/fork/materialise recompiles the graph” and proposed tightening
the delta path. Instrumenting it first — :registry/delta-seed-fns and
:registry/delta-recompiled-fns in compile-runtime/delta-recompile!,
:registry/rebuilt-fns in rebuild! — killed the hypothesis outright.
Whole unit suite, 1777 tests:
| counter | value |
|---|---|
:registry/delta-recompile |
365 |
:registry/delta-seed-fns |
477 |
:registry/delta-recompiled-fns |
478 |
:registry/rebuild |
146 |
:registry/rebuilt-fns |
52 678 |
Blast ratio 478/477 = 1.002. A delta that names one changed fn recompiles one fn. The inverse-closure walk is not over-reaching, the shared-ancestor fan-out people worry about does not happen, and there is no headroom there to take.
99.1 % of all compile work in the suite is full rebuilds (52 678 of
53 156 fns). So the question was never “are deltas tight” but “why 146
rebuilds”. Of those, 17 come from :registry/invalidate-full — a write
that answered unknown and dropped the registry. The other ~129 are
cold-start compiles: one per test namespace that builds its own ctx.
That is the fixture design (per-NS logical DB + own ctx buys parallel
isolation), not a product defect, and registry-test shows the pattern
in miniature — ONE rebuild of 5665 fns, then 21 deltas totalling 123
fns.
Where that leaves the two audiences:
:registry/
invalidate-full’s note (bb perf prints notes under a breached
budget; perf/runs/unit.edn carries them) counts the graphden call
frames that asked for it and, for the CRUD path, the write shape
([entity-type keys] — crud.entities/invalidate! passes it as the
third argument of invalidate-graph-cache!). See § 2026-09-10 below
for the breakdown that produced and what it fixed.And a caution for whoever picks up registry-test’s 91 s: its 62 s of
test time is not the compile — that lives in its 27.8 s fixture. The
tests themselves are storage writes and materialisation. Measure before
optimising; this note exists because the last person didn’t.
Both found by an audit and MEASURED before fixing, on the first-party
corpus parsed without a DB (packages/records/parse-module over all
seven packages: 9043 fns, 16381 bindings, 5902 list items):
| path | before | after | cause |
|---|---|---|---|
deps/incremental-update, 1 changed fn (every CRUD write via delta-recompile!) |
13.3 ms | 0.007 ms | began with index-graph over the whole graph; now reads the compile’s own lookups |
compile-all (8867 fns) |
4343 ms | 2199 ms | renames/inheritance-descendants rebuilt a parent→children index per call — 339 calls, 2206 ms; now :children-by-fn in build-lookups |
public-free-entries over every named fn |
5841 ms | 1139 ms | same helper, 758 calls |
Same shape as the compute-fn-typed-fn-ids O(n²) that build-lookups
now precomputes as :fn-typed-fn-ids: a pure function of fn-map
recomputed per caller. The fix is the same too — compute it once INSIDE
build-lookups, never as a process-global memo (a memo that outlives
its lookups returns a stale set for a different fn-map).
Leaf-time top frames (collapsed) from a 63-second
exec/execute :probe-call run over 2400-row dataset:
5500 PersistentHashMap$BitmapIndexedNode.assoc
2942 PersistentHashMap$NodeSeq.kvreduce
2111 StringLatin1.replace
1963 PersistentArrayMap.indexOf
1908 Var.getRawRoot # dynamic var deref (call-cache)
1886 RT.nthFrom
1403 G1BarrierSetRuntime::write_ref_array_post_entry
1387 ConcurrentHashMap.get
1180 oop_disjoint_arraycopy # GC scavenging
1152 Util.hasheq
991 G1ParScanThreadState::trim_queue_to_threshold
806 Keyword.intern
The cost is smeared evenly over PersistentHashMap.assoc and
kvreduce — there is no single hot frame. Each graph-node
execution allocates a handful of small maps (free-args, aug,
ref-thunk env), and a single iteration of :map :_rv-resolve-one
hits 10+ nodes. With 2400 outer iterations
× N inner iterations, that compounds to millions of map
allocations and gives the GC a lot to clean up
(G1BarrierSetRuntime, oop_disjoint_arraycopy,
trim_queue_to_threshold are all collector frames in the
top 12).
Two point-fixes were attempted; neither gave measurable speedup, and one made things WORSE:
reduce-kv in build-closure when aug is empty
— tiny composed fns (:get, :assoc, :vec) end with
empty aug. Replaced unconditional reduce-kv fa-base aug
with (if (seq aug) (reduce-kv …) fa-base). No measurable
change — the surviving :ref-binding fns still have
non-empty aug, and they dominate the call graph.snake->kw over a ConcurrentHashMap —
decode-row does (keyword (str/replace col "_" "-")) per
(row, column). A cache that returns interned keywords on
hit should remove the Keyword.intern + String.replace
cost. In practice the run got slower (~270 s), so the
bottleneck must be elsewhere; reverted.Lesson: when the diagnosis is “smear, not a hot frame”, shrinking ONE allocation site is wasted effort — the allocator + GC are paying the cost across the whole call graph. The real fix has to change the allocation PROFILE of the whole executor, not optimise individual helpers.
The executor pipeline needs to drop allocations on the hot path. Candidate work, roughly in order:
build-closure
— the arg-map builder loop in compile-fn’s closure already uses
transient/assoc!/persistent!. The env-binding merge loop
still uses persistent assoc, but it only runs when env-n > 0
(uncommon).make-ref-entry wraps every :ref binding in a
rt/thunk so resolve-arg can decide at call time
whether to deref. For impls that we know force ALL their
args (the common case), the thunk wrap is pure overhead.
Compile-time classification of “this impl forces arg X”
lets us bind the resolved value directly.HashMap not an Atom
— ::call-cache in ctx is a java.util.HashMap installed once
per top-level execute; nested siblings inherit it through ctx.
No per-write CAS or persistent-map churn.9d79b29d] Pool/skip the
per-call volatile! — now skipped entirely for fns with no
env-bindings (the common case; the volatile is only read by
env-binding delays). Measured noise-level, kept for cleanliness.
Fully pooling it for the env-n > 0 case is still open but
below the noise floor.Expected total: bring per-graph-node overhead from ~10 µs
down to <500 ns. Target ~50× speedup on the resolve-versioned-
rows fixture, which would make :resolve-versioned-rows over
the full graph fit in ~1 s.
Allocate this work only after a fresh clj-async-profiler
run confirms the smear is still in the same place after the
eager-compile refactor.
Historically CI hit the 600 s tests ceiling under in-JVM
parallel runs (4 concurrent NS bootstraps × ~38 s caused
memory pressure). The eager-compile refactor + shared golden
DB + DRY memo + rich-types snapshot fix brought walls to
~6:30–7:00 for all 1400+ tests / 0 failures. Ceiling now at
900 s. The bb test-parallel 4 worker-isolation workaround
is no longer needed.
A prior handoff flagged “~177 MB of live-set growth over a
suite run (95 → 272 MB after a full GC), unexplained.” It is
not a leak — it is the fixed working set of a fully-loaded
test JVM. A run boots the whole graph (252 base-fns + 2728
fn-defs compiled), builds the process-global registries
(base-fns, rich-types, compile-all closures) and the golden
bootstrap; that footprint loads early and plateaus — it
does not grow proportionally to the test count. Measured
end-of-:unit-suite live-set: 257 MB, stable across
identical runs. Mirrors the demo-instance note in the JVM
flags (“live set 125 MB after a full GC … not a leak”).
The one real per-NS leak — test namespaces that create a
storage and never sp/close it, so its HikariCP pool outlives
the NS — was measured and is within GC noise at the test
pool-size of 2: closing every tracked storage moved the
end-state 260.7 → 257.0 MB (~3.7 MB). An earlier ad-hoc probe
overstated it 5× by using the default pool-size (10). A
suite-end backstop now closes every tracked storage
(shared-container/register-storage! + close-all-storages!
in the plugin post-run) — cheap insurance against the
footprint growing if the pool size is raised or many leaking
NSes are added, not a fix for H.
Do not re-investigate H as a leak. Re-measure before/after with the real pool-size (2), not an ad-hoc probe, if the question resurfaces.
build-hof-translation + apply-hof-translation shipped on
refactor/slot-id-keyed-runtime add per-HOF-wrap-time work:
build-hof-translation runs at COMPILE time (once per HOF
binding’s arg-builder construction). Inside is a single
(deep-free-ext-entries r-fn-id lookups) call (memoised via
:deep-free-ext-entries-cache in lookups) plus a reduce
filter — O(R-walker-entries), typically 1-20 entries.apply-hof-translation runs at RUNTIME (per HOF callable
invocation). The empty-translation case short-circuits and
returns fa unchanged — the COMMON case for fns that don’t
surface their ext-names through HOF boundaries. The non-
empty case is reduce-kv over translation entries (typically
1-5) with two contains? checks + one rt/thunk? check per
entry.Worst-case per-HOF-wrap runtime overhead: ~5 entries × ~10 ns per entry = ~50 ns. Per-request HOF wraps are O(log handler- chain-depth), typically 3-5. Total per-request: ~250 ns.
Versus the actual per-request work — HTTP parsing, DB query, graph-compose response, JSON encode — typically 10-50 ms per endpoint per current measurements. The translation overhead is 5-6 orders of magnitude below the per-request budget.
Smoke + e2e regression checked after ship: no measurable
slowdown on any of the 11 smoke checks or the 51 e2e tests.
No explicit benchmark added since the overhead is below
bb rebuild round-trip noise (rebuild itself is ~3 minutes).
Attributing every :registry/invalidate-full (the note above) over the
unit suite gave 24 events, none of them a product-path warm clear:
| caller | warm? | seeded? | n |
|---|---|---|---|
crud.entities/invalidate! — :fn writes carrying :id |
no | yes | 9 |
crud.entities/invalidate! — :binding / :binding-list-item writes |
no | yes | 2 |
crud.entities-test, fn-execution-test, compile-runtime-test — tests calling the 1-arity on purpose |
mixed | no | 9 |
| the shared-container fixture’s bootstrap | mixed | mixed | 4 |
“Seeded but cold” is the whole product story: a write names its fns
(affected-fn-ids answered), but the ctx had no compiled registry and no
reverse-deps index yet — a fresh per-namespace fixture ctx, or a branch ctx
no request had compiled — so the delta branch could not run and the
:else fell through, counting a clear of nothing. It costs nothing (the
next request compiles once either way) and it was inflating the metric that
is supposed to track the ~5 s / ~50 s stalls. The counter is now split:
:registry/invalidate-full counts only WARM clears (a compiled registry was
dropped), :registry/invalidate-cold the no-ops. The attribution note
keys carry {:warm? :seeded? :write [entity-type keys]} so the next
investigation reads the shape off perf/runs/unit.edn instead of
re-instrumenting.
The perf trend (advisory) had graph-entities-tree at ~4× its 2026-08-27
units while every other scenario held. Not the database (one round trip, as
budgeted): the sidebar’s per-namespace kind counts — :type-count /
:fn-count, so the fn / types filters keep unloaded
namespaces visible — were computed by annotating the ROLE of every fn on
every paint (roled-fns in crud.entities.list, ~5k compute-fn-role
calls). The counts are a pure function of the graph snapshot and the
rich-types snapshot, so tree-kinds memoises them per branch on those two
identities (the same freshness check the graph lint uses — a write replaces
the snapshot object); the per-namespace diagnostic counts stay per request,
because they move without a graph write. :sidebar/tree-kinds-computed
counts the recomputations and perf/budgets.edn holds it at 1 for the
three-paint scenario.
The platform-tests namespace was ~760 s of every landing gate: 368 tests at
~2.1 s each, against tests that each run in milliseconds. A JFR profile of
run-tests! put 98% of the samples under resolve-execution-graph, reached
twice per test — the free-arg surface (free-arg-slot-map-cached) and the
run’s graph hash (graph-hash-cached). The batch resolver answers one root,
but gets there by loading EVERY fn / fn-slot / binding / list-item of the
branch with their version overlays (~0.8 s on the shipped graph) and walking
the root’s closure in memory. Both memos above it are per root, so each new
test paid two whole-branch loads. The same cost lands on a tenant’s
[Run all], and a cold Run of any fn paid it twice.
resolution/call-with-graph-load-memo shares the load inside a scope,
keyed on [org decorated? branch-id graph-epoch] — the epoch is bumped
before every graph-shaped write, so a test that edits the graph is followed
by a reload, not a stale read; a handle with no pool anywhere beneath is
never memoised. The epoch is read through graph-epoch/epoch-handle: the
first cut read it off the handle it held, and on the cloud stack
(Versioned(OrgScoped(Postgres)), no :pool on the decorator) the memo
never switched on, so a tenant’s [Run all] still paid two loads per test.
The org in scope and whether the base is decorated sit in the key because
the tenancy decorator filters reads by org — an org-scoped read and a raw
one never share an entry.
run-tests! scopes the whole run, apply-execute’s plan scopes its two
lookups. Measured on 40 platform tests, one JVM, no contention:
1638 ms → 69 ms per test. Scope-bound on purpose: holding a whole graph
across requests would duplicate what the compiled registry already holds.
Covered by versioning.storage.core-test/graph-load-memo-shares-one-branch-load-per-epoch-test
and, for the decorated stack, versioning.storage.decorated-stack-test/graph-load-memo-works-under-a-decorator-and-keys-on-the-org.