Graphden documentation

Runtime slot-id-keyed refactor (#104)

🔧 Internal engineering record — not user documentation. A contributor-facing design record, indexed from CLAUDE.md and outside the reader path (see docs/README.md).

Design record. The hybrid slot-id + name fa in §4/§7 is the FINAL runtime design, not a transitional state. This doc is kept as the design rationale for #104 — live code points here (compile/renames.clj deep-free-ext-entries, its test, ARCHITECTURE.md § Runtime fa). The “Problem” / “Workaround today” narrative below is HISTORICAL context for how the collision was closed, not an open issue.

Branch: refactor/slot-id-keyed-runtime (merged). Forked from dca0e740.

1. Problem

:body and similar names collide in the runtime free-args map (fa) when two semantically distinct slots reachable from one fn-def happen to share an ext-name. Concrete repro from app/page/fns.edn:

;; :html-page-handler :parent :html-ok-response
;;                    :args {:body :html-page-rendered}
;; (binds Ring HTTP body to rendered HTML string)
;;
;; Inside :html-page-rendered, an inline anon :parent :html-page
;; surfaces a free :body slot (HTML <body> content, hiccup tree).

Both slots’ ext-name is :body. Caller passes {:body [:p "ok"]}. Test fails:

expected: <body><p>ok</p></body>
actual:   <body>&lt;html lang=&quot;en&quot;&gt;...&lt;title&gt;X&lt;/title&gt;...&lt;/html&gt;</body>

The <body> tag receives an HTML-escaped rendered page string instead of caller’s hiccup. The two slots collide through fa[:body] and the wrong value flows to the inner consumer.

Workaround today: :as :page-body rename forces caller to pass :page-body; inner inline anon’s slot exposes as :page-body (different ext-name), no collision. Inline comment in app/page/fns.edn documents the workaround.

2. Root cause

Storage and parser hold full slot-id identity (binding row stores slot-id UUID, :body resolved at sync time). Runtime fa is name-keyed, dropping identity. Two distinct slots with the same name collide.

Layer Identity model
fns.edn (author input) name-keyed (human-readable)
Parser/sync resolves names → slot-id; stores IDs
Storage (DB rows) slot-id, fn-id, binding fields all UUIDs
Runtime fa name-keyed map ← bug surface
arg-builder reads (get fa ext-name)
env-builder writes (assoc fa env-name v)
apply-rename-aliases copies by name chain
build-ref-renames name → name translation
HOF closure capture by name

3. Architectural conclusion

The name → ID resolution boundary belongs at sync time (parser writes binding rows with slot-id) and at the public API boundary (execute-by-name translates user’s name-keyed args to slot-id-keyed fa). Everything past sync should be slot-id-keyed.

This matches CLAUDE.md principle: Slots are global identities (one-shot creation, immutable). Bindings overlay them per (fn, slot) pair. Storage realizes this; runtime currently doesn’t.

4. Target architecture

┌─────────────────────────────────────────────────────────────┐
│ AUTHOR INPUT (fns.edn)              — name-keyed            │
│   {:body :html-page-rendered}                                │
│                          │ parser resolves                   │
│                          ▼                                   │
│ STORAGE (DB rows)                   — slot-id-keyed          │
│   binding{fn-id, slot-id, ref-fn-id}                         │
│                          │                                   │
│                          ▼                                   │
│ PUBLIC API ENTRY (execute-by-name, make-single-arg-callable) │
│   {:body val} → translate via walker → {<slot-uuid> val}     │
│   (ambiguity → :execution-error/ambiguous-arg-name)          │
│                          │                                   │
│                          ▼                                   │
│ RUNTIME fa                          — slot-id-keyed          │
│   {<slot-uuid-1> val-1, <slot-uuid-2> val-2, …}              │
│                                                              │
│   arg-builder       → (get fa <slot-uuid>)                  │
│   env-builder       → (assoc fa <slot-uuid> v)              │
│   per-ref translate → {callee-sid → caller-sid} per ref     │
│   HOF closure capture → translation via ref-fn-id linkage   │
│   cache-projection  → set of slot-uuid                      │
│                          │                                   │
│                          ▼                                   │
│ PUBLIC API EXIT / EDITOR DISPLAY    — translate back to names│
│   slot-id-keyed result → name-keyed for human consumption    │
└─────────────────────────────────────────────────────────────┘

5. Why incremental dual-key failed

The dropped refactor/slot-id-runtime-fa tried to write slot-id keys ALONGSIDE name keys, so old name readers and new slot-id readers could coexist during transition. Three problems killed it:

  1. arg-builder’s bnd.slot-id is chain-leaf (base-fn root slot id). Chain-leaf slot-ids are GLOBAL through base-fn — every fn whose root is :get shares the same UUID for :m/:k/:default. Caller’s value written at chain-leaf pollutes all unrelated consumers using that base-fn.

  2. Closest-own-rename slot-id (Phase 0.5 walker) is LOCAL to a fn — caller’s enrichment under that doesn’t match inner reader’s chain-leaf.

  3. Per-ref translation is required to keep each fn’s slot-id namespace separate. Without it, naive shared slot-id space collides.

Conclusion: no incremental dual-key bridging works. The cutover must flip the whole runtime to slot-id-keyed at once.

6. Implementation plan

Multi-week, single branch. Each phase ends green for the full test suite — no half-states merged into popovers/develop.

Phase 1 — Walker (foundation)

Rewrite from scratch (old branch’s walker was tied to incremental approach):

Acceptance: walker tests green; bb test green (no behaviour change yet — walker has no consumer).

Phase 2 — Public API translator

Acceptance: all tests green; public API behavior is observably unchanged in Phase 2.

Phase 3 — Per-ref slot-id translation (mostly obsoleted by the transitive walker)

Original plan: build {callee-slot-id → caller-slot-id} per :ref binding + :seq :ref item, apply at the ref-call boundary; replaces today’s name-keyed build-ref-renames.

Outcome: the transitive walker shipped in Phase 1 (deep-free-ext-entries) emits CHAIN-LEAF slot-ids for every surface free arg, including ones reached by walking into ref-targets. Chain-leaf slot-ids are invariant — when F refs R, F’s walker entries for R-surface use the SAME slot-ids R’s own walker would emit. The Phase 2 boundary translator writes the value under those slot-ids directly. So a per-ref slot-id translation table is structurally always identity-skipped — it has nothing to copy.

This contrasts with the legacy name-keyed build-ref-renames, which IS load-bearing today: names CHANGE through renames, so caller’s fa[:outer] has to be copied to callee’s fa[:inner] at each ref call. Slot-ids don’t change — the walker’s slot-id field for an entry is the consumer’s bnd.slot-id regardless of whose ext-name it surfaces under.

Outcome: no helper, no wiring. The transitive walker (deep-free-ext-entries) makes per-ref slot-id translation always identity-skip — there’s nothing for it to copy. The name-keyed build-ref-renames stays as the load-bearing cross-fn cascade path. An earlier build-ref-slot-renames scaffold + tests was deleted alongside the Phase 5 ext cleanup (see acceptance note below).

Acceptance: no code added.

Phase 4 — Rename-aware slot-id readers (hybrid fa)

Implemented.

First attempt at Phase 4 used the chain-leaf slot-id as the reader’s fa key. That failed because chain-leaf ids are GLOBAL through their base-fn — every fn composed on :assoc shares the same UUID for :value, every :get shares the same :m/:k/:default. The :image test (with two inline-anon :assoc calls each with their own {:as :src} / {:as :alt} renames) and ex-pair-greet cleanly surfaced the bug.

Insight: rename-aware slot-ids already exist in storage. Phase 6c made the parser create a NEW slot row on each renaming fn with :source-slot-id chaining back to the chain-leaf. Each {:as :name} rename is a distinct slot-id. Two inline-anons of the same :assoc with different :as renames each have their OWN rename slot id — distinct, no collision.

The runtime just wasn’t reading at those slot-ids — it was indexing by the chain-leaf. Phase 4 wires in l/effective-reader-slot-id fn-id slot-id lookups — walks the reader’s inheritance chain looking for the closest own-slot whose :source-slot-id chain transitively reaches the chain-leaf. Found → use that rename-slot id; not found → fall back to chain-leaf. Used by:

Hybrid fa: the boundary translator writes slot-id keys for caller args; env-builder, hof-wrap lambda-args, and build-ref-renames slow path continue to write name keys. Readers prefer slot-id and fall back to name. This is the architecture, not a transitional kludge — the two key spaces serve different needs:

Tests verifying the #104 collision class:

What didn’t change at Phase 4 — app/page/fns.edn’s :as :page-body rename stayed because Phase 4 alone couldn’t close the runtime collision; only the boundary translator and rename-aware readers shipped. Phase 5 (HOF wrap-time slot-id translation) + parser disambiguation (commit 38c3fc6e) together let the workaround drop. See § Phase 5 for the runtime-side mechanism.

Acceptance: bb test 1591 / 6334 green; the :image, ex-pair-greet, refinements-test synthetic cases that exposed the collision class all pass.

Phase 5 — HOF wrap-time slot-id translation (conservative)

Implemented.

Diagnosis from the Phase 4 page-body bug attempt: removing the :as :page-body rename surfaces #104 at runtime even though parser-side disambiguation (commit 38c3fc6e) routes the bindings to the right slots at sync time. Mechanism:

  1. Caller invokes :html-page-route with :body [:p "ok"].
  2. Walker for :html-page-route stops at the :handler HOF boundary — translate-named-args writes only the name key, no slot-id matches reach the deeper page-body slot.
  3. Inside :html-page-handler, the :body :html-page-rendered binding is an env-binding (Ring-body rename slot is non-root). env-builder writes fa[:body] = <thunk> BY NAME, overwriting the caller’s [:p "ok"].
  4. The deep seq-item-builder at :_html-body-children-head’s positional :body slot misses its slot-id key and falls back to fa[:body] — hits the env-binding thunk, forces it, renders the HTML page string into the inner <body> tag (escaped).

Three coordinated pieces close it:

  1. hof-wrap accepts a translation table built by r/build-hof-translation. At wrap time (inside the HOF arg-builder), apply-hof-translation copies fa[ext-name] → fa[R-slot-id] for every R-side surface entry whose ext-name is NOT a lambda-param. Wired at BOTH HOF call sites — arg-builder’s :ref :is-fn case AND env-arg-builder’s :ref :is-fn case. This lets caller args past F’s HOF surface (e.g. :body for the deep page-body slot when F’s walker stops at the outer :handler) land under R’s slot-id namespace BEFORE R’s inner env-builder fires.
  2. Lambda-params excluded from translation — lambda values come from the per-call lambda-args merge after the wrap-time fa* capture. Pre-translating them would freeze the wrap-time value across iterations and break collection HOFs (:filter / :map :item).
  3. Thunks skipped at copy (commit dcc11101). An outer non-HOF env-binding writes rt/thunk under fa[name] whose body calls call-with-cache on a ref. Copying such a thunk under R’s slot-id key would let R’s inner reader find it via the Phase 4 slot-id path, force it, and re-enter call-with-cache on the SAME ref-id BEFORE the parent’s cache stores — infinite recursion. The :types-candidates chain hit this (env-bindings :validation / :parsed are deferred thunks; the inner :map / :filter HOFs trip the loop). Caller-supplied free args are plain values and copy as before; env-binding deferred values stay on their existing name-fallback path. The thunk-skip originally surfaced only in bb rebuild’s smoke test; it is now covered by unit regressions in compile_eager_test.clj (apply-hof-translation-skips-thunks-test, apply-hof-translation-mixed-thunk-and-value-test, plus the falsy-presence tests). The full HTTP-graph-chain path (a live request tripping the inner HOFs) remains exercised only by the rebuild smoke.

The page-body fix works because: at the outer :html-page-route level, the :handler env-binding is a HOF that wraps :html-page-handler. The HOF env-builder fires AT WRAP-CONSTRUCTION TIME (when the route is built) and captures fa-ref. When the wrapped callable is later INVOKED (a request hits), apply-hof-translation runs against the captured fa — caller’s :body is still under the name key (:html-page-handler’s own Ring-body env-binding hasn’t fired yet — it lives one level deeper). Translation copies fa[:body] → fa[<page-body-sid>]. Now when the inner :html-page-handler chain runs, its Ring-body env-binding overwrites fa[:body] with a thunk, but the page-body slot’s reader uses its own slot-id — fa[<page-body-sid>] still has caller’s value. Collision dodged.

What DIDN’T flip:

Acceptance: bb test green; app/page/fns.edn :as :page-body removed (page workaround landed alongside).

Phase 6 — Workaround sweep

7. What’s shipped, what’s not

The hybrid runtime fa described in § 4 + Phase 4 / 5 is the FINAL design, not a transitional state. Two key spaces (slot-id + name) serve different needs:

An owned slot whose immediate source has the same name is structural forwarding, not a semantic rename. Its slot ID and source FK remain intact; name-facing walks, public surface aliases and runtime alias generation do not rename the rest of its source chain. A genuine cross-name rename still translates that chain. This prevents a declared input from filling a generic value or coll route belonging to a sibling computation.

Browser plans carry optional lambdaSlots metadata for the same per-call routing. Existing plans without that field retain their previous behavior; regenerate built-in and published plans with the updated compiler when shipping this runtime change.

An alternative was considered (env-builder slot-id-only + cross-fn rename slot-id translation + readers drop name fallback) and dropped. Two attempts failed at runtime; the marginal benefit (closing 2 remaining :on-throw :const workaround sites) didn’t justify the multi-day risk.

8. Out of scope

9. Addendum — inherited renames

Two facts established while closing the lesson-15 :wrap-custom-script descendant-binding bug (a value bound as :body silently vanished):