The decision-fixed reference for distributing graphden as modules; the wider org/tenancy framing is TENANCY_SEAM.md. The business/license framing is open-core AGPL: no app-store commission, and cloud-shared users cannot ship custom Clojure impls.
Nothing here starts from zero. The registry (
:package-version+ publish/list/fetch/install), the round-trip exporter, the tenancy addon (orgs / users / grants / RLS / effect-gate / FaaS), and branch versioning are already shipped. This document records what those give us, the exact gaps, the architectural decisions that close them, and the task plan.
Two distribution channels (data vs code) plus the “swap a core component” case:
| Kind | Contents | Distribution channel | Who installs in cloud | Security boundary |
|---|---|---|---|---|
| 1. fns-only | only fns.edn (compositions over existing impls) |
our registry — EDN over HTTP → graph rows | the tenant themselves (it’s data, not code) | effect-gate (a fns-package can only compose base-fns; what it may do is bounded by :allowed-effects, not by the absence of impl) |
| 2. impl+fns | impls.clj (Clojure primitives) + optional fns.edn |
deps.edn git-dep / Maven coordinate → classpath | nobody — we build it into the cloud image; tenants may only add its fns as a Type-1 reference | the impl is our code, compiled into our executor |
| 3. core-swap | Clojure code replacing a core component (storage / auth / schema / executor / type-system / versioning) | deps.edn git-dep + Integrant addon config (GRAPHDEN_ADDON_CONFIGS) |
n/a — self-hosted / cloud-operator only | it is the security/enforcement layer; cannot itself be a graph-package |
Load-bearing invariant: every fns-package transitively depends on at least one impl-package, because the eldest ancestor of any fn is a base-fn with an impl. That transitive dependency becomes a declared package dependency (see § 4.4).
Verified against code on this branch. File references are anchors for the implementation tasks.
:package-version entity — src/graphden/schema/packages/schema.clj.
Immutable, content-hashed EDN snapshot of a namespace subtree’s fn-defs +
declared dependencies. Columns: name, version, ns-root, fns (jsonb
bundle), dependencies (jsonb), content-hash, published-at, plus the
org-scoped-registry pair org-id / public? (packages spec §5, below). NOT
versioned (immutable by contract — UNIQUE (name, version) at the DB, and
the publish path rejects re-publishing an existing pair before the insert).:package-version is an
org-scoped entity in the tenancy addon (default-scoped-entities): a
tenant’s publish stamps org-id and is invisible to every other org —
browse, resolve and install all read through the scoped storage. The
explicit public? opt-in (the publish popover checkbox / public in the
publish JSON body) makes a row platform-visible while writes stay own-only,
so the publisher keeps provenance and can revoke public visibility. The
flag is normalised at write time: a platform-tier publish
(single-tenant / operator) always lands public? = true, so readers key on
the flag alone, never on the org value. Postgres RLS mirrors both rules
(an extra "public?" IS TRUE SELECT arm on package_version; own-only
writes). Legacy NULL-org rows read as the shared tier. Package NAMES: a
public name is registry-wide, first come first served — another org’s
publish under it (public or private) is refused name-taken; private
names are per org (MARKETPLACE.md § 2, Names).resources/packages/registry/registry/{fns.edn,impls.clj}:
export-namespace, publish-package, list-package-versions,
fetch-package-version, install-package, wired to
POST /api/packages/publish, GET /api/packages,
GET /api/packages/:name/:version, POST /api/packages/install.:export-namespace GRAPH fn-def
(registry/registry/fns.edn) assembles the publishable fns.edn-shaped
bundle by composing export/export-graph (the records↔EDN codec that
stays in src/graphden/packages/export.clj) with
:namespace-external-deps / secret-path stripping / keyword encoding.
This is the reverse of the EDN→rows sync and is the core of both publish
and self-hosted extract; export.clj keeps only the codec + the
external-deps dependency-analysis pass.src/graphden/packages/loader.clj reads packages from the classpath
(io/resource "packages/<name>/package.edn"), topo-sorts by
:dependencies, loads each module’s fns.edn (+ optional impls.clj via
eval).:app/packages init-key
(src/graphden/system/core.clj:204) concatenates :package-names with
:extra-package-names. A package placed on the classpath under
packages/<name>/ and named in either list is loaded — this is the Type-2
hook (§ 5).resources/system-dev.edn → ["core" "storage" "web" "app-base" "app" "registry" "mcp" "examples"].(namespace, name) via
graphden.packages.records.ids/fn-id. This is the linchpin for versioned
materialization (§ 4.2): the same fn name under a different namespace gets a
different, stable UUID with no bookkeeping.OrgScopedStorage decorator (own+public read,
own write, stamps org-id) under VersionedStorage, plus Postgres RLS.
The storage/RLS policy lives in the private graphden-tenancy addon;
core keeps the tenancy/context.clj seam.:read-granted namespaces —
the addon’s tenancy/grant.clj (workspace). Grants are
(subject, capability, namespace); graph capabilities include
#{:read :view-impl :write :execute :admin :bind-args :append-list}.public org — the shared tenant; OrgScopedStorage returns
own-org + public rows, so a public-org package is visible to every tenant
without copying. This is the foundation of reference-install (§ 4.2).record-effect! throws when an effect is outside the
request’s :allowed-effects. The locked anonymous tier excludes
:env, :io, :network, :process, :raw-sql and :cross-org;
registered free and paid tiers add guarded network access, and the
dedicated tier adds persistent processes. See PLANS.md.
The runtime gate — not “no impl” — is the cloud security boundary
(TENANCY_SEAM.md § Effect gate).ExecutionContext, per-branch version rows, merge via
:branch-merge records (no row copy — source versions become visible via
resolution), branch-local? for runtime-config fns. docs/VERSIONING.md.The original gap list below records what the branch set out to close. Those mechanisms and the later moderation surface are now shipped (§ 9).
:dependencies (Task 2) — package.edn accepts
constraints; the loader resolves them (§ 4.4).install-package now installs by
REFERENCE (materialize-once under <ns>@<version> + a :package-install
pin), NOT by copying rows — the “install = grant of visibility” model.:package-install pin entity + update/rollback ref-rewrite (Tasks 3+4).mathx
via git coord (§ 5.1).extension_seam_test).GET /api/export/graph, § 9).
Moderation subsequently shipped through the marketplace (§ 9);
self-hosted↔self-hosted direct exchange needs no moderation queue.Decision. A published package version is materialised once into the
public org under a version-qualified namespace (<ns-root>@<version>,
e.g. web.components@1.3.0). Every tenant sees those rows via
OrgScopedStorage (own+public read). A tenant that “uses” a package fn holds a
reference (ref-fn-id) from their own fn into the public-org fn — they do
not copy the package’s rows into their own graph.
Why. This is the only model that satisfies the stated requirement — “no
copying package data into each project’s DB, just references” — while keeping
per-project independent versions. It reuses the already-shipped org-scoping:
public-org rows are already visible cross-tenant without duplication.
Deterministic fn-id(ns, name) makes each version’s fns distinct and stable
for free.
Rejected — per-tenant copy (today’s install-package). Simple and makes
per-project versions “free” via branches, but duplicates every package’s rows
into every project and is “fork by default.” Kept only as the explicit
copy-on-write fork path (§ 4.5), not as install.
Decision. Changing a project’s pinned package version rewrites only that
project’s own package-referencing bindings (on the current branch) to point
at the new version’s namespace fn-ids — computed with the deterministic
fn-id(<ns-root>@<new-version>, fn-name). Package rows are never touched. The
rewrite runs on a branch, so update/rollback reuse branch-staging and
merge/revert.
Why. The compiler resolves references as concrete UUIDs looked up in
child-callables (compile_eager.clj:238-240,431-436); the compile cache key
(compile_eager.clj:729-741) is graph-shape + base-fns with no per-request
pin dimension. True name-late-binding (variant A) would add a symbolic-ref
edge type, a compiler pre-pass, and a pin dimension to the cache key — a
hot-path change that violates principle #2 (minimal entities) and #7 (locality)
to buy O(1) rollback that branch-revert already provides. Variant B needs no
compiler change and no new edge type: it is a bounded, branch-scoped rewrite
of the user’s own data (which must change semantics on a version switch anyway).
Rejected — variant A (symbolic {:pkg :fn} ref resolved through a per-branch
pin map in a compile-all* pre-pass). Cleaner in the abstract (Unison-style
content-addressed indirection, pin-only version switch), but the seam analysis
shows it touches parse + compile + cache-key + type-check + cycle-constraints.
Recorded here so a future reader does not re-attempt it without weighing the
hot-path cost. If per-fn version coexistence within a single project is ever
needed (two versions of the same package live in one graph), revisit A — B
cannot express that.
Decision. An impl-package is an ordinary Clojure library: its own git repo
with packages/<name>/ resources (package.edn + fns.edn + impls.clj) and
a deps.edn. Self-hosters add its coordinate. A data manifest
executor-packages.edn (list of {:name "name" :lib group/lib :coord …}) is spliced by
build.clj into deps.edn :extra-deps and the loader’s package-name list,
so the operator edits one data file and never touches executor code.
Why. The loader already reads from the classpath and already has the
extra-package-names seam; git-deps is the Clojure-native distribution. This
manifest is distinct from the tenancy addon manifest (GRAPHDEN_ADDON_CONFIGS,
which injects Integrant config — a different concern, § 6). Polylith is a
monorepo-organisation tool, not required for swappability (protocols + Integrant
already provide it); submodules are strictly worse than git-deps.
extend-builder; storage-schema extension is already the answerDecision. Core components are swapped by implementing their protocol
(StorageCRUD / ExecutionGraph / data-schema-protocol / AuthProvider)
and wiring the implementation through Integrant via the addon config manifest.
The user’s specific worry — “a new storage backend must know about every index
and extra table other modules added” — is already solved by
data-schema-protocol + extend-builder: each module contributes its
entities/fields to a shared schema builder (that is how :package-version,
:org, :grant were added), and a storage backend consumes the same builder
output to create its tables/indexes/equivalents. There is therefore a single
storage interface through which modules add fields/tables/indexes. We document
the seams and prove one swap; we do not commit to making every core
component swap-clean now (principle #6 — no unnecessary expressiveness), and we
honestly mark the tightly-coupled trio (executor ↔ storage ↔ editor) as
“documented coupling,” not “clean swap.”
The distributable unit is the immutable :package-version bundle (§ 2.1). It is
the same artifact whether it lives in our registry (cloud) or is written to an
.edn file (self-hosted). Content-addressed by content-hash over the fn
bundle. This is the “download as EDN” format the user asked about — a single
self-describing EDN map, not a directory of files (nested namespaces are carried
as :namespace on each fn-def, reconstructed on install).
POST /api/packages/publish {name, version, ns-root}
→ export-namespace serialises the live subtree → publish-package hashes +
inserts the :package-version row. Already shipped.export-namespace returns the
bundle; the user writes it to their git repo as fns.edn. Registry optional.publish-package diffs the bundle against the newest published version
BELOW the candidate (graphden.packages.compat/breaking-changes — removed
public fn-defs, removed / narrowed / newly-required args, dropped bindings,
changed renames or parents, widened returns, a grown or dropped :expects-effects contract,
reshaped type-rows; _-private fn-defs never count) and diffs the two
bundles’ :package-dependencies (incompatible-dependency-bumps — a
dependency that left its previous caret range is :dependency-incompatible,
the one upstream change the fn-def diff cannot see; a platform upgrade
stays invisible to it). A break is refused with
{:ok false :reason "breaking-change" :previous "1.1.0" :changes [{:kind :arg-removed
:fn :shape :arg :nick :old :text} …]} unless the candidate leaves the previous
version’s caret range (^1.1.0 → a major bump; below 1.0 a minor bump) — the
range a ^ / ~> consumer auto-advances into. Additions, optional slots,
wider args and narrower returns are compatible. The guard is graph
(:_pub-breaking? in registry/registry/fns.edn) over two pure base-fns,
:breaking-changes-between, :incompatible-dependency-bumps and
:semver-compatible?. The editor’s panel publish shows the same
outcome — the notice is rendered from the envelope (Published …,
already exists, breaks 1.1.0: fn-removed wave), not from the form.public org under <ns-root>@<version>, syncing its fn-defs with
sync-fns-to-storage! against the public-org, versioned-ns namespace. Rows
live once, globally. The unqualified <ns-root> may carry a “latest” alias
(a thin convenience, not required for correctness).POST /api/packages/install {name, version} (branch-scoped, as
today) becomes:
:package-install pin row (branch-id, package-name, version);:read on <ns-root>@<version> so the package enters the workspace.
No fn rows are copied into the project.ref-fn-id into the public-org fn — a reference, not a copy.POST /api/packages/update with {name, version}:
materialise the new version, repoint the pin,
rewrite the branch’s own bindings that reference the old version’s ns to
the new version’s fn-ids (AD-2), re-grant. Do it on a test branch, verify,
merge to main.↑ <latest> available badge on the row, and the ↑ input prefilled with it,
so the update is one click. No periodic job and nothing stored: the panel
is the check.package.edn :dependencies
accepts version constraints — {"core" ">=1.5.0", "web" "~>2.1"} alongside
legacy bare names (bare ≡ any). The loader validates the classpath version
against every constraint at boot (graphden.packages.semver +
validate-dep-constraints!), throwing :packages/version-conflict early.^ / ~> therefore never auto-advance into a
break the registry could see.install / fork /
materialize accept an exact version, a semver constraint (>=1.1, ~>1.2,
*), or latest; resolve-version picks the highest published match and
pins/materialises that concrete version. Unsatisfiable → not-found.missing-dependencies before any write.install auto-resolves a package’s
package dependencies. Publish now records them: export-namespace reverse-
maps each external dep fn’s namespace — the versioned ones (<ns-root>@<v>,
materialised by another installed package) → the published
:package-version (name+version) that owns them — into a new
:package-dependencies field on the row ({:name :version} list;
platform-only packages get []). install-package then pulls those
packages FIRST, depth-first with a [name version] cycle guard — since the
graph decomposition it is a :fix worklist loop of graph fn-defs (the
:_inst-* chain in registry/registry/fns.edn) over the base-fn primitives
(:resolve-package-version / :missing-package-dependencies /
:package-version-materialized? / :materialize-package-fns /
:package-upsert-pin) — so a package’s cross-package refs
resolve without manual ordering. Best-effort on ns-root sharing: if several
packages publish the same (ns-root, version), the first registry match is
recorded (they materialise the same rows). The name-based
missing-dependencies precondition still guards genuinely-absent platform
fns after the package pull.Intentional modification of a public package = copy its subtree into the user’s
own namespace (this is where today’s copy-install-package logic is retained,
renamed to a fork operation). Only then do rows get duplicated, and only into
the forker’s own org — a deliberate act, not the default.
Refused on a package-owned namespace. The copy syncs at the ORIGINAL
namespace on the deterministic (namespace, name) ids — the same ids the boot
loader writes. When that namespace is a package synced from disk on the
serving instance, those ids are platform-owned (graphden.packages.owned):
the fork would land on rows crud.package-guard keeps API-read-only, the
“copies” would not be editable, and the next boot’s sync would overwrite them.
:fork-package therefore refuses the whole fork with
{:ok false :reason "package-owned" :owned [<names>]} (the panel renders the
names + where the fix belongs) — a partial copy of the un-owned remainder is
not a usable package. Same predicate the MCP upsert-fn-defs guard consults
(:platform-owned-def-names, core/system). Editing a built-in package is a
fns.edn change; fork serves packages that arrived through the registry
(another org’s public version, a mirror from another graphden), whose
namespace is not built into the instance.
packages/<name>/
resources on the classpath. No registry row — distribution is the Clojure
dependency graph, versioned by git tag / Maven version.resources/executor-packages.edn manifest — the operator’s single data
file. Each entry {:name "telegram" :lib com.acme/graphden-telegram :coord {…}}
(:coord is a deps.edn git-dep or :mvn/version). Read by
graphden.packages.manifest; two consumers so the operator edits ONE file:
build.clj merges extra-deps into the uberjar basis → the external
package’s resources are bundled;:app/packages appends package-names to the loaded list → the loader
picks them up at runtime.:local/root override, § 15) to deps.edn
/ a gitignored deps.local.edn. The manifest drives the build + load;
the dev CLI classpath is driven by deps.edn as usual.mathx, Track B): the standalone
repo graphden/graphden-mathx
(@8337147) is a real out-of-tree Type-2 package — a :gcd base-fn
(ops/impls.clj defbase + the impls var linking it) plus a
:gcd-with-12 composed fn-def (ops/fns.edn). It is pulled in by a git
coord, not a local path:
executor-packages.edn + deps.edn root :deps list
{:git/url "https://github.com/Graphden/graphden-mathx.git" :git/sha "8337147…"}.
bb rebuild clones the package straight from GitHub into the uberjar
(build.clj merges the manifest coord into the basis). The running instance
loads it (base-fn count 249 → 250, mathx.ops synced) and
POST /api/execute of :gcd-with-12 {b 18} returns 6.:test + :dev aliases carry
:override-deps {mathx/mathx {:local/root "external-packages/mathx"}}, so
bb test / bb dev resolve the identical in-tree copy — no repo access,
no ssh-agent, offline. external-packages/mathx/ is kept in-tree as the
override source (and matches the pushed repo at @8337147 — a byte-level
match pkg-copy-sync enforces in bb ci).bb rebuild / bb check / any non-:test/:dev invocation
resolves it anonymously, no credentials or ssh-agent. (If you fork it into a
private repo, add the build host key as a read-only deploy key.)manifest-test/external-mathx-package-loads-impl-and-fn-def (loader
layer, offline via override).defbase gotcha proven here: the macro rewrites every occurrence of an
arg symbol into a resolve-arg call, so an impl must not shadow its arg
names in a loop/let (mathx’s gcd loops on x/y, not a/b).bb rebuild
loads the new package’s base-fns + fn-defs. Update = bump the git-sha/version
in the manifest.Permissive: an impl-package may depend on other packages (impl or fns) — the loader already topo-sorts. We do not forbid mixed impl/fns packages up front (per the stated preference); the minimalism guidance (one impl OR one composition concern per package) stays a recommendation in PACKAGES.md, not a hard rule. Revisit only if a concrete problem appears.
StorageCRUD + ExecutionGraph; wire via
Integrant. VersionedStorage / OrgScopedStorage prove the decorator stack.data-schema-protocol + extend-builder: a module
adds entities/fields/indexes to the shared builder; the storage backend
materialises them. This is the “single storage interface that lets modules
add tables/fields/indexes” the design question asked for. A new backend
(e.g. a non-Postgres store) consumes the same builder and provides its own
equivalents (indexes → its indexing primitive, etc.).graphden.auth.provider/AuthProvider protocol + :auth/provider
Integrant key (default SingleTokenAuthProvider; tenancy addon overrides).GRAPHDEN_ADDON_CONFIGS” pattern.extend-builder keeps storage swap-clean despite this;
the editor coupling is accepted, not abstracted (principle #6).The honest answer to the original design question (“can we replace storage / executor / types / versioning / permissions?”). “Proven” = shipped code + a test exercises the seam.
| Component | Seam | Injected via | Status | Proof |
|---|---|---|---|---|
| Storage decorator | wrap the :app/storage chain |
Integrant key + addon config | proven | VersionedStorage, OrgScopedStorage (tenancy addon tests) |
| Storage backend | StorageCRUD + ExecutionGraph + Constraints |
override the backend init-key | partial | protocol exists; only Postgres implements the recursive-CTE ExecutionGraph. Swapping to a non-Postgres store = real work (see § 14) |
| Schema extension | data-schema-protocol extend-builder |
chain an extend-builder fn |
proven | extension-seam-test (arbitrary third-party entity accepted) + :package-install storage round-trip (registry-test) |
| Auth | AuthProvider protocol |
:auth/provider Integrant key |
proven | SingleTokenAuthProvider default; tenancy TokenAuthProvider override |
| Tenancy / org isolation | optional addon | GRAPHDEN_ADDON_CONFIGS |
proven | the whole tenancy addon (orgs / RLS / effect-gate) |
| Executor core | ExecutionGraph result contract |
— | documented, unproven | protocol-bounded; nobody has replaced it |
| Type system | — | — | coupled | reaches into the editor (rich-types, overlays) — a documented coupling, not a clean seam |
| Versioning decorator | it’s a storage decorator | omit VersionedStorage from the chain |
documented, unproven | mechanically a decorator, but the branch-router assumes it — untested to remove |
“Prove one” (fresh): test/graphden/schema/extension_seam_test.clj — a
deliberately third-party extend-builder adds a novel :widget entity chained
after the core graph schema; the built schema carries it with its declared
fields AND still carries the core :fn. Combined with the :package-install
storage round-trip (a builder-added entity that a real Postgres backend
materialises + CRUDs), this proves the “a new module adds its own tables
without touching the storage backend” claim end-to-end. We deliberately do NOT
attempt to make every core component swap-clean (principle #6) — the matrix
above says exactly which are proven vs documented vs coupled.
| Entity | Change | Versioned? | Rationale |
|---|---|---|---|
:package-version |
org-id + public? (nullable, packages spec §5) |
no | immutable artifact (shipped); org-RLS is a sanctioned column case (fn-metadata rule) — the registry row must be filterable before any graph materialisation exists |
:package-install |
new — (id, branch-id, package-name, version, org-id); UNIQUE (branch-id, package-name) |
no | desired-state pin, same class as :service; per-branch so it lands in the branch’s compile context with no new cache dimension |
package.edn :dependencies |
value may be {"name" "constraint"} |
— | version constraints (Task 2); bare string still accepted |
:package-install is not versioned (like :service, :fn-execution,
:package-version) — it is runtime desired-state, not graph semantics. It
carries org-id and is a tenant-forbidden-adjacent decision: a tenant may
read/write their own org’s pins (to install/update packages in their project)
but not others’ — so it is org-scoped, not in tenant-forbidden-entities.
No change to :binding / :binding-list-item: references stay ref-fn-id
UUIDs (AD-2). No symbolic-ref column.
API (extends the shipped /api/packages/*):
POST /api/packages/publish — unchanged (+ optional materialise-to-public step).GET /api/packages — registry index (+ latest-version surfacing).POST /api/packages/install — now writes a pin + grant (reference model).POST /api/packages/update — change pinned version and rewrite own refs (update/rollback).POST /api/packages/fork — copy-on-write into the user’s namespace (§ 4.5).GET /api/packages/installed — this branch’s pins (for the editor).GET /api/packages/:name/:version — full bundle (export / self-hosted pull) — shipped.DELETE /api/packages/withdraw?name=&version= — withdraw a published
version — shipped. A release is immutable, not eternal: publishing the wrong
namespace or the wrong number used to be permanent. Refused with 409
still-installed while any branch pins the package at that version (a
pin resolves through its own version row — a branch that moved on to 1.0.1
does not keep 1.0.0 alive), 404 no-such-version when the pair is
unknown. The
materialised <ns-root>@<version> copy is NOT removed — it is ordinary graph
content by then, deleted like any other namespace.Marketplace (shipped 2026-09, MARKETPLACE.md): discovery
over this registry — cards with description / category / tags, ratings and
reviews, install counts, every version; the same :package-version row
also carries editor THEMES and KEYBOARD LAYOUTS (:kind), applied per user.
The chip below keeps the quick “what is pinned here” view and links to the
surface.
Editor (shipped): a Build-surface “packages” context-bar chip
(#gd-pkg-chip, sibling of the workspace/branch chips) opens a browse/install
browser — install is a build act, so it lives with the project context, not on
the Organization admin pane. The browser lists the current branch’s installed
pins with their version, a per-row update/rollback version input (↑) +
uninstall (×), and the registry as a nested <details> browse (Install / Fork
per version) — the browse <details> also carries a remote-install form
(source URL / name / version) posting to the same panel-install route with
source, so pulling a package from another graphden’s registry (e.g.
graphden.dev → a self-hosted install, § 13) is one form away; the remote
bearer stays the server’s GRAPHDEN_REGISTRY_TOKEN, never a browser value
(and rides only dials to GRAPHDEN_REGISTRY_URL’s origin).
Server-rendered via GET /partials/packages-panel; the chip +
popover lifecycle lives in editor-shell-chips.js (gdRevealPkgChip / gdOpenPkgPop,
gated on the window.API registry probe). Publish is NOT on the chip — it is
a namespace authoring action (packages spec §3); its <details> is excluded from
the panel root and the namespace-scoped affordance is the ⬆ on a namespace row
(MARKETPLACE.md); the /api/packages/panel-publish POST
route stays live for API callers. Each installed row also carries an
↑ <latest> available badge when its pin is behind the registry, with the
version input prefilled to that version (§ 4.3).
status field on :package-version
(pending → approved / rejected), on by GRAPHDEN_MARKETPLACE_MODERATION;
the operator’s queue is Platform → Moderation. See
MARKETPLACE.md § 8.
Self-hosted → self-hosted sharing needs none (direct git/EDN exchange).GET /api/export/graph (auth-required) returns the WHOLE graph (current
branch/scope) as an EDN migration bundle {:fns [fn-def …] :namespaces
[dotted …]} — the “download my whole project” artifact. It reuses
export/export-graph (the same records→fn-defs machinery as
export-namespace, just unscoped) via the :export-graph base-fn +
:edn-ok-response. EDN, not JSON, so fn-def keywords (:parent :add,
refinement chips, …) round-trip faithfully. Import is idempotent-additive:
because fn-ids are deterministic from (namespace, name), syncing the bundle
onto a booted self-hosted install re-writes the platform fn-defs as no-ops and
only adds the caller’s own fns — so no platform-namespace filtering is needed.
Refs that can’t be spelled as readable keywords - a duplicated name
in a version-materialized @-ns or the ROOT (nil) ns - ride the
bundle as #graphden/ref "lib@1-2-0.sub/name" / #graphden/ref
"/name" tagged literals (records.wire); the package loader and
the remote-bundle reader decode them back to the same qualified
keywords. Tests: export-test/export-graph-bundle-shape,
export-test/{roundtrip-unspellable-ns-duplicates,wire-edn-text-roundtrip},
registry-test/export-graph-base-fn-and-handler.
:override-kind :secret-path bindings) are stripped by default and
manifested in :secrets + :secret-paths-included? on the bundle — the
publisher sees a stripped-secrets notice, the installer gets the manifest
back as :needs-definition in the install envelope (persisted on the
:package-version row’s nullable :secrets column) plus a Packages-panel
notice. ?include-secret-paths=true on GET /api/export/graph opts in for
org-internal migration. Full rationale + round-trip form:
SECRETS.md § Sharing / export policy.
Tests: export-test/{roundtrip-secret-path,strip-secret-paths-policy},
registry-test/publish-carries-secrets-manifest-install-reports-needs-definition.Historical implementation sequence, retained for its dependencies and checkpoint rationale. Current contributor commands follow dev/wtq/AGENT.md.
:package-install pin
(Task 3) — schema + materialise step + pin CRUD; install writes a pin+grant
instead of copying.executor-packages.edn +
build.clj splice + docs.v0.1.0 is the consumable
coordinate a downstream deps.edn pins (:git/tag "v0.1.0" :git/sha …) —
graphden-cloud does exactly this. The editor/app is already an opt-in
package chosen via :package-names (drop "app" for headless — proven by
headless_boot_test). Does not split the monorepo — it emits the
artifact; app/editor stays as an optional package. Remaining polish
(credential-gated): a Clojars com.graphden/graphden-core jar via
b/jar + deps-deploy — a versioned Maven coordinate that ships built
classes (not the whole repo). Not needed to consume graphden today.GET /api/export/graph) and the later marketplace moderation queue (§ 9).Agent checkpoints use bb wt up for an isolated live instance and
bb wt test for focused tests; the serialized landing gate owns shared
rebuilds. See dev/wtq/AGENT.md.
fork; migrate demo seeders to the pin model on a clean
DB (declarative-sync-needs-clean-baseline rule).sync-fns-to-storage! uses.:allowed-effects) is what actually protects
tenants. Any new install path must run under the tenant’s gated request scope.Two consumption models. A works today (graphden is an application you run). B is the target for anyone customising the core — it is the only fork-free way to swap storage / auth / core-impls, and it is how our own cloud is assembled (§ 16). Both are supported; they differ in who owns the build.
.env as described in DEPLOYMENT.md.bb rebuild builds the jar/image and starts the bundled Postgres stack.
A clean clone has no prebuilt executor image for docker compose up alone.http://localhost:9002 (container port 8080).For an external database or direct jar, follow DEPLOYMENT.md and set
JDBC_URL, DB_USERNAME and DB_PASSWORD. Contributor agents use their
claimed worktree’s bb wt up and its printed isolated URL instead.
Customising here means editing the checkout / building a custom image — a mild fork with a rebase burden. Fine for a one-off; for a product, use Track B.
The developer creates a thin project that depends on graphden as a git-dep
plus whatever they swap in. No fork, and — for git-deps — no published
artifact needed: the Clojure CLI clones the repo at a sha and puts its
:paths (src + resources) on the classpath. The dependency key is an
arbitrary local identifier (it need not match any published coordinate).
mkdir my-graphden && cd my-graphdendeps.edn — graphden (git-dep) + your addon(s):
{:paths ["src" "resources"]
:deps {com.graphden/graphden {:git/url "https://github.com/you/graphden"
:git/sha "<sha>"} ; graphden itself
acme/graphden-sqlite {:git/url "…" :git/sha "…"}}} ; your backend
During active dev of the addon, override to {:local/root "../graphden-sqlite"}
in a gitignored deps.local.edn (§ 15) — edit-in-place, no push/pull loop.
resources/my-config.edn — an Aero fragment overriding the seam + choosing
packages. Omit "app" to run headless (no editor / default web-server —
its resources ride the classpath but are not loaded):
{:app/storage {:base #ig/ref :sqlite/backend}
:sqlite/backend {:db-path "/data/graphden.db" :schema #ig/ref :db/schema}
:graphden/require [acme.graphden.sqlite]
:app/packages {:package-names ["core" "storage" "web"]}} ; drop "app" for headless
Run — reuse graphden’s -main:
GRAPHDEN_ADDON_CONFIGS=my-config.edn clojure -M -m graphden.executor-runtime.core
(or your own three-line -main calling graphden.system/start! for full control.)
:pg-storage references.
Replace or omit those in the consumer’s selected component graph.Headless boot (["core" "storage" "web"], no app) is verified end-to-end by
test/graphden/system/headless_boot_test.clj — the packages load + sync and the
executor evaluates a fn with the editor absent.
Caveat (git-deps pulls the whole repo). A :git/url on this repo brings the
editor’s resources/packages/app/ along; omitting "app" from :package-names
leaves them present-but-unloaded. A cleaner core-only dep (:deps/root on a
core/ subdir) or a Clojars library-jar (a com.graphden/graphden-core
Maven coordinate via b/jar + deps-deploy) is a later polish — needed only
for a versioned public artifact, not to consume graphden today.
Live example of the reverse direction (an external package pulled INTO a
graphden build). graphden/graphden-mathx
is exactly a Track-B-style thin repo, consumed the other way round: this repo
lists it by git coord in deps.edn + executor-packages.edn, and bb rebuild
clones it into the uberjar. See § 5.1 “Proven end-to-end via a real git coord”
for the full wiring (git coord for build/prod, :override-deps local for
hermetic bb test/bb dev, and the private-repo ssh-agent access note).
| Type | Channel | How self-hosted installs | Rebuild? |
|---|---|---|---|
| 1 fns | data | boot-bundle under resources/packages/, or POST /api/packages/install (from THIS install’s registry) |
no — runtime, on a branch, then merge |
| 2 impl+fns | code (git-dep) | add coordinate to executor-packages.edn → bb rebuild |
yes |
| 3 core-swap | code + Integrant | add dep + config fragment in GRAPHDEN_ADDON_CONFIGS → restart |
yes |
Type-1 sources: (a) bundled in the build (resources/packages/<name>/,
loaded at boot like core/web/app); (b) published into THIS install’s registry
(POST /api/packages/publish) and installed from it (POST /api/packages/install
takes {name, version} and reads the local :package-version row —
fetch-package-version is the server side of GET /api/packages/:name/:version,
not a remote client). All sync onto a branch → test → merge; update = install
newer version + merge; rollback = revert / re-install older.
Runtime bundle import: POST /api/import/graph?target=<branch> accepts
an exported .edn bundle (the GET /api/export/graph shape, or a bare
fn-def vector) and applies it to a NAMED branch — ?create=true forks the
branch (caller stamped owner, owner write-policy), ?prune=true gives
snapshot semantics (branch tombstones for fns the bundle no longer contains;
referenced ones are kept + reported), and package-owned defs are skipped +
reported rather than rewritten. ?target=, not ?branch= — the latter is
the branch-router’s request-scope selector. The exporter’s format IS the
import format, so cloud→self-host (and back) is one download + one POST,
then the normal diff → review → merge flow.
Remote-registry pull: POST /api/packages/install with
{name, version, source} mirrors the CONCRETE version from the remote
registry at source (its GET /api/packages/:name/:version?format=edn
face — the EDN wire keeps fn-def keywords intact; the bearer for the
remote comes from GRAPHDEN_REGISTRY_TOKEN and is sent ONLY when source
has the origin of the configured GRAPHDEN_REGISTRY_URL — source is
caller-chosen, so any other host is dialed without it), stores it as a LOCAL
:package-version row (never re-marked public) under the REQUESTED
(name, version) — a remote serving a row that names another pair is
refused as remote-bundle-mismatch, and the store runs under the same
package-name lock and public-name holder check as a publish (name-taken),
since install needs no :publish-packages and the key is registry-wide
(docs/MARKETPLACE.md § 2) — and then the normal
install worklist materializes + pins it. Missing dependencies mirror the
same way, one worklist step at a time. A non-concrete spec (latest or a semver range) is resolved against
the remote’s version list first, then the concrete version is mirrored. So
“self-host pulls a public package from the cloud” is: a free cloud
account’s token in GRAPHDEN_REGISTRY_TOKEN, GRAPHDEN_REGISTRY_URL=https://graphden.dev
and one install call with source: https://graphden.dev. Every remote dial
(index, version, marketplace card — and the hub’s push/pull) is one
:http-request fn-def, so in a restricted (tenant) execution it is
egress-checked at every hop, rate-capped and byte-capped like any other
outbound call.
Ordering rule: a fns-package transitively depends on an impl-package (§ 1). So install impl-dependencies first (Type-2, rebuild), then the fns-package (Type-1, runtime). Missing deps are rejected before any write.
Two different mechanisms, because storage and core-impls live at different layers. Concrete steps in § 12 (Track B) and § 5 (Type-2); the decision matrix:
| Motivation | Mechanism |
|---|---|
| different DB / storage engine | Integrant seam :app/storage — protocol impl + :db/schema builder (§ 12 Track B, § 6.1) |
| a base-fn impl needs replacement, interface is fine | trusted consumer bootstrap wiring; the :overrides package syntax below is a proposal, not supported loader behavior |
| rewrite a whole graph subsystem | replace a module — your fns in the same namespaces (same deterministic fn-id) |
| fork graphden wholesale | edit the monorepo checkout (AGPL-native) |
The :overrides syntax below is a design sketch. The loader does not
interpret this key, and package sync rejects duplicate base-fn names with
:packages/base-fn-name-collision (or a duplicate identity pair). Adding a
second :add declaration therefore does not install an override. A
consumer replacing trusted implementations needs explicit bootstrap wiring;
ordinary package dependencies alone do not provide this policy. Proposed syntax:
;; packages/my-core/arithmetic/impls.clj
(defbase add [nums] (my-faster-sum nums)) ; same contract, better impl
(def impls {:add add})
;; packages/my-core/package.edn
{:name "my-core" :dependencies ["core"] :overrides ["core"] :modules ["arithmetic"]}
The intended design would retain upstream’s fns.edn graph structure
while replacing its Clojure closure; that package policy remains unimplemented.
The default one-argument system.interface/start! initializes the whole
merged Integrant config. Overriding :app/storage alone leaves
:db/postgres, the PostgreSQL notification/lock components and
:exec/context’s :pg-storage reference configured. A non-Postgres
consumer must replace or omit that wiring, or explicitly start a selected
component subset with compatible dependencies. Implementing ExecutionGraph
and the needed invalidation/locking contracts remains backend work; the
headless package test does not prove a non-Postgres deployment.
We do NOT split the monorepo. “Distributable packages” means giving
third parties a clean boundary to publish into and giving self-hosters a way
to add packages — not breaking up our first-party code. First-party packages
(core, web, storage, app/editor) stay in the monorepo and ship as the
build artifact (Task 7 publishes it; the monorepo can emit multiple artifacts).
examples package.:local/root
override in a gitignored deps.local.edn — edit the package’s working copy in
place, bb rebuild, no push/pull/SHA-bump. The SHA bump in the manifest is
the release step, not the dev loop. This replaces git submodules (rejected).bb rebuild
(unchanged from today, the common case). External package you own → :local/root
override, fix locally, then push + bump. Third-party you don’t own → fork +
:local/root, PR upstream. Only the release has multi-repo friction.:local/root +
the manifest solve the multi-repo-pain the design question raised. Revisit
only if internal component sprawl becomes a real problem.The rule above, made concrete. A separate repo is paid ONLY where independence is real; modularity itself comes from packages + protocols/Integrant, in-tree.
| Repo | Kind | How it relates to the monorepo | Access |
|---|---|---|---|
graphden/graphden (this) |
monorepo | core / web / storage / app-base / app-editor / registry / mcp — all co-evolving first-party, in ONE tree. Emits the uberjar; can emit a graphden-core artifact (Task 7). |
public (AGPL-3.0) |
graphden/graphden-mathx |
external Type-2 pkg | pulled IN by git coord (deps.edn + executor-packages.edn); in-tree copy at external-packages/mathx for offline test (§ 5.1). |
public |
graphden/graphden-examples |
extracted dev pkg | the pedagogical examples package moved OUT; in-tree at external-packages/examples, on the classpath only via the :dev/:test :extra-paths (never prod). |
public |
graphden/graphden-tenancy |
private addon | the multi-tenant POLICY (grants/RLS/quotas/users + the tenancy-admin package), extracted 2026-08-03; pulled into graphden-cloud as a git-dep. Core keeps only the SPI seam (tenancy/context.clj, TENANCY_SEAM.md). |
private |
graphden/graphden-cloud |
thin consumer | depends on graphden and graphden-tenancy as git-deps, turns the tenancy addon on, adds cloud modules (usage-metering …). NOT a fork (§ 16). |
private |
| future private cloud modules | closed addons | attach to graphden-cloud via GRAPHDEN_ADDON_CONFIGS (billing / metering sinks / at-scale routing). |
proprietary |
What’s NOT extracted, on purpose: the Postgres storage impl (a swap seam exists — § 6.3 — but the default first-party backend co-evolves with schema / versioning / executor, so it stays in-tree); a non-Postgres backend is external work in the consumer’s addon, not a monorepo split.
In-tree first-party PACKAGE split (still one repo). The app monolith was
decomposed into cohesive packages without a repo split — modularity from
packages, not repos:
app-base — the reitit route-building vocabulary (:route /
:route-with-middleware, the per-method + auth route templates, app.common
helpers). Deps core+web, NO app dep, so the tenancy addon AND registry
reach the templates without dragging in the editor. tenancy-admin now depends
on app-base, not app.registry (in-graph publish/install/fork/export) and mcp (the /mcp
JSON-RPC AI endpoint) are OPTIONAL top-level packages: drop either from
:package-names and the app still boots (the editor hides its Packages panel
via a window.API probe; the endpoints 404). app’s router no longer
references their routes.:namespace string
(app.common / app.registry / app.mcp) — identity is uuid-v5(namespace,
name), so only the package DIRECTORY moved; ids + refs are byte-identical.:request, which cannot serve app HTTP routes; it’s tenancy-only). Each
optional package carries a :_registry-ring-response / :_mcp-ring-response
handler that graphden.system.branch-router resolves TOLERANTLY and serves
PER-BRANCH (fresh + invalidation-aware) alongside the main handler.Also deliberately kept in the app package (NOT moved to examples, despite
looking like demo content): app/contact-demo — the /demo/contact page.
It is the canonical NON-TRIVIAL app-namespace FIXTURE the distribution machinery
is tested against end-to-end: packages/export_test + packages/registry_test
export / publish / install / fork app.contact-demo through the shared
core+web+app golden; the edit-packages-panel e2e publishes it from the
browser; and tools/browser-test/contact-demo-smoke.js is the live-route smoke
for the components + submit-form dispatcher pipeline. It ships as a small demo
AND doubles as that cross-pyramid fixture — moving it to examples would fork
the fixture from the golden those tests build and drag the whole (heavy)
examples package into their bootstrap. So the demo route is intentional and the
smoke depends on it; it stays.
Our cloud is not a fork — it is the same self-hosted core + our addons +
a restriction policy (“cloud vs self-hosted = mode, not code, one binary” —
TENANCY_SEAM.md). Internally we operate the cloud as self-hosted admins
(platform-admin behind a VPN). Our cloud build is literally the best dogfood of
Track B (§ 12): a thin project depending on graphden-core + the addons.
The core mechanisms, tenancy policy and cloud-business modules have different ownership:
| Layer | What | Where | Openness |
|---|---|---|---|
| Open mechanisms + identity | org context, auth-provider/effect-gate/storage seams, opt-in accounts | this repo | AGPL |
| Tenancy policy | orgs / grants / RLS / plans / tenant FaaS routing | graphden-tenancy | separate proprietary addon; core remains single-org without it |
| Private cloud modules | billing / metering / at-scale routing / admin dashboards / managed-model gateway | separate closed repos | proprietary; we don’t publish them |
Private cloud modules attach via the same mechanism as any addon
(GRAPHDEN_ADDON_CONFIGS + :graphden/require + Integrant-seam overrides) —
there is no special “cloud” machinery; they are ordinary addons we keep private.
After this plan the assembly model is fully real: consumable core (Task 7) +
tenancy addon (shipped) + swap seams (Tasks 5–6) → cloud = core + tenancy +
[private modules] + policy. What remains beyond this plan is writing the
private modules (billing etc.); they plug into the socket unchanged.
The org-id columns exist in core (NULL means public/unowned); the
addon supplies ownership stamping and isolation policy. Accounts can be
enabled independently on a single-org self-host. This split is documented
in TENANCY_SEAM.md; consuming the same core does not
imply that the proprietary tenancy addon is part of the AGPL repository.