Three layers, cheapest first. Usage rollups and operational alerts use
counts and durations rather than tenant arguments, results or error
bodies. The org-scoped failure viewer reads sanitised errors from TTL’d
:fn-execution rows. Separately, the opt-in feedback intake stores
user-submitted report text and the built-in alerter can forward clipped
feedback to the configured operator channel (§ 3b and § 4).
Every terminal execution increments a pre-aggregated :usage-stat row
(hour bucket, org, fn, status) with a run count + summed duration
(graphden.crud.fn-execution.stats/bump!). Bounded by distinct keys,
not traffic; 90-day retention. Surfaced three ways: the 7d: N runs
· M failed · avg K ms strip in the editor’s execute-history panel
(:usage-fn-stats); the editor’s Stats sidebar section
(GET /partials/stats — org-scoped headline totals, per-day trend
table, top-fns table; each org sees only its own workspace, except
that a platform-tier principal or a :view-all-stats capability
delegate additionally gets the cross-org by-org rollup —
usage-all-org-stats returns [] for everyone else); org-stats /
org-totals for tooling; and GET /api/orgs/usage.csv?days=30 — the
same by-org rollup as CSV, the operator’s report for invoices and the
founding-orgs register (a tenant context gets the header only). See
EXECUTION.md § Usage rollups.
The interactive tutorial posts three events per lesson to
POST /api/tour/progress — started, one step per advance, and
finished — which bump process counters and therefore ride out on
/metrics and /metrics/prometheus with everything else:
graphden_counters_tour_started_01 # denominator
graphden_counters_tour_step_01_3 # reached step 3
graphden_counters_tour_finished_01 # numerator
A scrape is what makes them a funnel: the counters are process-local and
reset with the JVM, while Prometheus keeps the series across restarts
(increase(...[7d])). Without a scrape they still answer “since this
process started”.
Nothing identifying is recorded — a two-digit lesson id, a step index, one of three words. No account, no session, no per-user path. The route is OPEN, because the landing demo’s anonymous session is precisely the population whose drop-off is worth knowing, and both counter base-fns validate their input rather than trusting it: a caller cannot name a counter (the map is process-global and unbounded, so that would be a memory-growth vector).
The editor’s ✕ failed filter chip (Explorer) focuses on, and the Inspector’s Runs tab lists, the current branch view’s
UNRESOLVED failures (:unresolved-failure-counts → GET /api/failures
for the chip; :recent-failures for the Runs tab’s block), reading the
already-scrubbed :error/:error-data off the audit rows. A failure
clears when the fn ships a new version, when the same version later runs
to success, or when it’s dismissed (✕ Dismiss in the Runs tab →
POST /partials/execute-history/ack; Dismiss all under the chip →
POST /api/failures/ack-all; :failure-ack / :failure-ack-all);
visibility is branch-chain-scoped.
See EXECUTION.md § Error log.
GET /metrics/prometheus exposes OpenMetrics gauges, including the
runtime counters — notably graphden_counters_http_server_error
(uncaught 5xx, incremented at the error boundary) and the registry /
compile structural counters. Point a Prometheus at it and route
Alertmanager to your channel. Example rules (high 5xx rate, target
down) + a compose overlay ship in the cloud deployment repo
(deploy/prometheus/ — see its README); a self-hoster scrapes the same
endpoint. This path owns infra-level alerting (availability, latency,
resource).
/metrics and /metrics/prometheus are auth-required when auth is
active (they disclose JVM version / CPU / live load-average / restart
windows — recon material), so a Prometheus scrape must authenticate:
give the scrape job a bearer via authorization.credentials_file (a
file holding a valid token). When auth is OFF (no AUTH_TOKEN — a local
dev box) they stay open. /health is always public (const-only). The
cloud additionally refuses /metrics at the TLS edge (Caddy), so it is
never reachable publicly there regardless of auth.
For domain conditions Prometheus doesn’t naturally see, graphden ships a
small in-process alerter (:exec/alert-scheduler). Dormant unless a
channel is configured (the Telegram pair or the webhook). When on,
every ALERT_PERIOD_MS (default 5 min) it evaluates:
error-ratio (default 0.5) with ≥ min-runs
(default 10) runs;server-error counter delta ≥
server-error-min (default 20) since the last check;feedback_reports table. No cooldown: every report
is one event, the tick already batches them. This is the intake’s
only notification path — the request handler must stay inside the
cloud’s request-level effect gate (no :network), and the Telegram
token is a secret that lives here, in Clojure config, never in a
graph read.Delivery has two native channels (graphden.monitoring.alerts/alert-request
picks one, Telegram winning if both are set):
GRAPHDEN_ALERT_TELEGRAM_TOKEN + _CHAT and the
alerter POSTs {chat_id, text} straight to the Bot API
sendMessage endpoint. No relay. Create a bot with
@BotFather for the token; get the chat id
by messaging the bot then reading
https://api.telegram.org/bot<TOKEN>/getUpdates (or use a group’s
numeric -100… id). The text is clipped under Telegram’s 4096-character
limit — the operational alerts lead the batch, so only feedback text is
cut (the full reports stay in feedback_reports).GRAPHDEN_ALERT_WEBHOOK and it POSTs
{"text": "…"} (Slack / Mattermost / any JSON {…} sink).A per-key cooldown (default 1 h) means a sustained incident pages
once, not every tick. The decision policy AND the channel selection are
pure (graphden.monitoring.alerts, unit-tested); the scheduler
(graphden.system.init.alerter) only does the reads + the POST.
| env | default | meaning |
|---|---|---|
GRAPHDEN_ALERT_TELEGRAM_TOKEN |
(empty) | Bot API token — with _CHAT, enables native Telegram |
GRAPHDEN_ALERT_TELEGRAM_CHAT |
(empty) | target chat id (user or -100… group) |
GRAPHDEN_ALERT_WEBHOOK |
(empty) | generic {text} webhook (used when no Telegram pair) |
ALERT_PERIOD_MS |
300000 |
evaluation cadence |
The alerter is off only when no channel is configured. Tune the
thresholds via the :exec/alert-scheduler :config map if the defaults
don’t fit your traffic.
The editor carries a Report a problem form (the account menu’s “Report a problem” row). It is how a user on ANY instance — local, self-hosted, cloud — files a bug/idea/question without needing an account anywhere.
The flow deliberately does not depend on the local backend being
healthy: the browser POSTs directly to the intake URL (cross-origin
— the intake answers with Access-Control-Allow-Origin: *; the body
goes as text/plain so the POST stays a CORS simple request, no
preflight). When even that fails (offline), the form degrades to a
“download the report JSON + open a GitHub issue” fallback.
Everything attached is opt-in-by-checkbox and shown before sending:
build hashes + browser/env info, the client-side uncaught-JS-error ring
buffer, optionally the current location (instance URL, branch, open fn)
and, when signed in, the current branch view’s recent failed executions
as fn names + counts (the failed-runs lens’s GET /api/failures read —
no error text). Nothing is sent without an
explicit submit.
Two env vars (see DEPLOYMENT § Environment Variables):
| env | default | meaning |
|---|---|---|
GRAPHDEN_FEEDBACK_URL |
(blank ⇒ official intake) | What GET /api/feedback/config announces to this instance’s editor. The literal off ⇒ the feedback affordance is hidden |
GRAPHDEN_FEEDBACK_INTAKE |
(empty) | Non-blank ⇒ THIS instance is an intake: open POST /api/feedback accepts reports |
The intake itself is pure graph (app.feedback, no new core entity
type): reports land in a plain feedback_reports SQL table on the
intake’s own Postgres (created idempotently on first report); the
built-in alerter (§ 3b) picks the new rows up on its next tick and pings
the operator when a channel is configured. Both routes read their
settings (GRAPHDEN_FEEDBACK_URL, GRAPHDEN_FEEDBACK_INTAKE) through
:deploy-config — the boot snapshot, no :env effect — because on the
cloud they are hit from signed-in and demo sessions, which run under the
request-level effect gate. Abuse posture for an open write route: unarmed
by default, hard size caps per stored field, a honeypot field (filled ⇒
pretend success, store nothing), and fixed-window caps counted off the
table itself (200/hour globally, 20/hour per client IP). Report text is
attacker-controlled input — any triage surface must render it as TEXT.
Triage is dogfooded: on the intake instance, query feedback_reports
from the editor (a :pg-query fn) or psql; the status column
(new → whatever workflow you like) is yours to update.