Introduction
This document is the plan for Phase 7 of the Arcology Hypermedia Publishing Platform: metrics and monitoring for the Arcology Web Server. Today the server is observable only through println access lines (logRequest / logError) and a static /health responder that cannot fail. Phase 7 adds:
a Prometheus
GET /metricsendpoint (Micrometer +PrometheusMeterRegistry),per-request counters and timers with
site,path,status,ua_class, andreferer_typelabels,structured JSON access logging on the existing SLF4J backend,
a real health check (database, cache dir, org dir) with a 503 on degradation,
a Grafana dashboard + alert rule set (anti-viral, render p90, health).
Stack choice: Micrometer via Ktor's built-in MicrometerMetrics plugin for the standard HTTP timer + JVM binders, plus a small ArcologyMetrics facade registering Arcology's own counters/timers on the same registry. A hand-rolled counter plugin was considered and rejected — it would re-implement percentile histograms and exposition glue for no benefit.
Design Decisions
Locked in with the developer:
Micrometer + Prometheus registry, not a bespoke metrics implementation. The Ktor plugin (
ktor-server-metrics-micrometer, ktor 3.2.3) provides the built-inktor.http.server.requeststimer;micrometer-core/micrometer-registry-prometheusat 1.17.0 provide the registry, histogram buckets, and JVM binders.Referer classification ships as a
referer_typelabel on the request counter (direct internal search social external).Per-path label granularity, not route-pattern collapse. The URL space is the site's own bounded content (the
published_routestable), not unbounded{uuid}routes, so real paths are defensible. The mandatory mitigation is that unmatched requests — 404 probes, scanner noise — must not mint series; they collapse tounknown. See Cardinality Notes.Structured JSON logging:
@Serializablelog events serialized withkotlinx-serializationand emitted through the already-presentslf4j-simplebackend under thearcology.accesslogger name. No logback/logstash dependency until ops needs outgrow stderr./metricsis unauthenticated; it is blocked from the public internet at the nginx layer.
Metric Catalog
| Metric | Type | Labels | Source | |
|---|---|---|---|---|
arcology_requests_total | Counter | site, path, status, ua_class, referer_type | request instrumentation point | |
arcology_request_duration_seconds | Timer | site, path | servePath / servePathLocalhost | |
arcology_node_render_duration_seconds | Timer | (none) | cache-miss renderNodeBody | |
arcology_cache_hits_total | Counter | cache = page \ | sitemap | HtmlCache / SitemapGraphBuilder |
arcology_cache_misses_total | Counter | cache = page \ | sitemap | same |
ktor.http.server.requests | Timer | route, method, status, throwable | Ktor MicrometerMetrics plugin | |
jvm_memory_used_bytes etc. | Gauges | (binder-managed) | JvmMemoryMetrics | |
jvm_gc_pause_seconds | Timer | (binder-managed) | JvmGcMetrics | |
process_cpu_usage | Gauge | (binder-managed) | ProcessorMetrics | |
jvm_threads_live | Gauge | (binder-managed) | JvmThreadMetrics | |
process_uptime_seconds | Gauge | (binder-managed) | UptimeMetrics |
The Ktor plugin's built-in timer keeps its modest default tags (route pattern, method, status); the Arcology-specific dimensions live on the custom counter and page timer, where servePath already knows site and path — no post-hoc header sniffing in a plugin hook.
Architecture
Four pieces:
metrics.org (this file) — UA classifier, referer classifier,
AccessLogEvent+ structured logger,ArcologyMetricsfacade,MetricsConfiguration; all with tests. Tangles toMetrics.ktandMetricsTest.kt.Edits to server.org — plugin install,
/metricsroute, health check upgrade,logRequest/logErrorreplacement, cache/render instrumentation,ServeCommandwiring.Edits to index.org — mark Phase 7 items DONE as they land.
web/dashboard.json— Grafana dashboard + alert provisioning. Operational config, not tangled application code.
Dependencies
[versions]
micrometer = "1.17.0"
[libraries]
micrometer-core = { module = "io.micrometer:micrometer-core", version.ref = "micrometer" }
micrometer-registry-prometheus = { module = "io.micrometer:micrometer-registry-prometheus", version.ref = "micrometer" }
ktor-server-metrics-micrometer = { group = "io.ktor", name = "ktor-server-metrics-micrometer", version.ref = "ktor" }ktor-server-metrics-micrometer and micrometer-registry-prometheus (which pulls micrometer-core) go into the root build.gradle.kts jvmMain source set next to the other Ktor server dependencies. Per AGENTS.md axiom 6: make update-deps then nix-build before anything else.
UserAgentBucketer
An ordered classifier with ~contains~-style matching — no Yauaa, no ua-parser; the output value set is deliberately a tiny fixed enum because it is the metric label. The pattern corpus is built from community classification lists: getarcis/well-known-bots (MIT, 650 categorized patterns — the SCRAPER category supplies the HTTP-client block, AI_CRAWLER the LLM block, SOCIAL the social block) and monperrus/crawler-user-agents (MIT). Feed-reader names were cherry-picked from that list's SEARCH_ENGINE category (where feed fetchers live) and headless-browser automation / uptime monitors fold into BOT.
Six buckets, first match wins:
LLM— AI crawlers and agents (GPTBot, ClaudeBot, PerplexityBot, …)SOCIAL— link-preview bots (Twitterbot, Slackbot, Mastodon, …)FEED— feed readers and fetchers (Feedly, Miniflux, FreshRSS, …)HTTP_CLIENT— libraries and CLI tools (curl, wget, python-requests, aiohttp, Go-http-client, Postman, …). Neither "browser" nor a crawler: usually a human debugging or a script.BOT— search crawlers, SEO crawlers, archivers, headless-browser automation, uptime monitorsBROWSER— default for everything unmatched
Anchoring matters on the library patterns (^curl, ^Java/, ^PHP/ …) — unanchored substrings like httpx or aiohttp are safe precisely because no browser sends them, but ~^~-anchored ones follow the source list where overmatch is possible.
package computer.whatthefuck.arcology.publishing
import computer.whatthefuck.arcology.database.RoamRepository
import io.micrometer.core.instrument.Counter
import io.micrometer.core.instrument.Timer
import io.micrometer.prometheusmetrics.PrometheusConfig
import io.micrometer.prometheusmetrics.PrometheusMeterRegistry
import kotlinx.coroutines.runBlocking
import kotlinx.serialization.Serializable
import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
import org.slf4j.LoggerFactory
enum class UaClass { BROWSER, FEED, HTTP_CLIENT, BOT, LLM, SOCIAL }The Pattern Table
The classifier's pattern corpus lives in this org table — the source of truth, editable without touching Kotlin. Each row is one pattern: Pattern is a Kotlin/Java regex fragment (~contains~-matched against the user-agent), Class is the UaClass label, and Group is a comment marker: non-empty on the first row of a section and emitted as a // ── group ── comment in the generated Kotlin. Order in the table order in the classifier priority.
| Pattern | Class | Group |
|---|---|---|
| GPTBot | LLM | LLM / AI crawlers (well-known-bots AI_CRAWLER) |
| OAI-SearchBot | LLM | |
| ChatGPT-User | LLM | |
| ClaudeBot | LLM | |
| Claude-Web | LLM | |
| Claude-SearchBot | LLM | |
| Claude-User | LLM | |
| anthropic-ai | LLM | |
| PerplexityBot | LLM | |
| Perplexity-User | LLM | |
| CCBot | LLM | |
| Bytespider | LLM | |
| meta-externalagent | LLM | |
| Diffbot | LLM | |
| YouBot | LLM | |
| cohere-ai | LLM | |
| Amazonbot | LLM | |
| FacebookBot | LLM | |
| Applebot-Extended | LLM | |
| Google-Extended | LLM | |
| AISearchBot | LLM | |
| AI2Bot | LLM | |
| aiHitBot | LLM | |
| FriendlyCrawler | LLM | |
| ImagesiftBot | LLM | |
| img2dataset | LLM | |
| omgili | LLM | |
| VelenPublicWebCrawler | LLM | |
| facebookexternalhit | SOCIAL | social media preview bots (well-known-bots SOCIAL) |
| Facebot | SOCIAL | |
| Twitterbot | SOCIAL | |
| LinkedInBot | SOCIAL | |
| Slackbot | SOCIAL | |
| Slack-ImgProxy | SOCIAL | |
| TelegramBot | SOCIAL | |
| SOCIAL | ||
| mastodon | SOCIAL | |
| Mastodon | SOCIAL | |
| Lemmy | SOCIAL | |
| Discordbot | SOCIAL | |
| Pinterest(?:bot)? | SOCIAL | |
| redditbot | SOCIAL | |
| Viber | SOCIAL | |
| Snap URL Preview Service | SOCIAL | |
| SkypeUriPreview | SOCIAL | |
| MicrosoftPreview | SOCIAL | |
| BingPreview | SOCIAL | |
| Yahoo Link Preview | SOCIAL | |
| Google Web Preview | SOCIAL | |
| FlipboardProxy | SOCIAL | |
| Embedly | SOCIAL | |
| Iframely | SOCIAL | |
| bitlybot | SOCIAL | |
| TikTokSpider | SOCIAL | |
| KeybaseBot | SOCIAL | |
| Feedly | FEED | feed readers and fetchers (SEARCH_ENGINE feed subset) |
| Inoreader | FEED | |
| Miniflux | FEED | |
| FreshRSS | FEED | |
| NetNewsWire | FEED | |
| Reeder | FEED | |
| Feedfetcher-Google | FEED | |
| Tiny Tiny RSS | FEED | |
| BazQux | FEED | |
| NewsBlur | FEED | |
| Feedbin | FEED | |
| Feedspot | FEED | |
| SimplePie | FEED | |
| theoldreader | FEED | |
| g2reader-bot | FEED | |
| SerendeputyBot | FEED | |
| Refindbot | FEED | |
| Superfeedr | FEED | |
| rssbot | FEED | |
| RSSingBot | FEED | |
| Atom Feed Robot | FEED | |
| Fever | FEED | |
| Netvibes | FEED | |
| page2rss | FEED | |
| iTMS | FEED | |
| ^curl | HTTP_CLIENT | HTTP client libraries and CLI tools (well-known-bots SCRAPER) |
| [wW]get | HTTP_CLIENT | |
| python-requests | HTTP_CLIENT | |
| aiohttp | HTTP_CLIENT | |
| httpx | HTTP_CLIENT | |
| Python-urllib | HTTP_CLIENT | |
| Scrapy | HTTP_CLIENT | |
| ^PHP/ | HTTP_CLIENT | |
| ^PHP-Curl-Class | HTTP_CLIENT | |
| phpcrawl | HTTP_CLIENT | |
| libwww-perl | HTTP_CLIENT | |
| Pcore-HTTP | HTTP_CLIENT | |
| ^Ruby | HTTP_CLIENT | |
| MetaInspector | HTTP_CLIENT | |
| ^Java/ | HTTP_CLIENT | |
| ^Apache-HttpClient | HTTP_CLIENT | |
| AHC/ | HTTP_CLIENT | |
| okhttp | HTTP_CLIENT | |
| HttpUrlConnection | HTTP_CLIENT | |
| Jetty | HTTP_CLIENT | |
| crawler4j | HTTP_CLIENT | |
| Go-http-client | HTTP_CLIENT | |
| sindresorhus/got | HTTP_CLIENT | |
| axios | HTTP_CLIENT | |
| node-fetch | HTTP_CLIENT | |
| ^HTTPie/ | HTTP_CLIENT | |
| ^[iI]nsomnia | HTTP_CLIENT | |
| PostmanRuntime/ | HTTP_CLIENT | |
| ^Postman[ /] | HTTP_CLIENT | |
| check_http | HTTP_CLIENT | |
| HTTrack | HTTP_CLIENT | |
| python-opengraph | HTTP_CLIENT | |
| newspaper/ | HTTP_CLIENT | |
| Googlebot | BOT | search crawlers, SEO, archives, automation, monitoring |
| GoogleOther | BOT | |
| Google-Safety | BOT | |
| Google-InspectionTool | BOT | |
| Google-CloudVertexBot | BOT | |
| APIs-Google | BOT | |
| AdsBot-Google | BOT | |
| Mediapartners-Google | BOT | |
| Storebot-Google | BOT | |
| Google-Read-Aloud | BOT | |
| Chrome-Lighthouse | BOT | |
| bingbot | BOT | |
| Adidxbot | BOT | |
| msnbot | BOT | |
| DuckDuckBot | BOT | |
| YandexBot | BOT | |
| yandex.com/bots | BOT | |
| YandexRenderResourcesBot | BOT | |
| Slurp | BOT | |
| Y!J | BOT | |
| Teoma | BOT | |
| Baiduspider | BOT | |
| Applebot | BOT | |
| PetalBot | BOT | |
| MojeekBot | BOT | |
| Qwantify | BOT | |
| StractBot | BOT | |
| SeznamBot | BOT | |
| Sogou | BOT | |
| Mail.RU_Bot | BOT | |
| 360Spider | BOT | |
| AhrefsBot | BOT | |
| MJ12bot | BOT | |
| SemrushBot | BOT | |
| dotbot | BOT | |
| Serpstatbot | BOT | |
| SeobilityBot | BOT | |
| rogerbot | BOT | |
| Screaming Frog SEO Spider | BOT | |
| sistrix | BOT | |
| sitebulb | BOT | |
| OnCrawl | BOT | |
| Nutch | BOT | |
| archive.org_bot | BOT | |
| heritrix | BOT | |
| CloudFlare-AlwaysOnline | BOT | |
| Algolia Crawler | BOT | |
| AddSearchBot | BOT | |
| exabot | BOT | |
| yacybot | BOT | |
| meta-externalads | BOT | |
| [Cc]ypress | BOT | headless browser automation (AUTOMATED) |
| HeadlessChrome | BOT | |
| PhantomJS | BOT | |
| [Pp]laywright | BOT | |
| [Pp]uppeteer | BOT | |
| [Ss]elenium | BOT | |
| [Ww]ebdriver | BOT | |
| UptimeRobot | BOT | uptime / monitoring (MONITORING) |
| [pP]ingdom | BOT | |
| Zabbix | BOT | |
| Uptime-Kuma | BOT | |
| Datadog | BOT | |
| StatusCake | BOT | |
| Checkly | BOT | |
| Better Uptime | BOT | |
| SentryUptimeBot | BOT | |
| Amazon Route 53 Health | BOT | |
| Cloudflare-Healthchecks | BOT | |
| changedetection | BOT | |
| deadlinkchecker | BOT | |
| W3C_Validator | BOT | |
| W3C_CSS_Validator | BOT | |
| W3C-checklink | BOT | |
| FeedValidator | BOT | |
| Validator.nu | BOT | |
| (?i)bot | BOT | generic bot markers (case-insensitive catch-alls, last) |
| (?i)crawl | BOT | |
| (?i)spider | BOT | |
| (?i)slurp | BOT |
Lua generator
This lua block (the :eval arroyo pattern) reads ua-patterns-table and emits the Kotlin listOf entries. The Group column marks section boundaries: a non-empty value emits a // ── group ── comment line before the row. Patterns are escaped for Kotlin string literals (backslash and quote). The block carries :tangle no — it's only reachable through the <<ua-patterns()>> noweb call in the bucketer below, where the parentheses trigger evaluation with the table binding.
local function esc(s)
s = tostring(s or "")
s = s:gsub('\\', '\\\\')
s = s:gsub('"', '\\"')
return s
end
local VALID = { LLM=true, SOCIAL=true, FEED=true, HTTP_CLIENT=true, BOT=true, BROWSER=true }
local lines = {}
for _, row in ipairs(tbl) do
local pattern = row[1] or ""
local cls = row[2] or ""
local group = row[3] or ""
if pattern ~= "" and cls ~= "" then
if not VALID[cls] then
error("ua-patterns-table: invalid UaClass '" .. cls .. "' for pattern '" .. pattern .. "'")
end
if group ~= "" then
table.insert(lines, " // \xe2\x94\x80\xe2\x94\x80 " .. esc(group) .. " \xe2\x94\x80\xe2\x94\x80")
end
table.insert(lines, string.format(' "%s" to UaClass.%s,', esc(pattern), cls))
end
end
return table.concat(lines, "\n")/**
* Ordered pattern list: first match wins. contains-matching so new agents
* slot in by adding a row to ua-patterns-table. Unknown → BROWSER (the
* "human" bucket). Generated from the [[ua-patterns-table]] via the
* [[ua-patterns]] lua generator — edit the table, not this list.
*
* Corpus built from getarcis/well-known-bots (MIT) — SCRAPER category for
* the HTTP-client block, AI_CRAWLER for LLM, SOCIAL for social previews,
* feed fetchers cherry-picked from its SEARCH_ENGINE category — plus
* monperrus/crawler-user-agents (MIT). ^-anchors preserved where the source
* list anchors (curl, Java/, PHP/, Ruby, Apache-HttpClient, HTTPie,
* Insomnia, Postman).
*/
object UserAgentBucketer {
private val patterns: List<Pair<Regex, UaClass>> = listOf(
<<ua-patterns()>>
).map { (pattern, cls) -> Regex(pattern) to cls }
fun classify(userAgent: String?): UaClass {
if (userAgent.isNullOrBlank()) return UaClass.BROWSER
return patterns.firstOrNull { (regex, _) -> regex.containsMatchIn(userAgent) }?.second
?: UaClass.BROWSER
}
}Referer Classification
Small fixed enumeration, same shape as UserAgentBucketer:
enum class RefererType { DIRECT, INTERNAL, SEARCH, SOCIAL, EXTERNAL }
object RefererClassifier {
private val searchHosts = listOf(
"google.", "bing.com", "duckduckgo.com", "kagi.com", "search.brave.com", "ecosia.org"
)
private val socialHosts = listOf(
"twitter.com", "x.com", "t.co", "mastodon.", "fosstodon.org",
"news.ycombinator.com", "reddit.com", "lobste.rs", "facebook.com",
"linkedin.com", "bsky.app", "stribika"
)
fun classify(referer: String?, requestHost: String?): RefererType {
if (referer.isNullOrBlank()) return RefererType.DIRECT
val ref = referer.trim().lowercase()
val host = requestHost?.substringBefore(":")?.trim()?.lowercase()
if (host != null) {
for (scheme in listOf("http://", "https://")) {
if (ref == "$scheme$host/" || ref.startsWith("$scheme$host/")) return RefererType.INTERNAL
}
}
if (searchHosts.any { it in ref }) return RefererType.SEARCH
if (socialHosts.any { it in ref }) return RefererType.SOCIAL
return RefererType.EXTERNAL
}
}Structured JSON Access Log
The single instrumentation point is the existing logRequest — it already runs at every terminal response (the only gap is /static/{path...}, which gains a call). It grows a startedAt parameter, computes the request's ua_class / referer_type, increments arcology_requests_total, and emits one JSON line. logError routes through SLF4J at warn with the same JSON shape.
/**
* One JSON line per request under the "arcology.access" logger. The shape is
* the log-aggregation contract: jq-able fields for site/path/status/duration
* plus the classifier labels that mirror the Prometheus counter tags. The
* raw [userAgent] rides along (null when the client sent none) so the
* classifier's decision can be re-derived from the log when new agents show
* up — the label stays tiny, the evidence stays queryable. [remoteHost] is
* the TCP peer address and [forwardedFor] the raw =X-Forwarded-For= header
* (null when absent): behind nginx the peer is the proxy, so the chain's
* first hop is the client.
*/
@Serializable
data class AccessLogEvent(
val timestamp: String,
val method: String,
val path: String,
val host: String?,
val site: String?,
val status: Int,
val durationMs: Long,
val uaClass: String,
val refererType: String,
val remoteHost: String? = null,
val forwardedFor: String? = null,
val userAgent: String? = null,
val extra: String = ""
)
object AccessLog {
private val log = LoggerFactory.getLogger("arcology.access")
private val json = Json { encodeDefaults = true }
fun request(event: AccessLogEvent) {
log.info(json.encodeToString(event))
}
fun error(uri: String, host: String?, message: String, throwable: Throwable?) {
val stack = throwable?.let { "${it::class.simpleName}: ${it.message}" } ?: ""
val event = AccessLogEvent(
timestamp = java.time.Instant.now().toString(),
method = "GET",
path = uri,
host = host,
site = null,
status = 500,
durationMs = 0,
uaClass = UaClass.BROWSER.name,
refererType = RefererType.DIRECT.name,
userAgent = null,
extra = "error $message $stack".trim()
)
log.warn(json.encodeToString(event))
}
}The Json { encodeDefaults = true } instance and the ~@Serializable~-data-class-over-SLF4J pattern match how SitemapGraph.kt already serializes payloads.
Request Instrumentation and Path Labels
The counter lives in the instrumentation point described above; servePath / servePathLocalhost contribute the site and path label values via the server's existing context (they know both precisely, unlike a generic plugin hook).
*Path label normalization rules* (cardinality control, see Cardinality Notes):
The
pathlabel is the request path with the query string dropped./attachment/{hash}-{size}.{ext}collapses to/attachment— the 64-hex source hash is per-attachment cardinality and worthless as a label./{path...}unmatched by the route table (404s, scanner noise) →unknown.Localhost mode labels
pathwith the fullSITE/pathkey, matching how localhost routes.Fixed internal routes keep their literal shape:
/,/sitemap,/sitemap.json,/tags,/tags/{tag},/opml.xml,/health,/metrics,/sites.css,/404,/arcology/node/{nodeId}(internal-only, low traffic).ua_classandreferer_typecome from the classifiers above;statusis the response code.
/**
* Normalizes a request path to a bounded-cardinality metric label value.
* Per the design decision: real published paths get per-path granularity,
* but the unbounded cases (attachment hashes, static-asset names, node-id
* previews, scanner noise on the catch-all) collapse to fixed values so
* crawlers cannot mint time series. The catch-all 404 branch passes
* pathOverride="unknown" at the call site; this function handles the
* path-shape rules for everything else.
*/
fun normalizePathLabel(path: String): String {
val p = path.substringBefore('?').trim()
if (p.isEmpty()) return "/"
// attachment URLs carry a 64-hex source hash per variant — collapse them.
if (p.startsWith("/attachment/")) return "/attachment"
if (p.startsWith("/arcology/attachment/")) return "/arcology/attachment"
// static assets and node previews carry per-file/per-node names
if (p.startsWith("/static/")) return "/static"
if (p.startsWith("/arcology/node/")) return "/arcology/node"
return p
}Cardinality Notes
Honest math for a personal-scale Prometheus: 10^2–10^3 real paths × 4 statuses × 5 ua classes × 5 referer types. Realistically only observed combinations mint series (a few thousand); the unbounded cases are exactly the ones the normalization rules collapse (unknown, /attachment). If the series count ever balloons, the escape hatch is demoting referer_type (derivable from logs) or bucketing long-tail paths into overflow behind a bounded cache — deferred until it's actually a problem.
Metrics Facade and Registry
ArcologyMetrics is a small facade so the server holds one nullable dependency instead of raw Micrometer calls at every site; null (the default) makes every operation a no-op, which keeps the four existing ArcologyServer construction sites (including three tests) working unchanged.
The request counter and page timer are looked up per request via registry.counter(name, tags) / registry.timer(name, tags) — Micrometer dedupes registered meters by name+tags, so no explicit per-tag registration is needed.
/**
* Histogram config shared by every Arcology timer: server-side histogram
* buckets so quantiles aggregate in PromQL (client-side percentiles are a
* known aggregation footgun). Applied per-meter via the builder API rather
* than as a registry-wide MeterFilter — filters must precede meter
* registration on the shared registry.
*/
fun timerBuilder(name: String, description: String): Timer.Builder =
Timer.builder(name)
.description(description)
.publishPercentileHistogram()
/**
* Arcology-specific meters on the shared Prometheus registry. Request
* counting and page timing live here (called from the server's
* instrumentation point); cache and node-render meters are helpers for the
* servePath / SitemapGraphBuilder instrumentation.
*/
class ArcologyMetrics(val registry: PrometheusMeterRegistry) {
fun incRequest(
site: String?,
path: String,
status: Int,
uaClass: UaClass,
refererType: RefererType
) {
Counter.builder("arcology_requests_total")
.description("Requests served by the Arcology web server")
.tag("site", site ?: "localhost")
.tag("path", path)
.tag("status", status.toString())
.tag("ua_class", uaClass.name)
.tag("referer_type", refererType.name)
.register(registry)
.increment()
}
fun pageTimer(site: String?, path: String): Timer =
timerBuilder("arcology_request_duration_seconds", "Time to build/serve a page")
.tag("site", site ?: "localhost")
.tag("path", path)
.register(registry)
/**
* Lazily registered on first use: the server module may install
* Ktor's MicrometerMetrics plugin (which registers a registry-wide
* MeterFilter) after this object is constructed, and filters must
* precede meter registration.
*/
val nodeRenderTimer: Timer by lazy {
timerBuilder("arcology_node_render_duration_seconds", "Time to parse + render one node body (cache miss)")
.register(registry)
}
fun cacheHit(cache: String) {
Counter.builder("arcology_cache_hits_total")
.tag("cache", cache)
.register(registry)
.increment()
}
fun cacheMiss(cache: String) {
Counter.builder("arcology_cache_misses_total")
.tag("cache", cache)
.register(registry)
.increment()
}
}
object MetricsConfiguration {
/**
* The shared, empty Prometheus registry. The JVM binders are bound by
* Ktor's MicrometerMetrics plugin (its default =meterBinders= list)
* during ~module()~ install — binding them here too would double-register
* the same gauges (duplicate-registration warning, second copy dropped),
* and binding anything here before the plugin installs would make the
* plugin's registry-wide MeterFilter a late filter (which warns). The
* ordering contract: bare registry → plugin install (binders + filter) →
* any Arcology meters (all lazy/per-request).
*/
fun registry(): PrometheusMeterRegistry =
PrometheusMeterRegistry(PrometheusConfig.DEFAULT)
}The Ktor plugin config (in server.org's module()) sets a percentilesHistogram(true) distributionStatisticConfig so Ktor's built-in request-timer quantiles aggregate in PromQL (client-side percentiles are a known aggregation footgun). Ordering is the whole game for the plugin's registry-wide MeterFilter: it registers at install time, so nothing may be registered on the registry before module() runs — MetricsConfiguration.registry is deliberately bare (the plugin's default binder list does the JVM binding) and every Arcology meter registers per-request or lazy (ArcologyMetrics.nodeRenderTimer).
Cache and Render Metrics
Exactly two hit/miss decision points exist for page HTML — the htmlCache.getPage branches in servePath / servePathLocalhost — plus the fast-path and locked-miss in SitemapGraphBuilder.json. Each increments arcology_cache_{hits,misses}_total with cache=page or cache=sitemap. The sidebar cache hit branch is a deliberate no-op today and stays uninstrumented.
The page-build timer wraps the whole servePath / servePathLocalhost call (label: site, path); arcology_node_render_duration_seconds wraps cache-miss renderNodeBody work so render-p90 distinguishes "slow render" from "cache hit serving".
Health Check
GET /health upgrades from a static OK to a real check:
Database: a lightweight read through the roam repository (
getNodeAncestorson a sentinel ID returns a list — exercises the SQLDelight driver end-to-end without assuming any table content).Cache dir: writable probe file, deleted after the check.
Org dir: exists and is a directory.
Response: 200 with {"status":"ok"} or 503 with {"status":"degraded","checks":{...}} — serialized with the same kotlinx-serialization instance as the access log. This gives the health alert teeth (a 503 fails an HTTP-level probe; up drops to 0).
/**
* Health check probe results. The status strings are the JSON contract the
* Grafana alert (up == 0 via HTTP probe) and any external monitor rely on.
*/
object HealthCheck {
@Serializable
data class Report(val status: String, val checks: Map<String, String>)
fun check(
repository: RoamRepository,
cacheDir: java.io.File,
orgDir: String
): Report {
val checks = linkedMapOf<String, String>()
checks["database"] = try {
runBlocking { repository.getNodeAncestors("health-probe-sentinel") }
"ok"
} catch (e: Exception) {
"fail: ${e::class.simpleName}: ${e.message}"
}
checks["cache_dir"] = try {
val probe = java.io.File.createTempFile("health-probe", ".tmp", cacheDir)
probe.writeText("ok")
probe.delete()
"ok"
} catch (e: Exception) {
"fail: ${e::class.simpleName}: ${e.message}"
}
checks["org_dir"] = run {
val dir = java.io.File(orgDir)
if (dir.exists() && dir.isDirectory) "ok"
else "fail: missing or not a directory"
}
val degraded = checks.values.any { it != "ok" }
return Report(
status = if (degraded) "degraded" else "ok",
checks = checks
)
}
}The report is exposed by the GET /health handler in server.org: 200 + {"status":"ok"} on health, 503 + full report on degradation.
server.org Edits
install(MicrometerMetrics)inmodule()with the sharedPrometheusMeterRegistryand apercentilesHistogram(true)distributionStatisticConfig. The plugin binds the JVM binders (its default list — MetricsConfiguration.registry is deliberately bare so binders aren't double-registered) and registers its MeterFilter before any meter exists: MetricsConfiguration.registry is empty and every Arcology meter is per-request orlazy.New route
GET /metrics→call.respondText(prometheusRegistry.scrape()).Add
"metrics"to the/{path...}catch-all exclusion list.Every route handler captures
val start = TimeSource.Monotonic.markNow()at entry; the terminallogRequestcall becomeslogRequest(start, method, uri, host, status, extra), which increments the counter, records the timer, and emits the JSON access line. The ~~/static~ route gains its missing logging call.servePath/servePathLocalhost: page timer around the call body, cache hit/miss counters in thegetPagebranches, node-render timer around cache-missrenderNodeBody.ArcologyServergainsmetrics: ArcologyMetrics? = null(default keeps the three test construction sites compiling).ServeCommand: construct thePrometheusMeterRegistry+ArcologyMetrics, wire/metrics.
Grafana Dashboard
web/dashboard.json, provisioned per the layout in index.org's Phase 7:
| Row | Panels |
|---|---|
| 1 | counts served per site since restart; counts in last 24h; site hits per minute |
| 2 | "human" per-page hits (ua_class="BROWSER"); "bot" per-page hits (ua_class!="BROWSER"); feed-reader hits |
| 3 | page render time histogram; render quantiles p25/p50/p75/p95/p99/max |
| 4 | memory usage; CPU time; cache hit ratio (derived hits / (hits + misses)); GC pause, thread count |
Alert Rules
Anti-viral: a page's 5-minute request rate exceeds 3× its trailing-24h hourly average (non-bot, non-library traffic — humans and their feed readers only): #+begin_src promql rate(arcology_requests_total{ua_class!~"BOT|LLM|HTTP_CLIENT"}[5m]) > 3 avg_over_time(rate(arcology_requests_total{ua_class!~"BOT|LLM|HTTP_CLIENT"}[1h])[24h:1h]) #+end_src 2. Render p90: page build p90 over 5 minutes above 2s: #+begin_src promql histogram_quantile(0.9, rate(arcology_request_duration_seconds_bucket[5m])) > 2.0 #+end_src 3. Health*:
up{job="arcology"} == 0for > 1m (the 503-on-degraded health check makes an HTTP probe fail honestly).
Tests
All in this file, following the project's table-driven / fake-repository patterns:
UserAgentBucketerTest— known bots, feed readers, LLM crawlers, HTTP client libraries (curl/wget/requests/aiohttp/Go/Java/okhttp/axios/Postman…), social previews, browsers; empty/blank UA; the generic(?i)botcatch-all; curl-impersonation stays BROWSER.RefererClassifierTest— direct, internal (scheme + host matching incl. port stripping), search, social, external.AccessLogEventTest— JSON round-trip,encodeDefaultsshape, field presence, raw user-agent round-trip, remote-host / X-Forwarded-For round-trip.ArcologyMetricsTest— counters/timers registered with expected tag sets; scrape output containsarcology_requests_totalandjvm_memory_used_bytesin valid Prometheus text format.Path-label normalization tests — attachment names collapse to
/attachment, query strings dropped, unmatched →unknown.
Run with make test-only class=... against the root :jvmTest.
Classifier tests
package computer.whatthefuck.arcology.publishing
import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrueclass UserAgentBucketerTest {
@Test
fun `classifies known LLM crawlers`() {
assertEquals(UaClass.LLM, UserAgentBucketer.classify("GPTBot/1.0 (+https://openai.com/gptbot)"))
assertEquals(UaClass.LLM, UserAgentBucketer.classify("ClaudeBot/1.0"))
assertEquals(UaClass.LLM, UserAgentBucketer.classify("Mozilla/5.0 (compatible; PerplexityBot/1.0)"))
assertEquals(UaClass.LLM, UserAgentBucketer.classify("CCBot/2.0"))
assertEquals(UaClass.LLM, UserAgentBucketer.classify("meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"))
}
@Test
fun `classifies http client libraries and cli tools as HTTP_CLIENT`() {
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("curl/8.5.0"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Wget/1.21.4"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("python-requests/2.31.0"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Python/3.12 aiohttp/3.9.1"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("python-httpx/0.27.0"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Go-http-client/2.0"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Java/17.0.1"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Apache-HttpClient/4.5.13 (Java/17.0.1)"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("okhttp/4.11.0"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("axios/1.6.0"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("PostmanRuntime/7.36.0"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("HTTPie/3.2.2"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Scrapy/2.11 (+https://scrapy.org)"))
assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("HTTrack/3.0"))
}
@Test
fun `curl-impersonating browsers stay BROWSER`() {
// curl-impersonate spoofs real browser UA strings; a UA that contains
// no library token is a browser by classification.
assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36"
))
}
@Test
fun `classifies known feed readers`() {
assertEquals(UaClass.FEED, UserAgentBucketer.classify("Feedly/1.0 (+http://www.feedly.com/fetcher.html)"))
assertEquals(UaClass.FEED, UserAgentBucketer.classify("Miniflux/2.2.0"))
assertEquals(UaClass.FEED, UserAgentBucketer.classify("FreshRSS/1.20 (Linux; https://freshrss.org)"))
assertEquals(UaClass.FEED, UserAgentBucketer.classify("NetNewsWire/6.1 (Mac)"))
}
@Test
fun `classifies known search bots`() {
assertEquals(UaClass.BOT, UserAgentBucketer.classify("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"))
assertEquals(UaClass.BOT, UserAgentBucketer.classify("bingbot/2.0"))
assertEquals(UaClass.BOT, UserAgentBucketer.classify("Mozilla/5.0 (compatible; YandexBot/3.0)"))
assertEquals(UaClass.BOT, UserAgentBucketer.classify("Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"))
assertEquals(UaClass.BOT, UserAgentBucketer.classify("HeadlessChrome/126.0"))
}
@Test
fun `classifies social preview bots`() {
assertEquals(UaClass.SOCIAL, UserAgentBucketer.classify("facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"))
assertEquals(UaClass.SOCIAL, UserAgentBucketer.classify("Twitterbot/1.0"))
assertEquals(UaClass.SOCIAL, UserAgentBucketer.classify("Slackbot-LinkExpanding 1.0"))
}
@Test
fun `generic bot marker catches unknown crawlers`() {
assertEquals(UaClass.BOT, UserAgentBucketer.classify("some-new-crawler/1.0 (spider)"))
assertEquals(UaClass.BOT, UserAgentBucketer.classify("MysteryBot/2.0"))
}
@Test
fun `browsers and unknown agents classify as BROWSER`() {
assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(null))
assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(""))
assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(" "))
assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(
"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0"
))
assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 Safari/605.1.15"
))
}
}class RefererClassifierTest {
@Test
fun `null blank referer is DIRECT`() {
assertEquals(RefererType.DIRECT, RefererClassifier.classify(null, "example.com"))
assertEquals(RefererType.DIRECT, RefererClassifier.classify("", "example.com"))
assertEquals(RefererType.DIRECT, RefererClassifier.classify(" ", "example.com"))
}
@Test
fun `same host with either scheme is INTERNAL`() {
assertEquals(RefererType.INTERNAL, RefererClassifier.classify("https://example.com/some/page", "example.com"))
assertEquals(RefererType.INTERNAL, RefererClassifier.classify("http://example.com/", "example.com"))
// request host with a port is stripped
assertEquals(RefererType.INTERNAL, RefererClassifier.classify("https://example.com/x", "example.com:8080"))
}
@Test
fun `search engines classify as SEARCH`() {
assertEquals(RefererType.SEARCH, RefererClassifier.classify("https://www.google.com/search?q=x", "example.com"))
assertEquals(RefererType.SEARCH, RefererClassifier.classify("https://duckduckgo.com/?q=x", "example.com"))
assertEquals(RefererType.SEARCH, RefererClassifier.classify("https://kagi.com/search?q=x", "example.com"))
}
@Test
fun `social platforms classify as SOCIAL`() {
assertEquals(RefererType.SOCIAL, RefererClassifier.classify("https://news.ycombinator.com/item?id=1", "example.com"))
assertEquals(RefererType.SOCIAL, RefererClassifier.classify("https://t.co/abc", "example.com"))
assertEquals(RefererType.SOCIAL, RefererClassifier.classify("https://bsky.app/profile/x", "example.com"))
}
@Test
fun `other origins classify as EXTERNAL`() {
assertEquals(RefererType.EXTERNAL, RefererClassifier.classify("https://someblog.example.net/links", "example.com"))
// a different scheme that isn't a known host
assertEquals(RefererType.EXTERNAL, RefererClassifier.classify("android-app://com.someapp", "example.com"))
}
}Path normalization and access log tests
class NormalizePathLabelTest {
@Test
fun `attachment urls collapse to fixed label`() {
val hash = "a".repeat(64)
assertEquals("/attachment", normalizePathLabel("/attachment/$hash-1080.jpg"))
assertEquals("/attachment", normalizePathLabel("/attachment/$hash-0.pdf"))
assertEquals("/arcology/attachment", normalizePathLabel("/arcology/attachment/$hash-2160.jpg/html"))
}
@Test
fun `static and node-preview urls collapse to fixed labels`() {
assertEquals("/static", normalizePathLabel("/static/sitemap/sigma.min.js"))
assertEquals("/arcology/node", normalizePathLabel("/arcology/node/some-node-id"))
}
@Test
fun `query strings are dropped`() {
assertEquals("/404", normalizePathLabel("/404?node=some-id"))
}
@Test
fun `empty path normalizes to slash`() {
assertEquals("/", normalizePathLabel(""))
assertEquals("/", normalizePathLabel("?foo=bar"))
}
@Test
fun `real paths pass through unchanged`() {
assertEquals("/garden/index", normalizePathLabel("/garden/index"))
assertEquals("/tags/tech", normalizePathLabel("/tags/tech"))
}
}
class AccessLogEventTest {
@Test
fun `access log event serializes to compact JSON`() {
val event = AccessLogEvent(
timestamp = "2026-09-14T00:00:00Z",
method = "GET",
path = "/garden/index",
host = "arcology.garden",
site = "garden",
status = 200,
durationMs = 12,
uaClass = "BROWSER",
refererType = "DIRECT"
)
val json = Json { encodeDefaults = true }.encodeToString(event)
assertTrue(json.contains("\"status\":200"))
assertTrue(json.contains("\"uaClass\":\"BROWSER\""))
assertTrue(json.contains("\"site\":\"garden\""))
// extra has a default: encoded (encodeDefaults) with empty string
assertTrue(json.contains("\"extra\":\"\""))
// userAgent defaults to null and is encoded as such
assertTrue(json.contains("\"userAgent\":null"))
}
@Test
fun `raw user agent rides along in the log event`() {
val ua = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
val event = AccessLogEvent(
timestamp = "2026-09-14T00:00:00Z",
method = "GET",
path = "/garden/index",
host = "arcology.garden",
site = "garden",
status = 200,
durationMs = 12,
uaClass = "BOT",
refererType = "DIRECT",
userAgent = ua
)
val json = Json { encodeDefaults = true }.encodeToString(event)
assertTrue(json.contains("\"uaClass\":\"BOT\""))
assertTrue(json.contains("\"userAgent\":\"$ua\""))
}
@Test
fun `remote host and forwarded-for ride along in the log event`() {
val event = AccessLogEvent(
timestamp = "2026-09-14T00:00:00Z",
method = "GET",
path = "/garden/index",
host = "arcology.garden",
site = "garden",
status = 200,
durationMs = 12,
uaClass = "BROWSER",
refererType = "DIRECT",
remoteHost = "127.0.0.1",
forwardedFor = "203.0.113.7, 10.0.0.2"
)
val json = Json { encodeDefaults = true }.encodeToString(event)
// Behind nginx the peer is the proxy; the XFF chain's first hop is
// the client. Both fields present in the JSON contract.
assertTrue(json.contains("\"remoteHost\":\"127.0.0.1\""))
assertTrue(json.contains("\"forwardedFor\":\"203.0.113.7, 10.0.0.2\""))
}
}Metrics facade and health check tests
class ArcologyMetricsTest {
@Test
fun `request counter and page timer register with expected tags`() {
val metrics = ArcologyMetrics(MetricsConfiguration.registry())
metrics.incRequest("garden", "/index", 200, UaClass.BROWSER, RefererType.DIRECT)
metrics.incRequest("garden", "/index", 404, UaClass.BOT, RefererType.EXTERNAL)
metrics.pageTimer("garden", "/index").record(java.time.Duration.ofMillis(5))
val scrape = metrics.registry.scrape()
assertTrue(scrape.contains("arcology_requests_total"))
assertTrue(scrape.contains("site=\"garden\""))
assertTrue(scrape.contains("path=\"/index\""))
assertTrue(scrape.contains("ua_class=\"BROWSER\""))
assertTrue(scrape.contains("ua_class=\"BOT\""))
assertTrue(scrape.contains("referer_type=\"DIRECT\""))
assertTrue(scrape.contains("arcology_request_duration_seconds"))
}
@Test
fun `cache counters and node render timer register`() {
val metrics = ArcologyMetrics(MetricsConfiguration.registry())
metrics.cacheHit("page")
metrics.cacheMiss("page")
metrics.cacheHit("sitemap")
metrics.nodeRenderTimer.record(java.time.Duration.ofMillis(5))
val scrape = metrics.registry.scrape()
assertTrue(scrape.contains("arcology_cache_hits_total{cache=\"page\"}"))
assertTrue(scrape.contains("arcology_cache_misses_total{cache=\"page\"}"))
assertTrue(scrape.contains("cache=\"sitemap\""))
assertTrue(scrape.contains("arcology_node_render_duration_seconds"))
}
@Test
fun `registry exposes JVM binder metrics`() {
// MetricsConfiguration.registry() is deliberately bare — the JVM
// binders are the Ktor plugin's default meterBinders at install
// time (see MetricsConfiguration docs). Binding the same set here
// verifies the binder classes still expose the dashboard metrics.
val registry = MetricsConfiguration.registry()
listOf(
io.micrometer.core.instrument.binder.jvm.JvmMemoryMetrics(),
io.micrometer.core.instrument.binder.jvm.JvmGcMetrics(),
io.micrometer.core.instrument.binder.system.ProcessorMetrics(),
io.micrometer.core.instrument.binder.jvm.JvmThreadMetrics(),
io.micrometer.core.instrument.binder.system.UptimeMetrics()
).forEach { it.bindTo(registry) }
val scrape = registry.scrape()
assertTrue(scrape.contains("jvm_memory_used_bytes"))
assertTrue(scrape.contains("process_cpu_usage"))
assertTrue(scrape.contains("process_uptime_seconds"))
}
}
class HealthCheckTest {
private val healthyRepo = EmptyRoamRepository()
@Test
fun `all checks pass on healthy inputs`() {
val cacheDir = java.nio.file.Files.createTempDirectory("health-cache").toFile()
val orgDir = java.nio.file.Files.createTempDirectory("health-org").toFile()
val report = HealthCheck.check(healthyRepo, cacheDir, orgDir.absolutePath)
assertEquals("ok", report.status)
assertEquals("ok", report.checks["database"])
assertEquals("ok", report.checks["cache_dir"])
assertEquals("ok", report.checks["org_dir"])
}
@Test
fun `missing org dir degrades the report`() {
val cacheDir = java.nio.file.Files.createTempDirectory("health-cache").toFile()
val report = HealthCheck.check(healthyRepo, cacheDir, "/does/not/exist")
assertEquals("degraded", report.status)
assertTrue(report.checks["org_dir"]!!.startsWith("fail"))
}
@Test
fun `failing database degrades the report`() {
val cacheDir = java.nio.file.Files.createTempDirectory("health-cache").toFile()
val orgDir = java.nio.file.Files.createTempDirectory("health-org").toFile()
val brokenRepo = object : EmptyRoamRepository() {
override suspend fun getNodeAncestors(nodeId: String): List<String> {
throw IllegalStateException("database is closed")
}
}
val report = HealthCheck.check(brokenRepo, cacheDir, orgDir.absolutePath)
assertEquals("degraded", report.status)
assertTrue(report.checks["database"]!!.startsWith("fail"))
}
}Implementation Order
TOML +
build.gradle.ktsdeps →make update-deps→nix-build. (DONE: micrometer 1.17.0, ktor-server-metrics-micrometer, make update-deps + nix-build passed.)web/metrics.orgclassifiers + logger + facade + tests; tangle.web/server.orgedits; tangle;make build&&make test.web/dashboard.json+ alert rules.Update index.org status marks.
Tangle Targets
Metrics.kt
<<metrics-preamble>>
<<ua-bucketer>>
<<referer-classifier>>
<<access-log-event>>
<<path-normalization>>
<<metrics-facade>>
<<health-check>>MetricsTest.kt
<<metrics-test-prelude>>
<<metrics-test>>
<<metrics-test-end>>Related Modules
Arcology's Web Server — instrumentation point,
logRequest,ServeCommandwiringArcology's Web Publishing Layer —
DomainMapsite resolution for thesitelabelSitemap Graph and Tag Index —
SitemapGraphBuildercache instrumentationThe Arcology Hypermedia Publishing Platform — phase plan and dashboard layout