Arcology Engine

Arcology Web Metrics and Monitoring

Contents

Introduction

This document is the plan for Phase 7 of the Arcology Hypermedia Publishing Platform: metrics and monitoring for the Arcology Web Server. Today the server is observable only through println access lines (logRequest / logError) and a static /health responder that cannot fail. Phase 7 adds:

  • a Prometheus GET /metrics endpoint (Micrometer + PrometheusMeterRegistry),

  • per-request counters and timers with site, path, status, ua_class, and referer_type labels,

  • structured JSON access logging on the existing SLF4J backend,

  • a real health check (database, cache dir, org dir) with a 503 on degradation,

  • a Grafana dashboard + alert rule set (anti-viral, render p90, health).

Stack choice: Micrometer via Ktor's built-in MicrometerMetrics plugin for the standard HTTP timer + JVM binders, plus a small ArcologyMetrics facade registering Arcology's own counters/timers on the same registry. A hand-rolled counter plugin was considered and rejected — it would re-implement percentile histograms and exposition glue for no benefit.

Design Decisions

Locked in with the developer:

  1. Micrometer + Prometheus registry, not a bespoke metrics implementation. The Ktor plugin (ktor-server-metrics-micrometer, ktor 3.2.3) provides the built-in ktor.http.server.requests timer; micrometer-core / micrometer-registry-prometheus at 1.17.0 provide the registry, histogram buckets, and JVM binders.

  2. Referer classification ships as a referer_type label on the request counter (direct internal search social external).

  3. Per-path label granularity, not route-pattern collapse. The URL space is the site's own bounded content (the published_routes table), not unbounded {uuid} routes, so real paths are defensible. The mandatory mitigation is that unmatched requests — 404 probes, scanner noise — must not mint series; they collapse to unknown. See Cardinality Notes.

  4. Structured JSON logging: @Serializable log events serialized with kotlinx-serialization and emitted through the already-present slf4j-simple backend under the arcology.access logger name. No logback/logstash dependency until ops needs outgrow stderr.

  5. /metrics is unauthenticated; it is blocked from the public internet at the nginx layer.

Metric Catalog

Metric Type Labels Source
arcology_requests_total Counter site, path, status, ua_class, referer_type request instrumentation point
arcology_request_duration_seconds Timer site, path servePath / servePathLocalhost
arcology_node_render_duration_seconds Timer (none) cache-miss renderNodeBody
arcology_cache_hits_total Counter cache = page \ sitemap HtmlCache / SitemapGraphBuilder
arcology_cache_misses_total Counter cache = page \ sitemap same
ktor.http.server.requests Timer route, method, status, throwable Ktor MicrometerMetrics plugin
jvm_memory_used_bytes etc. Gauges (binder-managed) JvmMemoryMetrics
jvm_gc_pause_seconds Timer (binder-managed) JvmGcMetrics
process_cpu_usage Gauge (binder-managed) ProcessorMetrics
jvm_threads_live Gauge (binder-managed) JvmThreadMetrics
process_uptime_seconds Gauge (binder-managed) UptimeMetrics

The Ktor plugin's built-in timer keeps its modest default tags (route pattern, method, status); the Arcology-specific dimensions live on the custom counter and page timer, where servePath already knows site and path — no post-hoc header sniffing in a plugin hook.

Architecture

Four pieces:

  • metrics.org (this file) — UA classifier, referer classifier, AccessLogEvent + structured logger, ArcologyMetrics facade, MetricsConfiguration; all with tests. Tangles to Metrics.kt and MetricsTest.kt.

  • Edits to server.org — plugin install, /metrics route, health check upgrade, logRequest / logError replacement, cache/render instrumentation, ServeCommand wiring.

  • Edits to index.org — mark Phase 7 items DONE as they land.

  • web/dashboard.json — Grafana dashboard + alert provisioning. Operational config, not tangled application code.

Dependencies

toml#+name: toml-deps
[versions]
micrometer = "1.17.0"

[libraries]
micrometer-core = { module = "io.micrometer:micrometer-core", version.ref = "micrometer" }
micrometer-registry-prometheus = { module = "io.micrometer:micrometer-registry-prometheus", version.ref = "micrometer" }
ktor-server-metrics-micrometer = { group = "io.ktor", name = "ktor-server-metrics-micrometer", version.ref = "ktor" }

ktor-server-metrics-micrometer and micrometer-registry-prometheus (which pulls micrometer-core) go into the root build.gradle.kts jvmMain source set next to the other Ktor server dependencies. Per AGENTS.md axiom 6: make update-deps then nix-build before anything else.

UserAgentBucketer

An ordered classifier with ~contains~-style matching — no Yauaa, no ua-parser; the output value set is deliberately a tiny fixed enum because it is the metric label. The pattern corpus is built from community classification lists: getarcis/well-known-bots (MIT, 650 categorized patterns — the SCRAPER category supplies the HTTP-client block, AI_CRAWLER the LLM block, SOCIAL the social block) and monperrus/crawler-user-agents (MIT). Feed-reader names were cherry-picked from that list's SEARCH_ENGINE category (where feed fetchers live) and headless-browser automation / uptime monitors fold into BOT.

Six buckets, first match wins:

  • LLM — AI crawlers and agents (GPTBot, ClaudeBot, PerplexityBot, …)

  • SOCIAL — link-preview bots (Twitterbot, Slackbot, Mastodon, …)

  • FEED — feed readers and fetchers (Feedly, Miniflux, FreshRSS, …)

  • HTTP_CLIENT — libraries and CLI tools (curl, wget, python-requests, aiohttp, Go-http-client, Postman, …). Neither "browser" nor a crawler: usually a human debugging or a script.

  • BOT — search crawlers, SEO crawlers, archivers, headless-browser automation, uptime monitors

  • BROWSER — default for everything unmatched

Anchoring matters on the library patterns (^curl, ^Java/, ^PHP/ …) — unanchored substrings like httpx or aiohttp are safe precisely because no browser sends them, but ~^~-anchored ones follow the source list where overmatch is possible.

kotlin#+name: metrics-preamble
package computer.whatthefuck.arcology.publishing

import computer.whatthefuck.arcology.database.RoamRepository
import io.micrometer.core.instrument.Counter
import io.micrometer.core.instrument.Timer
import io.micrometer.prometheusmetrics.PrometheusConfig
import io.micrometer.prometheusmetrics.PrometheusMeterRegistry
import kotlinx.coroutines.runBlocking
import kotlinx.serialization.Serializable
import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
import org.slf4j.LoggerFactory

enum class UaClass { BROWSER, FEED, HTTP_CLIENT, BOT, LLM, SOCIAL }

The Pattern Table

The classifier's pattern corpus lives in this org table — the source of truth, editable without touching Kotlin. Each row is one pattern: Pattern is a Kotlin/Java regex fragment (~contains~-matched against the user-agent), Class is the UaClass label, and Group is a comment marker: non-empty on the first row of a section and emitted as a // ── group ── comment in the generated Kotlin. Order in the table order in the classifier priority.

Pattern Class Group
GPTBot LLM LLM / AI crawlers (well-known-bots AI_CRAWLER)
OAI-SearchBot LLM
ChatGPT-User LLM
ClaudeBot LLM
Claude-Web LLM
Claude-SearchBot LLM
Claude-User LLM
anthropic-ai LLM
PerplexityBot LLM
Perplexity-User LLM
CCBot LLM
Bytespider LLM
meta-externalagent LLM
Diffbot LLM
YouBot LLM
cohere-ai LLM
Amazonbot LLM
FacebookBot LLM
Applebot-Extended LLM
Google-Extended LLM
AISearchBot LLM
AI2Bot LLM
aiHitBot LLM
FriendlyCrawler LLM
ImagesiftBot LLM
img2dataset LLM
omgili LLM
VelenPublicWebCrawler LLM
facebookexternalhit SOCIAL social media preview bots (well-known-bots SOCIAL)
Facebot SOCIAL
Twitterbot SOCIAL
LinkedInBot SOCIAL
Slackbot SOCIAL
Slack-ImgProxy SOCIAL
TelegramBot SOCIAL
WhatsApp SOCIAL
mastodon SOCIAL
Mastodon SOCIAL
Lemmy SOCIAL
Discordbot SOCIAL
Pinterest(?:bot)? SOCIAL
redditbot SOCIAL
Viber SOCIAL
Snap URL Preview Service SOCIAL
SkypeUriPreview SOCIAL
MicrosoftPreview SOCIAL
BingPreview SOCIAL
Yahoo Link Preview SOCIAL
Google Web Preview SOCIAL
FlipboardProxy SOCIAL
Embedly SOCIAL
Iframely SOCIAL
bitlybot SOCIAL
TikTokSpider SOCIAL
KeybaseBot SOCIAL
Feedly FEED feed readers and fetchers (SEARCH_ENGINE feed subset)
Inoreader FEED
Miniflux FEED
FreshRSS FEED
NetNewsWire FEED
Reeder FEED
Feedfetcher-Google FEED
Tiny Tiny RSS FEED
BazQux FEED
NewsBlur FEED
Feedbin FEED
Feedspot FEED
SimplePie FEED
theoldreader FEED
g2reader-bot FEED
SerendeputyBot FEED
Refindbot FEED
Superfeedr FEED
rssbot FEED
RSSingBot FEED
Atom Feed Robot FEED
Fever FEED
Netvibes FEED
page2rss FEED
iTMS FEED
^curl HTTP_CLIENT HTTP client libraries and CLI tools (well-known-bots SCRAPER)
[wW]get HTTP_CLIENT
python-requests HTTP_CLIENT
aiohttp HTTP_CLIENT
httpx HTTP_CLIENT
Python-urllib HTTP_CLIENT
Scrapy HTTP_CLIENT
^PHP/ HTTP_CLIENT
^PHP-Curl-Class HTTP_CLIENT
phpcrawl HTTP_CLIENT
libwww-perl HTTP_CLIENT
Pcore-HTTP HTTP_CLIENT
^Ruby HTTP_CLIENT
MetaInspector HTTP_CLIENT
^Java/ HTTP_CLIENT
^Apache-HttpClient HTTP_CLIENT
AHC/ HTTP_CLIENT
okhttp HTTP_CLIENT
HttpUrlConnection HTTP_CLIENT
Jetty HTTP_CLIENT
crawler4j HTTP_CLIENT
Go-http-client HTTP_CLIENT
sindresorhus/got HTTP_CLIENT
axios HTTP_CLIENT
node-fetch HTTP_CLIENT
^HTTPie/ HTTP_CLIENT
^[iI]nsomnia HTTP_CLIENT
PostmanRuntime/ HTTP_CLIENT
^Postman[ /] HTTP_CLIENT
check_http HTTP_CLIENT
HTTrack HTTP_CLIENT
python-opengraph HTTP_CLIENT
newspaper/ HTTP_CLIENT
Googlebot BOT search crawlers, SEO, archives, automation, monitoring
GoogleOther BOT
Google-Safety BOT
Google-InspectionTool BOT
Google-CloudVertexBot BOT
APIs-Google BOT
AdsBot-Google BOT
Mediapartners-Google BOT
Storebot-Google BOT
Google-Read-Aloud BOT
Chrome-Lighthouse BOT
bingbot BOT
Adidxbot BOT
msnbot BOT
DuckDuckBot BOT
YandexBot BOT
yandex.com/bots BOT
YandexRenderResourcesBot BOT
Slurp BOT
Y!J BOT
Teoma BOT
Baiduspider BOT
Applebot BOT
PetalBot BOT
MojeekBot BOT
Qwantify BOT
StractBot BOT
SeznamBot BOT
Sogou BOT
Mail.RU_Bot BOT
360Spider BOT
AhrefsBot BOT
MJ12bot BOT
SemrushBot BOT
dotbot BOT
Serpstatbot BOT
SeobilityBot BOT
rogerbot BOT
Screaming Frog SEO Spider BOT
sistrix BOT
sitebulb BOT
OnCrawl BOT
Nutch BOT
archive.org_bot BOT
heritrix BOT
CloudFlare-AlwaysOnline BOT
Algolia Crawler BOT
AddSearchBot BOT
exabot BOT
yacybot BOT
meta-externalads BOT
[Cc]ypress BOT headless browser automation (AUTOMATED)
HeadlessChrome BOT
PhantomJS BOT
[Pp]laywright BOT
[Pp]uppeteer BOT
[Ss]elenium BOT
[Ww]ebdriver BOT
UptimeRobot BOT uptime / monitoring (MONITORING)
[pP]ingdom BOT
Zabbix BOT
Uptime-Kuma BOT
Datadog BOT
StatusCake BOT
Checkly BOT
Better Uptime BOT
SentryUptimeBot BOT
Amazon Route 53 Health BOT
Cloudflare-Healthchecks BOT
changedetection BOT
deadlinkchecker BOT
W3C_Validator BOT
W3C_CSS_Validator BOT
W3C-checklink BOT
FeedValidator BOT
Validator.nu BOT
(?i)bot BOT generic bot markers (case-insensitive catch-alls, last)
(?i)crawl BOT
(?i)spider BOT
(?i)slurp BOT

Lua generator

This lua block (the :eval arroyo pattern) reads ua-patterns-table and emits the Kotlin listOf entries. The Group column marks section boundaries: a non-empty value emits a // ── group ── comment line before the row. Patterns are escaped for Kotlin string literals (backslash and quote). The block carries :tangle no — it's only reachable through the <<ua-patterns()>> noweb call in the bucketer below, where the parentheses trigger evaluation with the table binding.

lua#+name: ua-patterns:eval arroyo:var tbl=ua-patterns-table:tangle no:comments none
local function esc(s)
  s = tostring(s or "")
  s = s:gsub('\\', '\\\\')
  s = s:gsub('"', '\\"')
  return s
end
local VALID = { LLM=true, SOCIAL=true, FEED=true, HTTP_CLIENT=true, BOT=true, BROWSER=true }
local lines = {}
for _, row in ipairs(tbl) do
  local pattern = row[1] or ""
  local cls = row[2] or ""
  local group = row[3] or ""
  if pattern ~= "" and cls ~= "" then
    if not VALID[cls] then
      error("ua-patterns-table: invalid UaClass '" .. cls .. "' for pattern '" .. pattern .. "'")
    end
    if group ~= "" then
      table.insert(lines, "        // \xe2\x94\x80\xe2\x94\x80 " .. esc(group) .. " \xe2\x94\x80\xe2\x94\x80")
    end
    table.insert(lines, string.format('        "%s" to UaClass.%s,', esc(pattern), cls))
  end
end
return table.concat(lines, "\n")
kotlin#+name: ua-bucketer:noweb yes
/**
 * Ordered pattern list: first match wins. contains-matching so new agents
 * slot in by adding a row to ua-patterns-table. Unknown → BROWSER (the
 * "human" bucket). Generated from the [[ua-patterns-table]] via the
 * [[ua-patterns]] lua generator — edit the table, not this list.
 *
 * Corpus built from getarcis/well-known-bots (MIT) — SCRAPER category for
 * the HTTP-client block, AI_CRAWLER for LLM, SOCIAL for social previews,
 * feed fetchers cherry-picked from its SEARCH_ENGINE category — plus
 * monperrus/crawler-user-agents (MIT). ^-anchors preserved where the source
 * list anchors (curl, Java/, PHP/, Ruby, Apache-HttpClient, HTTPie,
 * Insomnia, Postman).
 */
object UserAgentBucketer {
    private val patterns: List<Pair<Regex, UaClass>> = listOf(
<<ua-patterns()>>
    ).map { (pattern, cls) -> Regex(pattern) to cls }

    fun classify(userAgent: String?): UaClass {
        if (userAgent.isNullOrBlank()) return UaClass.BROWSER
        return patterns.firstOrNull { (regex, _) -> regex.containsMatchIn(userAgent) }?.second
            ?: UaClass.BROWSER
    }
}

Referer Classification

Small fixed enumeration, same shape as UserAgentBucketer:

kotlin#+name: referer-classifier
enum class RefererType { DIRECT, INTERNAL, SEARCH, SOCIAL, EXTERNAL }

object RefererClassifier {
    private val searchHosts = listOf(
        "google.", "bing.com", "duckduckgo.com", "kagi.com", "search.brave.com", "ecosia.org"
    )
    private val socialHosts = listOf(
        "twitter.com", "x.com", "t.co", "mastodon.", "fosstodon.org",
        "news.ycombinator.com", "reddit.com", "lobste.rs", "facebook.com",
        "linkedin.com", "bsky.app", "stribika"
    )

    fun classify(referer: String?, requestHost: String?): RefererType {
        if (referer.isNullOrBlank()) return RefererType.DIRECT
        val ref = referer.trim().lowercase()
        val host = requestHost?.substringBefore(":")?.trim()?.lowercase()
        if (host != null) {
            for (scheme in listOf("http://", "https://")) {
                if (ref == "$scheme$host/" || ref.startsWith("$scheme$host/")) return RefererType.INTERNAL
            }
        }
        if (searchHosts.any { it in ref }) return RefererType.SEARCH
        if (socialHosts.any { it in ref }) return RefererType.SOCIAL
        return RefererType.EXTERNAL
    }
}

Structured JSON Access Log

The single instrumentation point is the existing logRequest — it already runs at every terminal response (the only gap is /static/{path...}, which gains a call). It grows a startedAt parameter, computes the request's ua_class / referer_type, increments arcology_requests_total, and emits one JSON line. logError routes through SLF4J at warn with the same JSON shape.

kotlin#+name: access-log-event
/**
 * One JSON line per request under the "arcology.access" logger. The shape is
 * the log-aggregation contract: jq-able fields for site/path/status/duration
 * plus the classifier labels that mirror the Prometheus counter tags. The
 * raw [userAgent] rides along (null when the client sent none) so the
 * classifier's decision can be re-derived from the log when new agents show
 * up — the label stays tiny, the evidence stays queryable. [remoteHost] is
 * the TCP peer address and [forwardedFor] the raw =X-Forwarded-For= header
 * (null when absent): behind nginx the peer is the proxy, so the chain's
 * first hop is the client.
 */
@Serializable
data class AccessLogEvent(
    val timestamp: String,
    val method: String,
    val path: String,
    val host: String?,
    val site: String?,
    val status: Int,
    val durationMs: Long,
    val uaClass: String,
    val refererType: String,
    val remoteHost: String? = null,
    val forwardedFor: String? = null,
    val userAgent: String? = null,
    val extra: String = ""
)

object AccessLog {
    private val log = LoggerFactory.getLogger("arcology.access")
    private val json = Json { encodeDefaults = true }

    fun request(event: AccessLogEvent) {
        log.info(json.encodeToString(event))
    }

    fun error(uri: String, host: String?, message: String, throwable: Throwable?) {
        val stack = throwable?.let { "${it::class.simpleName}: ${it.message}" } ?: ""
        val event = AccessLogEvent(
            timestamp = java.time.Instant.now().toString(),
            method = "GET",
            path = uri,
            host = host,
            site = null,
            status = 500,
            durationMs = 0,
            uaClass = UaClass.BROWSER.name,
            refererType = RefererType.DIRECT.name,
            userAgent = null,
            extra = "error $message $stack".trim()
        )
        log.warn(json.encodeToString(event))
    }
}

The Json { encodeDefaults = true } instance and the ~@Serializable~-data-class-over-SLF4J pattern match how SitemapGraph.kt already serializes payloads.

Request Instrumentation and Path Labels

The counter lives in the instrumentation point described above; servePath / servePathLocalhost contribute the site and path label values via the server's existing context (they know both precisely, unlike a generic plugin hook).

*Path label normalization rules* (cardinality control, see Cardinality Notes):

  • The path label is the request path with the query string dropped.

  • /attachment/{hash}-{size}.{ext} collapses to /attachment — the 64-hex source hash is per-attachment cardinality and worthless as a label.

  • /{path...} unmatched by the route table (404s, scanner noise) → unknown.

  • Localhost mode labels path with the full SITE/path key, matching how localhost routes.

  • Fixed internal routes keep their literal shape: /, /sitemap, /sitemap.json, /tags, /tags/{tag}, /opml.xml, /health, /metrics, /sites.css, /404, /arcology/node/{nodeId} (internal-only, low traffic).

  • ua_class and referer_type come from the classifiers above; status is the response code.

kotlin#+name: path-normalization
/**
 * Normalizes a request path to a bounded-cardinality metric label value.
 * Per the design decision: real published paths get per-path granularity,
 * but the unbounded cases (attachment hashes, static-asset names, node-id
 * previews, scanner noise on the catch-all) collapse to fixed values so
 * crawlers cannot mint time series. The catch-all 404 branch passes
 * pathOverride="unknown" at the call site; this function handles the
 * path-shape rules for everything else.
 */
fun normalizePathLabel(path: String): String {
    val p = path.substringBefore('?').trim()
    if (p.isEmpty()) return "/"
    // attachment URLs carry a 64-hex source hash per variant — collapse them.
    if (p.startsWith("/attachment/")) return "/attachment"
    if (p.startsWith("/arcology/attachment/")) return "/arcology/attachment"
    // static assets and node previews carry per-file/per-node names
    if (p.startsWith("/static/")) return "/static"
    if (p.startsWith("/arcology/node/")) return "/arcology/node"
    return p
}

Cardinality Notes

Honest math for a personal-scale Prometheus: 10^2–10^3 real paths × 4 statuses × 5 ua classes × 5 referer types. Realistically only observed combinations mint series (a few thousand); the unbounded cases are exactly the ones the normalization rules collapse (unknown, /attachment). If the series count ever balloons, the escape hatch is demoting referer_type (derivable from logs) or bucketing long-tail paths into overflow behind a bounded cache — deferred until it's actually a problem.

Metrics Facade and Registry

ArcologyMetrics is a small facade so the server holds one nullable dependency instead of raw Micrometer calls at every site; null (the default) makes every operation a no-op, which keeps the four existing ArcologyServer construction sites (including three tests) working unchanged.

The request counter and page timer are looked up per request via registry.counter(name, tags) / registry.timer(name, tags) — Micrometer dedupes registered meters by name+tags, so no explicit per-tag registration is needed.

kotlin#+name: metrics-facade
/**
 * Histogram config shared by every Arcology timer: server-side histogram
 * buckets so quantiles aggregate in PromQL (client-side percentiles are a
 * known aggregation footgun). Applied per-meter via the builder API rather
 * than as a registry-wide MeterFilter — filters must precede meter
 * registration on the shared registry.
 */
fun timerBuilder(name: String, description: String): Timer.Builder =
    Timer.builder(name)
        .description(description)
        .publishPercentileHistogram()

/**
 * Arcology-specific meters on the shared Prometheus registry. Request
 * counting and page timing live here (called from the server's
 * instrumentation point); cache and node-render meters are helpers for the
 * servePath / SitemapGraphBuilder instrumentation.
 */
class ArcologyMetrics(val registry: PrometheusMeterRegistry) {

    fun incRequest(
        site: String?,
        path: String,
        status: Int,
        uaClass: UaClass,
        refererType: RefererType
    ) {
        Counter.builder("arcology_requests_total")
            .description("Requests served by the Arcology web server")
            .tag("site", site ?: "localhost")
            .tag("path", path)
            .tag("status", status.toString())
            .tag("ua_class", uaClass.name)
            .tag("referer_type", refererType.name)
            .register(registry)
            .increment()
    }

    fun pageTimer(site: String?, path: String): Timer =
        timerBuilder("arcology_request_duration_seconds", "Time to build/serve a page")
            .tag("site", site ?: "localhost")
            .tag("path", path)
            .register(registry)

    /**
     * Lazily registered on first use: the server module may install
     * Ktor's MicrometerMetrics plugin (which registers a registry-wide
     * MeterFilter) after this object is constructed, and filters must
     * precede meter registration.
     */
    val nodeRenderTimer: Timer by lazy {
        timerBuilder("arcology_node_render_duration_seconds", "Time to parse + render one node body (cache miss)")
            .register(registry)
    }

    fun cacheHit(cache: String) {
        Counter.builder("arcology_cache_hits_total")
            .tag("cache", cache)
            .register(registry)
            .increment()
    }

    fun cacheMiss(cache: String) {
        Counter.builder("arcology_cache_misses_total")
            .tag("cache", cache)
            .register(registry)
            .increment()
    }
}

object MetricsConfiguration {
    /**
     * The shared, empty Prometheus registry. The JVM binders are bound by
     * Ktor's MicrometerMetrics plugin (its default =meterBinders= list)
     * during ~module()~ install — binding them here too would double-register
     * the same gauges (duplicate-registration warning, second copy dropped),
     * and binding anything here before the plugin installs would make the
     * plugin's registry-wide MeterFilter a late filter (which warns). The
     * ordering contract: bare registry → plugin install (binders + filter) →
     * any Arcology meters (all lazy/per-request).
     */
    fun registry(): PrometheusMeterRegistry =
        PrometheusMeterRegistry(PrometheusConfig.DEFAULT)
}

The Ktor plugin config (in server.org's module()) sets a percentilesHistogram(true) distributionStatisticConfig so Ktor's built-in request-timer quantiles aggregate in PromQL (client-side percentiles are a known aggregation footgun). Ordering is the whole game for the plugin's registry-wide MeterFilter: it registers at install time, so nothing may be registered on the registry before module() runs — MetricsConfiguration.registry is deliberately bare (the plugin's default binder list does the JVM binding) and every Arcology meter registers per-request or lazy (ArcologyMetrics.nodeRenderTimer).

Cache and Render Metrics

Exactly two hit/miss decision points exist for page HTML — the htmlCache.getPage branches in servePath / servePathLocalhost — plus the fast-path and locked-miss in SitemapGraphBuilder.json. Each increments arcology_cache_{hits,misses}_total with cache=page or cache=sitemap. The sidebar cache hit branch is a deliberate no-op today and stays uninstrumented.

The page-build timer wraps the whole servePath / servePathLocalhost call (label: site, path); arcology_node_render_duration_seconds wraps cache-miss renderNodeBody work so render-p90 distinguishes "slow render" from "cache hit serving".

Health Check

GET /health upgrades from a static OK to a real check:

  • Database: a lightweight read through the roam repository (getNodeAncestors on a sentinel ID returns a list — exercises the SQLDelight driver end-to-end without assuming any table content).

  • Cache dir: writable probe file, deleted after the check.

  • Org dir: exists and is a directory.

Response: 200 with {"status":"ok"} or 503 with {"status":"degraded","checks":{...}} — serialized with the same kotlinx-serialization instance as the access log. This gives the health alert teeth (a 503 fails an HTTP-level probe; up drops to 0).

kotlin#+name: health-check
/**
 * Health check probe results. The status strings are the JSON contract the
 * Grafana alert (up == 0 via HTTP probe) and any external monitor rely on.
 */
object HealthCheck {
    @Serializable
    data class Report(val status: String, val checks: Map<String, String>)

    fun check(
        repository: RoamRepository,
        cacheDir: java.io.File,
        orgDir: String
    ): Report {
        val checks = linkedMapOf<String, String>()
        checks["database"] = try {
            runBlocking { repository.getNodeAncestors("health-probe-sentinel") }
            "ok"
        } catch (e: Exception) {
            "fail: ${e::class.simpleName}: ${e.message}"
        }
        checks["cache_dir"] = try {
            val probe = java.io.File.createTempFile("health-probe", ".tmp", cacheDir)
            probe.writeText("ok")
            probe.delete()
            "ok"
        } catch (e: Exception) {
            "fail: ${e::class.simpleName}: ${e.message}"
        }
        checks["org_dir"] = run {
            val dir = java.io.File(orgDir)
            if (dir.exists() && dir.isDirectory) "ok"
            else "fail: missing or not a directory"
        }
        val degraded = checks.values.any { it != "ok" }
        return Report(
            status = if (degraded) "degraded" else "ok",
            checks = checks
        )
    }
}

The report is exposed by the GET /health handler in server.org: 200 + {"status":"ok"} on health, 503 + full report on degradation.

server.org Edits

  1. install(MicrometerMetrics) in module() with the shared PrometheusMeterRegistry and a percentilesHistogram(true) distributionStatisticConfig. The plugin binds the JVM binders (its default list — MetricsConfiguration.registry is deliberately bare so binders aren't double-registered) and registers its MeterFilter before any meter exists: MetricsConfiguration.registry is empty and every Arcology meter is per-request or lazy.

  2. New route GET /metricscall.respondText(prometheusRegistry.scrape()).

  3. Add "metrics" to the /{path...} catch-all exclusion list.

  4. Every route handler captures val start = TimeSource.Monotonic.markNow() at entry; the terminal logRequest call becomes logRequest(start, method, uri, host, status, extra), which increments the counter, records the timer, and emits the JSON access line. The ~~/static~ route gains its missing logging call.

  5. servePath / servePathLocalhost: page timer around the call body, cache hit/miss counters in the getPage branches, node-render timer around cache-miss renderNodeBody.

  6. ArcologyServer gains metrics: ArcologyMetrics? = null (default keeps the three test construction sites compiling).

  7. ServeCommand: construct the PrometheusMeterRegistry + ArcologyMetrics, wire /metrics.

Grafana Dashboard

web/dashboard.json, provisioned per the layout in index.org's Phase 7:

Row Panels
1 counts served per site since restart; counts in last 24h; site hits per minute
2 "human" per-page hits (ua_class="BROWSER"); "bot" per-page hits (ua_class!="BROWSER"); feed-reader hits
3 page render time histogram; render quantiles p25/p50/p75/p95/p99/max
4 memory usage; CPU time; cache hit ratio (derived hits / (hits + misses)); GC pause, thread count

Alert Rules

  1. Anti-viral: a page's 5-minute request rate exceeds 3× its trailing-24h hourly average (non-bot, non-library traffic — humans and their feed readers only): #+begin_src promql rate(arcology_requests_total{ua_class!~"BOT|LLM|HTTP_CLIENT"}[5m]) > 3 avg_over_time(rate(arcology_requests_total{ua_class!~"BOT|LLM|HTTP_CLIENT"}[1h])[24h:1h]) #+end_src 2. Render p90: page build p90 over 5 minutes above 2s: #+begin_src promql histogram_quantile(0.9, rate(arcology_request_duration_seconds_bucket[5m])) > 2.0 #+end_src 3. Health*: up{job="arcology"} == 0 for > 1m (the 503-on-degraded health check makes an HTTP probe fail honestly).

Tests

All in this file, following the project's table-driven / fake-repository patterns:

  • UserAgentBucketerTest — known bots, feed readers, LLM crawlers, HTTP client libraries (curl/wget/requests/aiohttp/Go/Java/okhttp/axios/Postman…), social previews, browsers; empty/blank UA; the generic (?i)bot catch-all; curl-impersonation stays BROWSER.

  • RefererClassifierTest — direct, internal (scheme + host matching incl. port stripping), search, social, external.

  • AccessLogEventTest — JSON round-trip, encodeDefaults shape, field presence, raw user-agent round-trip, remote-host / X-Forwarded-For round-trip.

  • ArcologyMetricsTest — counters/timers registered with expected tag sets; scrape output contains arcology_requests_total and jvm_memory_used_bytes in valid Prometheus text format.

  • Path-label normalization tests — attachment names collapse to /attachment, query strings dropped, unmatched → unknown.

Run with make test-only class=... against the root :jvmTest.

Classifier tests

kotlin#+name: metrics-test-prelude
package computer.whatthefuck.arcology.publishing

import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
kotlin#+name: metrics-test:noweb-ref metrics-test
class UserAgentBucketerTest {
    @Test
    fun `classifies known LLM crawlers`() {
        assertEquals(UaClass.LLM, UserAgentBucketer.classify("GPTBot/1.0 (+https://openai.com/gptbot)"))
        assertEquals(UaClass.LLM, UserAgentBucketer.classify("ClaudeBot/1.0"))
        assertEquals(UaClass.LLM, UserAgentBucketer.classify("Mozilla/5.0 (compatible; PerplexityBot/1.0)"))
        assertEquals(UaClass.LLM, UserAgentBucketer.classify("CCBot/2.0"))
        assertEquals(UaClass.LLM, UserAgentBucketer.classify("meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"))
    }

    @Test
    fun `classifies http client libraries and cli tools as HTTP_CLIENT`() {
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("curl/8.5.0"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Wget/1.21.4"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("python-requests/2.31.0"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Python/3.12 aiohttp/3.9.1"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("python-httpx/0.27.0"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Go-http-client/2.0"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Java/17.0.1"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Apache-HttpClient/4.5.13 (Java/17.0.1)"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("okhttp/4.11.0"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("axios/1.6.0"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("PostmanRuntime/7.36.0"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("HTTPie/3.2.2"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("Scrapy/2.11 (+https://scrapy.org)"))
        assertEquals(UaClass.HTTP_CLIENT, UserAgentBucketer.classify("HTTrack/3.0"))
    }

    @Test
    fun `curl-impersonating browsers stay BROWSER`() {
        // curl-impersonate spoofs real browser UA strings; a UA that contains
        // no library token is a browser by classification.
        assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36"
        ))
    }

    @Test
    fun `classifies known feed readers`() {
        assertEquals(UaClass.FEED, UserAgentBucketer.classify("Feedly/1.0 (+http://www.feedly.com/fetcher.html)"))
        assertEquals(UaClass.FEED, UserAgentBucketer.classify("Miniflux/2.2.0"))
        assertEquals(UaClass.FEED, UserAgentBucketer.classify("FreshRSS/1.20 (Linux; https://freshrss.org)"))
        assertEquals(UaClass.FEED, UserAgentBucketer.classify("NetNewsWire/6.1 (Mac)"))
    }

    @Test
    fun `classifies known search bots`() {
        assertEquals(UaClass.BOT, UserAgentBucketer.classify("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"))
        assertEquals(UaClass.BOT, UserAgentBucketer.classify("bingbot/2.0"))
        assertEquals(UaClass.BOT, UserAgentBucketer.classify("Mozilla/5.0 (compatible; YandexBot/3.0)"))
        assertEquals(UaClass.BOT, UserAgentBucketer.classify("Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"))
        assertEquals(UaClass.BOT, UserAgentBucketer.classify("HeadlessChrome/126.0"))
    }

    @Test
    fun `classifies social preview bots`() {
        assertEquals(UaClass.SOCIAL, UserAgentBucketer.classify("facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"))
        assertEquals(UaClass.SOCIAL, UserAgentBucketer.classify("Twitterbot/1.0"))
        assertEquals(UaClass.SOCIAL, UserAgentBucketer.classify("Slackbot-LinkExpanding 1.0"))
    }

    @Test
    fun `generic bot marker catches unknown crawlers`() {
        assertEquals(UaClass.BOT, UserAgentBucketer.classify("some-new-crawler/1.0 (spider)"))
        assertEquals(UaClass.BOT, UserAgentBucketer.classify("MysteryBot/2.0"))
    }

    @Test
    fun `browsers and unknown agents classify as BROWSER`() {
        assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(null))
        assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(""))
        assertEquals(UaClass.BROWSER, UserAgentBucketer.classify("   "))
        assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(
            "Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0"
        ))
        assertEquals(UaClass.BROWSER, UserAgentBucketer.classify(
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 Safari/605.1.15"
        ))
    }
}
kotlin#+name: metrics-test:noweb-ref metrics-test
class RefererClassifierTest {
    @Test
    fun `null blank referer is DIRECT`() {
        assertEquals(RefererType.DIRECT, RefererClassifier.classify(null, "example.com"))
        assertEquals(RefererType.DIRECT, RefererClassifier.classify("", "example.com"))
        assertEquals(RefererType.DIRECT, RefererClassifier.classify("   ", "example.com"))
    }

    @Test
    fun `same host with either scheme is INTERNAL`() {
        assertEquals(RefererType.INTERNAL, RefererClassifier.classify("https://example.com/some/page", "example.com"))
        assertEquals(RefererType.INTERNAL, RefererClassifier.classify("http://example.com/", "example.com"))
        // request host with a port is stripped
        assertEquals(RefererType.INTERNAL, RefererClassifier.classify("https://example.com/x", "example.com:8080"))
    }

    @Test
    fun `search engines classify as SEARCH`() {
        assertEquals(RefererType.SEARCH, RefererClassifier.classify("https://www.google.com/search?q=x", "example.com"))
        assertEquals(RefererType.SEARCH, RefererClassifier.classify("https://duckduckgo.com/?q=x", "example.com"))
        assertEquals(RefererType.SEARCH, RefererClassifier.classify("https://kagi.com/search?q=x", "example.com"))
    }

    @Test
    fun `social platforms classify as SOCIAL`() {
        assertEquals(RefererType.SOCIAL, RefererClassifier.classify("https://news.ycombinator.com/item?id=1", "example.com"))
        assertEquals(RefererType.SOCIAL, RefererClassifier.classify("https://t.co/abc", "example.com"))
        assertEquals(RefererType.SOCIAL, RefererClassifier.classify("https://bsky.app/profile/x", "example.com"))
    }

    @Test
    fun `other origins classify as EXTERNAL`() {
        assertEquals(RefererType.EXTERNAL, RefererClassifier.classify("https://someblog.example.net/links", "example.com"))
        // a different scheme that isn't a known host
        assertEquals(RefererType.EXTERNAL, RefererClassifier.classify("android-app://com.someapp", "example.com"))
    }
}

Path normalization and access log tests

kotlin#+name: metrics-test:noweb-ref metrics-test
class NormalizePathLabelTest {
    @Test
    fun `attachment urls collapse to fixed label`() {
        val hash = "a".repeat(64)
        assertEquals("/attachment", normalizePathLabel("/attachment/$hash-1080.jpg"))
        assertEquals("/attachment", normalizePathLabel("/attachment/$hash-0.pdf"))
        assertEquals("/arcology/attachment", normalizePathLabel("/arcology/attachment/$hash-2160.jpg/html"))
    }

    @Test
    fun `static and node-preview urls collapse to fixed labels`() {
        assertEquals("/static", normalizePathLabel("/static/sitemap/sigma.min.js"))
        assertEquals("/arcology/node", normalizePathLabel("/arcology/node/some-node-id"))
    }

    @Test
    fun `query strings are dropped`() {
        assertEquals("/404", normalizePathLabel("/404?node=some-id"))
    }

    @Test
    fun `empty path normalizes to slash`() {
        assertEquals("/", normalizePathLabel(""))
        assertEquals("/", normalizePathLabel("?foo=bar"))
    }

    @Test
    fun `real paths pass through unchanged`() {
        assertEquals("/garden/index", normalizePathLabel("/garden/index"))
        assertEquals("/tags/tech", normalizePathLabel("/tags/tech"))
    }
}

class AccessLogEventTest {
    @Test
    fun `access log event serializes to compact JSON`() {
        val event = AccessLogEvent(
            timestamp = "2026-09-14T00:00:00Z",
            method = "GET",
            path = "/garden/index",
            host = "arcology.garden",
            site = "garden",
            status = 200,
            durationMs = 12,
            uaClass = "BROWSER",
            refererType = "DIRECT"
        )
        val json = Json { encodeDefaults = true }.encodeToString(event)
        assertTrue(json.contains("\"status\":200"))
        assertTrue(json.contains("\"uaClass\":\"BROWSER\""))
        assertTrue(json.contains("\"site\":\"garden\""))
        // extra has a default: encoded (encodeDefaults) with empty string
        assertTrue(json.contains("\"extra\":\"\""))
        // userAgent defaults to null and is encoded as such
        assertTrue(json.contains("\"userAgent\":null"))
    }

    @Test
    fun `raw user agent rides along in the log event`() {
        val ua = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
        val event = AccessLogEvent(
            timestamp = "2026-09-14T00:00:00Z",
            method = "GET",
            path = "/garden/index",
            host = "arcology.garden",
            site = "garden",
            status = 200,
            durationMs = 12,
            uaClass = "BOT",
            refererType = "DIRECT",
            userAgent = ua
        )
        val json = Json { encodeDefaults = true }.encodeToString(event)
        assertTrue(json.contains("\"uaClass\":\"BOT\""))
        assertTrue(json.contains("\"userAgent\":\"$ua\""))
    }

    @Test
    fun `remote host and forwarded-for ride along in the log event`() {
        val event = AccessLogEvent(
            timestamp = "2026-09-14T00:00:00Z",
            method = "GET",
            path = "/garden/index",
            host = "arcology.garden",
            site = "garden",
            status = 200,
            durationMs = 12,
            uaClass = "BROWSER",
            refererType = "DIRECT",
            remoteHost = "127.0.0.1",
            forwardedFor = "203.0.113.7, 10.0.0.2"
        )
        val json = Json { encodeDefaults = true }.encodeToString(event)
        // Behind nginx the peer is the proxy; the XFF chain's first hop is
        // the client. Both fields present in the JSON contract.
        assertTrue(json.contains("\"remoteHost\":\"127.0.0.1\""))
        assertTrue(json.contains("\"forwardedFor\":\"203.0.113.7, 10.0.0.2\""))
    }
}

Metrics facade and health check tests

kotlin#+name: metrics-test:noweb-ref metrics-test
class ArcologyMetricsTest {
    @Test
    fun `request counter and page timer register with expected tags`() {
        val metrics = ArcologyMetrics(MetricsConfiguration.registry())
        metrics.incRequest("garden", "/index", 200, UaClass.BROWSER, RefererType.DIRECT)
        metrics.incRequest("garden", "/index", 404, UaClass.BOT, RefererType.EXTERNAL)
        metrics.pageTimer("garden", "/index").record(java.time.Duration.ofMillis(5))

        val scrape = metrics.registry.scrape()
        assertTrue(scrape.contains("arcology_requests_total"))
        assertTrue(scrape.contains("site=\"garden\""))
        assertTrue(scrape.contains("path=\"/index\""))
        assertTrue(scrape.contains("ua_class=\"BROWSER\""))
        assertTrue(scrape.contains("ua_class=\"BOT\""))
        assertTrue(scrape.contains("referer_type=\"DIRECT\""))
        assertTrue(scrape.contains("arcology_request_duration_seconds"))
    }

    @Test
    fun `cache counters and node render timer register`() {
        val metrics = ArcologyMetrics(MetricsConfiguration.registry())
        metrics.cacheHit("page")
        metrics.cacheMiss("page")
        metrics.cacheHit("sitemap")
        metrics.nodeRenderTimer.record(java.time.Duration.ofMillis(5))

        val scrape = metrics.registry.scrape()
        assertTrue(scrape.contains("arcology_cache_hits_total{cache=\"page\"}"))
        assertTrue(scrape.contains("arcology_cache_misses_total{cache=\"page\"}"))
        assertTrue(scrape.contains("cache=\"sitemap\""))
        assertTrue(scrape.contains("arcology_node_render_duration_seconds"))
    }

    @Test
    fun `registry exposes JVM binder metrics`() {
        // MetricsConfiguration.registry() is deliberately bare — the JVM
        // binders are the Ktor plugin's default meterBinders at install
        // time (see MetricsConfiguration docs). Binding the same set here
        // verifies the binder classes still expose the dashboard metrics.
        val registry = MetricsConfiguration.registry()
        listOf(
            io.micrometer.core.instrument.binder.jvm.JvmMemoryMetrics(),
            io.micrometer.core.instrument.binder.jvm.JvmGcMetrics(),
            io.micrometer.core.instrument.binder.system.ProcessorMetrics(),
            io.micrometer.core.instrument.binder.jvm.JvmThreadMetrics(),
            io.micrometer.core.instrument.binder.system.UptimeMetrics()
        ).forEach { it.bindTo(registry) }
        val scrape = registry.scrape()
        assertTrue(scrape.contains("jvm_memory_used_bytes"))
        assertTrue(scrape.contains("process_cpu_usage"))
        assertTrue(scrape.contains("process_uptime_seconds"))
    }
}

class HealthCheckTest {
    private val healthyRepo = EmptyRoamRepository()

    @Test
    fun `all checks pass on healthy inputs`() {
        val cacheDir = java.nio.file.Files.createTempDirectory("health-cache").toFile()
        val orgDir = java.nio.file.Files.createTempDirectory("health-org").toFile()
        val report = HealthCheck.check(healthyRepo, cacheDir, orgDir.absolutePath)
        assertEquals("ok", report.status)
        assertEquals("ok", report.checks["database"])
        assertEquals("ok", report.checks["cache_dir"])
        assertEquals("ok", report.checks["org_dir"])
    }

    @Test
    fun `missing org dir degrades the report`() {
        val cacheDir = java.nio.file.Files.createTempDirectory("health-cache").toFile()
        val report = HealthCheck.check(healthyRepo, cacheDir, "/does/not/exist")
        assertEquals("degraded", report.status)
        assertTrue(report.checks["org_dir"]!!.startsWith("fail"))
    }

    @Test
    fun `failing database degrades the report`() {
        val cacheDir = java.nio.file.Files.createTempDirectory("health-cache").toFile()
        val orgDir = java.nio.file.Files.createTempDirectory("health-org").toFile()
        val brokenRepo = object : EmptyRoamRepository() {
            override suspend fun getNodeAncestors(nodeId: String): List<String> {
                throw IllegalStateException("database is closed")
            }
        }
        val report = HealthCheck.check(brokenRepo, cacheDir, orgDir.absolutePath)
        assertEquals("degraded", report.status)
        assertTrue(report.checks["database"]!!.startsWith("fail"))
    }
}
kotlin#+name: metrics-test-end

Implementation Order

  1. TOML + build.gradle.kts deps → make update-depsnix-build. (DONE: micrometer 1.17.0, ktor-server-metrics-micrometer, make update-deps + nix-build passed.)

  2. web/metrics.org classifiers + logger + facade + tests; tangle.

  3. web/server.org edits; tangle; make build && make test.

  4. web/dashboard.json + alert rules.

  5. Update index.org status marks.

Tangle Targets

Metrics.kt

kotlin#+name: metrics-assembly:tangle ../src/jvmMain/kotlin/computer/whatthefuck/arcology/publishing/Metrics.kt:noweb yes
<<metrics-preamble>>

<<ua-bucketer>>

<<referer-classifier>>

<<access-log-event>>

<<path-normalization>>

<<metrics-facade>>

<<health-check>>

MetricsTest.kt

kotlin#+name: metrics-test-assembly:tangle ../src/jvmTest/kotlin/computer/whatthefuck/arcology/publishing/MetricsTest.kt:noweb yes
<<metrics-test-prelude>>

<<metrics-test>>

<<metrics-test-end>>

Related Modules