Arcology Engine

The Arcology Hypermedia Publishing Platform

Contents

Introduction

The Arcology web publishing system is a "sync-to-publish" platform that serves org-roam content on the web. Files published locally appear on the web via Syncthing without git commits or manual deployment.

Core Principles:

  • Publish org-mode headings with ARCOLOGY_KEY property to web URLs

  • Multiple headings can publish to the same URL (e.g., multiple journal entries tagged to arcology/journal appear in order there)

  • Headings expire based on ARCOLOGY_EXPIRE property

  • RSS/Atom feeds aggregate entries by path

  • Cross-domain link rewriting for multi-site publishing

Publishing Model

Route Table Model

  • Single route table mapping paths to node IDs

  • Multiple headings can share the same path

  • Multiple entries rendered on the page at same heading level, sorted by timestamp (from ID or SCHEDULED)

  • Expired entries excluded from route table lookups

Publishing Key Format

  • Format: SITE/rest/of/path

  • Example: lionsrear/index → thelionsrear.com/index

  • Example: blog/2025/post → blog.example.com/2025/post

  • SITE prefix stripped for URL path, except localhost still exposes entire paths:

  • Example: blog/2025/post → localhost/blog/2025/post

Property System

  • File-level properties (=#+ARCOLOGY_:=) - apply to file as default, deprecated existing data format. - Heading-level properties (=:ARCOLOGY_:=) - override file defaults

  • Publish control via tags: :publish:, :ARCHIVE:

Web Server (Ktor)

Routes

Method Path Description
GET / Sitemap index
GET /sitemap SigmaJS graph of published pages
GET /sitemap.json Graph data (nodes + edges)
GET /tags Tag index
GET /tags/{tag} Pages with a tag (HTMX partial)
GET /sitemap.xml All published paths
GET /{path} Published page (list or single)
GET /{path}#{nodeId} Specific node
GET /{path}.xml Atom feed (FeedPublisher)
GET /opml.xml OPML list of the site's feeds
GET /arcology/node/{nodeId} Internal preview (unpublished)
GET /health Health check
GET /robots.txt Robot exclusion

HTMX Progressive Enhancement

  • Base HTML is full page render

  • HTMX endpoints for dynamic content:

    • =hx-get="/arcology/node/{id}/children"= - Load children on demand

    • =hx-get="/arcology/node/{id}/parent"= - Load parent on demand (navigate "up" in a document)

    • =hx-get="/arcology/node/{id}/backlinks"= - Load backlinks

    • =hx-post="/arcology/comment"= - Submit comment

Feature Implementations

MVP:

Phase 1: Core Publishing

Phase 1 is implemented across four literate files plus a prerequisite fix to the orgmode-kmp parser library:

  • publishing.org — Publishing.sq schema, PublishKey, DomainMap, RouteTable, RouteTableBuilder, PublishingRepository, ArcologyPublishingPlugin

  • renderer.org — OrgHtmlRenderer (AST → HTML string builder), LinkResolver

  • server.org — HtmlCache, ArcologyServer (Ktor), ServeCommand (CLI)

  • domains.org — domain map configuration table + lua :eval arroyo generator

DONE RouteTable data structures

DONE DomainMap for SITE → domain mapping

DONE Database queries for publishing nodes

DONE PublishKey parsing (SITE/path)

DONE RouteTableBuilder with timestamp sorting

DONE HtmlCache with hash-based storage

DONE Basic Renderer (node → HTML)

DONE Ktor server skeleton with routes

DONE Domain map configuration

DONE 404 handling for unknown sites

Phase 3: Advanced Rendering

Phase 3 is implemented across five literate files:

  • templates.org — Pebble templates (app.peb, page.peb, wide.peb, sidebar.peb, 404.peb, sitemap.peb, feed.peb, robots.peb) + arcology.css base stylesheet

  • server.org — PageModel, SidebarService, HtmlCache.putSidebar, ArcologyServer Pebble install, serveSitesCss, renderNodeBody rewrite

  • renderer.org — footnote → sidenote rendering, collectHeadings() TOC accumulator

  • publishing.org — DomainMap JSON rewrite with SiteMeta (title, cssFile, linkColor)

  • domains.org — single site-meta table (SITE, Title, Css File, Link Color, Domains) + lua → domains.json

DONE Mustache template engine → Pebble (decision resolved)

DONE Template inheritance → Pebble {% extends %}/{% block %}

DONE Partial templates → Pebble {% include %}

DONE Configuration options → ARCOLOGY_PAGE_TEMPLATE + site meta table

DONE Custom template per node → ARCOLOGY_PAGE_TEMPLATE heading/file property

DONE phase 3 feedback

  • DONE inline elements aren't rendered correctly, no bold, verbatim, code... → regex post-processing in OrgHtmlRenderer (InlineMarkupPatterns ported from app)

  • DONE sidebar links don't route to the correct domain → SidebarService takes currentSite, builds absolute https://<domain>/<path> for cross-site backlinks

  • DONE nodes don't render their sub-headings or sub-nodes and oughta → file-level nodes use renderDocument (renders preface + all top-level sections); heading-level nodes recurse via renderChunks(section.body). Published sub-headings get =class="published-node"= on <hN> (currently unstyled, hook for future CSS).

  • DONE / on a non-localhost domain in the domain map should open SITE/index not the auto-generated sitemap → GET / checks host → servePath("index", site), falls back to sitemap if no route

  • DONE begin_html blocks don't export the raw html → parseHTMLBlock implemented in orgmode-kmp/OrgBlock.kt (was commented out); OrgHTMLBlock data class already existed; renderer already passed chunk.body through

  • DONE inline html tags should work as well: bold text → regex in applyInlineMarkup extracts =...= and passes raw (org text between bookends still gets markup)

  • DONE ROAM_REFS external references just show up as <a target="_blank" href="http">http</a> → SidebarService was taking r.second (type) not r.first (URL); fixed

Phase 4: Attachment Handling

Phase 4 is implemented across the existing literate files:

  • publishing.org — published_attachments schema (denormalized: one row per node/source-hash/size, with type as the extension for direct URL mapping), PublishedAttachment model, PublishingRepository attachment queries, ArcologyAttachmentPlugin indexer plugin, AttachmentCrusher (ImageMagick magick subprocess, two sizes 512/2048, verbatim-copy fallback)

  • server.org — GET /attachment/{hash}-{size}.{ext} (content-addressed, strong ETag, immutable), GET /arcology/attachment/{name}/html HTMX fragment, ConditionalHeaders plugin, weak =ETag=s on pages, CrossDomainLinkResolver.resolveAttachment

  • renderer.org — LinkResolver.resolveAttachment + AttachmentRef, file: / attachment: image links render <img> with =hx-get=/hx-trigger="load"/hx-swap="outerHTML"=

  • templates.org — htmx.min.js static asset + script include in app.peb

The attachment cache dir defaults to $ARCOLOGY_ATTACHMENT_DIR or /tmp/arcology-cache/attachments (matching arcology-django's env var); both the indexer (FlowFileIndexerFactory) and the server (ServeCommand) read it.

Org-attach-dir resolution reuses AttachmentResolver's probing of the org-attach-id-to-path-function-list formats: the timestamp format data/{id[0..5]}/{id[6..]} (org-roam temporally-unique IDs) is tried first, then the UUID format data/{id[0..1]}/{id[2..]}.

DONE Database cache of all attachments in indexer plugin

  • a heading's attachment dir if it has ATTACH tag, and dir exists. directory resolving that mimics as best as we can org-attach-dir function[fn:1].

  • attachment table w/ source_path, crushed_path, source_hash

DONE indexer plugin oughta also do the file crushing....

DONE follow arcology-django's arcology.models.Attachment basically

DONE ETag conditional get if-none-match etc

strong etags for the attachments based on the hash, and also weak etags for the html cache hash too

DONE HTMX progressive enhancement

load a small-sized image in the <img> tag but have hx enhancements to swap in a larger res (but still crushed) file

Phase 5: Feed Publishing

Phase 5 is implemented in a new literate file plus wiring in the existing ones. The semantic model: many files can publish to one feed, the same way many headings can publish to one page URL.

  • feeds.org — Feeds.sq schema (feeds declarer rows keyed (route_key, file), feed_entries), =FeedModel=/=FeedEntryModel=, FeedRepository, PubdateParser (org timestamp → RFC-3339 UTC), ArcologyFeedPlugin (indexer plugin), FeedPublisher (serve-time aggregation)

  • server.org — shared node-body pipeline extracted to NodeRendering.kt (loadNodeParseResult / renderParsedNode, reused by FeedPublisher for one-parse-N-renders), GET /{path}.xml branch in the catch-all route, PageModel.feeds populated per-site for =<link rel="alternate">= autodiscovery, ServeCommand wiring

  • templates.org — feed.peb filled in (published, =link rel="self"=)

  • FlowFileIndexerFactory — ArcologyFeedPlugin registered

DONE Feed data models

DONE Atom feed pebble template

The Django feed.xml template ported to Pebble; no Kotlin Atom DSL needed.

DONE FeedPublisher for aggregating entries

Property rollup at index time (PUBDATE headings join the feed scoped to themselves, nearest ancestor-or-self, else file-level feeds); path-aggregation at serve time (nodes publishing to the anchor page's ARCOLOGY_KEY path with a PUBDATE); direct subscription (any file mentioning the feed). Strict PUBDATE, EXCLUDE_TAGS filtering, dedupe by node ID, top 10 by pubdate.

DONE /{path}.xml routes

Django-faithful: the ARCOLOGY_FEED value's path is the literal URL path, not a /feed/ prefix.

DONE /opml.xml route

generate an OPML file of the given site's feeds

Phase 6: Syncthing client

Phase 6 is implemented in a new literate file in the roam cluster, since the watcher serves any consumer of the indexer, not just the web server:

  • roam/syncthing.org — SyncthingClient interface (commonMain, platform-split like the indexer), SyncthingWatchService (folder resolution, cursor-seeded long-poll of =GET /rest/events?events=ItemFinished=, delete-wins dedupe, event-gap → rescan, backoff on transient failures), KtorSyncthingClient (jvmMain), SyncCommand (the arcology sync CLI command), and SyncthingWatchServiceTest

  • roam/indexer.org / roam/models.org — DatabaseFactory now sets PRAGMA busy_timeout + WAL so the watch process can write while serve reads

  • app/cli.org — sync registered alongside serve

The arcology sync command finds the Syncthing folder matching --org-dir (or --folder override), runs one full indexing pass at startup (hash-skip makes it cheap), then long-polls ItemFinished events and re-indexes each changed org file via the indexer's single-file path — which runs every publishing plugin, so routes, feeds, and attachments stay current without restarts.

Phase 7: Metrics/monitoring

Implemented in Arcology Web Metrics and Monitoring plus wiring in server.org:

  • metrics.org — UserAgentBucketer (org table + lua :eval arroyo generator → ordered pattern list; six buckets BROWSER/FEED/HTTP_CLIENT/BOT/LLM/SOCIAL built from getarcis/well-known-bots + monperrus/crawler-user-agents), RefererClassifier (DIRECT/INTERNAL/SEARCH/SOCIAL/EXTERNAL), AccessLogEvent + AccessLog (structured JSON via kotlinx-serialization over SLF4J on the arcology.access logger, raw user-agent included), normalizePathLabel (per-path labels with attachment/static/node-preview collapse), ArcologyMetrics facade (arcology_requests_total, arcology_request_duration_seconds, arcology_node_render_duration_seconds, arcology_cache_{hits,misses}_total), MetricsConfiguration (Prometheus registry + JVM binders), HealthCheck (database cache dir org dir probe)

  • server.org — MicrometerMetrics plugin install with percentilesHistogram(true), GET /metrics (blocked publicly at nginx), upgraded GET health (200 ok 503 degraded JSON), logRequest now counts + times + emits the JSON access line, per-path page timers around servePath / servePathLocalhost, cache hit/miss counters, ServeCommand registry wiring, SitemapGraphBuilder sitemap-cache counters, OPML builder https fix

  • dashboard.json — Grafana dashboard (per-site counts, human/bot/feed per-page, render histogram + quantiles, memory/CPU/cache-ratio/GC) + anti-viral render-p90 health alerts

DONE prometheus /metrics endpoint

  • user-agent parser/bucketer

  • labels for site key, arcology_key, status, agent type, referer

  • timers and counters

DONE Health check endpoint

DONE Logging configuration

DONE grafana dashboards and alert templates

alerts:

  • "anti-viral" alert: a particular page has gone viral and driving more traffic to the site

  • render time p90

  • health check failures

dashboard:

  • row

    • counts served per site since last restart

    • counts served in last 24h

    • site hits per minute

  • row

    • "human" per-page hits

    • "bot" per-page hits

  • row

    • page render time histogram

    • page render quantile line graph (p25, p50, p95, p99, max)

  • row (process stats)

    • memory usage

    • cpu time

    • cache hit ratio

    • gc/java internals?

Phase 8: CCE deployment manifests

web/deployment.org:

NEXT nixosModule in flake

  • write ARROYO_NIXOS_MODULE that uses the flake.nix nixosModule and provides options to configure it

    • service which starts web server

    • service which starts the sync watcher

    • nginx virtualHost with proxyPass shit on the wobserver

    • re-add to cce-wobserver/nginx-common.org's hosts so that https virtual host works w/ acme

NEXT a ~/nix ARROYO_NIXOS_MODULE that uses the flake nixosModule and configures it

  • expose in flake.nix nixosModules

  • options for CLI options

  • domains.json file inclusion

  • secrets for syncthing api key

  • the ARROYO_HOME_MODULE that include arcology2 in PATH

  • the existing ARROYO_EMACS_MODULE init configuration from arroyo/emacs-integration.org

Phase 2

Phase 9: Comments

  • simply storing comments in the arcology.db will violate the "org-mode is the source of truth" rules.

  • writing arbitrary user content back to my org-directory seems insane

  • arcology-django had a "split path" where browser-content would go to its own "writable.sqlite" and the arcology.db could be blown away, consider something similar here.

NEXT Comment data model

NEXT Comment storage in org format

NEXT Comment service interface

NEXT Comment form with HTMX

NEXT Comment moderation UI

NEXT Email notification for new comments

NEXT ARCOLOGY_EXPIRE property parsing (heading or file property or keyword containing an org inactive timestamp)

NEXT RenderedPage with expiresAt field

NEXT Expired check middleware

NEXT Cache cleanup for expired entries

NEXT List pages update when items expire

Phase 11: Federation

NEXT Webmention sender (discovery + POST)

NEXT ActivityPub/Fediverse publisher

NEXT Webmention storage in org format

NEXT Reply handling

Polish

NEXT Concurrent rendering

NEXT Memory usage profiling

NEXT Investigate runBlocking proliferation in ArcologyServer

The per-request DB-lookup refactor (Arcology's Web Server) replaced the startup-built in-memory RouteTable with per-request PublishingRepository queries, fixing the "new route 404s until restart" bug. The trade-off is two new runBlocking { … } sites per request (one for the route lookup, one for the publishedIndex used by the resolver and sidebar). The existing renderNodeBody already blocks on getNodeById and a file read+reparse on cache miss, so the route SELECT is cheap by comparison and the pattern matches the rest of the server — but every runBlocking blocks a Ktor CIO dispatcher thread. For single-user traffic this is fine; if the server ever sees concurrent load, switch the handlers to suspend fun ApplicationCall handlers (Ktor's pipeline.invokeConvertSuspending / suspend fun ApplicationCall.process) and drop runBlocking entirely so the dispatcher threads stay cooperative under load. The query count per request is bounded (one getPublishedRoutesByPath or getRoutesByPath for the page, one getPublishedRoutes for the resolver+sidebar index), so the surface area for this conversion is small.

DONE Sitemap graph and tag index (SigmaJS)

Implemented in sitemap.org — a port of arcology-django's sitemap module: GET /sitemap renders a SigmaJS force-directed graph of all published pages (nodes colored by site linkColor, sized by link counts, seeded by a loc hash, laid out by ForceAtlas2 in a browser worker, with hover-neighbor highlighting and click-to-navigate), GET /sitemap.json serves the graphology-shaped JSON from a content-keyed in-memory cache with a strong ETag, and GET tags GET /tags/{tag} provide the HTMX tag index with tag-cloud font sizing. SigmaJS v2.3.1 and graphology 0.24.1 are vendored minified under static/sitemap/ — no Node build chain.

NEXT XML sitemap generation (sitemap.xml)

NEXT robots.txt generation

Technical Decisions

Rendering Strategy

  • Lazy rendering: render on-demand when content hash changes

  • Hash-based cache invalidation (same content = same hash)

  • This is not a "static site generator", it's more like a content-addressed store.

  • Simple file system cache, not SQLite BLOB, systemd.tmpfiles can handle cache eviction.

Property Inheritance

  • Heading properties override file properties

  • Tags control publish status (:publish:, :draft:, :archive:)

  • No separate ARCOLOGY_STATUS property needed

Timestamp Source

  • Primary: Timestamp in ID, lexically sorted, using temporally-unique IDs (e.g., 20251220T231404)

  • Fallback: SCHEDULED property

Expiry Behavior

  • Per-heading expiry via ARCOLOGY_EXPIRE

  • Expired entries show 404

  • List pages show fewer entries as items expire

  • RSS only includes unexpired entries

Cross-Domain Links

  • =xxx= rewritten to full URLs when linking to other sites

  • 404 page for unknown sites

  • Domain map configuration for SITE → domain mapping

Notes

Future Considerations

  • Webmention storage as org headings

  • ActivityPub federated threads

  • Email-to-publish workflow

  • Pull request style publishing (draft/review/published tags)

  • Interactive pages from org tables

  • Transclusion feeds

Known Gaps

  • Comment moderation workflow

  • Webmention discovery mechanism

  • ActivityPub implementation details

Footnotes

[fn:1] /nix/store/*-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.8.8/org-attach.el

Contents
Pages Linking Here
Page Tags