Arcology Engine

Arcology Web Publishing Layer

Contents

Introduction

This document is the source of truth for the Arcology web publishing data layer. It defines the database schema, domain models, indexer plugin, and route table builder that turn org-roam metadata into a publishable route table.

The publishing layer is a plugin to the The Indexer Pipeline (responsible for parsing org files and building the database) and the The Web Server (Which renders Hypertext to Hyper Text Markup Language and delivers it over Hyper Text Transfer Protocol)

The publishing model is simple:

  • Every heading or file with an ARCOLOGY_KEY property publishes to a matching URL path

  • The key format is SITE/rest/of/path — the SITE prefix determines which domain serves it

  • Multiple headings can share the same path (e.g., multiple journal entries tagged to arcology/journal)

  • Entries on the same path are sorted by timestamp (from the ID or SCHEDULED)

  • Tags control visibility: :publish: (default), :draft: (hidden), :archive: (hidden)

An ArcologyPublishingPlugin runs at index time, extracts ARCOLOGY_KEY from the parsed document, and populates the published_routes table. The web server queries this table to build a route table at startup. Its sibling ArcologyAttachmentPlugin crushes and indexes attachments for =:ATTACH:=-tagged nodes into published_attachments.

Database Schema

The published_routes table is a denormalized view of publishing metadata, populated by the indexer plugin. It joins node IDs to their SITE/path keys, file paths, titles, timestamps, and publish status.

sql#+name: publishing-schema
-- Publishing: route table for web publishing
-- Populated by ArcologyPublishingPlugin at index time.
-- Each row represents one heading or file-level node that has an ARCOLOGY_KEY.
CREATE TABLE IF NOT EXISTS published_routes (
    node_id TEXT NOT NULL,
    site TEXT NOT NULL,
    path TEXT NOT NULL,
    file TEXT NOT NULL,
    title TEXT,
    timestamp TEXT,
    is_draft INTEGER NOT NULL DEFAULT 0,
    is_archived INTEGER NOT NULL DEFAULT 0,
    PRIMARY KEY (node_id, site, path),
    FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);

CREATE INDEX IF NOT EXISTS idx_published_routes_path ON published_routes (path);
CREATE INDEX IF NOT EXISTS idx_published_routes_site ON published_routes (site);
sql#+name: publishing-queries
selectAllPublishedRoutes:
SELECT * FROM published_routes ORDER BY path, timestamp;

selectRoutesByPath:
SELECT * FROM published_routes WHERE path = ? ORDER BY timestamp;

selectRoutesBySite:
SELECT * FROM published_routes WHERE site = ? ORDER BY path, timestamp;

selectPublishedRoutes:
SELECT * FROM published_routes WHERE is_draft = 0 AND is_archived = 0 ORDER BY path, timestamp;

selectPublishedRoutesByPath:
SELECT * FROM published_routes WHERE path = ? AND is_draft = 0 AND is_archived = 0 ORDER BY timestamp;

insertPublishedRoute:
INSERT OR REPLACE INTO published_routes (node_id, site, path, file, title, timestamp, is_draft, is_archived)
VALUES (?, ?, ?, ?, ?, ?, ?, ?);

deletePublishedRoutesByFile:
DELETE FROM published_routes WHERE file = ?;

countPublishedRoutes:
SELECT COUNT(*) FROM published_routes;

Attachment Cache Schema

Crushed attachments are cached on disk, content-addressed by the SHA-256 of the original file's bytes. The published_attachments table is the denormalized index of that cache: one row per (node, attachment, size variant), mirroring arcology.models.Attachment from arcology-django but with the per-size rows exploded out so that serving is a pure function of the URL — /attachment/{source_hash}-{size}.{type} — with no database lookup on the hot path.

  • size is the crush target in pixels (512, 2048) for crushable images, or 0 for verbatim copies of uncrushable files.

  • type is the lowercased file extension without the dot; it drives the URL suffix and the served content type.

  • source_path is the attachment's path relative to the org root (e.g. data/202607/xxxx/foo.jpg), kept so file: and attachment: links can be resolved back to a row.

  • file is the source org file, mirroring published_routes, so rows are replaced wholesale on re-index and dropped when the file is removed.

sql#+name: attachments-schema
-- Publishing: crushed attachment cache index.
-- Populated by ArcologyAttachmentPlugin at index time. One row per size variant.
-- Crushed files live on disk at {ATTACHMENT_DIR}/{source_hash}-{size}.{type}.
CREATE TABLE IF NOT EXISTS published_attachments (
    node_id TEXT NOT NULL,
    source_path TEXT NOT NULL,
    source_hash TEXT NOT NULL,
    size INTEGER NOT NULL,
    type TEXT NOT NULL,
    file TEXT NOT NULL,
    PRIMARY KEY (node_id, source_hash, size)
);

CREATE INDEX IF NOT EXISTS idx_published_attachments_node ON published_attachments (node_id);
CREATE INDEX IF NOT EXISTS idx_published_attachments_path ON published_attachments (source_path);
sql#+name: attachments-queries
selectPublishedAttachmentsByNode:
SELECT * FROM published_attachments WHERE node_id = ? ORDER BY source_path, size;

selectPublishedAttachmentsByPath:
SELECT * FROM published_attachments WHERE source_path = ?;

selectPublishedAttachmentsByBasename:
SELECT * FROM published_attachments WHERE source_path LIKE '%/' || ? ORDER BY source_path, size;

insertPublishedAttachment:
INSERT OR REPLACE INTO published_attachments (node_id, source_path, source_hash, size, type, file)
VALUES (?, ?, ?, ?, ?, ?);

deletePublishedAttachmentsByFile:
DELETE FROM published_attachments WHERE file = ?;

countPublishedAttachments:
SELECT COUNT(*) FROM published_attachments;

ArcologyPublishingPlugin

This indexer plugin extracts ARCOLOGY_KEY from parsed org files and populates the published_routes table. It follows the same pattern as ArroyoIndexerPlugin.

Extraction Sources

ARCOLOGY_KEY can appear in three places:

  1. Preamble keyword lines#+ARCOLOGY_KEY: site/path in the file preamble. These are file-scoped (assigned to the file-level node if it has an ID).

  2. Section body keyword lines#+ARCOLOGY_KEY: site/path under a heading. Also file-scoped in org semantics. Both preamble and section keyword lines are hoisted onto document.keywordLines by the parser, and exposed on ParseResult.Success via result.document?.keywordLines.

  3. Heading :PROPERTIES: drawers:ARCOLOGY_KEY: site/path in a heading's property drawer. These are heading-scoped (assigned to that heading's node ID).

Timestamp Extraction

The timestamp for sorting entries on the same path comes from:

  • Primary: The node's ID if it matches the org-roam format YYYYMMDDTHHMMSS (e.g., 20260710T100000)

  • Fallback: The node's SCHEDULED date

Tag-Based Visibility

Tags on the node control whether it appears in the published route table, using org-mode's existing export-control semantics:

  • :noexport: / :NOEXPORT: tag → excluded from public routes

  • :ARCHIVE: tag → excluded from public routes (org-mode's archive tag, uppercase)

  • All other tags (including :Archive:, :publish:, :draft:, etc.) → published (default)

The distinction between :ARCHIVE: (uppercase, org-mode's archive tag) and :Archive: (mixed case, personal knowledge archive) is intentional — :Archive: is a personal organization tag that does NOT control publishing. The exclude set is case-sensitive so that :Archive: survives.

The same exclude set is shared by three call sites so they cannot drift:

  1. ArcologyPublishingPlugin — drops the route from the route table at index time.

  2. SidebarService — drops backlinks and tag-cloud entries from the sidebar at request time.

  3. OrgHtmlRenderer — elides the heading and its sub-tree from rendered HTML at render time, so that non-node sub-headings carrying the tag are also suppressed (route-table filtering only catches headings that bear an ARCOLOGY_KEY).

Exclude tags constant

kotlin#+name: exclude-tags
package computer.whatthefuck.arcology.publishing

/**
 * Tags that suppress both route-table publishing and HTML rendering of a heading.
 *
 * Case-sensitive on purpose: `:ARCHIVE:` (org-mode's archive tag) is excluded,
 * while `:Archive:` (a personal knowledge-archive organization tag) is not.
 * Both lowercase `:noexport:` and uppercase `:NOEXPORT:` are excluded since
 * org-mode users write the tag either way and both should hide the heading.
 */
val EXCLUDE_TAGS: Set<String> = setOf("noexport", "NOEXPORT", "ARCHIVE")

Plugin class: declaration and helpers

kotlin#+name: plugin-preamble
package computer.whatthefuck.arcology.publishing

import computer.whatthefuck.arcology.domain.NodeProperty
import computer.whatthefuck.arcology.indexer.IndexerPlugin
import computer.whatthefuck.arcology.parser.ParseResult
import xyz.lepisma.orgmode.OrgChunk
kotlin#+name: plugin-core:noweb yes
class ArcologyPublishingPlugin(
    private val repository: PublishingRepository,
    private val debug: Boolean = false
) : IndexerPlugin {

    private data class PublishKeySource(
        val nodeId: String?,
        val rawKey: String,
        val source: String
    )

    <<plugin-onFileIndexed>>

    override suspend fun onFileRemoved(filePath: String) {
        repository.deleteRoutesByFile(filePath)
    }

    private fun extractTimestampFromId(nodeId: String): String? {
        val regex = Regex("^(\\d{8})T(\\d{6})")
        val match = regex.find(nodeId) ?: return null
        return match.groupValues[0]
    }
}

onFileIndexed: the extraction pipeline

The method has three phases: delete existing routes for the file, collect all ARCOLOGY_KEY sources, then resolve and insert each route.

kotlin#+name: plugin-onFileIndexed:noweb yes
override suspend fun onFileIndexed(result: ParseResult.Success) {
    val filePath = result.file.path
    <<plugin-phase-delete>>
    <<plugin-phase-collect>>
    <<plugin-phase-insert>>
}

Phase 1: Delete existing routes

kotlin#+name: plugin-phase-delete:noweb-ref plugin-phase-delete
repository.deleteRoutesByFile(filePath)

Phase 2: Collect ARCOLOGY_KEY sources and build lookups

Keyword lines (from preamble and sections) are assigned to the file-level node. Properties-drawer keys are assigned to their heading's node ID. We also build a node map and tag map for the resolution phase.

kotlin#+name: plugin-phase-collect:noweb-ref plugin-phase-collect
val allKeys = mutableListOf<PublishKeySource>()
val fileLevelNodeId = result.nodes.firstOrNull { it.level == 0 }?.id

// 1. Keyword lines — from result.document.keywordLines (hoisted by parser from preamble + preface + sections)
result.document?.keywordLines?.forEach { kwLine ->
    if (kwLine.keyword == "ARCOLOGY_KEY" && kwLine.value.isNotEmpty()) {
        allKeys.add(PublishKeySource(fileLevelNodeId, kwLine.value, "keyword-line"))
    }
}

// 2. :ARCOLOGY_KEY: in heading/file :PROPERTIES: drawers (from nodeProperties)
result.nodeProperties.forEach { prop ->
    if (prop.key == "ARCOLOGY_KEY" && !prop.value.isNullOrEmpty()) {
        allKeys.add(PublishKeySource(prop.nodeId, prop.value, "properties-drawer"))
    }
}

// Build a node lookup for tag and timestamp resolution
val nodeMap = result.nodes.associateBy { it.id }
val tagsByNode = result.tags.groupBy { it.nodeId }.mapValues { it.value.map { tag -> tag.tag } }

if (debug) {
    println("[ArcologyPublishingPlugin] $filePath: ${allKeys.size} ARCOLOGY_KEY sources found, ${result.document?.keywordLines?.size ?: 0} keyword lines total, ${result.nodes.size} nodes, fileLevelNodeId=$fileLevelNodeId")
}

Phase 3: Resolve and insert routes

For each key source: parse the key, resolve the node ID, check tags for archive exclusion, extract the timestamp, and insert the route entry.

kotlin#+name: plugin-phase-insert:noweb-ref plugin-phase-insert
var inserted = 0
for (source in allKeys) {
    val publishKey = PublishKey.parse(source.rawKey)
    if (publishKey == null) {
        if (debug) println("[ArcologyPublishingPlugin]   SKIP: invalid key '${source.rawKey}' (source=${source.source})")
        continue
    }
    val nodeId = source.nodeId ?: fileLevelNodeId
    if (nodeId == null) {
        if (debug) println("[ArcologyPublishingPlugin]   SKIP: no nodeId for key '${source.rawKey}' (source=${source.source})")
        continue
    }
    val node = nodeMap[nodeId]
    if (node == null) {
        if (debug) println("[ArcologyPublishingPlugin]   SKIP: node '$nodeId' not in nodeMap for key '${source.rawKey}' (source=${source.source})")
        continue
    }

    val tags = tagsByNode[nodeId] ?: emptyList()
    val isArchived = tags.any { it in EXCLUDE_TAGS }

    val timestamp = extractTimestampFromId(nodeId) ?: node.scheduled

    if (debug) {
        println("[ArcologyPublishingPlugin]   INSERT: site=${publishKey.site} path=${publishKey.path} nodeId=$nodeId title=${node.title} archived=$isArchived (source=${source.source})")
    }

    repository.insertRoute(RouteEntry(
        nodeId = nodeId,
        site = publishKey.site,
        path = publishKey.path,
        file = filePath,
        title = node.title,
        timestamp = timestamp,
        isDraft = false,
        isArchived = isArchived
    ))
    inserted++
}

if (debug && allKeys.isNotEmpty()) {
    println("[ArcologyPublishingPlugin] $filePath: inserted $inserted/${allKeys.size} routes")
}

Tests: ArcologyPublishingPlugin extraction, tags, and timestamps

kotlin#+name: publishing-plugin-test-prelude
package computer.whatthefuck.arcology.publishing

import computer.whatthefuck.arcology.domain.OrgFile
import computer.whatthefuck.arcology.domain.OrgNode
import computer.whatthefuck.arcology.domain.OrgTag
import computer.whatthefuck.arcology.domain.NodeProperty
import computer.whatthefuck.arcology.parser.ParseResult
import computer.whatthefuck.arcology.indexer.IndexerPlugin
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
import kotlinx.coroutines.runBlocking
import kotlin.time.Instant

class ArcologyPublishingPluginTest {

    private fun node(
        id: String, level: Int = 1, properties: Map<String, String> = emptyMap(),
        title: String? = "Test Node"
    ): OrgNode {
        return OrgNode(
            id = id, file = "test.org", level = level, position = 0,
            title = title, properties = properties
        )
    }

    private fun parseResultWith(
        nodes: List<OrgNode> = emptyList(),
        nodeProperties: List<NodeProperty> = emptyList(),
        tags: List<OrgTag> = emptyList()
    ): ParseResult.Success {
        return ParseResult.Success(
            file = OrgFile(path = "test.org", hash = "abc", accessTime = Instant.fromEpochSeconds(0), modificationTime = Instant.fromEpochSeconds(0)),
            nodes = nodes,
            links = emptyList(),
            tags = tags,
            aliases = emptyList(),
            refs = emptyList(),
            fileProperties = emptyList(),
            nodeProperties = nodeProperties,
            content = ""
        )
    }
kotlin#+name: publishing-plugin-test:noweb-ref publishing-plugin-test
    @Test
    fun `extracts ARCOLOGY_KEY from node properties`() = runBlocking {
        val plugin = ArcologyPublishingPlugin(RecordingRepository())
        val result = parseResultWith(
            nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "lionsrear/index"))),
            nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "lionsrear/index"))
        )
        val repo = RecordingRepository()
        ArcologyPublishingPlugin(repo).onFileIndexed(result)
        assertEquals(1, repo.inserted.size)
        val entry = repo.inserted.first()
        assertEquals("lionsrear", entry.site)
        assertEquals("index", entry.path)
        assertEquals("20260710T100000", entry.timestamp)
    }
kotlin#+name: publishing-plugin-test:noweb-ref publishing-plugin-test
    @Test
    fun `skips invalid publish keys`() = runBlocking {
        val result = parseResultWith(
            nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "no-slash"))),
            nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "no-slash"))
        )
        val repo = RecordingRepository()
        ArcologyPublishingPlugin(repo).onFileIndexed(result)
        assertEquals(0, repo.inserted.size)
    }
kotlin#+name: publishing-plugin-test:noweb-ref publishing-plugin-test
    @Test
    fun `draft tag does not exclude entries`() = runBlocking {
        val result = parseResultWith(
            nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
            nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path")),
            tags = listOf(OrgTag("20260710T100000", "draft"))
        )
        val repo = RecordingRepository()
        ArcologyPublishingPlugin(repo).onFileIndexed(result)
        assertEquals(1, repo.inserted.size)
        assertTrue(!repo.inserted.first().isArchived)
    }
kotlin#+name: publishing-plugin-test:noweb-ref publishing-plugin-test
    @Test
    fun `marks archived entries from ARCHIVE tag`() = runBlocking {
        val result = parseResultWith(
            nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
            nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path")),
            tags = listOf(OrgTag("20260710T100000", "ARCHIVE"))
        )
        val repo = RecordingRepository()
        ArcologyPublishingPlugin(repo).onFileIndexed(result)
        assertEquals(1, repo.inserted.size)
        assertTrue(repo.inserted.first().isArchived)
    }
kotlin#+name: publishing-plugin-test:noweb-ref publishing-plugin-test
    @Test
    fun `marks archived entries from noexport tag`() = runBlocking {
        val result = parseResultWith(
            nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
            nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path")),
            tags = listOf(OrgTag("20260710T100000", "noexport"))
        )
        val repo = RecordingRepository()
        ArcologyPublishingPlugin(repo).onFileIndexed(result)
        assertEquals(1, repo.inserted.size)
        assertTrue(repo.inserted.first().isArchived)
    }
kotlin#+name: publishing-plugin-test:noweb-ref publishing-plugin-test
    @Test
    fun `marks archived entries from uppercase NOEXPORT tag`() = runBlocking {
        val result = parseResultWith(
            nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
            nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path")),
            tags = listOf(OrgTag("20260710T100000", "NOEXPORT"))
        )
        val repo = RecordingRepository()
        ArcologyPublishingPlugin(repo).onFileIndexed(result)
        assertEquals(1, repo.inserted.size)
        assertTrue(repo.inserted.first().isArchived)
    }
kotlin#+name: publishing-plugin-test:noweb-ref publishing-plugin-test
    @Test
    fun `Archive tag does not archive entries`() = runBlocking {
        val result = parseResultWith(
            nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
            nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path")),
            tags = listOf(OrgTag("20260710T100000", "Archive"))
        )
        val repo = RecordingRepository()
        ArcologyPublishingPlugin(repo).onFileIndexed(result)
        assertEquals(1, repo.inserted.size)
        assertTrue(!repo.inserted.first().isArchived)
    }
kotlin#+name: publishing-plugin-test:noweb-ref publishing-plugin-test
    @Test
    fun `deletes existing routes before inserting`() = runBlocking {
        val result = parseResultWith(
            nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
            nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path"))
        )
        val repo = RecordingRepository()
        ArcologyPublishingPlugin(repo).onFileIndexed(result)
        assertEquals(1, repo.deletedFiles.size)
        assertEquals("test.org", repo.deletedFiles.first())
    }
kotlin#+name: publishing-plugin-test:noweb-ref publishing-plugin-test
    @Test
    fun `extracts timestamp from ID format`() = runBlocking {
        val result = parseResultWith(
            nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
            nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path"))
        )
        val repo = RecordingRepository()
        ArcologyPublishingPlugin(repo).onFileIndexed(result)
        assertEquals("20260710T100000", repo.inserted.first().timestamp)
    }
kotlin#+name: publishing-plugin-test-end
}
kotlin#+name: recording-repository
class RecordingRepository : PublishingRepository {
    val inserted = mutableListOf<RouteEntry>()
    val deletedFiles = mutableListOf<String>()

    override suspend fun getAllRoutes(): List<RouteEntry> = emptyList()
    override suspend fun getRoutesByPath(path: String): List<RouteEntry> = emptyList()
    override suspend fun getPublishedRoutes(): List<RouteEntry> = emptyList()
    override suspend fun getPublishedRoutesByPath(path: String): List<RouteEntry> = emptyList()
    override suspend fun insertRoute(entry: RouteEntry) { inserted.add(entry) }
    override suspend fun deleteRoutesByFile(file: String) { deletedFiles.add(file) }
    override suspend fun countRoutes(): Long = 0
    override suspend fun getAttachmentsByNode(nodeId: String): List<PublishedAttachment> = emptyList()
    override suspend fun getAttachmentsBySourcePath(sourcePath: String): List<PublishedAttachment> = emptyList()
    override suspend fun getAttachmentsByBasename(basename: String): List<PublishedAttachment> = emptyList()
    override suspend fun insertAttachment(entry: PublishedAttachment) {}
    override suspend fun deleteAttachmentsByFile(file: String) {}
    override suspend fun countAttachments(): Long = 0
}

Attachment Handling

Attachments are the web-side equivalent of arcology.models.Attachment in arcology-django: at index time, every =:ATTACH:=-tagged node's attachment directory is scanned (using AttachmentResolver's org-attach-dir emulation), each file is hashed and crushed — resized and re-encoded via ImageMagick magick, or copied verbatim when it can't be crushed — into a content-addressed cache directory, and one row per size variant is written to published_attachments.

The crush pipeline mirrors django's maybe_crush_file: magick SRC -auto-orient -strip -adaptive-resize {size}> -interlace Plane -quality 80 OUT, with the addition of the shrink-only > geometry modifier so small images are never upscaled, and a second size variant (512 and 2048) for the HTMX progressive enhancement (small image in the <img>, large swapped in on load). When magick is unavailable or a variant's crush fails, the file is copied verbatim under size 0= — attachment serving degrades gracefully and stays hermetic in test environments.

AttachmentCrusher

JVM-only: shells out to the magick binary. The commandRunner function is injectable so tests can fake ImageMagick without a process spawn.

kotlin#+name: attachment-crusher
package computer.whatthefuck.arcology.publishing

import java.io.File
import java.security.MessageDigest

private fun runCommand(command: List<String>) {
    val process = ProcessBuilder(command).start()
    val exitCode = process.waitFor()
    if (exitCode != 0) {
        val stderr = process.errorStream.bufferedReader().use { it.readText() }
        throw IllegalStateException("command '${command.first()}' exited $exitCode: ${stderr.trim()}")
    }
}

class AttachmentCrusher(
    private val cacheDir: File,
    private val commandRunner: (List<String>) -> Unit = ::runCommand
) {

    data class CrushedAttachment(val sourceHash: String, val type: String, val sizes: List<Int>)

    /** Extensions ImageMagick can crush; everything else is copied verbatim. */
    val crushableExtensions = setOf("jpg", "jpeg", "png", "webp", "avif")

    /** Crush variants, smallest first. The small variant is the initial <img> src. */
    val crushSizes = listOf(1080, 2160)

    init {
        if (!cacheDir.exists()) {
            cacheDir.mkdirs()
        }
    }

    private val magickAvailable: Boolean by lazy {
        try {
            commandRunner(listOf("magick", "-version"))
            true
        } catch (e: Exception) {
            false
        }
    }

    fun crush(source: File): CrushedAttachment {
        val hash = sha256Hex(source.readBytes())
        val ext = source.extension.lowercase()
        val sizes = if (ext in crushableExtensions && magickAvailable) {
            val succeeded = crushSizes.filter { size -> crushVariant(source, hash, ext, size) }
            if (succeeded.isEmpty()) {
                copyVerbatim(source, hash, ext)
                listOf(0)
            } else {
                succeeded
            }
        } else {
            copyVerbatim(source, hash, ext)
            listOf(0)
        }
        return CrushedAttachment(sourceHash = hash, type = ext, sizes = sizes)
    }

    fun filePathFor(sourceHash: String, size: Int, type: String): File =
        File(cacheDir, "$sourceHash-$size.$type")

    private fun crushVariant(source: File, hash: String, ext: String, size: Int): Boolean {
        val target = filePathFor(hash, size, ext)
        if (target.exists() && target.length() > 0) return true
        return try {
            commandRunner(listOf(
                "magick", source.absolutePath,
                "-auto-orient", "-strip",
                "-adaptive-resize", "${size}>",
                "-interlace", "Plane", "-quality", "80",
                target.absolutePath
            ))
            target.exists() && target.length() > 0
        } catch (e: Exception) {
            target.delete()
            false
        }
    }

    private fun copyVerbatim(source: File, hash: String, ext: String) {
        val target = filePathFor(hash, 0, ext)
        if (!target.exists()) {
            source.copyTo(target, overwrite = true)
        }
    }

    companion object {
        fun sha256Hex(bytes: ByteArray): String =
            MessageDigest.getInstance("SHA-256").digest(bytes).joinToString("") { "%02x".format(it) }

        /**
         * Default attachment cache dir, overridable with the ARCOLOGY_ATTACHMENT_DIR
         * environment variable (same variable arcology-django uses). The indexer
         * writes here; the web server serves from here.
         */
        fun defaultCacheDir(): File =
            File(System.getenv("ARCOLOGY_ATTACHMENT_DIR") ?: "/tmp/arcology-cache/attachments")
    }
}

ArcologyAttachmentPlugin

The indexer plugin. It follows the ArcologyPublishingPlugin pattern: rows for the file are deleted up front, then re-inserted. Attachment directories are resolved with AttachmentResolver, which mimics =org-attach-dir='s org-attach-id-to-path-function-list probing — the timestamp format data/{id[0..5]}/{id[6..]} is tried first (org-roam temporally-unique IDs), then the UUID format data/{id[0..1]}/{id[2..]}.

kotlin#+name: attachment-plugin-preamble
package computer.whatthefuck.arcology.publishing

import computer.whatthefuck.arcology.indexer.AttachmentResolver
import computer.whatthefuck.arcology.indexer.IndexerPlugin
import computer.whatthefuck.arcology.parser.ParseResult
import java.io.File
kotlin#+name: attachment-plugin-core:noweb yes
class ArcologyAttachmentPlugin(
    private val repository: PublishingRepository,
    private val crusher: AttachmentCrusher,
    private val attachmentResolver: AttachmentResolver,
    private val orgRoot: String,
    private val debug: Boolean = false
) : IndexerPlugin {

    override suspend fun onFileIndexed(result: ParseResult.Success) {
        repository.deleteAttachmentsByFile(result.file.path)

        val attachNodeIds = result.tags
            .filter { it.tag == "ATTACH" }
            .map { it.nodeId }
            .toSet()
        if (attachNodeIds.isEmpty()) return

        for (nodeId in attachNodeIds) {
            val attachments = try {
                attachmentResolver.resolveAttachments(nodeId, result.file.path)
            } catch (e: Exception) {
                if (debug) println("[ArcologyAttachmentPlugin] $result.file.path: failed to resolve attachments for $nodeId: ${e.message}")
                emptyList()
            }
            for (attachment in attachments) {
                val source = resolveSource(attachment.resolvedPath) ?: continue
                val crushed = try {
                    crusher.crush(source)
                } catch (e: Exception) {
                    if (debug) println("[ArcologyAttachmentPlugin] $result.file.path: failed to crush ${attachment.resolvedPath}: ${e.message}")
                    continue
                }
                for (size in crushed.sizes) {
                    repository.insertAttachment(PublishedAttachment(
                        nodeId = nodeId,
                        sourcePath = attachment.resolvedPath,
                        sourceHash = crushed.sourceHash,
                        size = size,
                        type = crushed.type,
                        file = result.file.path
                    ))
                }
            }
        }
    }

    override suspend fun onFileRemoved(filePath: String) {
        repository.deleteAttachmentsByFile(filePath)
    }

    private fun resolveSource(resolvedPath: String): File? {
        val candidate = if (resolvedPath.startsWith("/")) File(resolvedPath) else File(orgRoot, resolvedPath)
        return if (candidate.exists() && candidate.isFile) candidate else null
    }
}

Tests: AttachmentCrusher

kotlin#+name: attachment-crusher-test-prelude
package computer.whatthefuck.arcology.publishing

import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
import kotlin.test.assertFalse
import java.io.File
import java.nio.file.Files

class AttachmentCrusherTest {

    private fun tempCrusherDir(): File = Files.createTempDirectory("arcology-crush-test").toFile()

    private fun tempSource(ext: String, content: ByteArray = "source-bytes".toByteArray()): File {
        val dir = Files.createTempDirectory("arcology-crush-src").toFile()
        return File(dir, "photo.$ext").apply { writeBytes(content) }
    }

    /** Fake magick: succeeds for `-version`, writes a marker to the output path for crush commands. */
    private val fakeMagick: (List<String>) -> Unit = { command ->
        if (command.getOrNull(1) != "-version") {
            val output = command.last()
            File(output).writeBytes("crushed:${command[command.indexOf("-adaptive-resize") + 1]}".toByteArray())
        }
    }

    /** Fake magick that fails every real invocation (but probes healthy). */
    private val failingMagick: (List<String>) -> Unit = { command ->
        if (command.getOrNull(1) != "-version") {
            throw IllegalStateException("magick not found")
        }
    }
kotlin#+name: attachment-crusher-test:noweb-ref attachment-crusher-test
    @Test
    fun `crushes image to both size variants`() {
        val dir = tempCrusherDir()
        val source = tempSource("jpg")
        val crusher = AttachmentCrusher(dir, fakeMagick)
        val crushed = crusher.crush(source)
        assertEquals(2, crushed.sizes.size)
        assertTrue(crushed.sizes.containsAll(listOf(1080, 2160)))
        assertEquals("jpg", crushed.type)
        assertTrue(crusher.filePathFor(crushed.sourceHash, 1080, "jpg").exists())
        assertTrue(crusher.filePathFor(crushed.sourceHash, 2160, "jpg").exists())
    }
kotlin#+name: attachment-crusher-test:noweb-ref attachment-crusher-test
    @Test
    fun `crush output is content addressed by source hash`() {
        val dir = tempCrusherDir()
        val source = tempSource("png", content = "known-bytes".toByteArray())
        val crusher = AttachmentCrusher(dir, fakeMagick)
        val crushed = crusher.crush(source)
        assertEquals(AttachmentCrusher.sha256Hex("known-bytes".toByteArray()), crushed.sourceHash)
    }
kotlin#+name: attachment-crusher-test:noweb-ref attachment-crusher-test
    @Test
    fun `uncrushable extension copied verbatim with size zero`() {
        val dir = tempCrusherDir()
        val source = tempSource("pdf")
        val crusher = AttachmentCrusher(dir, fakeMagick)
        val crushed = crusher.crush(source)
        assertEquals(listOf(0), crushed.sizes)
        val verbatim = crusher.filePathFor(crushed.sourceHash, 0, "pdf")
        assertTrue(verbatim.exists())
        assertEquals("source-bytes", verbatim.readText())
    }
kotlin#+name: attachment-crusher-test:noweb-ref attachment-crusher-test
    @Test
    fun `falls back to verbatim when magick fails`() {
        val dir = tempCrusherDir()
        val source = tempSource("jpg")
        val crusher = AttachmentCrusher(dir, failingMagick)
        val crushed = crusher.crush(source)
        assertEquals(listOf(0), crushed.sizes)
        assertTrue(crusher.filePathFor(crushed.sourceHash, 0, "jpg").exists())
    }
kotlin#+name: attachment-crusher-test:noweb-ref attachment-crusher-test
    @Test
    fun `falls back to verbatim when magick binary is missing`() {
        val dir = tempCrusherDir()
        val source = tempSource("jpg")
        val crusher = AttachmentCrusher(dir, { _ -> throw java.io.IOException("no magick") })
        val crushed = crusher.crush(source)
        assertEquals(listOf(0), crushed.sizes)
        assertTrue(crusher.filePathFor(crushed.sourceHash, 0, "jpg").exists())
    }
kotlin#+name: attachment-crusher-test:noweb-ref attachment-crusher-test
    @Test
    fun `existing variant file is reused without re-crushing`() {
        val dir = tempCrusherDir()
        val source = tempSource("jpg")
        val crusher = AttachmentCrusher(dir, fakeMagick)
        val first = crusher.crush(source)
        // Drop the 2048 variant, then make magick fail for real invocations:
        // the existing 512 file must be reused untouched; the failed 2048 is dropped.
        crusher.filePathFor(first.sourceHash, 2160, "jpg").delete()
        val second = AttachmentCrusher(dir, failingMagick).crush(source)
        assertEquals(listOf(1080), second.sizes)
        assertEquals(first.sourceHash, second.sourceHash)
    }
kotlin#+name: attachment-crusher-test:noweb-ref attachment-crusher-test
    @Test
    fun `sha256Hex produces known digest`() {
        assertEquals(
            "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
            AttachmentCrusher.sha256Hex("abc".toByteArray())
        )
    }
kotlin#+name: attachment-crusher-test-end
}

Tests: ArcologyAttachmentPlugin

The fake filesystem implements only what AttachmentResolver touches: listFilesInDirectory returns canned listings keyed by directory, everything else is stubbed. The plugin test drives onFileIndexed with a hand-built ParseResult.Success, same as ArcologyPublishingPluginTest.

kotlin#+name: attachment-plugin-test-prelude
package computer.whatthefuck.arcology.publishing

import computer.whatthefuck.arcology.domain.OrgFile
import computer.whatthefuck.arcology.domain.OrgNode
import computer.whatthefuck.arcology.domain.OrgTag
import computer.whatthefuck.arcology.indexer.AttachmentResolver
import computer.whatthefuck.arcology.indexer.FileSystemInterface
import computer.whatthefuck.arcology.parser.ParseResult
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.emptyFlow
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
import kotlinx.coroutines.runBlocking
import kotlin.time.Instant

class ArcologyAttachmentPluginTest {

    /** Directory listing fake: maps "relative/dir" -> list of file paths. */
    private class FakeAttachmentFs(private val listings: Map<String, List<String>>) : FileSystemInterface {
        override suspend fun fileExists(path: String): Boolean = false
        override suspend fun readFile(path: String): String = ""
        override suspend fun writeFile(path: String, content: String) {}
        override suspend fun getLastModified(path: String): Instant = Instant.fromEpochSeconds(0)
        override fun listOrgFiles(path: String, recursive: Boolean): Flow<String> = emptyFlow()
        override suspend fun readIgnoreFile(rootPath: String): String? = null
        override suspend fun listFilesInDirectory(baseFilePath: String, relativeDir: String): List<String> =
            listings[relativeDir] ?: emptyList()
    }

    private class RecordingAttachmentRepository : PublishingRepository {
        val inserted = mutableListOf<PublishedAttachment>()
        val deletedFiles = mutableListOf<String>()
        override suspend fun getAllRoutes(): List<RouteEntry> = emptyList()
        override suspend fun getRoutesByPath(path: String): List<RouteEntry> = emptyList()
        override suspend fun getPublishedRoutes(): List<RouteEntry> = emptyList()
        override suspend fun getPublishedRoutesByPath(path: String): List<RouteEntry> = emptyList()
        override suspend fun insertRoute(entry: RouteEntry) {}
        override suspend fun deleteRoutesByFile(file: String) {}
        override suspend fun countRoutes(): Long = 0
        override suspend fun getAttachmentsByNode(nodeId: String): List<PublishedAttachment> = emptyList()
        override suspend fun getAttachmentsBySourcePath(sourcePath: String): List<PublishedAttachment> = emptyList()
        override suspend fun getAttachmentsByBasename(basename: String): List<PublishedAttachment> = emptyList()
        override suspend fun insertAttachment(entry: PublishedAttachment) { inserted.add(entry) }
        override suspend fun deleteAttachmentsByFile(file: String) { deletedFiles.add(file) }
        override suspend fun countAttachments(): Long = 0
    }

    private fun parseResultWith(
        nodeId: String,
        tags: List<OrgTag> = listOf(OrgTag(nodeId, "ATTACH")),
        filePath: String = "test.org"
    ): ParseResult.Success {
        return ParseResult.Success(
            file = OrgFile(path = filePath, hash = "abc", accessTime = Instant.fromEpochSeconds(0), modificationTime = Instant.fromEpochSeconds(0)),
            nodes = listOf(OrgNode(id = nodeId, file = filePath, level = 1, position = 0, title = "Node", properties = emptyMap())),
            links = emptyList(),
            tags = tags,
            aliases = emptyList(),
            refs = emptyList(),
            fileProperties = emptyList(),
            nodeProperties = emptyList(),
            content = ""
        )
    }

    /** Fake magick that writes a marker file to the command's output path. */
    private val fakeMagick: (List<String>) -> Unit = { command ->
        if (command.getOrNull(1) != "-version") {
            java.io.File(command.last()).writeBytes("crushed".toByteArray())
        }
    }
kotlin#+name: attachment-plugin-test:noweb-ref attachment-plugin-test
    @Test
    fun `inserts rows for ATTACH tagged node with both size variants`() = runBlocking {
        val nodeId = "20260710T100000"
        // AttachmentResolver probes data/{id[0..5]}/{id[6..]} first for timestamp IDs
        val attachDir = "data/202607/10T100000"
        val orgRoot = java.nio.file.Files.createTempDirectory("arcology-attach-org").toFile()
        java.io.File(orgRoot, "$attachDir").mkdirs()
        val source = java.io.File(orgRoot, "$attachDir/photo.jpg").apply { writeBytes("photo-bytes".toByteArray()) }

        val fs = FakeAttachmentFs(mapOf(attachDir to listOf(source.absolutePath)))
        val repo = RecordingAttachmentRepository()
        val crusher = AttachmentCrusher(
            java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(),
            fakeMagick
        )
        val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(fs), orgRoot.absolutePath)

        plugin.onFileIndexed(parseResultWith(nodeId))

        assertEquals(2, repo.inserted.size)
        val sizes = repo.inserted.map { it.size }.sorted()
        assertEquals(listOf(1080, 2160), sizes)
        val expectedHash = AttachmentCrusher.sha256Hex("photo-bytes".toByteArray())
        assertTrue(repo.inserted.all { it.sourceHash == expectedHash })
        assertTrue(repo.inserted.all { it.type == "jpg" })
        assertTrue(repo.inserted.all { it.nodeId == nodeId })
        assertTrue(repo.inserted.all { it.file == "test.org" })
        assertTrue(repo.inserted.all { it.sourcePath == source.absolutePath })
    }
kotlin#+name: attachment-plugin-test:noweb-ref attachment-plugin-test
    @Test
    fun `non-image attachment stored verbatim with size zero`() = runBlocking {
        val nodeId = "20260710T100000"
        val attachDir = "data/202607/10T100000"
        val orgRoot = java.nio.file.Files.createTempDirectory("arcology-attach-org").toFile()
        java.io.File(orgRoot, "$attachDir").mkdirs()
        val source = java.io.File(orgRoot, "$attachDir/doc.pdf").apply { writeBytes("pdf-bytes".toByteArray()) }

        val fs = FakeAttachmentFs(mapOf(attachDir to listOf(source.absolutePath)))
        val repo = RecordingAttachmentRepository()
        val crusher = AttachmentCrusher(
            java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(),
            fakeMagick
        )
        val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(fs), orgRoot.absolutePath)

        plugin.onFileIndexed(parseResultWith(nodeId))

        assertEquals(1, repo.inserted.size)
        assertEquals(0, repo.inserted.first().size)
        assertEquals("pdf", repo.inserted.first().type)
    }
kotlin#+name: attachment-plugin-test:noweb-ref attachment-plugin-test
    @Test
    fun `nodes without ATTACH tag yield no rows`() = runBlocking {
        val repo = RecordingAttachmentRepository()
        val crusher = AttachmentCrusher(java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(), fakeMagick)
        val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(FakeAttachmentFs(emptyMap())), "/tmp")

        plugin.onFileIndexed(parseResultWith("20260710T100000", tags = listOf(OrgTag("20260710T100000", "publish"))))

        assertEquals(0, repo.inserted.size)
    }
kotlin#+name: attachment-plugin-test:noweb-ref attachment-plugin-test
    @Test
    fun `re-index deletes existing rows for the file first`() = runBlocking {
        val repo = RecordingAttachmentRepository()
        val crusher = AttachmentCrusher(java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(), fakeMagick)
        val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(FakeAttachmentFs(emptyMap())), "/tmp")

        plugin.onFileIndexed(parseResultWith("20260710T100000"))

        assertEquals(1, repo.deletedFiles.size)
        assertEquals("test.org", repo.deletedFiles.first())
    }
kotlin#+name: attachment-plugin-test:noweb-ref attachment-plugin-test
    @Test
    fun `onFileRemoved deletes rows for the file`() = runBlocking {
        val repo = RecordingAttachmentRepository()
        val crusher = AttachmentCrusher(java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(), fakeMagick)
        val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(FakeAttachmentFs(emptyMap())), "/tmp")

        plugin.onFileRemoved("gone.org")

        assertEquals(listOf("gone.org"), repo.deletedFiles)
    }
kotlin#+name: attachment-plugin-test:noweb-ref attachment-plugin-test
    @Test
    fun `missing attachment directory yields no rows`() = runBlocking {
        val fs = FakeAttachmentFs(emptyMap())
        val repo = RecordingAttachmentRepository()
        val crusher = AttachmentCrusher(java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(), fakeMagick)
        val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(fs), "/tmp")

        plugin.onFileIndexed(parseResultWith("20260710T100000"))

        assertEquals(0, repo.inserted.size)
    }
kotlin#+name: attachment-plugin-test-end
}

Data Models

PublishKey

A publish key is the SITE/rest/of/path string from ARCOLOGY_KEY. This data class parses and validates that format.

kotlin#+name: publish-key
package computer.whatthefuck.arcology.publishing

data class PublishKey(
    val site: String,
    val path: String
) {
    companion object {
        fun parse(raw: String): PublishKey? {
            val trimmed = raw.trim()
            if (trimmed.isEmpty()) return null
            val slashIndex = trimmed.indexOf('/')
            if (slashIndex <= 0 || slashIndex == trimmed.length - 1) return null
            val site = trimmed.substring(0, slashIndex).trim()
            val path = trimmed.substring(slashIndex + 1).trim()
            if (site.isEmpty() || path.isEmpty()) return null
            return PublishKey(site, path)
        }
    }
}

Tests: PublishKey parsing

kotlin#+name: publish-key-test-prelude
package computer.whatthefuck.arcology.publishing

import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertNull

class PublishKeyTest {
kotlin#+name: publish-key-test:noweb-ref publish-key-test
    @Test
    fun `parse valid key with path`() {
        val key = PublishKey.parse("lionsrear/index")
        assertEquals("lionsrear", key?.site)
        assertEquals("index", key?.path)
    }
kotlin#+name: publish-key-test:noweb-ref publish-key-test
    @Test
    fun `parse key with nested path`() {
        val key = PublishKey.parse("blog/2025/my-post")
        assertEquals("blog", key?.site)
        assertEquals("2025/my-post", key?.path)
    }
kotlin#+name: publish-key-test:noweb-ref publish-key-test
    @Test
    fun `parse returns null for empty string`() {
        assertNull(PublishKey.parse(""))
    }
kotlin#+name: publish-key-test:noweb-ref publish-key-test
    @Test
    fun `parse returns null for missing slash`() {
        assertNull(PublishKey.parse("lionsrear"))
    }
kotlin#+name: publish-key-test:noweb-ref publish-key-test
    @Test
    fun `parse returns null for leading slash`() {
        assertNull(PublishKey.parse("/index"))
    }
kotlin#+name: publish-key-test:noweb-ref publish-key-test
    @Test
    fun `parse returns null for trailing slash`() {
        assertNull(PublishKey.parse("lionsrear/"))
    }
kotlin#+name: publish-key-test:noweb-ref publish-key-test
    @Test
    fun `parse trims whitespace`() {
        val key = PublishKey.parse("  lionsrear/index  ")
        assertEquals("lionsrear", key?.site)
        assertEquals("index", key?.path)
    }
kotlin#+name: publish-key-test-end
}

DomainMap

The domain map resolves SITE prefixes to domain names, and reverse-resolves domain names back to sites. It also carries per-site metadata (title, optional CSS file, optional link color) consumed by the page templates and the dynamic /sites.css endpoint. It loads from a domains.json file generated from an org table via an arroyo lua block (see domains.org). The JSON shape is:

json
{
  "sites": [
    {"key":"lionsrear","title":"The Lions Rear","cssFile":"arcology/css/lionsrear.css","linkColor":"#a64040","domains":["rix.si","thelionsrear.com"]}
  ]
}

fromMap is kept as a convenience for tests and callers that build a domain map in memory (single-domain sites, no metadata). The JSON load path is the production path.

Data classes and imports

kotlin#+name: domain-map-preamble
package computer.whatthefuck.arcology.publishing

import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.decodeFromString
import java.io.File
import java.io.InputStream

@Serializable
data class SiteMeta(
    val key: String,
    val title: String? = null,
    val cssFile: String? = null,
    val linkColor: String? = null,
    val hljsTheme: String? = null,
    val domains: List<String> = emptyList()
)

@Serializable
private data class DomainConfig(val sites: List<SiteMeta>)

Class body: resolution and metadata lookup

kotlin#+name: domain-map-class
class DomainMap private constructor(
    private val sites: List<SiteMeta>,
    private val domainToSite: Map<String, String>
) {
    fun resolve(site: String): String? = meta(site)?.domains?.firstOrNull()

    fun resolveAll(site: String): List<String> = meta(site)?.domains ?: emptyList()

    fun resolveSite(domain: String): String? = domainToSite[domain.lowercase()]

    fun meta(site: String): SiteMeta? = sites.firstOrNull { it.key == site }

    fun allSites(): Set<String> = sites.map { it.key }.toSet()

Companion object: loading from JSON and in-memory construction

kotlin#+name: domain-map-companion
    companion object {
        private val json = Json { ignoreUnknownKeys = true }

        fun load(file: File): DomainMap {
            file.inputStream().use { stream ->
                return load(stream)
            }
        }

        fun load(stream: InputStream): DomainMap {
            val text = stream.bufferedReader().use { it.readText() }
            return parse(text)
        }

        fun parse(text: String): DomainMap {
            val config: DomainConfig = json.decodeFromString(text)
            val domainToSite = mutableMapOf<String, String>()
            for (site in config.sites) {
                for (domain in site.domains) {
                    domainToSite[domain.lowercase()] = site.key
                }
            }
            return DomainMap(config.sites, domainToSite)
        }

        fun fromMap(map: Map<String, String>): DomainMap {
            val sites = map.map { (key, domain) ->
                val domains = domain.split(",").map { it.trim() }.filter { it.isNotEmpty() }
                SiteMeta(key = key, domains = domains)
            }
            val domainToSite = mutableMapOf<String, String>()
            for (site in sites) {
                for (domain in site.domains) {
                    domainToSite[domain.lowercase()] = site.key
                }
            }
            return DomainMap(sites, domainToSite)
        }
    }
}

Tests: DomainMap loading, resolution, and metadata

kotlin#+name: domain-map-test-prelude
package computer.whatthefuck.arcology.publishing

import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertNull
import kotlin.test.assertNotNull
import java.io.ByteArrayInputStream

class DomainMapTest {
kotlin#+name: domain-map-test:noweb-ref domain-map-test
    @Test
    fun `load from JSON stream`() {
        val input = ByteArrayInputStream("""
            {
              "sites": [
                {"key":"lionsrear","title":"The Lions Rear","cssFile":"arcology/css/lionsrear.css","linkColor":"#a64040","domains":["rix.si","thelionsrear.com"]},
                {"key":"arcology","title":"Arcology Engine","cssFile":"arcology/css/arcology.css","linkColor":"#4060a6","domains":["engine.arcology.garden"]}
              ]
            }
        """.trimIndent().toByteArray())
        val map = DomainMap.load(input)
        assertEquals("rix.si", map.resolve("lionsrear"))
        assertEquals("engine.arcology.garden", map.resolve("arcology"))
    }
kotlin#+name: domain-map-test:noweb-ref domain-map-test
    @Test
    fun `resolve returns null for unknown site`() {
        val map = DomainMap.fromMap(mapOf("lionsrear" to "thelionsrear.com"))
        assertNull(map.resolve("unknown"))
    }
kotlin#+name: domain-map-test:noweb-ref domain-map-test
    @Test
    fun `allSites returns all keys`() {
        val map = DomainMap.fromMap(mapOf(
            "lionsrear" to "thelionsrear.com",
            "arcology" to "arcology.garden"
        ))
        assertEquals(setOf("lionsrear", "arcology"), map.allSites())
    }
kotlin#+name: domain-map-test:noweb-ref domain-map-test
    @Test
    fun `meta returns site metadata from JSON`() {
        val input = ByteArrayInputStream("""
            {
              "sites": [
                {"key":"lionsrear","title":"The Lions Rear","cssFile":"arcology/css/lionsrear.css","linkColor":"#a64040","hljsTheme":"nord","domains":["rix.si","thelionsrear.com"]}
              ]
            }
        """.trimIndent().toByteArray())
        val map = DomainMap.load(input)
        val meta = map.meta("lionsrear")
        assertNotNull(meta)
        assertEquals("The Lions Rear", meta.title)
        assertEquals("arcology/css/lionsrear.css", meta.cssFile)
        assertEquals("#a64040", meta.linkColor)
        assertEquals("nord", meta.hljsTheme)
        assertEquals(listOf("rix.si", "thelionsrear.com"), meta.domains)
    }
kotlin#+name: domain-map-test:noweb-ref domain-map-test
    @Test
    fun `meta hljsTheme defaults to null when absent`() {
        val input = ByteArrayInputStream("""
            {
              "sites": [
                {"key":"lionsrear","title":"The Lions Rear","cssFile":"arcology/css/lionsrear.css","linkColor":"#a64040","domains":["rix.si"]}
              ]
            }
        """.trimIndent().toByteArray())
        val map = DomainMap.load(input)
        val meta = map.meta("lionsrear")
        assertNotNull(meta)
        assertNull(meta.hljsTheme)
    }
kotlin#+name: domain-map-test:noweb-ref domain-map-test
    @Test
    fun `meta returns null for unknown site`() {
        val map = DomainMap.fromMap(mapOf("lionsrear" to "thelionsrear.com"))
        assertNull(map.meta("unknown"))
    }
kotlin#+name: domain-map-test:noweb-ref domain-map-test
    @Test
    fun `fromMap produces SiteMeta with null title and cssFile`() {
        val map = DomainMap.fromMap(mapOf("lionsrear" to "thelionsrear.com"))
        val meta = map.meta("lionsrear")
        assertNotNull(meta)
        assertEquals("lionsrear", meta.key)
        assertNull(meta.title)
        assertNull(meta.cssFile)
        assertNull(meta.linkColor)
        assertNull(meta.hljsTheme)
        assertEquals(listOf("thelionsrear.com"), meta.domains)
    }
kotlin#+name: domain-map-test:noweb-ref domain-map-test
    @Test
    fun `resolveSite reverse-resolves domain to site`() {
        val input = ByteArrayInputStream("""
            {
              "sites": [
                {"key":"lionsrear","title":null,"cssFile":null,"linkColor":null,"domains":["rix.si","thelionsrear.com"]}
              ]
            }
        """.trimIndent().toByteArray())
        val map = DomainMap.load(input)
        assertEquals("lionsrear", map.resolveSite("rix.si"))
        assertEquals("lionsrear", map.resolveSite("THELIONSREAR.COM"))
        assertNull(map.resolveSite("unknown.example"))
    }
kotlin#+name: domain-map-test:noweb-ref domain-map-test
    @Test
    fun `resolveAll returns all domains for site`() {
        val input = ByteArrayInputStream("""
            {
              "sites": [
                {"key":"lionsrear","title":null,"cssFile":null,"linkColor":null,"domains":["rix.si","thelionsrear.com"]}
              ]
            }
        """.trimIndent().toByteArray())
        val map = DomainMap.load(input)
        assertEquals(listOf("rix.si", "thelionsrear.com"), map.resolveAll("lionsrear"))
        assertEquals(emptyList(), map.resolveAll("unknown"))
    }
kotlin#+name: domain-map-test:noweb-ref domain-map-test
    @Test
    fun `parse handles JSON with unknown fields without failing`() {
        val input = """
            {
              "sites": [
                {"key":"lionsrear","title":null,"cssFile":null,"linkColor":null,"domains":["thelionsrear.com"],"futureField":"ignored"}
              ]
            }
        """.trimIndent()
        val map = DomainMap.parse(input)
        assertEquals("thelionsrear.com", map.resolve("lionsrear"))
    }
kotlin#+name: domain-map-test-end
}

Route Table

The route table is the in-memory structure used by the web server to look up paths and render pages.

kotlin#+name: route-table
package computer.whatthefuck.arcology.publishing

data class RouteEntry(
    val nodeId: String,
    val site: String,
    val path: String,
    val file: String,
    val title: String?,
    val timestamp: String?,
    val isDraft: Boolean,
    val isArchived: Boolean
)

/**
 * One row of the published_attachments cache: a single size variant of a
 * crushed (or verbatim-copied) attachment. The crushed file lives on disk at
 * {ATTACHMENT_DIR}/{urlName}; `size` is the crush target in pixels, or 0 for
 * verbatim copies. See [[AttachmentCrusher]].
 */
data class PublishedAttachment(
    val nodeId: String,
    val sourcePath: String,
    val sourceHash: String,
    val size: Int,
    val type: String,
    val file: String
) {
    /** Content-addressed file name; the URL path is /attachment/{urlName}. */
    val urlName: String get() = "$sourceHash-$size.$type"
}

data class RouteTable(
    val routes: Map<String, List<RouteEntry>>
) {
    fun getEntries(path: String): List<RouteEntry> = routes[path] ?: emptyList()

    fun getEntriesByKey(fullKey: String): List<RouteEntry> {
        val slashIndex = fullKey.indexOf('/')
        if (slashIndex <= 0) return emptyList()
        val site = fullKey.substring(0, slashIndex)
        val path = fullKey.substring(slashIndex + 1)
        return routes[path]?.filter { it.site == site } ?: emptyList()
    }

    fun allKeys(): Set<String> = routes.values.flatten().map { "${it.site}/${it.path}" }.toSet()

    fun allPaths(): Set<String> = routes.keys

    val totalRoutes: Int get() = routes.values.sumOf { it.size }
}

PublishingRepository

The repository interface and SQLDelight implementation for the published_routes table.

Interface

kotlin#+name: publishing-repo-interface
interface PublishingRepository {
    suspend fun getAllRoutes(): List<RouteEntry>
    suspend fun getRoutesByPath(path: String): List<RouteEntry>
    suspend fun getPublishedRoutes(): List<RouteEntry>
    suspend fun getPublishedRoutesByPath(path: String): List<RouteEntry>
    suspend fun insertRoute(entry: RouteEntry)
    suspend fun deleteRoutesByFile(file: String)
    suspend fun countRoutes(): Long

    suspend fun getAttachmentsByNode(nodeId: String): List<PublishedAttachment>
    suspend fun getAttachmentsBySourcePath(sourcePath: String): List<PublishedAttachment>
    suspend fun getAttachmentsByBasename(basename: String): List<PublishedAttachment>
    suspend fun insertAttachment(entry: PublishedAttachment)
    suspend fun deleteAttachmentsByFile(file: String)
    suspend fun countAttachments(): Long
}

Implementation: class declaration, dispatcher, and row mapper

kotlin#+name: publishing-repo-impl-core
class PublishingRepositoryImpl(
    private val database: ArcologyDatabase
) : PublishingRepository {

    @OptIn(ExperimentalCoroutinesApi::class)
    private val dbDispatcher = Dispatchers.IO.limitedParallelism(1)

    private fun computer.whatthefuck.arcology.db.Published_routes.toRouteEntry(): RouteEntry {
        return RouteEntry(
            nodeId = node_id,
            site = site,
            path = path,
            file = file_,
            title = title,
            timestamp = timestamp,
            isDraft = is_draft != 0L,
            isArchived = is_archived != 0L
        )
    }

    private fun computer.whatthefuck.arcology.db.Published_attachments.toPublishedAttachment(): PublishedAttachment {
        return PublishedAttachment(
            nodeId = node_id,
            sourcePath = source_path,
            sourceHash = source_hash,
            size = size.toInt(),
            type = type,
            file = file_
        )
    }

Read methods

kotlin#+name: publishing-repo-impl-body:noweb-ref publishing-repo-impl-body
    override suspend fun getAllRoutes(): List<RouteEntry> {
        return withContext(Dispatchers.IO) {
            database.publishingQueries.selectAllPublishedRoutes().executeAsList().map { it.toRouteEntry() }
        }
    }

    override suspend fun getRoutesByPath(path: String): List<RouteEntry> {
        return withContext(Dispatchers.IO) {
            database.publishingQueries.selectRoutesByPath(path).executeAsList().map { it.toRouteEntry() }
        }
    }

    override suspend fun getPublishedRoutes(): List<RouteEntry> {
        return withContext(Dispatchers.IO) {
            database.publishingQueries.selectPublishedRoutes().executeAsList().map { it.toRouteEntry() }
        }
    }

    override suspend fun getPublishedRoutesByPath(path: String): List<RouteEntry> {
        return withContext(Dispatchers.IO) {
            database.publishingQueries.selectPublishedRoutesByPath(path).executeAsList().map { it.toRouteEntry() }
        }
    }

Write methods

kotlin#+name: publishing-repo-impl-body:noweb-ref publishing-repo-impl-body
    override suspend fun insertRoute(entry: RouteEntry) {
        database.publishingQueries.insertPublishedRoute(
            entry.nodeId,
            entry.site,
            entry.path,
            entry.file,
            entry.title,
            entry.timestamp,
            if (entry.isDraft) 1L else 0L,
            if (entry.isArchived) 1L else 0L
        )
    }

    override suspend fun deleteRoutesByFile(file: String) {
        database.publishingQueries.deletePublishedRoutesByFile(file)
    }

    override suspend fun countRoutes(): Long {
        return withContext(Dispatchers.IO) {
            database.publishingQueries.countPublishedRoutes().executeAsOne()
        }
    }

    override suspend fun getAttachmentsByNode(nodeId: String): List<PublishedAttachment> {
        return withContext(Dispatchers.IO) {
            database.publishingQueries.selectPublishedAttachmentsByNode(nodeId).executeAsList().map { it.toPublishedAttachment() }
        }
    }

    override suspend fun getAttachmentsBySourcePath(sourcePath: String): List<PublishedAttachment> {
        return withContext(Dispatchers.IO) {
            database.publishingQueries.selectPublishedAttachmentsByPath(sourcePath).executeAsList().map { it.toPublishedAttachment() }
        }
    }

    override suspend fun getAttachmentsByBasename(basename: String): List<PublishedAttachment> {
        return withContext(Dispatchers.IO) {
            database.publishingQueries.selectPublishedAttachmentsByBasename(basename).executeAsList().map { it.toPublishedAttachment() }
        }
    }

    override suspend fun insertAttachment(entry: PublishedAttachment) {
        database.publishingQueries.insertPublishedAttachment(
            entry.nodeId,
            entry.sourcePath,
            entry.sourceHash,
            entry.size.toLong(),
            entry.type,
            entry.file
        )
    }

    override suspend fun deleteAttachmentsByFile(file: String) {
        database.publishingQueries.deletePublishedAttachmentsByFile(file)
    }

    override suspend fun countAttachments(): Long {
        return withContext(Dispatchers.IO) {
            database.publishingQueries.countPublishedAttachments().executeAsOne()
        }
    }
kotlin#+name: publishing-repo-impl-end
}

RouteTableBuilder

Builds a RouteTable from the PublishingRepository by querying published (non-draft, non-archived) routes, grouping them by path, and sorting by timestamp.

kotlin#+name: route-table-builder
package computer.whatthefuck.arcology.publishing

class RouteTableBuilder {
    suspend fun build(repository: PublishingRepository): RouteTable {
        val routes = repository.getPublishedRoutes()
        val grouped = routes.groupBy { it.path }
        return RouteTable(routes = grouped)
    }
}

Tests: RouteTableBuilder grouping and filtering

kotlin#+name: route-table-builder-test-prelude
package computer.whatthefuck.arcology.publishing

import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
import kotlinx.coroutines.runBlocking

class RouteTableBuilderTest {

    private fun routeEntry(
        nodeId: String, site: String, path: String, timestamp: String,
        isDraft: Boolean = false, isArchived: Boolean = false
    ): RouteEntry {
        return RouteEntry(
            nodeId = nodeId, site = site, path = path, file = "test.org",
            title = "Test", timestamp = timestamp, isDraft = isDraft, isArchived = isArchived
        )
    }
kotlin#+name: route-table-builder-test:noweb-ref route-table-builder-test
    @Test
    fun `build groups routes by path`() = runBlocking {
        val repo = FakePublishingRepository(listOf(
            routeEntry("node1", "site", "index", "20260701T100000"),
            routeEntry("node2", "site", "index", "20260702T100000"),
            routeEntry("node3", "site", "about", "20260703T100000")
        ))
        val table = RouteTableBuilder().build(repo)
        assertEquals(2, table.allPaths().size)
        assertEquals(2, table.getEntries("index").size)
        assertEquals(1, table.getEntries("about").size)
    }
kotlin#+name: route-table-builder-test:noweb-ref route-table-builder-test
    @Test
    fun `build excludes drafts and archived`() = runBlocking {
        val repo = FakePublishingRepository(listOf(
            routeEntry("node1", "site", "index", "20260701T100000", isDraft = false),
            routeEntry("node2", "site", "index", "20260702T100000", isDraft = true),
            routeEntry("node3", "site", "index", "20260703T100000", isArchived = true)
        ))
        val table = RouteTableBuilder().build(repo)
        assertEquals(1, table.getEntries("index").size)
        assertEquals("node1", table.getEntries("index").first().nodeId)
    }
kotlin#+name: route-table-builder-test:noweb-ref route-table-builder-test
    @Test
    fun `totalRoutes counts all entries`() = runBlocking {
        val repo = FakePublishingRepository(listOf(
            routeEntry("node1", "site", "index", "20260701T100000"),
            routeEntry("node2", "site", "about", "20260702T100000")
        ))
        val table = RouteTableBuilder().build(repo)
        assertEquals(2, table.totalRoutes)
    }
kotlin#+name: route-table-builder-test-end
}
kotlin#+name: fake-publishing-repo
class FakePublishingRepository(private val routes: List<RouteEntry>) : PublishingRepository {
    override suspend fun getAllRoutes(): List<RouteEntry> = routes
    override suspend fun getRoutesByPath(path: String): List<RouteEntry> = routes.filter { it.path == path }
    override suspend fun getPublishedRoutes(): List<RouteEntry> = routes.filter { !it.isDraft && !it.isArchived }
    override suspend fun getPublishedRoutesByPath(path: String): List<RouteEntry> =
        routes.filter { it.path == path && !it.isDraft && !it.isArchived }
    override suspend fun insertRoute(entry: RouteEntry) {}
    override suspend fun deleteRoutesByFile(file: String) {}
    override suspend fun countRoutes(): Long = routes.size.toLong()
    override suspend fun getAttachmentsByNode(nodeId: String): List<PublishedAttachment> = emptyList()
    override suspend fun getAttachmentsBySourcePath(sourcePath: String): List<PublishedAttachment> = emptyList()
    override suspend fun getAttachmentsByBasename(basename: String): List<PublishedAttachment> = emptyList()
    override suspend fun insertAttachment(entry: PublishedAttachment) {}
    override suspend fun deleteAttachmentsByFile(file: String) {}
    override suspend fun countAttachments(): Long = 0
}

Tangle Targets

Publishing.sq

sql#+name: publishing-sq-assembly:tangle ../src/commonMain/sqldelight/computer/whatthefuck/arcology/db/Publishing.sq:noweb yes
<<publishing-schema>>

<<publishing-queries>>

<<attachments-schema>>

<<attachments-queries>>

PublishKey.kt

kotlin#+name: publish-key-assembly:tangle ../src/commonMain/kotlin/computer/whatthefuck/arcology/publishing/PublishKey.kt:noweb yes
<<publish-key>>

DomainMap.kt

kotlin#+name: domain-map-assembly:tangle ../src/commonMain/kotlin/computer/whatthefuck/arcology/publishing/DomainMap.kt:noweb yes
<<domain-map-preamble>>

<<domain-map-class>>

<<domain-map-companion>>

RouteTable.kt

kotlin#+name: route-table-assembly:tangle ../src/commonMain/kotlin/computer/whatthefuck/arcology/publishing/RouteTable.kt:noweb yes
<<route-table>>

PublishingRepository.kt

kotlin#+name: publishing-repo-assembly:tangle ../src/commonMain/kotlin/computer/whatthefuck/arcology/publishing/PublishingRepository.kt:noweb yes
@file:OptIn(kotlin.time.ExperimentalTime::class)
package computer.whatthefuck.arcology.publishing

import computer.whatthefuck.arcology.db.ArcologyDatabase
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.ExperimentalCoroutinesApi
import kotlinx.coroutines.withContext

<<publishing-repo-interface>>

<<publishing-repo-impl-core>>
<<publishing-repo-impl-body>>
<<publishing-repo-impl-end>>

RouteTableBuilder.kt

kotlin#+name: route-table-builder-assembly:tangle ../src/commonMain/kotlin/computer/whatthefuck/arcology/publishing/RouteTableBuilder.kt:noweb yes
<<route-table-builder>>

ExcludeTags.kt

kotlin#+name: exclude-tags-assembly:tangle ../src/commonMain/kotlin/computer/whatthefuck/arcology/publishing/ExcludeTags.kt:noweb yes
<<exclude-tags>>

ArcologyPublishingPlugin.kt

kotlin#+name: publishing-plugin-assembly:tangle ../src/commonMain/kotlin/computer/whatthefuck/arcology/publishing/ArcologyPublishingPlugin.kt:noweb yes
<<plugin-preamble>>

<<plugin-core>>

PublishKeyTest.kt

kotlin#+name: publish-key-test-assembly:tangle ../src/commonTest/kotlin/computer/whatthefuck/arcology/publishing/PublishKeyTest.kt:noweb yes
<<publish-key-test-prelude>>
<<publish-key-test>>
<<publish-key-test-end>>

DomainMapTest.kt

kotlin#+name: domain-map-test-assembly:tangle ../src/commonTest/kotlin/computer/whatthefuck/arcology/publishing/DomainMapTest.kt:noweb yes
<<domain-map-test-prelude>>
<<domain-map-test>>
<<domain-map-test-end>>

RouteTableBuilderTest.kt

kotlin#+name: route-table-builder-test-assembly:tangle ../src/commonTest/kotlin/computer/whatthefuck/arcology/publishing/RouteTableBuilderTest.kt:noweb yes
<<route-table-builder-test-prelude>>
<<route-table-builder-test>>
<<route-table-builder-test-end>>
<<fake-publishing-repo>>

ArcologyPublishingPluginTest.kt

kotlin#+name: publishing-plugin-test-assembly:tangle ../src/commonTest/kotlin/computer/whatthefuck/arcology/publishing/ArcologyPublishingPluginTest.kt:noweb yes
<<publishing-plugin-test-prelude>>
<<publishing-plugin-test>>
<<publishing-plugin-test-end>>
<<recording-repository>>

AttachmentCrusher.kt

kotlin#+name: attachment-crusher-assembly:tangle ../src/jvmMain/kotlin/computer/whatthefuck/arcology/publishing/AttachmentCrusher.kt:noweb yes
<<attachment-crusher>>

ArcologyAttachmentPlugin.kt

kotlin#+name: attachment-plugin-assembly:tangle ../src/jvmMain/kotlin/computer/whatthefuck/arcology/publishing/ArcologyAttachmentPlugin.kt:noweb yes
<<attachment-plugin-preamble>>

<<attachment-plugin-core>>

AttachmentCrusherTest.kt

kotlin#+name: attachment-crusher-test-assembly:tangle ../src/jvmTest/kotlin/computer/whatthefuck/arcology/publishing/AttachmentCrusherTest.kt:noweb yes
<<attachment-crusher-test-prelude>>
<<attachment-crusher-test>>
<<attachment-crusher-test-end>>

ArcologyAttachmentPluginTest.kt

kotlin#+name: attachment-plugin-test-assembly:tangle ../src/jvmTest/kotlin/computer/whatthefuck/arcology/publishing/ArcologyAttachmentPluginTest.kt:noweb yes
<<attachment-plugin-test-prelude>>
<<attachment-plugin-test>>
<<attachment-plugin-test-end>>

Related Modules