Introduction
This document is the source of truth for the Arcology web publishing data layer. It defines the database schema, domain models, indexer plugin, and route table builder that turn org-roam metadata into a publishable route table.
The publishing layer is a plugin to the The Indexer Pipeline (responsible for parsing org files and building the database) and the The Web Server (Which renders Hypertext to Hyper Text Markup Language and delivers it over Hyper Text Transfer Protocol)
The publishing model is simple:
Every heading or file with an
ARCOLOGY_KEYproperty publishes to a matching URL pathThe key format is
SITE/rest/of/path— the SITE prefix determines which domain serves itMultiple headings can share the same path (e.g., multiple journal entries tagged to
arcology/journal)Entries on the same path are sorted by timestamp (from the ID or
SCHEDULED)Tags control visibility:
:publish:(default),:draft:(hidden),:archive:(hidden)
An ArcologyPublishingPlugin runs at index time, extracts ARCOLOGY_KEY from the parsed document, and populates the published_routes table. The web server queries this table to build a route table at startup. Its sibling ArcologyAttachmentPlugin crushes and indexes attachments for =:ATTACH:=-tagged nodes into published_attachments.
Database Schema
The published_routes table is a denormalized view of publishing metadata, populated by the indexer plugin. It joins node IDs to their SITE/path keys, file paths, titles, timestamps, and publish status.
-- Publishing: route table for web publishing
-- Populated by ArcologyPublishingPlugin at index time.
-- Each row represents one heading or file-level node that has an ARCOLOGY_KEY.
CREATE TABLE IF NOT EXISTS published_routes (
node_id TEXT NOT NULL,
site TEXT NOT NULL,
path TEXT NOT NULL,
file TEXT NOT NULL,
title TEXT,
timestamp TEXT,
is_draft INTEGER NOT NULL DEFAULT 0,
is_archived INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (node_id, site, path),
FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_published_routes_path ON published_routes (path);
CREATE INDEX IF NOT EXISTS idx_published_routes_site ON published_routes (site);selectAllPublishedRoutes:
SELECT * FROM published_routes ORDER BY path, timestamp;
selectRoutesByPath:
SELECT * FROM published_routes WHERE path = ? ORDER BY timestamp;
selectRoutesBySite:
SELECT * FROM published_routes WHERE site = ? ORDER BY path, timestamp;
selectPublishedRoutes:
SELECT * FROM published_routes WHERE is_draft = 0 AND is_archived = 0 ORDER BY path, timestamp;
selectPublishedRoutesByPath:
SELECT * FROM published_routes WHERE path = ? AND is_draft = 0 AND is_archived = 0 ORDER BY timestamp;
insertPublishedRoute:
INSERT OR REPLACE INTO published_routes (node_id, site, path, file, title, timestamp, is_draft, is_archived)
VALUES (?, ?, ?, ?, ?, ?, ?, ?);
deletePublishedRoutesByFile:
DELETE FROM published_routes WHERE file = ?;
countPublishedRoutes:
SELECT COUNT(*) FROM published_routes;Attachment Cache Schema
Crushed attachments are cached on disk, content-addressed by the SHA-256 of the original file's bytes. The published_attachments table is the denormalized index of that cache: one row per (node, attachment, size variant), mirroring arcology.models.Attachment from arcology-django but with the per-size rows exploded out so that serving is a pure function of the URL — /attachment/{source_hash}-{size}.{type} — with no database lookup on the hot path.
sizeis the crush target in pixels (512, 2048) for crushable images, or0for verbatim copies of uncrushable files.typeis the lowercased file extension without the dot; it drives the URL suffix and the served content type.source_pathis the attachment's path relative to the org root (e.g.data/202607/xxxx/foo.jpg), kept sofile:andattachment:links can be resolved back to a row.fileis the source org file, mirroringpublished_routes, so rows are replaced wholesale on re-index and dropped when the file is removed.
-- Publishing: crushed attachment cache index.
-- Populated by ArcologyAttachmentPlugin at index time. One row per size variant.
-- Crushed files live on disk at {ATTACHMENT_DIR}/{source_hash}-{size}.{type}.
CREATE TABLE IF NOT EXISTS published_attachments (
node_id TEXT NOT NULL,
source_path TEXT NOT NULL,
source_hash TEXT NOT NULL,
size INTEGER NOT NULL,
type TEXT NOT NULL,
file TEXT NOT NULL,
PRIMARY KEY (node_id, source_hash, size)
);
CREATE INDEX IF NOT EXISTS idx_published_attachments_node ON published_attachments (node_id);
CREATE INDEX IF NOT EXISTS idx_published_attachments_path ON published_attachments (source_path);selectPublishedAttachmentsByNode:
SELECT * FROM published_attachments WHERE node_id = ? ORDER BY source_path, size;
selectPublishedAttachmentsByPath:
SELECT * FROM published_attachments WHERE source_path = ?;
selectPublishedAttachmentsByBasename:
SELECT * FROM published_attachments WHERE source_path LIKE '%/' || ? ORDER BY source_path, size;
insertPublishedAttachment:
INSERT OR REPLACE INTO published_attachments (node_id, source_path, source_hash, size, type, file)
VALUES (?, ?, ?, ?, ?, ?);
deletePublishedAttachmentsByFile:
DELETE FROM published_attachments WHERE file = ?;
countPublishedAttachments:
SELECT COUNT(*) FROM published_attachments;ArcologyPublishingPlugin
This indexer plugin extracts ARCOLOGY_KEY from parsed org files and populates the published_routes table. It follows the same pattern as ArroyoIndexerPlugin.
Extraction Sources
ARCOLOGY_KEY can appear in three places:
Preamble keyword lines —
#+ARCOLOGY_KEY: site/pathin the file preamble. These are file-scoped (assigned to the file-level node if it has an ID).Section body keyword lines —
#+ARCOLOGY_KEY: site/pathunder a heading. Also file-scoped in org semantics. Both preamble and section keyword lines are hoisted ontodocument.keywordLinesby the parser, and exposed onParseResult.Successviaresult.document?.keywordLines.Heading
:PROPERTIES:drawers —:ARCOLOGY_KEY: site/pathin a heading's property drawer. These are heading-scoped (assigned to that heading's node ID).
Timestamp Extraction
The timestamp for sorting entries on the same path comes from:
Primary: The node's ID if it matches the org-roam format
YYYYMMDDTHHMMSS(e.g.,20260710T100000)Fallback: The node's
SCHEDULEDdate
Tag-Based Visibility
Tags on the node control whether it appears in the published route table, using org-mode's existing export-control semantics:
:noexport:/:NOEXPORT:tag → excluded from public routes:ARCHIVE:tag → excluded from public routes (org-mode's archive tag, uppercase)All other tags (including
:Archive:,:publish:,:draft:, etc.) → published (default)
The distinction between :ARCHIVE: (uppercase, org-mode's archive tag) and :Archive: (mixed case, personal knowledge archive) is intentional — :Archive: is a personal organization tag that does NOT control publishing. The exclude set is case-sensitive so that :Archive: survives.
The same exclude set is shared by three call sites so they cannot drift:
ArcologyPublishingPlugin — drops the route from the route table at index time.
SidebarService — drops backlinks and tag-cloud entries from the sidebar at request time.
OrgHtmlRenderer — elides the heading and its sub-tree from rendered HTML at render time, so that non-node sub-headings carrying the tag are also suppressed (route-table filtering only catches headings that bear an
ARCOLOGY_KEY).
Exclude tags constant
package computer.whatthefuck.arcology.publishing
/**
* Tags that suppress both route-table publishing and HTML rendering of a heading.
*
* Case-sensitive on purpose: `:ARCHIVE:` (org-mode's archive tag) is excluded,
* while `:Archive:` (a personal knowledge-archive organization tag) is not.
* Both lowercase `:noexport:` and uppercase `:NOEXPORT:` are excluded since
* org-mode users write the tag either way and both should hide the heading.
*/
val EXCLUDE_TAGS: Set<String> = setOf("noexport", "NOEXPORT", "ARCHIVE")Plugin class: declaration and helpers
package computer.whatthefuck.arcology.publishing
import computer.whatthefuck.arcology.domain.NodeProperty
import computer.whatthefuck.arcology.indexer.IndexerPlugin
import computer.whatthefuck.arcology.parser.ParseResult
import xyz.lepisma.orgmode.OrgChunkclass ArcologyPublishingPlugin(
private val repository: PublishingRepository,
private val debug: Boolean = false
) : IndexerPlugin {
private data class PublishKeySource(
val nodeId: String?,
val rawKey: String,
val source: String
)
<<plugin-onFileIndexed>>
override suspend fun onFileRemoved(filePath: String) {
repository.deleteRoutesByFile(filePath)
}
private fun extractTimestampFromId(nodeId: String): String? {
val regex = Regex("^(\\d{8})T(\\d{6})")
val match = regex.find(nodeId) ?: return null
return match.groupValues[0]
}
}onFileIndexed: the extraction pipeline
The method has three phases: delete existing routes for the file, collect all ARCOLOGY_KEY sources, then resolve and insert each route.
override suspend fun onFileIndexed(result: ParseResult.Success) {
val filePath = result.file.path
<<plugin-phase-delete>>
<<plugin-phase-collect>>
<<plugin-phase-insert>>
}Phase 1: Delete existing routes
repository.deleteRoutesByFile(filePath)Phase 2: Collect ARCOLOGY_KEY sources and build lookups
Keyword lines (from preamble and sections) are assigned to the file-level node. Properties-drawer keys are assigned to their heading's node ID. We also build a node map and tag map for the resolution phase.
val allKeys = mutableListOf<PublishKeySource>()
val fileLevelNodeId = result.nodes.firstOrNull { it.level == 0 }?.id
// 1. Keyword lines — from result.document.keywordLines (hoisted by parser from preamble + preface + sections)
result.document?.keywordLines?.forEach { kwLine ->
if (kwLine.keyword == "ARCOLOGY_KEY" && kwLine.value.isNotEmpty()) {
allKeys.add(PublishKeySource(fileLevelNodeId, kwLine.value, "keyword-line"))
}
}
// 2. :ARCOLOGY_KEY: in heading/file :PROPERTIES: drawers (from nodeProperties)
result.nodeProperties.forEach { prop ->
if (prop.key == "ARCOLOGY_KEY" && !prop.value.isNullOrEmpty()) {
allKeys.add(PublishKeySource(prop.nodeId, prop.value, "properties-drawer"))
}
}
// Build a node lookup for tag and timestamp resolution
val nodeMap = result.nodes.associateBy { it.id }
val tagsByNode = result.tags.groupBy { it.nodeId }.mapValues { it.value.map { tag -> tag.tag } }
if (debug) {
println("[ArcologyPublishingPlugin] $filePath: ${allKeys.size} ARCOLOGY_KEY sources found, ${result.document?.keywordLines?.size ?: 0} keyword lines total, ${result.nodes.size} nodes, fileLevelNodeId=$fileLevelNodeId")
}Phase 3: Resolve and insert routes
For each key source: parse the key, resolve the node ID, check tags for archive exclusion, extract the timestamp, and insert the route entry.
var inserted = 0
for (source in allKeys) {
val publishKey = PublishKey.parse(source.rawKey)
if (publishKey == null) {
if (debug) println("[ArcologyPublishingPlugin] SKIP: invalid key '${source.rawKey}' (source=${source.source})")
continue
}
val nodeId = source.nodeId ?: fileLevelNodeId
if (nodeId == null) {
if (debug) println("[ArcologyPublishingPlugin] SKIP: no nodeId for key '${source.rawKey}' (source=${source.source})")
continue
}
val node = nodeMap[nodeId]
if (node == null) {
if (debug) println("[ArcologyPublishingPlugin] SKIP: node '$nodeId' not in nodeMap for key '${source.rawKey}' (source=${source.source})")
continue
}
val tags = tagsByNode[nodeId] ?: emptyList()
val isArchived = tags.any { it in EXCLUDE_TAGS }
val timestamp = extractTimestampFromId(nodeId) ?: node.scheduled
if (debug) {
println("[ArcologyPublishingPlugin] INSERT: site=${publishKey.site} path=${publishKey.path} nodeId=$nodeId title=${node.title} archived=$isArchived (source=${source.source})")
}
repository.insertRoute(RouteEntry(
nodeId = nodeId,
site = publishKey.site,
path = publishKey.path,
file = filePath,
title = node.title,
timestamp = timestamp,
isDraft = false,
isArchived = isArchived
))
inserted++
}
if (debug && allKeys.isNotEmpty()) {
println("[ArcologyPublishingPlugin] $filePath: inserted $inserted/${allKeys.size} routes")
}Tests: ArcologyPublishingPlugin extraction, tags, and timestamps
package computer.whatthefuck.arcology.publishing
import computer.whatthefuck.arcology.domain.OrgFile
import computer.whatthefuck.arcology.domain.OrgNode
import computer.whatthefuck.arcology.domain.OrgTag
import computer.whatthefuck.arcology.domain.NodeProperty
import computer.whatthefuck.arcology.parser.ParseResult
import computer.whatthefuck.arcology.indexer.IndexerPlugin
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
import kotlinx.coroutines.runBlocking
import kotlin.time.Instant
class ArcologyPublishingPluginTest {
private fun node(
id: String, level: Int = 1, properties: Map<String, String> = emptyMap(),
title: String? = "Test Node"
): OrgNode {
return OrgNode(
id = id, file = "test.org", level = level, position = 0,
title = title, properties = properties
)
}
private fun parseResultWith(
nodes: List<OrgNode> = emptyList(),
nodeProperties: List<NodeProperty> = emptyList(),
tags: List<OrgTag> = emptyList()
): ParseResult.Success {
return ParseResult.Success(
file = OrgFile(path = "test.org", hash = "abc", accessTime = Instant.fromEpochSeconds(0), modificationTime = Instant.fromEpochSeconds(0)),
nodes = nodes,
links = emptyList(),
tags = tags,
aliases = emptyList(),
refs = emptyList(),
fileProperties = emptyList(),
nodeProperties = nodeProperties,
content = ""
)
} @Test
fun `extracts ARCOLOGY_KEY from node properties`() = runBlocking {
val plugin = ArcologyPublishingPlugin(RecordingRepository())
val result = parseResultWith(
nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "lionsrear/index"))),
nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "lionsrear/index"))
)
val repo = RecordingRepository()
ArcologyPublishingPlugin(repo).onFileIndexed(result)
assertEquals(1, repo.inserted.size)
val entry = repo.inserted.first()
assertEquals("lionsrear", entry.site)
assertEquals("index", entry.path)
assertEquals("20260710T100000", entry.timestamp)
} @Test
fun `skips invalid publish keys`() = runBlocking {
val result = parseResultWith(
nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "no-slash"))),
nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "no-slash"))
)
val repo = RecordingRepository()
ArcologyPublishingPlugin(repo).onFileIndexed(result)
assertEquals(0, repo.inserted.size)
} @Test
fun `draft tag does not exclude entries`() = runBlocking {
val result = parseResultWith(
nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path")),
tags = listOf(OrgTag("20260710T100000", "draft"))
)
val repo = RecordingRepository()
ArcologyPublishingPlugin(repo).onFileIndexed(result)
assertEquals(1, repo.inserted.size)
assertTrue(!repo.inserted.first().isArchived)
} @Test
fun `marks archived entries from ARCHIVE tag`() = runBlocking {
val result = parseResultWith(
nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path")),
tags = listOf(OrgTag("20260710T100000", "ARCHIVE"))
)
val repo = RecordingRepository()
ArcologyPublishingPlugin(repo).onFileIndexed(result)
assertEquals(1, repo.inserted.size)
assertTrue(repo.inserted.first().isArchived)
} @Test
fun `marks archived entries from noexport tag`() = runBlocking {
val result = parseResultWith(
nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path")),
tags = listOf(OrgTag("20260710T100000", "noexport"))
)
val repo = RecordingRepository()
ArcologyPublishingPlugin(repo).onFileIndexed(result)
assertEquals(1, repo.inserted.size)
assertTrue(repo.inserted.first().isArchived)
} @Test
fun `marks archived entries from uppercase NOEXPORT tag`() = runBlocking {
val result = parseResultWith(
nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path")),
tags = listOf(OrgTag("20260710T100000", "NOEXPORT"))
)
val repo = RecordingRepository()
ArcologyPublishingPlugin(repo).onFileIndexed(result)
assertEquals(1, repo.inserted.size)
assertTrue(repo.inserted.first().isArchived)
} @Test
fun `Archive tag does not archive entries`() = runBlocking {
val result = parseResultWith(
nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path")),
tags = listOf(OrgTag("20260710T100000", "Archive"))
)
val repo = RecordingRepository()
ArcologyPublishingPlugin(repo).onFileIndexed(result)
assertEquals(1, repo.inserted.size)
assertTrue(!repo.inserted.first().isArchived)
} @Test
fun `deletes existing routes before inserting`() = runBlocking {
val result = parseResultWith(
nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path"))
)
val repo = RecordingRepository()
ArcologyPublishingPlugin(repo).onFileIndexed(result)
assertEquals(1, repo.deletedFiles.size)
assertEquals("test.org", repo.deletedFiles.first())
} @Test
fun `extracts timestamp from ID format`() = runBlocking {
val result = parseResultWith(
nodes = listOf(node("20260710T100000", level = 0, properties = mapOf("ARCOLOGY_KEY" to "site/path"))),
nodeProperties = listOf(NodeProperty("20260710T100000", "ARCOLOGY_KEY", "site/path"))
)
val repo = RecordingRepository()
ArcologyPublishingPlugin(repo).onFileIndexed(result)
assertEquals("20260710T100000", repo.inserted.first().timestamp)
}}class RecordingRepository : PublishingRepository {
val inserted = mutableListOf<RouteEntry>()
val deletedFiles = mutableListOf<String>()
override suspend fun getAllRoutes(): List<RouteEntry> = emptyList()
override suspend fun getRoutesByPath(path: String): List<RouteEntry> = emptyList()
override suspend fun getPublishedRoutes(): List<RouteEntry> = emptyList()
override suspend fun getPublishedRoutesByPath(path: String): List<RouteEntry> = emptyList()
override suspend fun insertRoute(entry: RouteEntry) { inserted.add(entry) }
override suspend fun deleteRoutesByFile(file: String) { deletedFiles.add(file) }
override suspend fun countRoutes(): Long = 0
override suspend fun getAttachmentsByNode(nodeId: String): List<PublishedAttachment> = emptyList()
override suspend fun getAttachmentsBySourcePath(sourcePath: String): List<PublishedAttachment> = emptyList()
override suspend fun getAttachmentsByBasename(basename: String): List<PublishedAttachment> = emptyList()
override suspend fun insertAttachment(entry: PublishedAttachment) {}
override suspend fun deleteAttachmentsByFile(file: String) {}
override suspend fun countAttachments(): Long = 0
}Attachment Handling
Attachments are the web-side equivalent of arcology.models.Attachment in arcology-django: at index time, every =:ATTACH:=-tagged node's attachment directory is scanned (using AttachmentResolver's org-attach-dir emulation), each file is hashed and crushed — resized and re-encoded via ImageMagick magick, or copied verbatim when it can't be crushed — into a content-addressed cache directory, and one row per size variant is written to published_attachments.
The crush pipeline mirrors django's maybe_crush_file: magick SRC -auto-orient -strip -adaptive-resize {size}> -interlace Plane -quality 80 OUT, with the addition of the shrink-only > geometry modifier so small images are never upscaled, and a second size variant (512 and 2048) for the HTMX progressive enhancement (small image in the <img>, large swapped in on load). When magick is unavailable or a variant's crush fails, the file is copied verbatim under size 0= — attachment serving degrades gracefully and stays hermetic in test environments.
AttachmentCrusher
JVM-only: shells out to the magick binary. The commandRunner function is injectable so tests can fake ImageMagick without a process spawn.
package computer.whatthefuck.arcology.publishing
import java.io.File
import java.security.MessageDigest
private fun runCommand(command: List<String>) {
val process = ProcessBuilder(command).start()
val exitCode = process.waitFor()
if (exitCode != 0) {
val stderr = process.errorStream.bufferedReader().use { it.readText() }
throw IllegalStateException("command '${command.first()}' exited $exitCode: ${stderr.trim()}")
}
}
class AttachmentCrusher(
private val cacheDir: File,
private val commandRunner: (List<String>) -> Unit = ::runCommand
) {
data class CrushedAttachment(val sourceHash: String, val type: String, val sizes: List<Int>)
/** Extensions ImageMagick can crush; everything else is copied verbatim. */
val crushableExtensions = setOf("jpg", "jpeg", "png", "webp", "avif")
/** Crush variants, smallest first. The small variant is the initial <img> src. */
val crushSizes = listOf(1080, 2160)
init {
if (!cacheDir.exists()) {
cacheDir.mkdirs()
}
}
private val magickAvailable: Boolean by lazy {
try {
commandRunner(listOf("magick", "-version"))
true
} catch (e: Exception) {
false
}
}
fun crush(source: File): CrushedAttachment {
val hash = sha256Hex(source.readBytes())
val ext = source.extension.lowercase()
val sizes = if (ext in crushableExtensions && magickAvailable) {
val succeeded = crushSizes.filter { size -> crushVariant(source, hash, ext, size) }
if (succeeded.isEmpty()) {
copyVerbatim(source, hash, ext)
listOf(0)
} else {
succeeded
}
} else {
copyVerbatim(source, hash, ext)
listOf(0)
}
return CrushedAttachment(sourceHash = hash, type = ext, sizes = sizes)
}
fun filePathFor(sourceHash: String, size: Int, type: String): File =
File(cacheDir, "$sourceHash-$size.$type")
private fun crushVariant(source: File, hash: String, ext: String, size: Int): Boolean {
val target = filePathFor(hash, size, ext)
if (target.exists() && target.length() > 0) return true
return try {
commandRunner(listOf(
"magick", source.absolutePath,
"-auto-orient", "-strip",
"-adaptive-resize", "${size}>",
"-interlace", "Plane", "-quality", "80",
target.absolutePath
))
target.exists() && target.length() > 0
} catch (e: Exception) {
target.delete()
false
}
}
private fun copyVerbatim(source: File, hash: String, ext: String) {
val target = filePathFor(hash, 0, ext)
if (!target.exists()) {
source.copyTo(target, overwrite = true)
}
}
companion object {
fun sha256Hex(bytes: ByteArray): String =
MessageDigest.getInstance("SHA-256").digest(bytes).joinToString("") { "%02x".format(it) }
/**
* Default attachment cache dir, overridable with the ARCOLOGY_ATTACHMENT_DIR
* environment variable (same variable arcology-django uses). The indexer
* writes here; the web server serves from here.
*/
fun defaultCacheDir(): File =
File(System.getenv("ARCOLOGY_ATTACHMENT_DIR") ?: "/tmp/arcology-cache/attachments")
}
}ArcologyAttachmentPlugin
The indexer plugin. It follows the ArcologyPublishingPlugin pattern: rows for the file are deleted up front, then re-inserted. Attachment directories are resolved with AttachmentResolver, which mimics =org-attach-dir='s org-attach-id-to-path-function-list probing — the timestamp format data/{id[0..5]}/{id[6..]} is tried first (org-roam temporally-unique IDs), then the UUID format data/{id[0..1]}/{id[2..]}.
package computer.whatthefuck.arcology.publishing
import computer.whatthefuck.arcology.indexer.AttachmentResolver
import computer.whatthefuck.arcology.indexer.IndexerPlugin
import computer.whatthefuck.arcology.parser.ParseResult
import java.io.Fileclass ArcologyAttachmentPlugin(
private val repository: PublishingRepository,
private val crusher: AttachmentCrusher,
private val attachmentResolver: AttachmentResolver,
private val orgRoot: String,
private val debug: Boolean = false
) : IndexerPlugin {
override suspend fun onFileIndexed(result: ParseResult.Success) {
repository.deleteAttachmentsByFile(result.file.path)
val attachNodeIds = result.tags
.filter { it.tag == "ATTACH" }
.map { it.nodeId }
.toSet()
if (attachNodeIds.isEmpty()) return
for (nodeId in attachNodeIds) {
val attachments = try {
attachmentResolver.resolveAttachments(nodeId, result.file.path)
} catch (e: Exception) {
if (debug) println("[ArcologyAttachmentPlugin] $result.file.path: failed to resolve attachments for $nodeId: ${e.message}")
emptyList()
}
for (attachment in attachments) {
val source = resolveSource(attachment.resolvedPath) ?: continue
val crushed = try {
crusher.crush(source)
} catch (e: Exception) {
if (debug) println("[ArcologyAttachmentPlugin] $result.file.path: failed to crush ${attachment.resolvedPath}: ${e.message}")
continue
}
for (size in crushed.sizes) {
repository.insertAttachment(PublishedAttachment(
nodeId = nodeId,
sourcePath = attachment.resolvedPath,
sourceHash = crushed.sourceHash,
size = size,
type = crushed.type,
file = result.file.path
))
}
}
}
}
override suspend fun onFileRemoved(filePath: String) {
repository.deleteAttachmentsByFile(filePath)
}
private fun resolveSource(resolvedPath: String): File? {
val candidate = if (resolvedPath.startsWith("/")) File(resolvedPath) else File(orgRoot, resolvedPath)
return if (candidate.exists() && candidate.isFile) candidate else null
}
}Tests: AttachmentCrusher
package computer.whatthefuck.arcology.publishing
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
import kotlin.test.assertFalse
import java.io.File
import java.nio.file.Files
class AttachmentCrusherTest {
private fun tempCrusherDir(): File = Files.createTempDirectory("arcology-crush-test").toFile()
private fun tempSource(ext: String, content: ByteArray = "source-bytes".toByteArray()): File {
val dir = Files.createTempDirectory("arcology-crush-src").toFile()
return File(dir, "photo.$ext").apply { writeBytes(content) }
}
/** Fake magick: succeeds for `-version`, writes a marker to the output path for crush commands. */
private val fakeMagick: (List<String>) -> Unit = { command ->
if (command.getOrNull(1) != "-version") {
val output = command.last()
File(output).writeBytes("crushed:${command[command.indexOf("-adaptive-resize") + 1]}".toByteArray())
}
}
/** Fake magick that fails every real invocation (but probes healthy). */
private val failingMagick: (List<String>) -> Unit = { command ->
if (command.getOrNull(1) != "-version") {
throw IllegalStateException("magick not found")
}
} @Test
fun `crushes image to both size variants`() {
val dir = tempCrusherDir()
val source = tempSource("jpg")
val crusher = AttachmentCrusher(dir, fakeMagick)
val crushed = crusher.crush(source)
assertEquals(2, crushed.sizes.size)
assertTrue(crushed.sizes.containsAll(listOf(1080, 2160)))
assertEquals("jpg", crushed.type)
assertTrue(crusher.filePathFor(crushed.sourceHash, 1080, "jpg").exists())
assertTrue(crusher.filePathFor(crushed.sourceHash, 2160, "jpg").exists())
} @Test
fun `crush output is content addressed by source hash`() {
val dir = tempCrusherDir()
val source = tempSource("png", content = "known-bytes".toByteArray())
val crusher = AttachmentCrusher(dir, fakeMagick)
val crushed = crusher.crush(source)
assertEquals(AttachmentCrusher.sha256Hex("known-bytes".toByteArray()), crushed.sourceHash)
} @Test
fun `uncrushable extension copied verbatim with size zero`() {
val dir = tempCrusherDir()
val source = tempSource("pdf")
val crusher = AttachmentCrusher(dir, fakeMagick)
val crushed = crusher.crush(source)
assertEquals(listOf(0), crushed.sizes)
val verbatim = crusher.filePathFor(crushed.sourceHash, 0, "pdf")
assertTrue(verbatim.exists())
assertEquals("source-bytes", verbatim.readText())
} @Test
fun `falls back to verbatim when magick fails`() {
val dir = tempCrusherDir()
val source = tempSource("jpg")
val crusher = AttachmentCrusher(dir, failingMagick)
val crushed = crusher.crush(source)
assertEquals(listOf(0), crushed.sizes)
assertTrue(crusher.filePathFor(crushed.sourceHash, 0, "jpg").exists())
} @Test
fun `falls back to verbatim when magick binary is missing`() {
val dir = tempCrusherDir()
val source = tempSource("jpg")
val crusher = AttachmentCrusher(dir, { _ -> throw java.io.IOException("no magick") })
val crushed = crusher.crush(source)
assertEquals(listOf(0), crushed.sizes)
assertTrue(crusher.filePathFor(crushed.sourceHash, 0, "jpg").exists())
} @Test
fun `existing variant file is reused without re-crushing`() {
val dir = tempCrusherDir()
val source = tempSource("jpg")
val crusher = AttachmentCrusher(dir, fakeMagick)
val first = crusher.crush(source)
// Drop the 2048 variant, then make magick fail for real invocations:
// the existing 512 file must be reused untouched; the failed 2048 is dropped.
crusher.filePathFor(first.sourceHash, 2160, "jpg").delete()
val second = AttachmentCrusher(dir, failingMagick).crush(source)
assertEquals(listOf(1080), second.sizes)
assertEquals(first.sourceHash, second.sourceHash)
} @Test
fun `sha256Hex produces known digest`() {
assertEquals(
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
AttachmentCrusher.sha256Hex("abc".toByteArray())
)
}}Tests: ArcologyAttachmentPlugin
The fake filesystem implements only what AttachmentResolver touches: listFilesInDirectory returns canned listings keyed by directory, everything else is stubbed. The plugin test drives onFileIndexed with a hand-built ParseResult.Success, same as ArcologyPublishingPluginTest.
package computer.whatthefuck.arcology.publishing
import computer.whatthefuck.arcology.domain.OrgFile
import computer.whatthefuck.arcology.domain.OrgNode
import computer.whatthefuck.arcology.domain.OrgTag
import computer.whatthefuck.arcology.indexer.AttachmentResolver
import computer.whatthefuck.arcology.indexer.FileSystemInterface
import computer.whatthefuck.arcology.parser.ParseResult
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.emptyFlow
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
import kotlinx.coroutines.runBlocking
import kotlin.time.Instant
class ArcologyAttachmentPluginTest {
/** Directory listing fake: maps "relative/dir" -> list of file paths. */
private class FakeAttachmentFs(private val listings: Map<String, List<String>>) : FileSystemInterface {
override suspend fun fileExists(path: String): Boolean = false
override suspend fun readFile(path: String): String = ""
override suspend fun writeFile(path: String, content: String) {}
override suspend fun getLastModified(path: String): Instant = Instant.fromEpochSeconds(0)
override fun listOrgFiles(path: String, recursive: Boolean): Flow<String> = emptyFlow()
override suspend fun readIgnoreFile(rootPath: String): String? = null
override suspend fun listFilesInDirectory(baseFilePath: String, relativeDir: String): List<String> =
listings[relativeDir] ?: emptyList()
}
private class RecordingAttachmentRepository : PublishingRepository {
val inserted = mutableListOf<PublishedAttachment>()
val deletedFiles = mutableListOf<String>()
override suspend fun getAllRoutes(): List<RouteEntry> = emptyList()
override suspend fun getRoutesByPath(path: String): List<RouteEntry> = emptyList()
override suspend fun getPublishedRoutes(): List<RouteEntry> = emptyList()
override suspend fun getPublishedRoutesByPath(path: String): List<RouteEntry> = emptyList()
override suspend fun insertRoute(entry: RouteEntry) {}
override suspend fun deleteRoutesByFile(file: String) {}
override suspend fun countRoutes(): Long = 0
override suspend fun getAttachmentsByNode(nodeId: String): List<PublishedAttachment> = emptyList()
override suspend fun getAttachmentsBySourcePath(sourcePath: String): List<PublishedAttachment> = emptyList()
override suspend fun getAttachmentsByBasename(basename: String): List<PublishedAttachment> = emptyList()
override suspend fun insertAttachment(entry: PublishedAttachment) { inserted.add(entry) }
override suspend fun deleteAttachmentsByFile(file: String) { deletedFiles.add(file) }
override suspend fun countAttachments(): Long = 0
}
private fun parseResultWith(
nodeId: String,
tags: List<OrgTag> = listOf(OrgTag(nodeId, "ATTACH")),
filePath: String = "test.org"
): ParseResult.Success {
return ParseResult.Success(
file = OrgFile(path = filePath, hash = "abc", accessTime = Instant.fromEpochSeconds(0), modificationTime = Instant.fromEpochSeconds(0)),
nodes = listOf(OrgNode(id = nodeId, file = filePath, level = 1, position = 0, title = "Node", properties = emptyMap())),
links = emptyList(),
tags = tags,
aliases = emptyList(),
refs = emptyList(),
fileProperties = emptyList(),
nodeProperties = emptyList(),
content = ""
)
}
/** Fake magick that writes a marker file to the command's output path. */
private val fakeMagick: (List<String>) -> Unit = { command ->
if (command.getOrNull(1) != "-version") {
java.io.File(command.last()).writeBytes("crushed".toByteArray())
}
} @Test
fun `inserts rows for ATTACH tagged node with both size variants`() = runBlocking {
val nodeId = "20260710T100000"
// AttachmentResolver probes data/{id[0..5]}/{id[6..]} first for timestamp IDs
val attachDir = "data/202607/10T100000"
val orgRoot = java.nio.file.Files.createTempDirectory("arcology-attach-org").toFile()
java.io.File(orgRoot, "$attachDir").mkdirs()
val source = java.io.File(orgRoot, "$attachDir/photo.jpg").apply { writeBytes("photo-bytes".toByteArray()) }
val fs = FakeAttachmentFs(mapOf(attachDir to listOf(source.absolutePath)))
val repo = RecordingAttachmentRepository()
val crusher = AttachmentCrusher(
java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(),
fakeMagick
)
val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(fs), orgRoot.absolutePath)
plugin.onFileIndexed(parseResultWith(nodeId))
assertEquals(2, repo.inserted.size)
val sizes = repo.inserted.map { it.size }.sorted()
assertEquals(listOf(1080, 2160), sizes)
val expectedHash = AttachmentCrusher.sha256Hex("photo-bytes".toByteArray())
assertTrue(repo.inserted.all { it.sourceHash == expectedHash })
assertTrue(repo.inserted.all { it.type == "jpg" })
assertTrue(repo.inserted.all { it.nodeId == nodeId })
assertTrue(repo.inserted.all { it.file == "test.org" })
assertTrue(repo.inserted.all { it.sourcePath == source.absolutePath })
} @Test
fun `non-image attachment stored verbatim with size zero`() = runBlocking {
val nodeId = "20260710T100000"
val attachDir = "data/202607/10T100000"
val orgRoot = java.nio.file.Files.createTempDirectory("arcology-attach-org").toFile()
java.io.File(orgRoot, "$attachDir").mkdirs()
val source = java.io.File(orgRoot, "$attachDir/doc.pdf").apply { writeBytes("pdf-bytes".toByteArray()) }
val fs = FakeAttachmentFs(mapOf(attachDir to listOf(source.absolutePath)))
val repo = RecordingAttachmentRepository()
val crusher = AttachmentCrusher(
java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(),
fakeMagick
)
val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(fs), orgRoot.absolutePath)
plugin.onFileIndexed(parseResultWith(nodeId))
assertEquals(1, repo.inserted.size)
assertEquals(0, repo.inserted.first().size)
assertEquals("pdf", repo.inserted.first().type)
} @Test
fun `nodes without ATTACH tag yield no rows`() = runBlocking {
val repo = RecordingAttachmentRepository()
val crusher = AttachmentCrusher(java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(), fakeMagick)
val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(FakeAttachmentFs(emptyMap())), "/tmp")
plugin.onFileIndexed(parseResultWith("20260710T100000", tags = listOf(OrgTag("20260710T100000", "publish"))))
assertEquals(0, repo.inserted.size)
} @Test
fun `re-index deletes existing rows for the file first`() = runBlocking {
val repo = RecordingAttachmentRepository()
val crusher = AttachmentCrusher(java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(), fakeMagick)
val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(FakeAttachmentFs(emptyMap())), "/tmp")
plugin.onFileIndexed(parseResultWith("20260710T100000"))
assertEquals(1, repo.deletedFiles.size)
assertEquals("test.org", repo.deletedFiles.first())
} @Test
fun `onFileRemoved deletes rows for the file`() = runBlocking {
val repo = RecordingAttachmentRepository()
val crusher = AttachmentCrusher(java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(), fakeMagick)
val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(FakeAttachmentFs(emptyMap())), "/tmp")
plugin.onFileRemoved("gone.org")
assertEquals(listOf("gone.org"), repo.deletedFiles)
} @Test
fun `missing attachment directory yields no rows`() = runBlocking {
val fs = FakeAttachmentFs(emptyMap())
val repo = RecordingAttachmentRepository()
val crusher = AttachmentCrusher(java.nio.file.Files.createTempDirectory("arcology-attach-cache").toFile(), fakeMagick)
val plugin = ArcologyAttachmentPlugin(repo, crusher, AttachmentResolver(fs), "/tmp")
plugin.onFileIndexed(parseResultWith("20260710T100000"))
assertEquals(0, repo.inserted.size)
}}Data Models
PublishKey
A publish key is the SITE/rest/of/path string from ARCOLOGY_KEY. This data class parses and validates that format.
package computer.whatthefuck.arcology.publishing
data class PublishKey(
val site: String,
val path: String
) {
companion object {
fun parse(raw: String): PublishKey? {
val trimmed = raw.trim()
if (trimmed.isEmpty()) return null
val slashIndex = trimmed.indexOf('/')
if (slashIndex <= 0 || slashIndex == trimmed.length - 1) return null
val site = trimmed.substring(0, slashIndex).trim()
val path = trimmed.substring(slashIndex + 1).trim()
if (site.isEmpty() || path.isEmpty()) return null
return PublishKey(site, path)
}
}
}Tests: PublishKey parsing
package computer.whatthefuck.arcology.publishing
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertNull
class PublishKeyTest { @Test
fun `parse valid key with path`() {
val key = PublishKey.parse("lionsrear/index")
assertEquals("lionsrear", key?.site)
assertEquals("index", key?.path)
} @Test
fun `parse key with nested path`() {
val key = PublishKey.parse("blog/2025/my-post")
assertEquals("blog", key?.site)
assertEquals("2025/my-post", key?.path)
} @Test
fun `parse returns null for empty string`() {
assertNull(PublishKey.parse(""))
} @Test
fun `parse returns null for missing slash`() {
assertNull(PublishKey.parse("lionsrear"))
} @Test
fun `parse returns null for leading slash`() {
assertNull(PublishKey.parse("/index"))
} @Test
fun `parse returns null for trailing slash`() {
assertNull(PublishKey.parse("lionsrear/"))
} @Test
fun `parse trims whitespace`() {
val key = PublishKey.parse(" lionsrear/index ")
assertEquals("lionsrear", key?.site)
assertEquals("index", key?.path)
}}DomainMap
The domain map resolves SITE prefixes to domain names, and reverse-resolves domain names back to sites. It also carries per-site metadata (title, optional CSS file, optional link color) consumed by the page templates and the dynamic /sites.css endpoint. It loads from a domains.json file generated from an org table via an arroyo lua block (see domains.org). The JSON shape is:
{
"sites": [
{"key":"lionsrear","title":"The Lions Rear","cssFile":"arcology/css/lionsrear.css","linkColor":"#a64040","domains":["rix.si","thelionsrear.com"]}
]
}fromMap is kept as a convenience for tests and callers that build a domain map in memory (single-domain sites, no metadata). The JSON load path is the production path.
Data classes and imports
package computer.whatthefuck.arcology.publishing
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.decodeFromString
import java.io.File
import java.io.InputStream
@Serializable
data class SiteMeta(
val key: String,
val title: String? = null,
val cssFile: String? = null,
val linkColor: String? = null,
val hljsTheme: String? = null,
val domains: List<String> = emptyList()
)
@Serializable
private data class DomainConfig(val sites: List<SiteMeta>)Class body: resolution and metadata lookup
class DomainMap private constructor(
private val sites: List<SiteMeta>,
private val domainToSite: Map<String, String>
) {
fun resolve(site: String): String? = meta(site)?.domains?.firstOrNull()
fun resolveAll(site: String): List<String> = meta(site)?.domains ?: emptyList()
fun resolveSite(domain: String): String? = domainToSite[domain.lowercase()]
fun meta(site: String): SiteMeta? = sites.firstOrNull { it.key == site }
fun allSites(): Set<String> = sites.map { it.key }.toSet()Companion object: loading from JSON and in-memory construction
companion object {
private val json = Json { ignoreUnknownKeys = true }
fun load(file: File): DomainMap {
file.inputStream().use { stream ->
return load(stream)
}
}
fun load(stream: InputStream): DomainMap {
val text = stream.bufferedReader().use { it.readText() }
return parse(text)
}
fun parse(text: String): DomainMap {
val config: DomainConfig = json.decodeFromString(text)
val domainToSite = mutableMapOf<String, String>()
for (site in config.sites) {
for (domain in site.domains) {
domainToSite[domain.lowercase()] = site.key
}
}
return DomainMap(config.sites, domainToSite)
}
fun fromMap(map: Map<String, String>): DomainMap {
val sites = map.map { (key, domain) ->
val domains = domain.split(",").map { it.trim() }.filter { it.isNotEmpty() }
SiteMeta(key = key, domains = domains)
}
val domainToSite = mutableMapOf<String, String>()
for (site in sites) {
for (domain in site.domains) {
domainToSite[domain.lowercase()] = site.key
}
}
return DomainMap(sites, domainToSite)
}
}
}Tests: DomainMap loading, resolution, and metadata
package computer.whatthefuck.arcology.publishing
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertNull
import kotlin.test.assertNotNull
import java.io.ByteArrayInputStream
class DomainMapTest { @Test
fun `load from JSON stream`() {
val input = ByteArrayInputStream("""
{
"sites": [
{"key":"lionsrear","title":"The Lions Rear","cssFile":"arcology/css/lionsrear.css","linkColor":"#a64040","domains":["rix.si","thelionsrear.com"]},
{"key":"arcology","title":"Arcology Engine","cssFile":"arcology/css/arcology.css","linkColor":"#4060a6","domains":["engine.arcology.garden"]}
]
}
""".trimIndent().toByteArray())
val map = DomainMap.load(input)
assertEquals("rix.si", map.resolve("lionsrear"))
assertEquals("engine.arcology.garden", map.resolve("arcology"))
} @Test
fun `resolve returns null for unknown site`() {
val map = DomainMap.fromMap(mapOf("lionsrear" to "thelionsrear.com"))
assertNull(map.resolve("unknown"))
} @Test
fun `allSites returns all keys`() {
val map = DomainMap.fromMap(mapOf(
"lionsrear" to "thelionsrear.com",
"arcology" to "arcology.garden"
))
assertEquals(setOf("lionsrear", "arcology"), map.allSites())
} @Test
fun `meta returns site metadata from JSON`() {
val input = ByteArrayInputStream("""
{
"sites": [
{"key":"lionsrear","title":"The Lions Rear","cssFile":"arcology/css/lionsrear.css","linkColor":"#a64040","hljsTheme":"nord","domains":["rix.si","thelionsrear.com"]}
]
}
""".trimIndent().toByteArray())
val map = DomainMap.load(input)
val meta = map.meta("lionsrear")
assertNotNull(meta)
assertEquals("The Lions Rear", meta.title)
assertEquals("arcology/css/lionsrear.css", meta.cssFile)
assertEquals("#a64040", meta.linkColor)
assertEquals("nord", meta.hljsTheme)
assertEquals(listOf("rix.si", "thelionsrear.com"), meta.domains)
} @Test
fun `meta hljsTheme defaults to null when absent`() {
val input = ByteArrayInputStream("""
{
"sites": [
{"key":"lionsrear","title":"The Lions Rear","cssFile":"arcology/css/lionsrear.css","linkColor":"#a64040","domains":["rix.si"]}
]
}
""".trimIndent().toByteArray())
val map = DomainMap.load(input)
val meta = map.meta("lionsrear")
assertNotNull(meta)
assertNull(meta.hljsTheme)
} @Test
fun `meta returns null for unknown site`() {
val map = DomainMap.fromMap(mapOf("lionsrear" to "thelionsrear.com"))
assertNull(map.meta("unknown"))
} @Test
fun `fromMap produces SiteMeta with null title and cssFile`() {
val map = DomainMap.fromMap(mapOf("lionsrear" to "thelionsrear.com"))
val meta = map.meta("lionsrear")
assertNotNull(meta)
assertEquals("lionsrear", meta.key)
assertNull(meta.title)
assertNull(meta.cssFile)
assertNull(meta.linkColor)
assertNull(meta.hljsTheme)
assertEquals(listOf("thelionsrear.com"), meta.domains)
} @Test
fun `resolveSite reverse-resolves domain to site`() {
val input = ByteArrayInputStream("""
{
"sites": [
{"key":"lionsrear","title":null,"cssFile":null,"linkColor":null,"domains":["rix.si","thelionsrear.com"]}
]
}
""".trimIndent().toByteArray())
val map = DomainMap.load(input)
assertEquals("lionsrear", map.resolveSite("rix.si"))
assertEquals("lionsrear", map.resolveSite("THELIONSREAR.COM"))
assertNull(map.resolveSite("unknown.example"))
} @Test
fun `resolveAll returns all domains for site`() {
val input = ByteArrayInputStream("""
{
"sites": [
{"key":"lionsrear","title":null,"cssFile":null,"linkColor":null,"domains":["rix.si","thelionsrear.com"]}
]
}
""".trimIndent().toByteArray())
val map = DomainMap.load(input)
assertEquals(listOf("rix.si", "thelionsrear.com"), map.resolveAll("lionsrear"))
assertEquals(emptyList(), map.resolveAll("unknown"))
} @Test
fun `parse handles JSON with unknown fields without failing`() {
val input = """
{
"sites": [
{"key":"lionsrear","title":null,"cssFile":null,"linkColor":null,"domains":["thelionsrear.com"],"futureField":"ignored"}
]
}
""".trimIndent()
val map = DomainMap.parse(input)
assertEquals("thelionsrear.com", map.resolve("lionsrear"))
}}Route Table
The route table is the in-memory structure used by the web server to look up paths and render pages.
package computer.whatthefuck.arcology.publishing
data class RouteEntry(
val nodeId: String,
val site: String,
val path: String,
val file: String,
val title: String?,
val timestamp: String?,
val isDraft: Boolean,
val isArchived: Boolean
)
/**
* One row of the published_attachments cache: a single size variant of a
* crushed (or verbatim-copied) attachment. The crushed file lives on disk at
* {ATTACHMENT_DIR}/{urlName}; `size` is the crush target in pixels, or 0 for
* verbatim copies. See [[AttachmentCrusher]].
*/
data class PublishedAttachment(
val nodeId: String,
val sourcePath: String,
val sourceHash: String,
val size: Int,
val type: String,
val file: String
) {
/** Content-addressed file name; the URL path is /attachment/{urlName}. */
val urlName: String get() = "$sourceHash-$size.$type"
}
data class RouteTable(
val routes: Map<String, List<RouteEntry>>
) {
fun getEntries(path: String): List<RouteEntry> = routes[path] ?: emptyList()
fun getEntriesByKey(fullKey: String): List<RouteEntry> {
val slashIndex = fullKey.indexOf('/')
if (slashIndex <= 0) return emptyList()
val site = fullKey.substring(0, slashIndex)
val path = fullKey.substring(slashIndex + 1)
return routes[path]?.filter { it.site == site } ?: emptyList()
}
fun allKeys(): Set<String> = routes.values.flatten().map { "${it.site}/${it.path}" }.toSet()
fun allPaths(): Set<String> = routes.keys
val totalRoutes: Int get() = routes.values.sumOf { it.size }
}PublishingRepository
The repository interface and SQLDelight implementation for the published_routes table.
Interface
interface PublishingRepository {
suspend fun getAllRoutes(): List<RouteEntry>
suspend fun getRoutesByPath(path: String): List<RouteEntry>
suspend fun getPublishedRoutes(): List<RouteEntry>
suspend fun getPublishedRoutesByPath(path: String): List<RouteEntry>
suspend fun insertRoute(entry: RouteEntry)
suspend fun deleteRoutesByFile(file: String)
suspend fun countRoutes(): Long
suspend fun getAttachmentsByNode(nodeId: String): List<PublishedAttachment>
suspend fun getAttachmentsBySourcePath(sourcePath: String): List<PublishedAttachment>
suspend fun getAttachmentsByBasename(basename: String): List<PublishedAttachment>
suspend fun insertAttachment(entry: PublishedAttachment)
suspend fun deleteAttachmentsByFile(file: String)
suspend fun countAttachments(): Long
}Implementation: class declaration, dispatcher, and row mapper
class PublishingRepositoryImpl(
private val database: ArcologyDatabase
) : PublishingRepository {
@OptIn(ExperimentalCoroutinesApi::class)
private val dbDispatcher = Dispatchers.IO.limitedParallelism(1)
private fun computer.whatthefuck.arcology.db.Published_routes.toRouteEntry(): RouteEntry {
return RouteEntry(
nodeId = node_id,
site = site,
path = path,
file = file_,
title = title,
timestamp = timestamp,
isDraft = is_draft != 0L,
isArchived = is_archived != 0L
)
}
private fun computer.whatthefuck.arcology.db.Published_attachments.toPublishedAttachment(): PublishedAttachment {
return PublishedAttachment(
nodeId = node_id,
sourcePath = source_path,
sourceHash = source_hash,
size = size.toInt(),
type = type,
file = file_
)
}Read methods
override suspend fun getAllRoutes(): List<RouteEntry> {
return withContext(Dispatchers.IO) {
database.publishingQueries.selectAllPublishedRoutes().executeAsList().map { it.toRouteEntry() }
}
}
override suspend fun getRoutesByPath(path: String): List<RouteEntry> {
return withContext(Dispatchers.IO) {
database.publishingQueries.selectRoutesByPath(path).executeAsList().map { it.toRouteEntry() }
}
}
override suspend fun getPublishedRoutes(): List<RouteEntry> {
return withContext(Dispatchers.IO) {
database.publishingQueries.selectPublishedRoutes().executeAsList().map { it.toRouteEntry() }
}
}
override suspend fun getPublishedRoutesByPath(path: String): List<RouteEntry> {
return withContext(Dispatchers.IO) {
database.publishingQueries.selectPublishedRoutesByPath(path).executeAsList().map { it.toRouteEntry() }
}
}Write methods
override suspend fun insertRoute(entry: RouteEntry) {
database.publishingQueries.insertPublishedRoute(
entry.nodeId,
entry.site,
entry.path,
entry.file,
entry.title,
entry.timestamp,
if (entry.isDraft) 1L else 0L,
if (entry.isArchived) 1L else 0L
)
}
override suspend fun deleteRoutesByFile(file: String) {
database.publishingQueries.deletePublishedRoutesByFile(file)
}
override suspend fun countRoutes(): Long {
return withContext(Dispatchers.IO) {
database.publishingQueries.countPublishedRoutes().executeAsOne()
}
}
override suspend fun getAttachmentsByNode(nodeId: String): List<PublishedAttachment> {
return withContext(Dispatchers.IO) {
database.publishingQueries.selectPublishedAttachmentsByNode(nodeId).executeAsList().map { it.toPublishedAttachment() }
}
}
override suspend fun getAttachmentsBySourcePath(sourcePath: String): List<PublishedAttachment> {
return withContext(Dispatchers.IO) {
database.publishingQueries.selectPublishedAttachmentsByPath(sourcePath).executeAsList().map { it.toPublishedAttachment() }
}
}
override suspend fun getAttachmentsByBasename(basename: String): List<PublishedAttachment> {
return withContext(Dispatchers.IO) {
database.publishingQueries.selectPublishedAttachmentsByBasename(basename).executeAsList().map { it.toPublishedAttachment() }
}
}
override suspend fun insertAttachment(entry: PublishedAttachment) {
database.publishingQueries.insertPublishedAttachment(
entry.nodeId,
entry.sourcePath,
entry.sourceHash,
entry.size.toLong(),
entry.type,
entry.file
)
}
override suspend fun deleteAttachmentsByFile(file: String) {
database.publishingQueries.deletePublishedAttachmentsByFile(file)
}
override suspend fun countAttachments(): Long {
return withContext(Dispatchers.IO) {
database.publishingQueries.countPublishedAttachments().executeAsOne()
}
}}RouteTableBuilder
Builds a RouteTable from the PublishingRepository by querying published (non-draft, non-archived) routes, grouping them by path, and sorting by timestamp.
package computer.whatthefuck.arcology.publishing
class RouteTableBuilder {
suspend fun build(repository: PublishingRepository): RouteTable {
val routes = repository.getPublishedRoutes()
val grouped = routes.groupBy { it.path }
return RouteTable(routes = grouped)
}
}Tests: RouteTableBuilder grouping and filtering
package computer.whatthefuck.arcology.publishing
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
import kotlinx.coroutines.runBlocking
class RouteTableBuilderTest {
private fun routeEntry(
nodeId: String, site: String, path: String, timestamp: String,
isDraft: Boolean = false, isArchived: Boolean = false
): RouteEntry {
return RouteEntry(
nodeId = nodeId, site = site, path = path, file = "test.org",
title = "Test", timestamp = timestamp, isDraft = isDraft, isArchived = isArchived
)
} @Test
fun `build groups routes by path`() = runBlocking {
val repo = FakePublishingRepository(listOf(
routeEntry("node1", "site", "index", "20260701T100000"),
routeEntry("node2", "site", "index", "20260702T100000"),
routeEntry("node3", "site", "about", "20260703T100000")
))
val table = RouteTableBuilder().build(repo)
assertEquals(2, table.allPaths().size)
assertEquals(2, table.getEntries("index").size)
assertEquals(1, table.getEntries("about").size)
} @Test
fun `build excludes drafts and archived`() = runBlocking {
val repo = FakePublishingRepository(listOf(
routeEntry("node1", "site", "index", "20260701T100000", isDraft = false),
routeEntry("node2", "site", "index", "20260702T100000", isDraft = true),
routeEntry("node3", "site", "index", "20260703T100000", isArchived = true)
))
val table = RouteTableBuilder().build(repo)
assertEquals(1, table.getEntries("index").size)
assertEquals("node1", table.getEntries("index").first().nodeId)
} @Test
fun `totalRoutes counts all entries`() = runBlocking {
val repo = FakePublishingRepository(listOf(
routeEntry("node1", "site", "index", "20260701T100000"),
routeEntry("node2", "site", "about", "20260702T100000")
))
val table = RouteTableBuilder().build(repo)
assertEquals(2, table.totalRoutes)
}}class FakePublishingRepository(private val routes: List<RouteEntry>) : PublishingRepository {
override suspend fun getAllRoutes(): List<RouteEntry> = routes
override suspend fun getRoutesByPath(path: String): List<RouteEntry> = routes.filter { it.path == path }
override suspend fun getPublishedRoutes(): List<RouteEntry> = routes.filter { !it.isDraft && !it.isArchived }
override suspend fun getPublishedRoutesByPath(path: String): List<RouteEntry> =
routes.filter { it.path == path && !it.isDraft && !it.isArchived }
override suspend fun insertRoute(entry: RouteEntry) {}
override suspend fun deleteRoutesByFile(file: String) {}
override suspend fun countRoutes(): Long = routes.size.toLong()
override suspend fun getAttachmentsByNode(nodeId: String): List<PublishedAttachment> = emptyList()
override suspend fun getAttachmentsBySourcePath(sourcePath: String): List<PublishedAttachment> = emptyList()
override suspend fun getAttachmentsByBasename(basename: String): List<PublishedAttachment> = emptyList()
override suspend fun insertAttachment(entry: PublishedAttachment) {}
override suspend fun deleteAttachmentsByFile(file: String) {}
override suspend fun countAttachments(): Long = 0
}Tangle Targets
Publishing.sq
<<publishing-schema>>
<<publishing-queries>>
<<attachments-schema>>
<<attachments-queries>>PublishKey.kt
<<publish-key>>DomainMap.kt
<<domain-map-preamble>>
<<domain-map-class>>
<<domain-map-companion>>RouteTable.kt
<<route-table>>PublishingRepository.kt
@file:OptIn(kotlin.time.ExperimentalTime::class)
package computer.whatthefuck.arcology.publishing
import computer.whatthefuck.arcology.db.ArcologyDatabase
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.ExperimentalCoroutinesApi
import kotlinx.coroutines.withContext
<<publishing-repo-interface>>
<<publishing-repo-impl-core>>
<<publishing-repo-impl-body>>
<<publishing-repo-impl-end>>RouteTableBuilder.kt
<<route-table-builder>>ExcludeTags.kt
<<exclude-tags>>ArcologyPublishingPlugin.kt
<<plugin-preamble>>
<<plugin-core>>PublishKeyTest.kt
<<publish-key-test-prelude>>
<<publish-key-test>>
<<publish-key-test-end>>DomainMapTest.kt
<<domain-map-test-prelude>>
<<domain-map-test>>
<<domain-map-test-end>>RouteTableBuilderTest.kt
<<route-table-builder-test-prelude>>
<<route-table-builder-test>>
<<route-table-builder-test-end>>
<<fake-publishing-repo>>ArcologyPublishingPluginTest.kt
<<publishing-plugin-test-prelude>>
<<publishing-plugin-test>>
<<publishing-plugin-test-end>>
<<recording-repository>>AttachmentCrusher.kt
<<attachment-crusher>>ArcologyAttachmentPlugin.kt
<<attachment-plugin-preamble>>
<<attachment-plugin-core>>AttachmentCrusherTest.kt
<<attachment-crusher-test-prelude>>
<<attachment-crusher-test>>
<<attachment-crusher-test-end>>ArcologyAttachmentPluginTest.kt
<<attachment-plugin-test-prelude>>
<<attachment-plugin-test>>
<<attachment-plugin-test-end>>Related Modules
The Indexer Pipeline — provides
IndexerPlugin,ParseResultData Models and Persistence Layer — provides
OrgNode,NodeProperty,OrgTagArroyo Core — the
ArroyoIndexerPluginpattern this followsWeb Server — consumes the route table and serves the attachment cache
Domain Map Configuration — the org table source for domain mappings