Introduction
This document is the source of truth for the Arcology Project's data models, database schema, and query layer. It is written using literate programming — the code blocks in this document tangle to the actual source files. The `.org` file is canonical; never edit the tangled `.kt` or `.sq` files directly.
The Roam module turns org-mode files into an AST and normalized metadata, persisting them in a SQLite database so they become queryable. It serves as the metadata cache between the indexer pipeline (which discovers and parses files) and downstream consumers (search, flashcards, tasks, export — documented separately).
Conceptually it is an "API" to a set of org-mode documents: the parser produces OrgNode, OrgLink, OrgTag etc. from raw org text, and this layer stores them, indexes them, and provides typed queries.
Model Flow
This section documents how data flows from the parser through domain models into the database.
Overview
To persist a single document in to the database:
Read an org file using platform-specific filesystem implementations (see the indexer platform layer)
Parse it with orgmode-kmp's
OrgFileParserThis returns a
ParseResult, typicallyParseResult.Successwith extracted dataThat data is inserted into the
RoamRepositoryDomain models are converted to SQLDelight row types and persisted via named queries and SQLite
Insertion Flow (Node Example)
OrgNode (domain)
↓ RoamRepository.insertNode()
↓ database.arcologyDatabaseQueries.insertNode(
id, file, level, pos, todo, priority, scheduled, deadline,
title,
properties.joinToString("\n") { "key: value" },
outlinePath.joinToString("/")
)
↓ SQLDelight generates: insertNode binding
↓ INSERT OR REPLACE INTO nodes (...) VALUES (...)
↓ SQLite stores the rowQuery Flow (Node Example)
RoamRepository.getNodeById(id)
↓ database.arcologyDatabaseQueries.selectNodeById(id).executeAsOneOrNull()
↓ SQLDelight generates: selectNodeById binding
↓ SELECT * FROM nodes WHERE id = ?
↓ Returns: computer.whatthefuck.arcology.db.Nodes
↓ toDomain() extension function
↓ OrgNode (domain model)FTS Staging Flow
Full-text search uses FTS5 virtual tables. Because individual inserts into FTS tables carry overhead, entries are staged during indexing and batch-processed:
Parse file → Extract node data
→ insertNode() → nodes table
→ insertFtsStaging() → fts_staging table (deferred)
Batch flush:
→ getFtsStagingBatch() → read staged entries
→ bulkInsertTitleFts() / bulkInsertContentFts() → FTS virtual tables
→ clearFtsStaging() → empty staging tableThis avoids FTS index rebuild overhead during individual file processing and lets the indexer control when the expensive bulk insert happens.
Core Domain Models
All core domain models live in Models.kt. Each represents an entity extracted from org-mode files by the parser.
The preamble sets up the file:
@file:OptIn(kotlin.time.ExperimentalTime::class)
package computer.whatthefuck.arcology.domain
import kotlin.time.InstantOrgFile
Represents an org-mode file tracked in the system. The hash supports incremental indexing — unchanged files are skipped. Access and modification times come from the filesystem.
data class OrgFile(
val path: String,
val title: String? = null,
val hash: String,
val accessTime: Instant,
val modificationTime: Instant
)Creation from Parser
The parser extracts file metadata from the org-mode file preamble:
path: The file path passed to the parsertitle: Extracted from#+TITLE:directive or filename fallbackhash: SHA256 hash of file content (computed in the parser)accessTime,modificationTime: Timestamps from the filesystem
OrgNode
A node represents a heading with an :ID: property — the fundamental unit of org-roam's zettelkasten.
data class OrgNode(
val id: String,
val file: String,
val level: Int,
val position: Int,
val todo: String? = null,
val priority: String? = null,
val scheduled: String? = null,
val deadline: String? = null,
val title: String? = null,
val properties: Map<String, String> = emptyMap(),
val outlinePath: List<String> = emptyList(),
val drawers: Map<String, List<Any>> = emptyMap(),
val parentNodeId: String? = null
)Creation from Parser
id: From:ID:property in the heading's properties drawerfile: The source org file pathlevel: Heading level (number of asterisks)position: Sequential byte position within the filetodo: TODO state (e.g., "TODO", "DONE")priority: Priority cookie (e.g., "#A", "#B")scheduled: Scheduled date from planning linedeadline: Deadline date from planning linetitle: Heading text without markup or tagsproperties: All key-value pairs from the:PROPERTIES:draweroutlinePath: List of ancestor heading titles (breadcrumb path)drawers: Content of named drawers (e.g.,:LOGBOOK:)
OrgLink
Represents a link between nodes or to external resources.
data class OrgLink(
val position: Int,
val fromNode: String,
val toNode: String? = null,
val type: String,
val properties: Map<String, String> = emptyMap()
)Creation from Parser
Extracted from org-mode links:
=...= links create internal links (
toNodeset)=...= links create file links (
toNodenull)=//...= links create external links (
toNodenull)type: The link protocol ("id", "file", "http", etc.)
OrgTag
A tag on a node, extracted from heading tags or the ROAM_TAGS property.
data class OrgTag(
val nodeId: String,
val tag: String
)Creation from Parser
Tags come from two sources:
Heading tags after
:(e.g.,* TODO Title :tag1:tag2:)The
ROAM_TAGSproperty (space-separated tags)
OrgAlias
An alternative name for a node, from the ROAM_ALIAS property.
data class OrgAlias(
val nodeId: String,
val alias: String
)OrgRef
An external reference from the ROAM_REFS property — typically URLs or citation keys.
data class OrgRef(
val nodeId: String,
val ref: String,
val type: String
)Properties (Node + File)
Arcology extends org-roam with separate property tables for efficient key-value queries. NodeProperty stores per-heading properties; FileProperty stores per-file properties (from #+KEYWORD: lines in the preamble).
data class NodeProperty(
val nodeId: String,
val key: String,
val value: String?
)data class FileProperty(
val file: String,
val key: String,
val value: String?
)GeoCoordinate
Geographic coordinate with latitude and longitude. Parsed from the GEO_COORDS heading property. Includes parsing, serialization, and KML conversion utilities.
/**
* Geographic coordinate with latitude and longitude.
* Used for storing and querying location data from GEO_COORDS heading properties.
*/
data class GeoCoordinate(
val latitude: Double,
val longitude: Double
) {
companion object {
/**
* Standard property key for storing coordinates in org-mode headings.
*/
const val PROPERTY_KEY = "GEO_COORDS"
/**
* Parse a coordinate string in "latitude,longitude" format.
* Returns null if the string is not a valid coordinate pair.
*/
fun parse(value: String): GeoCoordinate? {
val parts = value.trim().split(",")
if (parts.size != 2) return null
val lat = parts[0].trim().toDoubleOrNull() ?: return null
val lon = parts[1].trim().toDoubleOrNull() ?: return null
// Basic validation: latitude must be -90 to 90, longitude -180 to 180
if (lat < -90.0 || lat > 90.0 || lon < -180.0 || lon > 180.0) return null
return GeoCoordinate(lat, lon)
}
}
/**
* Serialize to "latitude,longitude" format for storage in org-mode properties.
*/
fun toPropertyValue(): String = "$latitude,$longitude"
/**
* Convert to KML coordinate format "longitude,latitude,altitude".
* KML uses lon,lat order (opposite of most GPS conventions).
*/
fun toKmlCoordinates(altitude: Double = 0.0): String = "$longitude,$latitude,$altitude"
}Full-Text Search Models
These models support the FTS5 full-text search system.
data class FtsSearchResult(
val rank: Double,
val nodeId: String
)FtsStagingEntry holds a node's searchable text during deferred batch insertion:
data class FtsStagingEntry(
val nodeId: String,
val title: String,
val tags: String,
val aliases: String,
val content: String
)FtsTitleEntry and FtsContentEntry are used for bulk inserts into the two FTS indexes (titles and content):
data class FtsTitleEntry(
val nodeId: String,
val title: String,
val tags: String,
val aliases: String
)data class FtsContentEntry(
val nodeId: String,
val title: String,
val content: String
)App-specific models
FailedFile
Tracks files that failed to parse so the indexer can skip them on subsequent runs, optionally retrying when the file hash changes.
data class FailedFile(
val path: String,
val errorMessage: String,
val failureCount: Int,
val firstFailedAt: Instant,
val lastFailedAt: Instant,
val fileHash: String?
)FileDiscoveryCache
Cached file discovery metadata for incremental indexing. Used on Android to avoid full SAF (Storage Access Framework) scans by tracking which files have already been seen.
/**
* Cached file discovery metadata for incremental indexing.
* Stores metadata about discovered files to avoid full filesystem scans.
*/
data class FileDiscoveryCache(
val uri: String,
val documentId: String,
val displayName: String,
val lastModified: Long,
val contentHash: String?,
val lastSeenAt: Instant
)Models.kt Composition
The complete Models.kt file is assembled from all the named blocks above using noweb:
<<models-preamble>>
<<model-org-file>>
<<model-org-node>>
<<model-org-link>>
<<model-org-tag>>
<<model-org-alias>>
<<model-org-ref>>
<<model-node-property>>
<<model-file-property>>
<<model-fts-search-result>>
<<model-failed-file>>
<<model-fts-staging-entry>>
<<model-fts-title-entry>>
<<model-fts-content-entry>>
<<model-file-discovery-cache>>
<<model-geo-coordinate>>Attachment Models
These models handle file attachments on nodes — images, videos, and generic files discovered during indexing for nodes tagged with ATTACH.
AttachmentType and OrgAttachment
package computer.whatthefuck.arcology.domain
enum class AttachmentType { IMAGE, VIDEO, FILE }
data class OrgAttachment(
val nodeId: String,
val resolvedPath: String,
val type: AttachmentType
)Attachments are discovered by the AttachmentResolver in the indexer pipeline (see indexer documentation). Each node with an ATTACH tag gets its attachment directory scanned.
PendingAttachment (Capture Flow)
Represents an attachment selected by the user but not yet captured — holds the URI and metadata before the file is copied into the data directory. The data class itself lives in commonMain since it has no platform dependencies.
On Android, constructing a PendingAttachment requires querying ContentResolver for filename (via OpenableColumns) and MIME type, then classifying by MIME with an extension-based fallback. This logic was formerly in a separate UriHelper utility class, but since it exists solely to construct PendingAttachment instances, it is now a single Context.createPendingAttachment() extension function in androidMain.
package computer.whatthefuck.arcology.domain
/**
* Represents a pending attachment that has been selected but not yet captured.
* Used in the capture flow to hold attachment information before copying to the data directory.
*
* @property uri The content:// or file:// URI from the file picker or share intent
* @property filename The original filename of the attachment
* @property mimeType The MIME type from ContentResolver
* @property type The attachment type (IMAGE, VIDEO, or FILE)
*/
data class PendingAttachment(
val uri: String,
val filename: String,
val mimeType: String,
val type: AttachmentType
)On Android, use context.createPendingAttachment(uri) to construct one from a Uri:
package computer.whatthefuck.arcology.domain
import android.content.Context
import android.net.Uri
import android.provider.OpenableColumns
/**
* Construct a [PendingAttachment] from a content:// or file:// URI by querying
* ContentResolver for filename, MIME type, and classifying by type.
*
* The filename comes from OpenableColumns.DISPLAY_NAME (fallback: last path segment).
* The type classification uses primary MIME type (~image/~, ~video/~) with an
* extension-based fallback for application/octet-stream URIs.
*/
suspend fun Context.createPendingAttachment(uri: Uri): PendingAttachment {
// Resolve filename — OpenableColumns for content:// URIs
val filename = if (uri.scheme == "content") {
contentResolver.query(uri, arrayOf(OpenableColumns.DISPLAY_NAME), null, null, null)?.use { cursor ->
val index = cursor.getColumnIndex(OpenableColumns.DISPLAY_NAME)
if (cursor.moveToFirst() && index != -1) cursor.getString(index) else null
}
} else {
null
} ?: uri.lastPathSegment ?: "unknown"
// Resolve MIME type
val mimeType = contentResolver.getType(uri) ?: "application/octet-stream"
// Classify by MIME type with extension-based fallback
val type = when {
mimeType.startsWith("image/") -> AttachmentType.IMAGE
mimeType.startsWith("video/") -> AttachmentType.VIDEO
mimeType == "application/octet-stream" -> {
val ext = filename.substringAfterLast(".", "").lowercase()
val imageExts = setOf("jpg", "jpeg", "png", "gif", "webp", "svg", "heic", "heif", "bmp", "tiff", "tif")
val videoExts = setOf("mp4", "webm", "mkv", "avi", "mov", "m4v")
when {
ext in imageExts -> AttachmentType.IMAGE
ext in videoExts -> AttachmentType.VIDEO
else -> AttachmentType.FILE
}
}
else -> AttachmentType.FILE
}
return PendingAttachment(
uri = uri.toString(),
filename = filename,
mimeType = mimeType,
type = type
)
}Database Schema
The database schema is defined in ArcologyDatabase.sq, a SQLDelight file that generates type-safe Kotlin query bindings. Tables are organized by domain.
Core Org-Roam Tables
The foundational tables mirror org-roam's data model: files, nodes, links, tags, aliases, refs, and a legacy citations table.
-- Core org-roam schema
CREATE TABLE IF NOT EXISTS files (
id INTEGER PRIMARY KEY AUTOINCREMENT,
file TEXT UNIQUE NOT NULL,
title TEXT,
hash TEXT NOT NULL,
atime INTEGER,
mtime INTEGER,
UNIQUE(file)
);
CREATE TABLE IF NOT EXISTS nodes (
id TEXT PRIMARY KEY,
file TEXT NOT NULL,
level INTEGER NOT NULL,
pos INTEGER NOT NULL,
todo TEXT,
priority TEXT,
scheduled TEXT,
deadline TEXT,
title TEXT,
properties TEXT,
olp TEXT,
FOREIGN KEY (file) REFERENCES files (file) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS aliases (
node_id TEXT NOT NULL,
alias TEXT NOT NULL,
PRIMARY KEY (node_id, alias),
FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS citations (
node_id TEXT NOT NULL,
cite_key TEXT NOT NULL,
pos INTEGER NOT NULL,
properties TEXT,
PRIMARY KEY (node_id, cite_key, pos),
FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS links (
pos INTEGER NOT NULL,
from_node TEXT NOT NULL,
to_node TEXT,
type TEXT NOT NULL,
properties TEXT,
PRIMARY KEY (pos, from_node),
FOREIGN KEY (from_node) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS refs (
node_id TEXT NOT NULL,
ref TEXT NOT NULL,
type TEXT NOT NULL,
PRIMARY KEY (node_id, ref, type),
FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS tags (
node_id TEXT NOT NULL,
tag TEXT NOT NULL,
PRIMARY KEY (node_id, tag),
FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
-- Node ancestor closure table for property inheritance
-- Each row means "ancestor_id is an ancestor of node_id" (including self)
-- No FK constraints — populated by indexer/plugin, not user data
CREATE TABLE IF NOT EXISTS node_ancestors (
node_id TEXT NOT NULL,
ancestor_id TEXT NOT NULL,
PRIMARY KEY (node_id, ancestor_id)
);Core Indexes
CREATE INDEX IF NOT EXISTS idx_nodes_file ON nodes (file);
CREATE INDEX IF NOT EXISTS idx_nodes_id ON nodes (id);
CREATE INDEX IF NOT EXISTS idx_links_from ON links (from_node);
CREATE INDEX IF NOT EXISTS idx_links_to ON links (to_node);
CREATE INDEX IF NOT EXISTS idx_tags_node ON tags (node_id);
CREATE INDEX IF NOT EXISTS idx_tags_tag ON tags (tag);
CREATE INDEX IF NOT EXISTS idx_refs_node ON refs (node_id);
CREATE INDEX IF NOT EXISTS idx_refs_ref ON refs (ref);
CREATE INDEX IF NOT EXISTS idx_aliases_node ON aliases (node_id);
CREATE INDEX IF NOT EXISTS idx_aliases_alias ON aliases (alias);
CREATE INDEX IF NOT EXISTS idx_files_hash ON files (hash);
CREATE INDEX IF NOT EXISTS idx_node_ancestors_ancestor ON node_ancestors (ancestor_id);File Queries
-- File operations
selectAllFiles:
SELECT * FROM files ORDER BY file;
selectFileByPath:
SELECT * FROM files WHERE file = ?;
selectFilesByHash:
SELECT * FROM files WHERE hash = ?;
selectFileTitle:
SELECT title FROM files WHERE file = ?;
insertFile:
INSERT OR REPLACE INTO files (file, title, hash, atime, mtime)
VALUES (?, ?, ?, ?, ?);
deleteFile:
DELETE FROM files WHERE file = ?;Node Queries
-- Node operations
selectAllNodes:
SELECT * FROM nodes ORDER BY file, pos;
selectNodeById:
SELECT * FROM nodes WHERE id = ?;
selectNodesByFile:
SELECT * FROM nodes WHERE file = ? ORDER BY pos;
selectNodesByTitle:
SELECT * FROM nodes WHERE title LIKE ? ORDER BY title;
selectNodesByFilePath:
SELECT * FROM nodes WHERE file LIKE ? ORDER BY file, pos;
insertNode:
INSERT OR REPLACE INTO nodes (id, file, level, pos, todo, priority, scheduled, deadline, title, properties, olp)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
deleteNode:
DELETE FROM nodes WHERE id = ?;
deleteNodesByFile:
DELETE FROM nodes WHERE file = ?;
-- Node ancestor closure table
insertNodeAncestor:
INSERT OR IGNORE INTO node_ancestors (node_id, ancestor_id) VALUES (?, ?);
deleteNodeAncestorsByFile:
DELETE FROM node_ancestors WHERE node_id IN (SELECT id FROM nodes WHERE file = ?);
selectNodeAncestors:
SELECT ancestor_id FROM node_ancestors WHERE node_id = ?;Link Queries
-- Link operations
selectLinksFrom:
SELECT * FROM links WHERE from_node = ?;
selectLinksTo:
SELECT * FROM links WHERE to_node = ?;
insertLink:
INSERT OR REPLACE INTO links (pos, from_node, to_node, type, properties)
VALUES (?, ?, ?, ?, ?);
deleteLinksFrom:
DELETE FROM links WHERE from_node = ?;
deleteLinksByFile:
DELETE FROM links WHERE from_node IN (SELECT id FROM nodes WHERE file = ?);Tag Queries
Includes flashcard-specific queries that cross-reference the flashcards table to filter tags by nodes that have flashcard content.
-- Tag operations
selectTagsByNode:
SELECT tag FROM tags WHERE node_id = ? ORDER BY tag;
selectTagsByNodes:
SELECT node_id, tag FROM tags WHERE node_id IN ? ORDER BY node_id, tag;
selectNodesByTag:
SELECT node_id FROM tags WHERE tag = ? ORDER BY node_id;
insertTag:
INSERT OR IGNORE INTO tags (node_id, tag) VALUES (?, ?);
deleteTagsByNode:
DELETE FROM tags WHERE node_id = ?;
selectAllTags:
SELECT DISTINCT tag FROM tags ORDER BY tag;
selectTagsWithCount:
SELECT tag, COUNT(*) AS count FROM tags GROUP BY tag ORDER BY count DESC, tag;
deleteTagsByFile:
DELETE FROM tags WHERE node_id IN (SELECT id FROM nodes WHERE file = ?);Alias Queries
-- Alias operations
selectAliasesByNode:
SELECT alias FROM aliases WHERE node_id = ? ORDER BY alias;
selectNodesByAlias:
SELECT node_id FROM aliases WHERE alias = ? ORDER BY node_id;
insertAlias:
INSERT OR IGNORE INTO aliases (node_id, alias) VALUES (?, ?);
deleteAliasesByNode:
DELETE FROM aliases WHERE node_id = ?;
deleteAliasesByFile:
DELETE FROM aliases WHERE node_id IN (SELECT id FROM nodes WHERE file = ?);Ref Queries
-- Reference operations
selectRefsByNode:
SELECT ref, type FROM refs WHERE node_id = ? ORDER BY ref;
selectRefsByNodes:
SELECT node_id, ref FROM refs WHERE node_id IN ? ORDER BY node_id, ref;
selectNodesByRef:
SELECT node_id, type FROM refs WHERE ref = ? ORDER BY node_id;
insertRef:
INSERT OR IGNORE INTO refs (node_id, ref, type) VALUES (?, ?, ?);
deleteRefsByNode:
DELETE FROM refs WHERE node_id = ?;
deleteRefsByFile:
DELETE FROM refs WHERE node_id IN (SELECT id FROM nodes WHERE file = ?);Arcology Property Extensions
These tables extend the core schema with per-file and per-heading key-value properties, enabling Arcology-specific metadata queries (publishing keywords, feed configuration, geo coordinates, etc.).
-- Arcology extensions
CREATE TABLE IF NOT EXISTS file_properties (
file TEXT NOT NULL,
key TEXT NOT NULL,
value TEXT,
PRIMARY KEY (file, key),
FOREIGN KEY (file) REFERENCES files (file) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS heading_properties (
node_id TEXT NOT NULL,
key TEXT NOT NULL,
value TEXT,
PRIMARY KEY (node_id, key),
FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
-- Indexes
CREATE INDEX IF NOT EXISTS idx_file_properties_file ON file_properties (file);
CREATE INDEX IF NOT EXISTS idx_file_properties_key ON file_properties (key);
CREATE INDEX IF NOT EXISTS idx_heading_properties_node ON heading_properties (node_id);
CREATE INDEX IF NOT EXISTS idx_heading_properties_key ON heading_properties (key);
CREATE INDEX IF NOT EXISTS idx_heading_properties_value ON heading_properties (value);-- File properties (Arcology extension)
selectFileProperties:
SELECT key, value FROM file_properties WHERE file = ? ORDER BY key;
selectFileProperty:
SELECT value FROM file_properties WHERE file = ? AND key = ?;
insertFileProperty:
INSERT OR REPLACE INTO file_properties (file, key, value) VALUES (?, ?, ?);
deleteFileProperties:
DELETE FROM file_properties WHERE file = ?;
-- Heading properties (Arcology extension)
selectHeadingProperties:
SELECT key, value FROM heading_properties WHERE node_id = ? ORDER BY key;
selectHeadingProperty:
SELECT value FROM heading_properties WHERE node_id = ? AND key = ?;
selectNodesByProperty:
SELECT node_id, value FROM heading_properties WHERE key = ? ORDER BY node_id;
insertHeadingProperty:
INSERT OR REPLACE INTO heading_properties (node_id, key, value) VALUES (?, ?, ?);
deleteHeadingProperties:
DELETE FROM heading_properties WHERE node_id = ?;
deleteHeadingProperty:
DELETE FROM heading_properties WHERE node_id = ? AND key = ?;
deleteHeadingPropertiesByFile:
DELETE FROM heading_properties WHERE node_id IN (SELECT id FROM nodes WHERE file = ?);Auxiliary Tables
Failed file tracking, attachment storage, and file discovery cache.
-- Track files that failed to parse (for skipping on future runs)
CREATE TABLE IF NOT EXISTS failed_files (
file TEXT PRIMARY KEY NOT NULL,
error_message TEXT NOT NULL,
failure_count INTEGER NOT NULL DEFAULT 1,
first_failed_at INTEGER NOT NULL,
last_failed_at INTEGER NOT NULL,
file_hash TEXT
);
-- Attachment support
CREATE TABLE IF NOT EXISTS attachments (
node_id TEXT NOT NULL,
resolved_path TEXT NOT NULL,
type TEXT NOT NULL,
PRIMARY KEY (node_id, resolved_path),
FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_attachments_node ON attachments (node_id);
CREATE INDEX IF NOT EXISTS idx_attachments_type ON attachments (type);
-- File discovery cache for incremental indexing (Phase 4 SAF optimization)
-- Stores metadata about discovered files to avoid full ContentProvider scans
CREATE TABLE IF NOT EXISTS file_discovery_cache (
uri TEXT PRIMARY KEY NOT NULL,
document_id TEXT NOT NULL,
display_name TEXT NOT NULL,
last_modified INTEGER NOT NULL,
content_hash TEXT,
last_seen_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_discovery_cache_modified ON file_discovery_cache (last_modified);
CREATE INDEX IF NOT EXISTS idx_discovery_cache_seen ON file_discovery_cache (last_seen_at);-- Failed files operations (track parse failures for skipping)
selectFailedFile:
SELECT * FROM failed_files WHERE file = ?;
selectAllFailedFiles:
SELECT * FROM failed_files ORDER BY last_failed_at DESC;
insertFailedFile:
INSERT INTO failed_files (file, error_message, failure_count, first_failed_at, last_failed_at, file_hash)
VALUES (?, ?, ?, ?, ?, ?);
updateFailedFile:
UPDATE failed_files
SET error_message = ?, failure_count = failure_count + 1, last_failed_at = ?, file_hash = ?
WHERE file = ?;
deleteFailedFile:
DELETE FROM failed_files WHERE file = ?;
deleteAllFailedFiles:
DELETE FROM failed_files;insertAttachment:
INSERT OR REPLACE INTO attachments (node_id, resolved_path, type)
VALUES (?, ?, ?);
selectAttachmentsByNode:
SELECT * FROM attachments WHERE node_id = ? ORDER BY resolved_path;
selectAttachmentsByType:
SELECT * FROM attachments WHERE type = ? ORDER BY resolved_path;
deleteAttachmentsByNode:
DELETE FROM attachments WHERE node_id = ?;-- File discovery cache operations
selectDiscoveryCacheAll:
SELECT * FROM file_discovery_cache ORDER BY uri;
selectDiscoveryCacheByUri:
SELECT * FROM file_discovery_cache WHERE uri = ?;
selectDiscoveryCacheByModified:
SELECT * FROM file_discovery_cache WHERE last_modified >= ?;
insertDiscoveryCache:
INSERT OR REPLACE INTO file_discovery_cache (uri, document_id, display_name, last_modified, content_hash, last_seen_at)
VALUES (?, ?, ?, ?, ?, ?);
updateDiscoveryCacheHash:
UPDATE file_discovery_cache SET content_hash = ?, last_seen_at = ? WHERE uri = ?;
deleteDiscoveryCacheByUri:
DELETE FROM file_discovery_cache WHERE uri = ?;
deleteDiscoveryCacheStale:
DELETE FROM file_discovery_cache WHERE last_seen_at < ?;
clearDiscoveryCache:
DELETE FROM file_discovery_cache;
countDiscoveryCache:
SELECT COUNT(*) FROM file_discovery_cache;Full-Text Search (FTS5)
FTS5 virtual tables power full-text search across node titles, tags, aliases, and content. The system uses staged batch insertion to avoid index rebuild overhead during individual file processing.
-- FTS5 Virtual Tables for full-text search
CREATE VIRTUAL TABLE IF NOT EXISTS nodes_fts_titles USING fts5(
node_id TEXT NOT NULL,
title TEXT NOT NULL,
tags TEXT NOT NULL,
aliases TEXT NOT NULL
);
CREATE VIRTUAL TABLE IF NOT EXISTS nodes_fts_content USING fts5(
node_id TEXT NOT NULL,
title TEXT NOT NULL,
content TEXT NOT NULL
);-- FTS Search operations with BM25 ranking
searchTitlesBM25:
SELECT node_id, bm25(nodes_fts_titles) AS rank
FROM nodes_fts_titles
WHERE nodes_fts_titles MATCH ?
ORDER BY bm25(nodes_fts_titles)
LIMIT ?;
searchContentBM25:
SELECT node_id, bm25(nodes_fts_content) AS rank
FROM nodes_fts_content
WHERE nodes_fts_content MATCH ?
ORDER BY bm25(nodes_fts_content)
LIMIT ?;
-- Simple FTS search (fallback without ranking)
searchTitles:
SELECT node_id FROM nodes_fts_titles
WHERE nodes_fts_titles MATCH ? LIMIT ?;
searchContent:
SELECT node_id FROM nodes_fts_content
WHERE nodes_fts_content MATCH ? LIMIT ?;
-- FTS maintenance operations
insertNodeTitleFts:
INSERT INTO nodes_fts_titles(node_id, title, tags, aliases) VALUES (?, ?, ?, ?);
insertNodeContentFts:
INSERT INTO nodes_fts_content(node_id, title, content) VALUES (?, ?, ?);
updateNodeTitleFts:
DELETE FROM nodes_fts_titles WHERE node_id = ?;
insertNodeTitleFtsAfterDelete:
INSERT INTO nodes_fts_titles(node_id, title, tags, aliases) VALUES (?, ?, ?, ?);
updateNodeContentFts:
DELETE FROM nodes_fts_content WHERE node_id = ?;
insertNodeContentFtsAfterDelete:
INSERT INTO nodes_fts_content(node_id, title, content) VALUES (?, ?, ?);
deleteNodeTitleFts:
DELETE FROM nodes_fts_titles WHERE node_id = ?;
deleteNodeContentFts:
DELETE FROM nodes_fts_content WHERE node_id = ?;
-- Legacy search operations (for compatibility)
searchNodes:
SELECT DISTINCT n.*
FROM nodes n
LEFT JOIN tags t ON n.id = t.node_id
LEFT JOIN aliases a ON n.id = a.node_id
LEFT JOIN heading_properties hp ON n.id = hp.node_id
WHERE n.title LIKE ?
OR t.tag LIKE ?
OR a.alias LIKE ?
OR hp.value LIKE ?
ORDER BY n.title;-- FTS staging table for deferred batch insertion
CREATE TABLE IF NOT EXISTS fts_staging (
node_id TEXT PRIMARY KEY NOT NULL,
title TEXT NOT NULL,
tags TEXT NOT NULL,
aliases TEXT NOT NULL,
content TEXT NOT NULL,
FOREIGN KEY (node_id) REFERENCES nodes (id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_fts_staging_node ON fts_staging (node_id);
-- FTS staging operations
insertFtsStaging:
INSERT OR REPLACE INTO fts_staging (node_id, title, tags, aliases, content)
VALUES (?, ?, ?, ?, ?);
selectFtsStagingBatch:
SELECT * FROM fts_staging ORDER BY node_id LIMIT ? OFFSET ?;
selectFtsStagingCount:
SELECT COUNT(*) FROM fts_staging;
clearFtsStaging:
DELETE FROM fts_staging;
-- Bulk FTS clear operations
clearNodesTitleFts:
DELETE FROM nodes_fts_titles;
clearNodesContentFts:
DELETE FROM nodes_fts_content;
selectNodesContentFtsCount:
SELECT COUNT(*) FROM nodes_fts_content;-- Recent nodes by file modification time
selectRecentNodes:
SELECT n.* FROM nodes n
JOIN files f ON n.file = f.file
ORDER BY f.mtime DESC, n.pos ASC
LIMIT ?;ArcologyDatabase.sq Composition
The complete SQL file is assembled from all named blocks in schema-definition order:
<<db-core-roam>>
<<db-core-indexes>>
<<db-file-queries>>
<<db-node-queries>>
<<db-link-queries>>
<<db-tag-queries>>
<<db-alias-queries>>
<<db-ref-queries>>
<<db-properties>>
<<db-property-queries>>
<<db-aux-schema>>
<<db-failed-file-queries>>
<<db-attachment-queries>>
<<db-discovery-cache-queries>>
<<db-fts-schema>>
<<db-fts-queries>>
<<db-fts-staging>>
<<db-recent-nodes>>
DONE Extract QuizRepository from RoamRepository
The RoamRepository interface was split as part of the quiz cluster documentation in quiz/models.org. Flashcard tables (see flashcards, flashcard_positions, flashcard_reviews) and their queries now live in Quiz.sq, tangled from there. The Kotlin interface boundary (`QuizRepository`) and SQLDelight implementation (`QuizRepositoryImpl`) are now documented alongside the quiz domain models.
The flashcard schema and queries that were previously in this file have been moved to quiz/models.org.
The Repository
The RoamRepository is the persistence layer's public API — a Kotlin interface with 175 methods backed by a SQLDelight implementation. It follows the Repository pattern common in FOSS Android apps: domain models in, SQLDelight operations out, with private toDomain() extension functions converting between the two.
Preamble
@file:OptIn(kotlin.time.ExperimentalTime::class)
package computer.whatthefuck.arcology.database
import computer.whatthefuck.arcology.db.ArcologyDatabase
import computer.whatthefuck.arcology.domain.*
import kotlinx.datetime.LocalDate
import kotlin.time.Clock
import kotlin.time.Instant
import kotlinx.coroutines.*
import kotlinx.coroutines.ExperimentalCoroutinesApiInterface
The interface is organized by domain area. All methods are suspend functions for coroutine-based async access.
interface RoamRepository {
// File operations
suspend fun getAllFiles(): List<OrgFile>
suspend fun getFileByPath(path: String): OrgFile?
suspend fun insertFile(file: OrgFile)
suspend fun deleteFile(path: String)
// Node operations
suspend fun getAllNodes(): List<OrgNode>
suspend fun getNodeById(id: String): OrgNode?
suspend fun getNodesByFile(file: String): List<OrgNode>
suspend fun getRecentNodes(limit: Long = 100): List<OrgNode>
suspend fun searchNodesByTitle(query: String): List<OrgNode>
suspend fun searchNodesByFilePath(query: String): List<OrgNode>
suspend fun insertNode(node: OrgNode)
suspend fun deleteNode(id: String)
// Node ancestor operations
suspend fun insertNodeAncestor(nodeId: String, ancestorId: String)
suspend fun deleteNodeAncestorsByFile(file: String)
suspend fun getNodeAncestors(nodeId: String): List<String>
// Link operations
suspend fun getLinksFrom(nodeId: String): List<OrgLink>
suspend fun getLinksTo(nodeId: String): List<OrgLink>
suspend fun insertLink(link: OrgLink)
suspend fun deleteLinksByFile(file: String)
// Tag operations
suspend fun getTagsByNode(nodeId: String): List<String>
suspend fun getTagsByNodes(nodeIds: List<String>): Map<String, List<String>>
suspend fun getNodesByTag(tag: String): List<String>
suspend fun getAllTags(): List<String>
suspend fun getTagsWithCount(): List<Pair<String, Long>>
suspend fun insertTag(tag: OrgTag)
suspend fun deleteTagsByFile(file: String)
// Ref operations
suspend fun getRefsByNode(nodeId: String): List<Pair<String, String>>
suspend fun getRefsByNodes(nodeIds: List<String>): Map<String, List<String>>
suspend fun insertRef(ref: OrgRef)
suspend fun deleteRefsByFile(file: String)
// Alias operations
suspend fun getAliasesByNode(nodeId: String): List<String>
suspend fun getNodesByAlias(alias: String): List<String>
suspend fun insertAlias(alias: OrgAlias)
suspend fun deleteAliasesByFile(file: String)
// Properties operations
suspend fun getHeadingProperties(nodeId: String): Map<String, String?>
suspend fun getHeadingProperty(nodeId: String, key: String): String?
suspend fun insertHeadingProperty(property: NodeProperty)
suspend fun deleteHeadingProperty(nodeId: String, key: String)
suspend fun deleteHeadingPropertiesByFile(file: String)
suspend fun getFileProperties(file: String): Map<String, String?>
suspend fun insertFileProperty(property: FileProperty)
suspend fun getNodesByPropertyKey(key: String): List<Pair<String, String?>>
// Geolocation operations
suspend fun getNodesWithLocation(): List<Pair<OrgNode, GeoCoordinate>>
// Search operations
suspend fun searchNodes(query: String): List<OrgNode>
// FTS search operations
suspend fun searchNodesByTitles(query: String, limit: Long = 50): List<String>
suspend fun searchNodesByContent(query: String, limit: Long = 50): List<String>
suspend fun searchNodesByTitlesBM25(query: String, limit: Long = 50): List<FtsSearchResult>
suspend fun searchNodesByContentBM25(query: String, limit: Long = 50): List<FtsSearchResult>
// FTS maintenance operations
suspend fun insertNodeToFts(node: OrgNode, tags: List<String>, aliases: List<String>, content: String = "")
suspend fun updateNodeInFts(node: OrgNode, tags: List<String>, aliases: List<String>, content: String = "")
suspend fun deleteNodeFromFts(nodeId: String)
suspend fun getFtsContentCount(): Long
// Failed file tracking (for skipping problematic files)
suspend fun getFailedFile(path: String): FailedFile?
suspend fun getAllFailedFiles(): List<FailedFile>
suspend fun insertFailedFile(failedFile: FailedFile)
suspend fun updateFailedFile(path: String, errorMessage: String, fileHash: String?)
suspend fun deleteFailedFile(path: String)
suspend fun deleteAllFailedFiles()
// FTS staging operations (for deferred batch insertion)
suspend fun insertFtsStaging(nodeId: String, title: String, tags: String, aliases: String, content: String)
suspend fun getFtsStagingBatch(limit: Long, offset: Long): List<FtsStagingEntry>
suspend fun getFtsStagingCount(): Long
suspend fun clearFtsStaging()
// Bulk FTS operations
suspend fun clearAllTitleFts()
suspend fun clearAllContentFts()
suspend fun bulkInsertTitleFts(entries: List<FtsTitleEntry>)
suspend fun bulkInsertContentFts(entries: List<FtsContentEntry>)
// Transaction operations
suspend fun <T> transaction(block: suspend () -> T): T
// Attachment operations
suspend fun getAttachmentsByNode(nodeId: String): List<OrgAttachment>
suspend fun getAttachmentsByType(type: String): List<OrgAttachment>
suspend fun insertAttachment(attachment: OrgAttachment)
suspend fun deleteAttachmentsByNode(nodeId: String)
// File discovery cache operations (for incremental indexing)
suspend fun getAllDiscoveryCache(): List<FileDiscoveryCache>
suspend fun getDiscoveryCacheByUri(uri: String): FileDiscoveryCache?
suspend fun getDiscoveryCacheModifiedSince(timestamp: Long): List<FileDiscoveryCache>
suspend fun insertDiscoveryCache(entry: FileDiscoveryCache)
suspend fun updateDiscoveryCacheHash(uri: String, contentHash: String?, lastSeenAt: Long)
suspend fun deleteDiscoveryCacheByUri(uri: String)
suspend fun deleteStaleDiscoveryCache(beforeTimestamp: Long)
suspend fun clearDiscoveryCache()
suspend fun countDiscoveryCache(): Long
}Implementation Class
class RoamRepositoryImpl(
private val database: ArcologyDatabase
) : RoamRepository {
// Single-threaded dispatcher for database operations to avoid deadlocks
// when runBlocking is used inside transactions
@OptIn(ExperimentalCoroutinesApi::class)
private val dbDispatcher = Dispatchers.IO.limitedParallelism(1)Core CRUD Operations
File, node, link, tag, alias, ref, and property operations. Most follow a simple pattern: delegate to the SQLDelight query, map results through toDomain().
Note on Dispatchers.IO usage: public-facing query methods use withContext(Dispatchers.IO) to ensure they're safe to call from the main thread. Internal-use methods (called from within transactions) skip the dispatcher switch since the caller already manages threading.
override suspend fun getAllFiles(): List<OrgFile> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectAllFiles().executeAsList().map { it.toDomain() }
}
}
override suspend fun getFileByPath(path: String): OrgFile? {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectFileByPath(path).executeAsOneOrNull()?.toDomain()
}
}
override suspend fun insertFile(file: OrgFile) {
// No withContext - caller must ensure background thread or call within transaction
database.arcologyDatabaseQueries.insertFile(
file.path,
file.title,
file.hash,
file.accessTime.epochSeconds,
file.modificationTime.epochSeconds
)
}
override suspend fun deleteFile(path: String) {
database.arcologyDatabaseQueries.deleteFile(path)
}
override suspend fun getAllNodes(): List<OrgNode> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectAllNodes().executeAsList().map { it.toDomain() }
}
}
override suspend fun getNodeById(id: String): OrgNode? {
return database.arcologyDatabaseQueries.selectNodeById(id).executeAsOneOrNull()?.toDomain()
}
override suspend fun getNodesByFile(file: String): List<OrgNode> {
return database.arcologyDatabaseQueries.selectNodesByFile(file).executeAsList().map { it.toDomain() }
}
override suspend fun getRecentNodes(limit: Long): List<OrgNode> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectRecentNodes(limit).executeAsList().map { it.toDomain() }
}
}
override suspend fun searchNodesByTitle(query: String): List<OrgNode> {
return database.arcologyDatabaseQueries.selectNodesByTitle("%$query%").executeAsList().map { it.toDomain() }
}
override suspend fun searchNodesByFilePath(query: String): List<OrgNode> {
return database.arcologyDatabaseQueries.selectNodesByFilePath("%$query%").executeAsList().map { it.toDomain() }
}
override suspend fun insertNode(node: OrgNode) {
database.arcologyDatabaseQueries.insertNode(
node.id,
node.file,
node.level.toLong(),
node.position.toLong(),
node.todo,
node.priority,
node.scheduled,
node.deadline,
node.title,
node.properties.entries.joinToString("\n") { "${it.key}: ${it.value}" },
node.outlinePath.joinToString("/")
)
}
override suspend fun deleteNode(id: String) {
database.arcologyDatabaseQueries.deleteNode(id)
}
override suspend fun insertNodeAncestor(nodeId: String, ancestorId: String) {
database.arcologyDatabaseQueries.insertNodeAncestor(nodeId, ancestorId)
}
override suspend fun deleteNodeAncestorsByFile(file: String) {
database.arcologyDatabaseQueries.deleteNodeAncestorsByFile(file)
}
override suspend fun getNodeAncestors(nodeId: String): List<String> {
return database.arcologyDatabaseQueries.selectNodeAncestors(nodeId).executeAsList()
}
override suspend fun getLinksFrom(nodeId: String): List<OrgLink> {
return database.arcologyDatabaseQueries.selectLinksFrom(nodeId).executeAsList().map { it.toDomain() }
}
override suspend fun getLinksTo(nodeId: String): List<OrgLink> {
return database.arcologyDatabaseQueries.selectLinksTo(nodeId).executeAsList().map { it.toDomain() }
}
override suspend fun insertLink(link: OrgLink) {
database.arcologyDatabaseQueries.insertLink(
link.position.toLong(),
link.fromNode,
link.toNode,
link.type,
link.properties.entries.joinToString("\n") { "${it.key}: ${it.value}" }
)
}
override suspend fun deleteLinksByFile(file: String) {
database.arcologyDatabaseQueries.deleteLinksByFile(file)
}
override suspend fun getTagsByNode(nodeId: String): List<String> {
return database.arcologyDatabaseQueries.selectTagsByNode(nodeId).executeAsList()
}
override suspend fun getTagsByNodes(nodeIds: List<String>): Map<String, List<String>> {
if (nodeIds.isEmpty()) return emptyMap()
return database.arcologyDatabaseQueries.selectTagsByNodes(nodeIds)
.executeAsList()
.groupBy({ it.node_id }, { it.tag })
}
override suspend fun getNodesByTag(tag: String): List<String> {
return database.arcologyDatabaseQueries.selectNodesByTag(tag).executeAsList()
}
override suspend fun getAllTags(): List<String> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectAllTags().executeAsList()
}
}
override suspend fun getTagsWithCount(): List<Pair<String, Long>> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectTagsWithCount().executeAsList()
.map { it.tag to it.count }
}
}
override suspend fun insertTag(tag: OrgTag) {
database.arcologyDatabaseQueries.insertTag(tag.nodeId, tag.tag)
}
override suspend fun deleteTagsByFile(file: String) {
database.arcologyDatabaseQueries.deleteTagsByFile(file)
}
override suspend fun getRefsByNode(nodeId: String): List<Pair<String, String>> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectRefsByNode(nodeId)
.executeAsList()
.map { it.ref to it.type }
}
}
override suspend fun getRefsByNodes(nodeIds: List<String>): Map<String, List<String>> {
if (nodeIds.isEmpty()) return emptyMap()
return database.arcologyDatabaseQueries.selectRefsByNodes(nodeIds)
.executeAsList()
.groupBy({ it.node_id }, { it.ref })
}
override suspend fun insertRef(ref: OrgRef) {
database.arcologyDatabaseQueries.insertRef(ref.nodeId, ref.ref, ref.type)
}
override suspend fun deleteRefsByFile(file: String) {
database.arcologyDatabaseQueries.deleteRefsByFile(file)
}
override suspend fun getAliasesByNode(nodeId: String): List<String> {
return database.arcologyDatabaseQueries.selectAliasesByNode(nodeId).executeAsList()
}
override suspend fun getNodesByAlias(alias: String): List<String> {
return database.arcologyDatabaseQueries.selectNodesByAlias(alias).executeAsList()
}
override suspend fun insertAlias(alias: OrgAlias) {
database.arcologyDatabaseQueries.insertAlias(alias.nodeId, alias.alias)
}
override suspend fun deleteAliasesByFile(file: String) {
database.arcologyDatabaseQueries.deleteAliasesByFile(file)
}
override suspend fun getHeadingProperties(nodeId: String): Map<String, String?> {
return database.arcologyDatabaseQueries.selectHeadingProperties(nodeId)
.executeAsList()
.associate { it.key to it.value_ }
}
override suspend fun getHeadingProperty(nodeId: String, key: String): String? {
return database.arcologyDatabaseQueries.selectHeadingProperty(nodeId, key).executeAsOneOrNull()?.value_
}
override suspend fun insertHeadingProperty(property: NodeProperty) {
database.arcologyDatabaseQueries.insertHeadingProperty(property.nodeId, property.key, property.value)
}
override suspend fun deleteHeadingProperty(nodeId: String, key: String) {
database.arcologyDatabaseQueries.deleteHeadingProperty(nodeId, key)
}
override suspend fun deleteHeadingPropertiesByFile(file: String) {
database.arcologyDatabaseQueries.deleteHeadingPropertiesByFile(file)
}
override suspend fun getFileProperties(file: String): Map<String, String?> {
return database.arcologyDatabaseQueries.selectFileProperties(file)
.executeAsList()
.associate { it.key to it.value_ }
}
override suspend fun insertFileProperty(property: FileProperty) {
database.arcologyDatabaseQueries.insertFileProperty(property.file, property.key, property.value)
}
override suspend fun getNodesByPropertyKey(key: String): List<Pair<String, String?>> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectNodesByProperty(key)
.executeAsList()
.map { it.node_id to it.value_ }
}
}
override suspend fun getNodesWithLocation(): List<Pair<OrgNode, GeoCoordinate>> {
return withContext(Dispatchers.IO) {
val nodeIdsWithCoords = database.arcologyDatabaseQueries
.selectNodesByProperty(GeoCoordinate.PROPERTY_KEY)
.executeAsList()
nodeIdsWithCoords.mapNotNull { row ->
val coord = row.value_?.let { GeoCoordinate.parse(it) } ?: return@mapNotNull null
val node = database.arcologyDatabaseQueries
.selectNodeById(row.node_id)
.executeAsOneOrNull()
?.toDomain() ?: return@mapNotNull null
node to coord
}
}
}
override suspend fun searchNodes(query: String): List<OrgNode> {
val searchTerm = "%$query%"
return database.arcologyDatabaseQueries.searchNodes(searchTerm, searchTerm, searchTerm, searchTerm)
.executeAsList().map { it.toDomain() }
}FTS Operations
Full-text search with BM25 ranking, FTS maintenance (insert/update/delete), staging for deferred batch insertion, and bulk operations for full reindexing.
// FTS search operations
override suspend fun searchNodesByTitles(query: String, limit: Long): List<String> {
return database.arcologyDatabaseQueries.searchTitles(query, limit)
.executeAsList()
}
override suspend fun searchNodesByContent(query: String, limit: Long): List<String> {
return database.arcologyDatabaseQueries.searchContent(query, limit)
.executeAsList()
}
override suspend fun searchNodesByTitlesBM25(query: String, limit: Long): List<FtsSearchResult> {
return database.arcologyDatabaseQueries.searchTitlesBM25(query, limit)
.executeAsList()
.map { FtsSearchResult(rank = it.rank ?: 0.0, nodeId = it.node_id ?: "") }
}
override suspend fun searchNodesByContentBM25(query: String, limit: Long): List<FtsSearchResult> {
return database.arcologyDatabaseQueries.searchContentBM25(query, limit)
.executeAsList()
.map { FtsSearchResult(rank = it.rank ?: 0.0, nodeId = it.node_id ?: "") }
}
// FTS maintenance operations
override suspend fun insertNodeToFts(node: OrgNode, tags: List<String>, aliases: List<String>, content: String) {
// Insert into title FTS
database.arcologyDatabaseQueries.insertNodeTitleFts(
node.id,
node.title ?: "",
tags.joinToString(" "),
aliases.joinToString(" ")
)
// Insert into content FTS if content provided
if (content.isNotEmpty()) {
database.arcologyDatabaseQueries.insertNodeContentFts(
node.id,
node.title ?: "",
content
)
}
}
override suspend fun updateNodeInFts(node: OrgNode, tags: List<String>, aliases: List<String>, content: String) {
// Update title FTS (delete + insert)
database.arcologyDatabaseQueries.updateNodeTitleFts(node.id)
database.arcologyDatabaseQueries.insertNodeTitleFtsAfterDelete(
node.id,
node.title ?: "",
tags.joinToString(" "),
aliases.joinToString(" ")
)
// Update content FTS if content provided
if (content.isNotEmpty()) {
database.arcologyDatabaseQueries.updateNodeContentFts(node.id)
database.arcologyDatabaseQueries.insertNodeContentFtsAfterDelete(
node.id,
node.title ?: "",
content
)
}
}
override suspend fun deleteNodeFromFts(nodeId: String) {
database.arcologyDatabaseQueries.deleteNodeTitleFts(nodeId)
database.arcologyDatabaseQueries.deleteNodeContentFts(nodeId)
}
// Failed file tracking
override suspend fun getFailedFile(path: String): FailedFile? {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectFailedFile(path).executeAsOneOrNull()?.toDomain()
}
}
override suspend fun getAllFailedFiles(): List<FailedFile> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectAllFailedFiles().executeAsList().map { it.toDomain() }
}
}
override suspend fun insertFailedFile(failedFile: FailedFile) {
withContext(dbDispatcher) {
database.arcologyDatabaseQueries.insertFailedFile(
failedFile.path,
failedFile.errorMessage,
failedFile.failureCount.toLong(),
failedFile.firstFailedAt.epochSeconds,
failedFile.lastFailedAt.epochSeconds,
failedFile.fileHash
)
}
}
override suspend fun updateFailedFile(path: String, errorMessage: String, fileHash: String?) {
withContext(dbDispatcher) {
database.arcologyDatabaseQueries.updateFailedFile(
errorMessage,
Clock.System.now().epochSeconds,
fileHash,
path
)
}
}
override suspend fun deleteFailedFile(path: String) {
database.arcologyDatabaseQueries.deleteFailedFile(path)
}
override suspend fun deleteAllFailedFiles() {
database.arcologyDatabaseQueries.deleteAllFailedFiles()
}
// FTS staging operations
override suspend fun insertFtsStaging(nodeId: String, title: String, tags: String, aliases: String, content: String) {
database.arcologyDatabaseQueries.insertFtsStaging(nodeId, title, tags, aliases, content)
}
override suspend fun getFtsStagingBatch(limit: Long, offset: Long): List<FtsStagingEntry> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectFtsStagingBatch(limit, offset)
.executeAsList()
.map { it.toFtsStagingDomain() }
}
}
override suspend fun getFtsStagingCount(): Long {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectFtsStagingCount().executeAsOne()
}
}
override suspend fun clearFtsStaging() {
database.arcologyDatabaseQueries.clearFtsStaging()
}
override suspend fun getFtsContentCount(): Long {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectNodesContentFtsCount().executeAsOne()
}
}
// Bulk FTS operations
override suspend fun clearAllTitleFts() {
database.arcologyDatabaseQueries.clearNodesTitleFts()
}
override suspend fun clearAllContentFts() {
database.arcologyDatabaseQueries.clearNodesContentFts()
}
override suspend fun bulkInsertTitleFts(entries: List<FtsTitleEntry>) {
entries.forEach { entry ->
database.arcologyDatabaseQueries.insertNodeTitleFts(
entry.nodeId,
entry.title,
entry.tags,
entry.aliases
)
}
}
override suspend fun bulkInsertContentFts(entries: List<FtsContentEntry>) {
entries.forEach { entry ->
database.arcologyDatabaseQueries.insertNodeContentFts(
entry.nodeId,
entry.title,
entry.content
)
}
}Attachment and Discovery Cache Operations
// Attachment operations
override suspend fun getAttachmentsByNode(nodeId: String): List<OrgAttachment> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectAttachmentsByNode(nodeId)
.executeAsList()
.map { it.toAttachmentDomain() }
}
}
override suspend fun getAttachmentsByType(type: String): List<OrgAttachment> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectAttachmentsByType(type)
.executeAsList()
.map { it.toAttachmentDomain() }
}
}
override suspend fun insertAttachment(attachment: OrgAttachment) {
database.arcologyDatabaseQueries.insertAttachment(
attachment.nodeId,
attachment.resolvedPath,
attachment.type.name
)
}
override suspend fun deleteAttachmentsByNode(nodeId: String) {
database.arcologyDatabaseQueries.deleteAttachmentsByNode(nodeId)
}
// File discovery cache operations
override suspend fun getAllDiscoveryCache(): List<FileDiscoveryCache> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectDiscoveryCacheAll()
.executeAsList()
.map { it.toDiscoveryCacheDomain() }
}
}
override suspend fun getDiscoveryCacheByUri(uri: String): FileDiscoveryCache? {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectDiscoveryCacheByUri(uri)
.executeAsOneOrNull()
?.toDiscoveryCacheDomain()
}
}
override suspend fun getDiscoveryCacheModifiedSince(timestamp: Long): List<FileDiscoveryCache> {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.selectDiscoveryCacheByModified(timestamp)
.executeAsList()
.map { it.toDiscoveryCacheDomain() }
}
}
override suspend fun insertDiscoveryCache(entry: FileDiscoveryCache) {
database.arcologyDatabaseQueries.insertDiscoveryCache(
entry.uri,
entry.documentId,
entry.displayName,
entry.lastModified,
entry.contentHash,
entry.lastSeenAt.epochSeconds
)
}
override suspend fun updateDiscoveryCacheHash(uri: String, contentHash: String?, lastSeenAt: Long) {
database.arcologyDatabaseQueries.updateDiscoveryCacheHash(contentHash, lastSeenAt, uri)
}
override suspend fun deleteDiscoveryCacheByUri(uri: String) {
database.arcologyDatabaseQueries.deleteDiscoveryCacheByUri(uri)
}
override suspend fun deleteStaleDiscoveryCache(beforeTimestamp: Long) {
database.arcologyDatabaseQueries.deleteDiscoveryCacheStale(beforeTimestamp)
}
override suspend fun clearDiscoveryCache() {
database.arcologyDatabaseQueries.clearDiscoveryCache()
}
override suspend fun countDiscoveryCache(): Long {
return withContext(Dispatchers.IO) {
database.arcologyDatabaseQueries.countDiscoveryCache().executeAsOne()
}
}Transaction Support
The transaction method uses a single-threaded dispatcher (dbDispatcher) to serialize all database operations. Within the transaction, runBlocking is used because SQLDelight's transactionWithResult expects a synchronous lambda. The single-threaded dispatcher ensures that nested runBlocking calls don't deadlock — only one thread ever enters the transaction.
override suspend fun <T> transaction(block: suspend () -> T): T {
// Use a single-threaded dispatcher to ensure all DB operations
// happen on one dedicated thread, avoiding contention.
// runBlocking without a dispatcher argument runs on the current thread.
return withContext(dbDispatcher) {
database.transactionWithResult {
runBlocking {
block()
}
}
}
}Closing the Implementation Class
}Domain Conversion Extensions
Private extension functions convert SQLDelight-generated row types to domain models. These live as top-level private functions in the same file so the repository implementation can access them while keeping the generated types decoupled from domain consumers.
// Extension functions to convert SQLDelight generated types to domain models
private fun computer.whatthefuck.arcology.db.Files.toDomain(): OrgFile {
return OrgFile(
path = file_,
title = title,
hash = hash,
accessTime = Instant.fromEpochSeconds(atime ?: 0),
modificationTime = Instant.fromEpochSeconds(mtime ?: 0)
)
}
private fun computer.whatthefuck.arcology.db.Nodes.toDomain(): OrgNode {
return OrgNode(
id = id,
file = file_,
level = level.toInt(),
position = pos.toInt(),
todo = todo,
priority = priority,
scheduled = scheduled,
deadline = deadline,
title = title,
properties = properties?.split("\n")?.associate {
val parts = it.split(": ", limit = 2)
parts[0] to (parts.getOrNull(1) ?: "")
} ?: emptyMap(),
outlinePath = olp?.split("/") ?: emptyList()
)
}
private fun computer.whatthefuck.arcology.db.Links.toDomain(): OrgLink {
return OrgLink(
position = pos.toInt(),
fromNode = from_node,
toNode = to_node,
type = type,
properties = properties?.split("\n")?.associate {
val parts = it.split(": ", limit = 2)
parts[0] to (parts.getOrNull(1) ?: "")
} ?: emptyMap()
)
}
private fun computer.whatthefuck.arcology.db.Failed_files.toDomain(): FailedFile {
return FailedFile(
path = file_,
errorMessage = error_message,
failureCount = failure_count.toInt(),
firstFailedAt = Instant.fromEpochSeconds(first_failed_at),
lastFailedAt = Instant.fromEpochSeconds(last_failed_at),
fileHash = file_hash
)
}
private fun computer.whatthefuck.arcology.db.Fts_staging.toFtsStagingDomain(): FtsStagingEntry {
return FtsStagingEntry(
nodeId = node_id,
title = title,
tags = tags,
aliases = aliases,
content = content
)
}
private fun computer.whatthefuck.arcology.db.Attachments.toAttachmentDomain(): OrgAttachment {
return OrgAttachment(
nodeId = node_id,
resolvedPath = resolved_path,
type = try { AttachmentType.valueOf(type) } catch (_: Exception) { AttachmentType.FILE }
)
}
private fun computer.whatthefuck.arcology.db.File_discovery_cache.toDiscoveryCacheDomain(): FileDiscoveryCache {
return FileDiscoveryCache(
uri = uri,
documentId = document_id,
displayName = display_name,
lastModified = last_modified,
contentHash = content_hash,
lastSeenAt = Instant.fromEpochSeconds(last_seen_at)
)
}RoamRepository.kt Composition
<<repo-preamble>>
<<repo-interface>>
<<repo-impl-header>>
<<repo-core-crud>>
<<repo-fts>>
<<repo-misc>>
<<repo-transaction>>
<<repo-impl-footer>>
<<repo-conversions>>Platform Driver Factory
A Kotlin expect class for platform-specific SQLDelight driver creation. Each platform (jvmMain, androidMain) provides an actual implementation — JVM uses in-memory SQLite for tests, Android uses the platform AndroidSqliteDriver.
package computer.whatthefuck.arcology.database
import app.cash.sqldelight.db.SqlDriver
expect class DatabaseDriverFactory {
fun createDriver(): SqlDriver
}NEXT Unify DatabaseDriverFactory and DatabaseFactory
The current codebase has two overlapping JDBC driver creation patterns: `DatabaseDriverFactory.jvm.kt` (the `actual` for the `expect class`, used by Android and JVM tests) and `DatabaseFactory.kt` (a standalone JVM object with its own schema initialization logic). They should be merged into a single JVM path. The `DatabaseFactory` file lives in `src/jvmMain` and is used by `roam/cli.org` and `roam/indexer-platform.org`; `DatabaseDriverFactory.jvm.kt` is the `actual` companion. Keep this as a follow-up — both paths work, so unification is cleanup, not a fix.
JVM Driver Factory
The `actual class` for `DatabaseDriverFactory` on the JVM. It creates a `JdbcSqliteDriver` — either in-memory (default `createDriver()`) or from a file path (`createDriver(databasePath: String)`). Both paths enable `PRAGMA foreign_keys = ON` and call `ArcologyDatabase.Schema.create()` to initialize the schema.
The in-memory variant is used by tests. The file-backed variant with path parameter is used by `FlowFileIndexerFactory` (see indexer-platform.org) and the `mcp` CLI command (see cli.org).
package computer.whatthefuck.arcology.database
import app.cash.sqldelight.db.SqlDriver
import app.cash.sqldelight.driver.jdbc.sqlite.JdbcSqliteDriver
import computer.whatthefuck.arcology.db.ArcologyDatabase
actual class DatabaseDriverFactory {
actual fun createDriver(): SqlDriver {
val driver = JdbcSqliteDriver(JdbcSqliteDriver.IN_MEMORY)
driver.execute(null, "PRAGMA foreign_keys = ON", 0)
ArcologyDatabase.Schema.create(driver)
return driver
}
fun createDriver(databasePath: String): SqlDriver {
val driver = JdbcSqliteDriver("jdbc:sqlite:$databasePath")
driver.execute(null, "PRAGMA foreign_keys = ON", 0)
ArcologyDatabase.Schema.create(driver)
return driver
}
}JVM Database Factory
A convenience object that wraps the driver logic with schema existence checking and `runBlocking` schema creation. Currently used by CLI commands that open or create a database at a given file path.
File-backed databases get `PRAGMA busy_timeout 5000` and `journal_mode WAL` so that concurrent CLI processes don't corrupt or fail each other's writes: `arcology sync` (syncthing.org) writes the index while `arcology serve` (server.org) reads it. In-memory databases skip the pragmas — they're single-owner by construction.
package computer.whatthefuck.arcology.database
import app.cash.sqldelight.db.SqlDriver
import app.cash.sqldelight.driver.jdbc.sqlite.JdbcSqliteDriver
import computer.whatthefuck.arcology.db.ArcologyDatabase
import kotlinx.coroutines.runBlocking
import java.util.Properties
object DatabaseFactory {
fun createDatabase(dbPath: String): ArcologyDatabase {
val driver: SqlDriver = JdbcSqliteDriver(
url = "jdbc:sqlite:$dbPath",
properties = Properties()
)
driver.execute(null, "PRAGMA foreign_keys = ON", 0)
// The sync watch command writes while the serve command reads the
// same file; without these, concurrent access fails fast or worse.
driver.execute(null, "PRAGMA busy_timeout = 5000", 0)
driver.execute(null, "PRAGMA journal_mode = WAL", 0)
// Initialize schema if needed
initializeSchema(driver, dbPath)
return ArcologyDatabase(driver)
}
fun createInMemoryDatabase(): ArcologyDatabase {
val driver: SqlDriver = JdbcSqliteDriver(
url = JdbcSqliteDriver.IN_MEMORY,
properties = Properties()
)
driver.execute(null, "PRAGMA foreign_keys = ON", 0)
runBlocking {
ArcologyDatabase.Schema.create(driver).await()
}
return ArcologyDatabase(driver)
}
private fun initializeSchema(driver: SqlDriver, dbPath: String) {
try {
// Every CREATE in the SQLDelight schema is IF NOT EXISTS, so running
// Schema.create is idempotent AND backfills tables added after a
// database was first created (e.g. published_routes, published_attachments).
runBlocking {
ArcologyDatabase.Schema.create(driver).await()
}
println("Database schema ensured at $dbPath")
} catch (e: Exception) {
println("Error during schema initialization: ${e.message}")
throw RuntimeException("Failed to initialize database schema", e)
}
}
}Android Driver Factory
The `actual class` for the Android platform. It wraps `AndroidxSqliteDriver` over a file-backed database at `context.getDatabasePath("arcology.db")`. The schema is passed to the driver at creation so `CREATE TABLE IF NOT EXISTS` in the SQLDelight schema handles initialization safely.
The class also exposes `exportDatabase(targetUri)` and `importDatabase(sourceUri)` for the user-facing backup/restore workflow. These use the Android ContentResolver to copy the database file (and its WAL/SHM siblings) via Storage Access Framework URIs. WAL is checkpointed before export to ensure all data is in the main file.
package computer.whatthefuck.arcology.database
import android.content.Context
import android.net.Uri
import android.util.Log
import androidx.sqlite.driver.bundled.BundledSQLiteDriver
import app.cash.sqldelight.async.coroutines.synchronous
import app.cash.sqldelight.db.SqlDriver
import com.eygraber.sqldelight.androidx.driver.AndroidxSqliteDriver
import com.eygraber.sqldelight.androidx.driver.AndroidxSqliteDatabaseType
import computer.whatthefuck.arcology.db.ArcologyDatabase
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.io.File
import java.io.InputStream
import java.io.OutputStream
private const val TAG = "ARCOLOGY_DB"
actual class DatabaseDriverFactory(private val context: Context) {
actual fun createDriver(): SqlDriver {
val dbFile = context.getDatabasePath("arcology.db")
val dbExists = dbFile.exists()
val dbSize = if (dbExists) dbFile.length() else 0L
Log.i(TAG, "ARCOLOGY_DB_INIT: Database file exists=$dbExists, size=$dbSize bytes")
// Schema now uses CREATE TABLE IF NOT EXISTS, so it's safe to always pass it
// AndroidxSqliteDriver will create tables only if they don't exist
Log.i(TAG, "ARCOLOGY_DB_INIT: Creating driver with schema (IF NOT EXISTS)")
return AndroidxSqliteDriver(
driver = BundledSQLiteDriver(),
databaseType = AndroidxSqliteDatabaseType.FileProvider {
dbFile.absolutePath
},
schema = ArcologyDatabase.Schema.synchronous()
)
}
/**
* Checkpoint WAL to ensure all data is committed to main database file.
*/
private fun checkpointWal() {
try {
val dbPath = context.getDatabasePath("arcology.db").absolutePath
// Open connection and checkpoint
android.database.sqlite.SQLiteDatabase.openDatabase(
dbPath,
null,
android.database.sqlite.SQLiteDatabase.OPEN_READONLY
).use { db ->
db.execSQL("PRAGMA wal_checkpoint(TRUNCATE)")
Log.i(TAG, "ARCOLOGY_DB_EXPORT: WAL checkpoint completed for $dbPath")
}
} catch (e: Exception) {
Log.w(TAG, "ARCOLOGY_DB_EXPORT: WAL checkpoint failed (may not exist): ${e.message}")
}
}
/**
* Export the database to a SAF URI location.
* Checkpoints WAL first to ensure all data is committed.
* @param targetUri The destination URI where the database will be copied
* @return true if export succeeded, false otherwise
*/
suspend fun exportDatabase(targetUri: Uri): Boolean = withContext(Dispatchers.IO) {
val databaseFile = context.getDatabasePath("arcology.db")
val databasePath = databaseFile.absolutePath
val contentResolver = context.contentResolver
try {
Log.i(TAG, "ARCOLOGY_DB_EXPORT: Starting export from $databasePath")
Log.i(TAG, "ARCOLOGY_DB_EXPORT: Source file size: ${databaseFile.length()} bytes")
// Checkpoint WAL to ensure all data is in main file
checkpointWal()
// Delete WAL and SHM files if they exist (they're now checkpointed)
File("$databasePath-wal").delete()
File("$databasePath-shm").delete()
contentResolver.openOutputStream(targetUri, "wt")?.use { outputStream ->
contentResolver.openInputStream(Uri.parse("file://$databasePath"))?.use { inputStream ->
val bytesCopied = inputStream.copyTo(outputStream)
Log.i(TAG, "ARCOLOGY_DB_EXPORT: Copied $bytesCopied bytes")
}
}
Log.i(TAG, "ARCOLOGY_DB_EXPORT: Export completed successfully to $targetUri")
true
} catch (e: Exception) {
Log.e(TAG, "ARCOLOGY_DB_EXPORT: Export failed: ${e.message}", e)
false
}
}
/**
* Import a database from a SAF URI location.
* Replaces the current database with the imported one.
* Deletes WAL/SHM files to ensure clean state.
* @param sourceUri The source URI to import from
* @return true if import succeeded, false otherwise
*/
suspend fun importDatabase(sourceUri: Uri): Boolean = withContext(Dispatchers.IO) {
val databaseFile = context.getDatabasePath("arcology.db")
val databasePath = databaseFile.absolutePath
val contentResolver = context.contentResolver
try {
Log.i(TAG, "ARCOLOGY_DB_IMPORT: Starting import from $sourceUri to $databasePath")
// Ensure the databases directory exists
databaseFile.parentFile?.mkdirs()
// Delete existing database and WAL/SHM files first
databaseFile.delete()
File("$databasePath-wal").delete()
File("$databasePath-shm").delete()
Log.i(TAG, "ARCOLOGY_DB_IMPORT: Deleted existing database files")
// Copy the imported database
val bytesCopied = contentResolver.openInputStream(sourceUri)?.use { inputStream ->
databaseFile.outputStream().use { outputStream ->
inputStream.copyTo(outputStream)
}
} ?: run {
Log.e(TAG, "ARCOLOGY_DB_IMPORT: Failed to open input stream for: $sourceUri")
return@withContext false
}
Log.i(TAG, "ARCOLOGY_DB_IMPORT: Copied $bytesCopied bytes")
Log.i(TAG, "ARCOLOGY_DB_IMPORT: Imported file size: ${databaseFile.length()} bytes")
// Verify the imported database has data
try {
android.database.sqlite.SQLiteDatabase.openDatabase(
databasePath,
null,
android.database.sqlite.SQLiteDatabase.OPEN_READONLY
).use { db ->
val nodeCount = db.rawQuery("SELECT COUNT(*) FROM nodes", null).use { cursor ->
if (cursor.moveToFirst()) cursor.getLong(0) else 0
}
val fileCount = db.rawQuery("SELECT COUNT(*) FROM files", null).use { cursor ->
if (cursor.moveToFirst()) cursor.getLong(0) else 0
}
Log.i(TAG, "ARCOLOGY_DB_IMPORT: Verification - $fileCount files, $nodeCount nodes")
if (fileCount == 0L && nodeCount == 0L) {
Log.w(TAG, "ARCOLOGY_DB_IMPORT: WARNING - Imported database appears to be empty!")
}
}
} catch (e: Exception) {
Log.e(TAG, "ARCOLOGY_DB_IMPORT: Verification failed: ${e.message}", e)
}
Log.i(TAG, "ARCOLOGY_DB_IMPORT: Import completed successfully")
true
} catch (e: Exception) {
Log.e(TAG, "ARCOLOGY_DB_IMPORT: Import failed: ${e.message}", e)
false
}
}
}Design Decisions
Repository Pattern with SQLDelight
The Repository pattern was chosen because it provides a clean abstraction over SQLDelight — domain models go in and out, generated query bindings stay internal. This is a common pattern in FOSS Android applications. SQLDelight was selected over Room or Exposed for its compile-time query verification and first-class Kotlin Multiplatform support (needed for both Android and JVM test targets).
FTS Staging for Batching
Inserting directly into FTS5 virtual tables is expensive because FTS rebuilds its index on each insert. The staging pattern decouples this:
During indexing, each node's searchable text goes into
fts_stagingAfter all nodes are indexed, a batch flush reads from staging and bulk-inserts into the FTS virtual tables
The staging table is cleared
This keeps individual file processing fast while still providing full-text search. The indexer controls when the batch flush happens (see the indexer pipeline).
Single-Threaded Transaction Dispatcher
The dbDispatcher Dispatchers.IO.limitedParallelism(1)= design ensures that all database operations serialize onto one thread. This matters because runBlocking is used inside transactionWithResult — without serialization, nested runBlocking calls from different coroutines could deadlock on the IO dispatcher's thread pool. A single dedicated thread eliminates this risk entirely.
Why Separate Task and Node Tables
Tasks (tasks table) are modeled separately from nodes (nodes table) because many TODO headings lack :ID: properties, making them ineligible as org-roam nodes. The task table stores its own primary key and cross-references node_id when available, with FOREIGN KEY cascade deletion to keep things consistent.
Tests
This section documents the test suites for the domain models and the repository layer. Both are Kotest DescribeSpec tests that exercise CRUD operations, edge cases, and property-based validation.
ModelsTest
Unit tests for the core domain models (=\=OrgFile\==, =\=OrgNode\==, =\=OrgLink\==, =\=OrgTag\==, =\=OrgAlias\==, =\=NodeProperty\==, =\=FileProperty\==). Covers construction, null handling, unicode, deep nesting, property-based testing, and copy semantics.
package computer.whatthefuck.arcology.domain
import computer.whatthefuck.arcology.fixtures.EdgeCaseTestData
import computer.whatthefuck.arcology.fixtures.TestData
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.shouldBe
import io.kotest.matchers.shouldNotBe
import io.kotest.matchers.collections.shouldBeEmpty
import io.kotest.matchers.collections.shouldContain
import io.kotest.matchers.collections.shouldHaveSize
import io.kotest.matchers.nulls.shouldBeNull
import io.kotest.matchers.nulls.shouldNotBeNull
import io.kotest.matchers.string.shouldContain
import io.kotest.property.Arb
import io.kotest.property.arbitrary.long
import io.kotest.property.arbitrary.string
import io.kotest.property.checkAll
import kotlinx.datetime.Instant
class ModelsTest : DescribeSpec({
describe("OrgFile") {
it("should create with all required properties") {
val file = TestData.sampleOrgFile
file.path shouldBe "/test/sample.org"
file.title shouldBe "Sample Org File"
file.hash shouldBe "abc123"
file.accessTime shouldBe TestData.sampleTimestamp
file.modificationTime shouldBe TestData.sampleTimestamp
}
it("should handle null title") {
val file = TestData.emptyOrgFile
file.title.shouldBeNull()
file.path.shouldNotBeNull()
file.hash.shouldNotBeNull()
}
it("should handle very long paths") {
val file = EdgeCaseTestData.fileWithLongPath
file.path.length shouldBe 92 // Verify we can handle long paths
file.title shouldBe "File with very long path"
}
it("should handle different timestamps") {
val file = OrgFile(
path = "/test.org",
hash = "hash",
accessTime = TestData.sampleTimestamp,
modificationTime = TestData.laterTimestamp
)
file.accessTime shouldNotBe file.modificationTime
file.modificationTime shouldBe TestData.laterTimestamp
}
it("should be equal when all properties match") {
val file1 = TestData.sampleOrgFile
val file2 = TestData.sampleOrgFile.copy()
file1 shouldBe file2
file1.hashCode() shouldBe file2.hashCode()
}
it("should handle property-based testing") {
checkAll<String, String>(10) { path, hash ->
val file = OrgFile(
path = path,
hash = hash,
accessTime = TestData.sampleTimestamp,
modificationTime = TestData.sampleTimestamp
)
file.path shouldBe path
file.hash shouldBe hash
}
}
}
describe("OrgNode") {
it("should create with minimal required properties") {
val node = OrgNode(
id = "test-id",
file = "/test.org",
level = 1,
position = 0
)
node.id shouldBe "test-id"
node.file shouldBe "/test.org"
node.level shouldBe 1
node.position shouldBe 0
node.todo.shouldBeNull()
node.title.shouldBeNull()
node.properties.isEmpty().shouldBe(true)
node.outlinePath.isEmpty().shouldBe(true)
}
it("should create with all properties populated") {
val node = TestData.childNode
node.id shouldBe "child-node-id"
node.todo shouldBe "TODO"
node.priority shouldBe "A"
node.scheduled shouldBe "2022-01-01"
node.deadline shouldBe "2022-01-15"
node.title shouldBe "Child Task"
node.properties.size shouldBe 1
node.properties.containsKey("EFFORT").shouldBe(true)
node.properties["EFFORT"] shouldBe "2h"
node.outlinePath.size shouldBe 2
}
it("should handle complex properties") {
val node = TestData.nodeWithComplexProperties
node.properties.size shouldBe 5
node.properties.containsKey("ROAM_ALIASES").shouldBe(true)
node.properties["ROAM_ALIASES"] shouldBe "alias1 alias2"
node.properties.containsKey("ROAM_TAGS").shouldBe(true)
node.properties["ROAM_TAGS"] shouldBe "tag1 tag2 tag3"
node.outlinePath.size shouldBe 3
}
it("should handle deep nesting") {
val node = EdgeCaseTestData.nodeWithMaxLevel
node.level shouldBe 20
node.position shouldBe 999
node.outlinePath.size shouldBe 20
node.outlinePath.first() shouldBe "Level 1"
node.outlinePath.last() shouldBe "Level 20"
}
it("should handle empty values") {
val node = EdgeCaseTestData.nodeWithEmptyValues
node.todo shouldBe ""
node.priority shouldBe ""
node.title shouldBe ""
node.properties.isEmpty().shouldBe(true)
node.outlinePath.isEmpty().shouldBe(true)
}
it("should handle unicode and special characters") {
val node = EdgeCaseTestData.nodeWithSpecialCharacters
node.id shouldContain "éñü"
node.title shouldContain "🎉"
node.title shouldContain "中文"
node.title shouldContain "العربية"
node.properties["UNICODE_PROP"] shouldContain "🚀"
node.properties["SYMBOLS"] shouldContain "!@#$%^&*()"
}
it("should copy with modifications") {
val original = TestData.rootNode
val modified = original.copy(title = "Modified Title", level = 2)
modified.title shouldBe "Modified Title"
modified.level shouldBe 2
modified.id shouldBe original.id // Unchanged
modified.properties shouldBe original.properties // Unchanged
}
}
describe("OrgLink") {
it("should create internal link") {
val link = TestData.sampleLink
link.fromNode shouldBe "root-node-id"
link.toNode shouldBe "child-node-id"
link.type shouldBe "internal"
link.position shouldBe 42
}
it("should create external link") {
val link = TestData.externalLink
link.fromNode shouldBe "child-node-id"
link.toNode.shouldBeNull()
link.type shouldBe "external"
link.properties.containsKey("url").shouldBe(true)
link.properties["url"] shouldBe "https://example.com"
}
it("should handle links without properties") {
val link = OrgLink(
position = 1,
fromNode = "source",
toNode = "target",
type = "simple"
)
link.properties.isEmpty().shouldBe(true)
link.toNode shouldBe "target"
}
}
describe("OrgTag") {
it("should create tag") {
val tag = TestData.projectTag
tag.nodeId shouldBe "root-node-id"
tag.tag shouldBe "project"
}
it("should handle unicode tags") {
val tag = OrgTag(nodeId = "test", tag = "项目")
tag.tag shouldBe "项目"
}
}
describe("OrgAlias") {
it("should create alias") {
val alias = TestData.rootAlias
alias.nodeId shouldBe "root-node-id"
alias.alias shouldBe "main-project"
}
}
describe("NodeProperty") {
it("should create node property") {
val property = TestData.customIdProperty
property.nodeId shouldBe "root-node-id"
property.key shouldBe "CUSTOM_ID"
property.value shouldBe "root"
}
it("should handle null values") {
val property = NodeProperty(nodeId = "test", key = "EMPTY", value = null)
property.value.shouldBeNull()
}
}
describe("FileProperty") {
it("should create file property") {
val property = TestData.titleFileProperty
property.file shouldBe "/test/sample.org"
property.key shouldBe "TITLE"
property.value shouldBe "Sample File"
}
}
})RoamRepositoryTest
Integration tests for the =\=RoamRepository\== implementation backed by an in-memory SQLite database. Covers file, node, link, tag, alias, and property CRUD operations, FTS staging, search, database state management, and edge cases (duplicate inserts, large property maps, long outline paths, unicode content).
package computer.whatthefuck.arcology.database
import computer.whatthefuck.arcology.domain.*
import computer.whatthefuck.arcology.fixtures.EdgeCaseTestData
import computer.whatthefuck.arcology.fixtures.TestData
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.collections.shouldBeEmpty
import io.kotest.matchers.collections.shouldContain
import io.kotest.matchers.collections.shouldHaveSize
import io.kotest.matchers.maps.shouldContainKey
import io.kotest.matchers.nulls.shouldBeNull
import io.kotest.matchers.nulls.shouldNotBeNull
import io.kotest.matchers.shouldBe
import kotlinx.coroutines.test.runTest
class RoamRepositoryTest : DescribeSpec({
describe("File operations") {
it("should insert and retrieve file") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val file = TestData.sampleOrgFile
repository.insertFile(file)
val retrieved = repository.getFileByPath(file.path)
retrieved.shouldNotBeNull()
retrieved.path shouldBe file.path
retrieved.title shouldBe file.title
retrieved.hash shouldBe file.hash
retrieved.accessTime shouldBe file.accessTime
retrieved.modificationTime shouldBe file.modificationTime
}
}
it("should return null for non-existent file") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val result = repository.getFileByPath("/non-existent.org")
result.shouldBeNull()
}
}
it("should get all files") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertFile(TestData.sampleOrgFile)
repository.insertFile(TestData.emptyOrgFile)
val files = repository.getAllFiles()
(files as List<OrgFile>).shouldHaveSize(2)
(files as List<OrgFile>).shouldContain(TestData.sampleOrgFile)
(files as List<OrgFile>).shouldContain(TestData.emptyOrgFile)
}
}
it("should delete file") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertFile(TestData.sampleOrgFile)
repository.deleteFile(TestData.sampleOrgFile.path)
val result = repository.getFileByPath(TestData.sampleOrgFile.path)
result.shouldBeNull()
}
}
it("should handle file with very long path") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val file = EdgeCaseTestData.fileWithLongPath
repository.insertFile(file)
val retrieved = repository.getFileByPath(file.path)
retrieved.shouldNotBeNull()
retrieved.path shouldBe file.path
}
}
}
describe("Node operations") {
it("should insert and retrieve node") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val node = TestData.rootNode
repository.insertNode(node)
val retrieved = repository.getNodeById(node.id)
retrieved.shouldNotBeNull()
retrieved.id shouldBe node.id
retrieved.title shouldBe node.title
retrieved.level shouldBe node.level
retrieved.properties shouldBe node.properties
retrieved.outlinePath shouldBe node.outlinePath
}
}
it("should get nodes by file") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode)
repository.insertNode(TestData.childNode)
val nodes = repository.getNodesByFile("/test/sample.org")
(nodes as List<OrgNode>).shouldHaveSize(2)
(nodes as List<OrgNode>).shouldContain(TestData.rootNode)
(nodes as List<OrgNode>).shouldContain(TestData.childNode)
}
}
it("should search nodes by title") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode.copy(title = "Project Planning"))
repository.insertNode(TestData.childNode.copy(title = "Task Implementation"))
val nodes = repository.searchNodesByTitle("Project")
(nodes as List<OrgNode>).shouldHaveSize(1)
nodes.first().title shouldBe "Project Planning"
}
}
it("should handle node with complex properties") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val node = TestData.nodeWithComplexProperties
repository.insertNode(node)
val retrieved = repository.getNodeById(node.id)
retrieved.shouldNotBeNull()
retrieved!!.properties shouldBe node.properties
retrieved.properties.size shouldBe 5
}
}
it("should delete node") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode)
repository.deleteNode(TestData.rootNode.id)
val result = repository.getNodeById(TestData.rootNode.id)
result.shouldBeNull()
}
}
it("should handle deep nesting") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val node = EdgeCaseTestData.nodeWithMaxLevel
repository.insertNode(node)
val retrieved = repository.getNodeById(node.id)
retrieved.shouldNotBeNull()
retrieved.level shouldBe 20
retrieved.outlinePath shouldHaveSize 20
}
}
it("should handle unicode content") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val node = EdgeCaseTestData.nodeWithSpecialCharacters
repository.insertNode(node)
val retrieved = repository.getNodeById(node.id)
retrieved.shouldNotBeNull()
retrieved.id shouldBe node.id
retrieved.title shouldBe node.title
}
}
}
describe("Link operations") {
it("should insert and retrieve links") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
// Insert nodes first
repository.insertNode(TestData.rootNode)
repository.insertNode(TestData.childNode)
val link = TestData.sampleLink
repository.insertLink(link)
val linksFrom = repository.getLinksFrom(link.fromNode)
linksFrom shouldHaveSize 1
linksFrom.first().fromNode shouldBe link.fromNode
linksFrom.first().toNode shouldBe link.toNode
linksFrom.first().type shouldBe link.type
}
}
it("should get links to node") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode)
repository.insertNode(TestData.childNode)
val link = TestData.sampleLink
repository.insertLink(link)
val linksTo = repository.getLinksTo(link.toNode!!)
linksTo shouldHaveSize 1
linksTo.first().toNode shouldBe link.toNode
}
}
it("should handle external links") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.childNode)
val link = TestData.externalLink
repository.insertLink(link)
val linksFrom = repository.getLinksFrom(link.fromNode)
linksFrom shouldHaveSize 1
linksFrom.first().toNode.shouldBeNull()
linksFrom.first().type shouldBe "external"
}
}
}
describe("Tag operations") {
it("should insert and retrieve tags") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode)
val tag = TestData.projectTag
repository.insertTag(tag)
val tags = repository.getTagsByNode(tag.nodeId)
tags shouldHaveSize 1
tags shouldContain tag.tag
}
}
it("should get nodes by tag") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode)
repository.insertNode(TestData.childNode)
repository.insertTag(TestData.projectTag)
repository.insertTag(TestData.urgentTag)
val nodes = repository.getNodesByTag("project")
nodes shouldHaveSize 1
nodes shouldContain TestData.projectTag.nodeId
}
}
it("should handle multiple tags per node") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.childNode)
repository.insertTag(TestData.urgentTag)
repository.insertTag(TestData.workTag)
val tags = repository.getTagsByNode(TestData.childNode.id)
tags shouldHaveSize 2
tags shouldContain "urgent"
tags shouldContain "work"
}
}
}
describe("Alias operations") {
it("should insert and retrieve aliases") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode)
val alias = TestData.rootAlias
repository.insertAlias(alias)
val aliases = repository.getAliasesByNode(alias.nodeId)
aliases shouldHaveSize 1
aliases shouldContain alias.alias
}
}
it("should get nodes by alias") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode)
val alias = TestData.rootAlias
repository.insertAlias(alias)
val nodes = repository.getNodesByAlias(alias.alias)
nodes shouldHaveSize 1
nodes shouldContain alias.nodeId
}
}
}
describe("Property operations") {
it("should insert and retrieve heading properties") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode)
val property = TestData.customIdProperty
repository.insertHeadingProperty(property)
val properties = repository.getHeadingProperties(property.nodeId)
properties.containsKey(property.key).shouldBe(true)
properties[property.key] shouldBe property.value
}
}
it("should get single heading property") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode)
repository.insertHeadingProperty(TestData.customIdProperty)
val value = repository.getHeadingProperty(
TestData.customIdProperty.nodeId,
TestData.customIdProperty.key
)
value shouldBe TestData.customIdProperty.value
}
}
it("should insert and retrieve file properties") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertFile(TestData.sampleOrgFile)
val property = TestData.titleFileProperty
repository.insertFileProperty(property)
val properties = repository.getFileProperties(property.file)
properties.containsKey(property.key).shouldBe(true)
properties[property.key] shouldBe property.value
}
}
it("should handle null property values") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
repository.insertNode(TestData.rootNode)
val property = NodeProperty(
nodeId = TestData.rootNode.id,
key = "NULL_PROP",
value = null
)
repository.insertHeadingProperty(property)
val value = repository.getHeadingProperty(property.nodeId, property.key)
value.shouldBeNull()
}
}
}
describe("Search operations") {
it("should search across nodes, tags, aliases, and properties") {
runTest {
val repository = DatabaseTestUtils.createPopulatedTestRepository()
// Search should find matches in titles, tags, aliases, and property values
val results = repository.searchNodes("project")
results.shouldNotBeNull()
// Should find nodes with "project" in title, tags, aliases, or properties
}
}
it("should return empty results for no matches") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val results = repository.searchNodes("nonexistent")
results.shouldBeEmpty()
}
}
}
describe("Database state management") {
it("should start with empty database") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
DatabaseTestUtils.assertDatabaseEmpty(repository)
}
}
it("should populate with test data") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
DatabaseTestUtils.populateWithTestData(repository)
val stats = DatabaseTestUtils.getTotalRecordCount(repository)
stats.filesCount shouldBe 2
stats.nodesCount shouldBe 3
stats.totalCount shouldBe stats.filesCount + stats.nodesCount +
stats.linksCount + stats.tagsCount + stats.aliasesCount +
stats.nodePropertiesCount + stats.filePropertiesCount
}
}
it("should clear database") {
runTest {
val repository = DatabaseTestUtils.createPopulatedTestRepository()
DatabaseTestUtils.clearDatabase(repository)
DatabaseTestUtils.assertDatabaseEmpty(repository)
}
}
}
describe("Error handling and edge cases") {
it("should handle inserting duplicate nodes") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val node = TestData.rootNode
repository.insertNode(node)
repository.insertNode(node.copy(title = "Updated Title"))
val retrieved = repository.getNodeById(node.id)
retrieved.shouldNotBeNull()
retrieved.title shouldBe "Updated Title" // Should update, not create duplicate
}
}
it("should handle very large properties map") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val largeProperties = (1..100).associate { "prop$it" to "value$it" }
val node = TestData.rootNode.copy(properties = largeProperties)
repository.insertNode(node)
val retrieved = repository.getNodeById(node.id)
retrieved.shouldNotBeNull()
retrieved!!.properties.size shouldBe 100
}
}
it("should handle very long outline paths") {
runTest {
val repository = DatabaseTestUtils.createTestRepository()
val longPath = (1..50).map { "Level $it with very long name that might cause issues" }
val node = TestData.rootNode.copy(outlinePath = longPath)
repository.insertNode(node)
val retrieved = repository.getNodeById(node.id)
retrieved.shouldNotBeNull()
retrieved!!.outlinePath.size shouldBe 50
}
}
}
})NEXT Investigate DatabaseTestUtilsInterface and expect/actual redundancy
The current design uses both a `DatabaseTestUtilsInterface` (with `expect object DatabaseTestUtils : DatabaseTestUtilsInterface`) — an interface + expect/actual double. This may be redundant: an `expect object` alone would provide the same shape. The interface exists for historical reasons; investigate whether `app/src/test/` or other non-KMP consumers depend on it, and whether the interface can be removed in favor of a plain `expect object DatabaseTestUtils`.
Test Infrastructure
Shared test infrastructure for setting up in-memory SQLite databases across JVM and Android test targets.
*Platform Contract*
`DatabaseStats` counts records across all tables — used by test assertions to verify empty/full states. `DatabaseTestUtilsInterface` (implemented by platform-specific `actual object DatabaseTestUtils`) provides:
`createTestRepository()` — fresh in-memory SQLite + wrapped `RoamRepository`
`createTestQuizRepository()` — fresh in-memory SQLite + wrapped `QuizRepository`
`populateWithTestData(repository)` — inserts sample files, nodes, links, tags, aliases, properties from `TestData` fixtures
`clearDatabase(repository)` — deletes all nodes and files (cascades per schema)
`getTotalRecordCount(repository)` — counts all entities across all tables
`assertDatabaseEmpty(repository)` — `require` that all tables are empty
`createPopulatedTestRepository()` — convenience combining create + populate
The common `expect object` declaration:
package computer.whatthefuck.arcology.database
import computer.whatthefuck.arcology.db.ArcologyDatabase
import computer.whatthefuck.arcology.domain.*
/**
* Platform-agnostic interface for database test utilities
*/
interface DatabaseTestUtilsInterface {
suspend fun createTestRepository(): RoamRepository
suspend fun createTestQuizRepository(): QuizRepository
suspend fun populateWithTestData(repository: RoamRepository)
suspend fun clearDatabase(repository: RoamRepository)
suspend fun getTotalRecordCount(repository: RoamRepository): DatabaseStats
suspend fun assertDatabaseEmpty(repository: RoamRepository)
suspend fun createPopulatedTestRepository(): RoamRepository
suspend fun createSharedTestDatabase(): ArcologyDatabase
}
/**
* Statistics about database content
*/
data class DatabaseStats(
val filesCount: Int,
val nodesCount: Int,
val linksCount: Int,
val tagsCount: Int,
val aliasesCount: Int,
val nodePropertiesCount: Int,
val filePropertiesCount: Int
) {
val totalCount = filesCount + nodesCount + linksCount + tagsCount + aliasesCount + nodePropertiesCount + filePropertiesCount
override fun toString(): String {
return "DatabaseStats(files=$filesCount, nodes=$nodesCount, links=$linksCount, " +
"tags=$tagsCount, aliases=$aliasesCount, nodeProps=$nodePropertiesCount, " +
"fileProps=$filePropertiesCount, total=$totalCount)"
}
}
// Platform-specific implementations
expect object DatabaseTestUtils : DatabaseTestUtilsInterface*JVM Implementation*
`DatabaseTestUtils` on the JVM creates a `JdbcSqliteDriver(JdbcSqliteDriver.IN_MEMORY)`, runs `ArcologyDatabase.Schema.create()`, and wraps it. Test data comes from `computer.whatthefuck.arcology.fixtures.TestData`:
package computer.whatthefuck.arcology.database
import computer.whatthefuck.arcology.db.ArcologyDatabase
import kotlinx.coroutines.runBlocking
import computer.whatthefuck.arcology.domain.*
import computer.whatthefuck.arcology.fixtures.TestData
import app.cash.sqldelight.db.SqlDriver
import app.cash.sqldelight.driver.jdbc.sqlite.JdbcSqliteDriver
/**
* JVM-specific implementation of database test utilities
*/
actual object DatabaseTestUtils : DatabaseTestUtilsInterface {
/**
* Creates an in-memory SQLite database for testing
*/
private fun createTestDatabase(): ArcologyDatabase {
val driver = createTestDriver()
return ArcologyDatabase(driver)
}
/**
* Creates a test SQLite driver with schema initialized
*/
private fun createTestDriver(): SqlDriver {
val driver = JdbcSqliteDriver(JdbcSqliteDriver.IN_MEMORY)
runBlocking {
ArcologyDatabase.Schema.create(driver).await()
}
return driver
}
/**
* Creates a test repository with in-memory database
*/
override suspend fun createTestRepository(): RoamRepository {
return RoamRepositoryImpl(createTestDatabase())
}
override suspend fun createTestQuizRepository(): QuizRepository {
return QuizRepositoryImpl(createTestDatabase())
}
/**
* Expose a raw database so a test can back two repositories (e.g.
* RoamRepositoryImpl + QuizRepositoryImpl) with the same in-memory
* SQLite — plugins like QuizIndexerPlugin join flashcards to nodes.
*/
override suspend fun createSharedTestDatabase(): ArcologyDatabase {
return createTestDatabase()
}
/**
* Populates a repository with test data
*/
override suspend fun populateWithTestData(repository: RoamRepository) {
// Insert test files
repository.insertFile(TestData.sampleOrgFile)
repository.insertFile(TestData.emptyOrgFile)
// Insert test nodes
TestData.allNodes.forEach { node ->
repository.insertNode(node)
}
// Insert test links
TestData.allLinks.forEach { link ->
repository.insertLink(link)
}
// Insert test tags
TestData.allTags.forEach { tag ->
repository.insertTag(tag)
}
// Insert test aliases
TestData.allAliases.forEach { alias ->
repository.insertAlias(alias)
}
// Insert test node properties
TestData.allNodeProperties.forEach { property ->
repository.insertHeadingProperty(property)
}
// Insert test file properties
TestData.allFileProperties.forEach { property ->
repository.insertFileProperty(property)
}
}
/**
* Clears all data from a database
*/
override suspend fun clearDatabase(repository: RoamRepository) {
// For test isolation, create A fresh repository instead of trying to clear data
// This is more reliable and simpler than manually deleting all relationships
// Each test should use createTestRepository() for proper isolation
// Note: In A real implementation, we would need individual delete methods
// for links, tags, aliases, and properties. For now, tests should create
// fresh repositories for isolation.
// Delete all nodes and files (cascading deletes should handle relationships)
val allNodes = repository.getAllNodes()
val allFiles = repository.getAllFiles()
allNodes.forEach { node ->
repository.deleteNode(node.id)
}
allFiles.forEach { file ->
repository.deleteFile(file.path)
}
}
/**
* Counts total records across all tables
*/
override suspend fun getTotalRecordCount(repository: RoamRepository): DatabaseStats {
val filesCount = repository.getAllFiles().size
val nodesCount = repository.getAllNodes().size
// Count links by getting from all nodes
val linksCount = repository.getAllNodes().sumOf { node ->
repository.getLinksFrom(node.id).size
}
// Count tags by getting from all nodes
val tagsCount = repository.getAllNodes().sumOf { node ->
repository.getTagsByNode(node.id).size
}
// Count aliases by getting from all nodes
val aliasesCount = repository.getAllNodes().sumOf { node ->
repository.getAliasesByNode(node.id).size
}
// Count properties by getting from all nodes
val nodePropertiesCount = repository.getAllNodes().sumOf { node ->
repository.getHeadingProperties(node.id).size
}
val filePropertiesCount = repository.getAllFiles().sumOf { file ->
repository.getFileProperties(file.path).size
}
return DatabaseStats(
filesCount = filesCount,
nodesCount = nodesCount,
linksCount = linksCount,
tagsCount = tagsCount,
aliasesCount = aliasesCount,
nodePropertiesCount = nodePropertiesCount,
filePropertiesCount = filePropertiesCount
)
}
/**
* Asserts that the database is empty
*/
override suspend fun assertDatabaseEmpty(repository: RoamRepository) {
val stats = getTotalRecordCount(repository)
require(stats.totalCount == 0) {
"Database is not empty: $stats"
}
}
/**
* Creates a repository pre-populated with test data
*/
override suspend fun createPopulatedTestRepository(): RoamRepository {
val repository = createTestRepository()
populateWithTestData(repository)
return repository
}
}*Android Implementation*
Android unit tests currently cannot instantiate an SQLite driver without an `Application`/`Context`, so this implementation throws `UnsupportedOperationException` with a helpful message directing the developer to JVM tests or connected Android testing. It stubs every method with the same logic.
Android unit tests are not fully supported — the test targets that run on device (`connectedAndroidTest`) are the proper future home for SQLite-backed tests on Android.
package computer.whatthefuck.arcology.database
import computer.whatthefuck.arcology.db.ArcologyDatabase
import computer.whatthefuck.arcology.domain.*
import computer.whatthefuck.arcology.fixtures.TestData
import app.cash.sqldelight.db.SqlDriver
import com.eygraber.sqldelight.androidx.driver.AndroidxSqliteDriver
/**
* Android-specific implementation of database test utilities.
*
* NOTE: Android unit tests are not fully supported. The tests will fail because
* Android-specific SQLite driver requires Android context which isn's available
* in unit test environment. Use JVM tests for database testing:
* ./gradlew jvmTest
*
* For Android integration testing, use:
* ./gradlew connectedAndroidTest
*/
actual object DatabaseTestUtils : DatabaseTestUtilsInterface {
/**
* Creates an in-memory SQLite database for testing
*/
private fun createTestDatabase(): ArcologyDatabase {
val driver = createTestDriver()
return ArcologyDatabase(driver)
}
/**
* Creates a test SQLite driver with schema initialized.
*
* This method throws UnsupportedOperationException because Android unit tests
* require Android context which isn's available in the test environment.
*/
private fun createTestDriver(): SqlDriver {
throw UnsupportedOperationException(
"Android unit tests are not fully supported. " +
"Use JVM tests for now: './gradlew jvmTest'. " +
"For Android testing, use './gradlew connectedAndroidTest' when available."
)
}
override suspend fun createTestRepository(): RoamRepository {
return RoamRepositoryImpl(createTestDatabase())
}
override suspend fun createTestQuizRepository(): QuizRepository {
return QuizRepositoryImpl(createTestDatabase())
}
override suspend fun createSharedTestDatabase(): ArcologyDatabase {
return createTestDatabase()
}
override suspend fun populateWithTestData(repository: RoamRepository) {
// Same implementation as JVM version
repository.insertFile(TestData.sampleOrgFile)
repository.insertFile(TestData.emptyOrgFile)
TestData.allNodes.forEach { node ->
repository.insertNode(node)
}
TestData.allLinks.forEach { link ->
repository.insertLink(link)
}
TestData.allTags.forEach { tag ->
repository.insertTag(tag)
}
TestData.allAliases.forEach { alias ->
repository.insertAlias(alias)
}
TestData.allNodeProperties.forEach { property ->
repository.insertHeadingProperty(property)
}
TestData.allFileProperties.forEach { property ->
repository.insertFileProperty(property)
}
}
override suspend fun clearDatabase(repository: RoamRepository) {
throw UnsupportedOperationException("Android unit tests not implemented - use JVM tests")
}
override suspend fun getTotalRecordCount(repository: RoamRepository): DatabaseStats {
val filesCount = repository.getAllFiles().size
val nodesCount = repository.getAllNodes().size
val linksCount = repository.getAllNodes().sumOf { node ->
repository.getLinksFrom(node.id).size
}
val tagsCount = repository.getAllNodes().sumOf { node ->
repository.getTagsByNode(node.id).size
}
val aliasesCount = repository.getAllNodes().sumOf { node ->
repository.getAliasesByNode(node.id).size
}
val nodePropertiesCount = repository.getAllNodes().sumOf { node ->
repository.getHeadingProperties(node.id).size
}
val filePropertiesCount = repository.getAllFiles().sumOf { file ->
repository.getFileProperties(file.path).size
}
return DatabaseStats(
filesCount = filesCount,
nodesCount = nodesCount,
linksCount = linksCount,
tagsCount = tagsCount,
aliasesCount = aliasesCount,
nodePropertiesCount = nodePropertiesCount,
filePropertiesCount = filePropertiesCount
)
}
override suspend fun assertDatabaseEmpty(repository: RoamRepository) {
val stats = getTotalRecordCount(repository)
require(stats.totalCount == 0) {
"Database is not empty: $stats"
}
}
override suspend fun createPopulatedTestRepository(): RoamRepository {
val repository = createTestRepository()
populateWithTestData(repository)
return repository
}
}Indexer Plugin Interface
Plugins extend the indexer pipeline by extracting domain-specific metadata after core org-roam data is stored. Each domain module (arroyo, quiz, agenda) provides its own plugin implementation.
package computer.whatthefuck.arcology.indexer
import computer.whatthefuck.arcology.parser.ParseResult
/**
* A plugin invoked by FlowFileIndexer after core org-roam metadata is stored.
* Each plugin extracts domain-specific data (tasks, flashcards, Arroyo keywords, etc.)
* from the parsed result and writes it to its repository.
*
* Plugins are run sequentially within the same transaction as core metadata,
* so they can safely delete and re-insert domain data per file.
*/
interface IndexerPlugin {
/**
* Called after core metadata (nodes, links, tags, properties) has been stored.
*
* @param result The parsed file content with all extracted metadata
*/
suspend fun onFileIndexed(result: ParseResult.Success)
/**
* Called when a file is removed from the index.
* Plugins should clean up any domain-specific data for this file.
* Default no-op so existing plugins don't need to implement it.
*
* @param filePath Path to the file being removed
*/
suspend fun onFileRemoved(filePath: String) {}
/**
* Called once after all files in a directory-indexing pass have been
* processed and committed, before FTS indexing runs. This is the hook
* for one-shot post-pass work that needs the full card/node set to be
* present (e.g. importing org-fc's review-history TSV, whose rows carry
* foreign keys to flashcard rows that the per-file [onFileIndexed] pass
* just wrote).
*
* Not invoked by the single-file [indexFile] path: that path does not
* establish the full flashcard set a TSV importer depends on.
*
* Default no-op so existing plugins don't need to implement it.
*
* @param rootPath The index root path (the argument passed to
* [FlowFileIndexer.indexDirectoryFlow]); empty string on Android
* where [FileSystemInterface] resolves relative paths against the
* selected tree URI.
*/
suspend fun onIndexingComplete(rootPath: String) {}
}
/**
* Concrete holder for the app's [IndexerPlugin] set, parameterized by the
* [FileSystemInterface] the plugins run against.
*
* Every [FlowFileIndexer] in the app should resolve an [IndexerPluginFactory]
* from DI and call [create] to get its plugin list. Constructing a
* [FlowFileIndexer] with a bare / ad-hoc plugin list risks advancing the
* `files.hash` without updating a plugin's domain tables (e.g.
* `task_headings`, `flashcards`, `arroyo_keywords`), after which the next
* full-directory index skips the file on hash match and locks in the stale
* data.
*
* This is a class rather than a function type so Koin can distinguish it from
* other `Function1`-shaped singles (e.g. the OrgDocumentEditor factory) under
* R8 type erasure — a function-typed `single<(AndroidFileSystem) -> ...>`
* collides with `(AndroidFileSystem) -> OrgDocumentEditor` and recurses into
* itself, causing a StackOverflowError.
*/
class IndexerPluginFactory(
private val plugins: (FileSystemInterface) -> List<IndexerPlugin>
) {
fun create(fileSystem: FileSystemInterface): List<IndexerPlugin> = plugins(fileSystem)
}Related Modules
The Indexer Pipeline — discovers and parses files, writes through the repository
arroyo/generators.org — ArroyoIndexerPlugin for ARROYO_* keyword extraction
The Indexer Platform Layer — filesystem abstractions for Android and JVM
The Org Document Editor — edits org files and writes node content back
The Indexer Platform Layer — filesystem abstractions for Android and JVM
The Org Document Editor — edits org files and writes node content back
arroyo/core.org — Phase 2 Arroyo code generation
Flashcard services (
src/.../flashcard/) — consume the repository for quiz functionality (now documented in quiz/flashcard.org)Quiz domain models and persistence (in quiz/models.org)
Task extraction (
src/.../tasks/) — consumes the repository for agenda views (to be documented separately)Search service (
src/.../search/) — wraps FTS queries with ranking (to be documented separately)