Introduction
This document is the source of truth for the Arcology Project's org-mode file parser. The parser bridges orgmode-kmp's low-level tokenizing and parsing with Arcology's domain models โ it turns raw org text into OrgNode, OrgLink, OrgTag, OrgAlias, and OrgRef objects that feed into the persistence layer.
The parser is consumed by the indexer pipeline (via FlowFileIndexer), its tests, and by CLI tools.
Parse Pipeline
The parser follows a three-phase pipeline:
raw org text string
orgmode-kmp's
OrgLexer(content).tokenize()presents a list of tokensorgmode-kmp's
parseWithDetailsin to an ASTextractDataFromDocumentextract domain modelsFile-level extraction (title, ID, properties,
ROAM_REFS)Recursive section walking to create
OrgNodelistLink extraction (
OrgInlineElem.Linkdiscovery)Tag extraction (heading tags +
ROAM_TAGS+ inherited filetags)Drawer collection (
LOGBOOK,REVIEW_DATA)Content extraction (FTS plain-text body)
ParseResult.Success(nodes, links, tags, aliases, refs, document, ...)
The Parser Class
The class is one open class OrgFileParser body assembled from named noweb blocks in the order given in the Tangle Targets section. The narrative below walks through each feature area in roughly the order the pipeline touches it, with the tests for each feature interleaved next to the code that implements it.
package computer.whatthefuck.arcology.parser
import computer.whatthefuck.arcology.domain.*
import computer.whatthefuck.arcology.utils.HashUtils
import computer.whatthefuck.arcology.utils.parseRoamAliases
import computer.whatthefuck.arcology.utils.parseRoamRefs
import kotlinx.datetime.Clock
import kotlinx.datetime.Instant
import xyz.lepisma.orgmode.OrgDocument
import xyz.lepisma.orgmode.OrgParseResult
import xyz.lepisma.orgmode.lexer.OrgLexer
import xyz.lepisma.orgmode.parseWithDetails
import xyz.lepisma.orgmode.OrgSection
import xyz.lepisma.orgmode.OrgHeading
import xyz.lepisma.orgmode.OrgInlineElem
import xyz.lepisma.orgmode.OrgChunk
import xyz.lepisma.orgmode.OrgBlock
import xyz.lepisma.orgmode.OrgList
import xyz.lepisma.orgmode.plainText
import xyz.lepisma.orgmode.formatInlineElemsToPlaintext
open class OrgFileParser {All the logic is described in the documents below
}OrgFileParser is declared open class rather than relying solely on interface injection. This allows test doubles to override the public parseFileContent() method.
For example, FlowOomTest injects a parser that throws OutOfMemoryError to verify the indexer's graceful degradation path. The FlowFileIndexer receives the parser as a constructor parameter, so the design supports both subclassing and DI.
Public API: parseFileContent
The single entry point. Tokenizes with OrgLexer, parses with parseWithDetails, and extracts domain models. Guards against blank content with an early empty-success return. Catches both tokenization and parsing exceptions, returning ParseResult.ParseError with a contextual error message that includes the problematic token position.
The parser uses orgmode-kmp's parseWithDetails() instead of the older parse(). This variant in my orgmode-kmp fork captures token position information, enabling detailed error messages critical for debugging parser errors in org files.
open fun parseFileContent(filePath: String, content: String, lastModified: Instant): ParseResult {
val hash = calculateHash(content)
val orgFile = OrgFile(
path = filePath,
title = null, // Will be extracted from document
hash = hash,
accessTime = lastModified,
modificationTime = lastModified
)
// Handle empty content specially
if (content.isBlank()) {
return ParseResult.Success(
file = orgFile,
nodes = emptyList(),
links = emptyList(),
tags = emptyList(),
aliases = emptyList(),
refs = emptyList(),
fileProperties = emptyList(),
nodeProperties = emptyList(),
nodeContents = emptyMap(),
document = null,
content = content
)
}
val tokens = try {
OrgLexer(content).tokenize()
} catch (e: Exception) {
return ParseResult.ParseError(filePath, "Failed to tokenize: ${e.message}")
}
val parseResult = try {
parseWithDetails(tokens)
} catch (e: Exception) {
return ParseResult.ParseError(filePath, "Failed to parse: ${e.message}")
}
return when (parseResult) {
is OrgParseResult.Success -> {
val extractedData = extractDataFromDocument(parseResult.document, filePath)
ParseResult.Success(
file = orgFile.copy(title = extractedData.fileTitle),
nodes = extractedData.nodes,
links = extractedData.links,
tags = extractedData.tags,
aliases = extractedData.aliases,
refs = extractedData.refs,
fileProperties = extractedData.fileProperties,
nodeProperties = extractedData.nodeProperties,
nodeContents = extractedData.nodeContents,
document = parseResult.document,
content = content
)
}
is OrgParseResult.Failure -> {
val errorContext = if (parseResult.tokens.isNotEmpty()) {
val tokenInfo = parseResult.tokens.first()
" at position ${parseResult.position} near '${tokenInfo.text.take(30)}'"
} else {
" at position ${parseResult.position}"
}
ParseResult.ParseError(filePath, "${parseResult.error}$errorContext")
}
}
}The OrgFileParserTest scaffold opens here; individual describe blocks are interleaved with the features they exercise and concatenated back into this file in document order via the parser-test-orgfileparser noweb-ref.
It uses the Shared Test Fixtures.
package computer.whatthefuck.arcology.parser
import computer.whatthefuck.arcology.fixtures.SampleOrgFiles
import computer.whatthefuck.arcology.fixtures.TestData
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.collections.shouldBeEmpty
import io.kotest.matchers.collections.shouldHaveSize
import io.kotest.matchers.nulls.shouldNotBeNull
import io.kotest.matchers.shouldBe
import io.kotest.matchers.shouldNotBe
import io.kotest.matchers.string.shouldContain
import io.kotest.matchers.string.shouldHaveLength
import io.kotest.matchers.types.shouldBeInstanceOf
import io.kotest.matchers.comparables.shouldBeLessThanOrEqualTo
class OrgFileParserTest : DescribeSpec({Basic parsing and parse-result-type coverage for the public entry point.
describe("Basic parsing") {
it("should parse simple org content") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
filePath = "/test/simple.org",
content = SampleOrgFiles.SIMPLE_ORG_CONTENT,
lastModified = TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
result.file.path shouldBe "/test/simple.org"
result.file.hash shouldHaveLength 64 // SHA-256 hash
}
it("should parse complex org content") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
filePath = "/test/complex.org",
content = SampleOrgFiles.COMPLEX_ORG_CONTENT,
lastModified = TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
result.file.path shouldBe "/test/complex.org"
// Currently stubbed, but structure should be there
result.nodes.shouldNotBeNull()
result.links.shouldNotBeNull()
result.tags.shouldNotBeNull()
}
it("should parse minimal org content") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
filePath = "/test/minimal.org",
content = SampleOrgFiles.MINIMAL_ORG_CONTENT,
lastModified = TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should handle empty org content") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
filePath = "/test/empty.org",
content = SampleOrgFiles.EMPTY_ORG_CONTENT,
lastModified = TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes.shouldBeEmpty()
result.links.shouldBeEmpty()
}
it("should handle malformed org content gracefully") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
filePath = "/test/malformed.org",
content = SampleOrgFiles.MALFORMED_ORG_CONTENT,
lastModified = TestData.sampleTimestamp
)
// Should not crash, might be Success with limited data or ParseError
when (result) {
is ParseResult.Success -> {
result.file.path shouldBe "/test/malformed.org"
}
is ParseResult.ParseError -> {
result.path shouldBe "/test/malformed.org"
result.error.shouldNotBeNull()
}
is ParseResult.FileNotFound -> {
// Should not happen in this test
throw AssertionError("Unexpected FileNotFound result")
}
}
}
}
describe("Performance and large content") {
it("should handle large org files") {
val parser = OrgFileParser()
val largeContent = SampleOrgFiles.createLargeOrgContent(100) // 100 nodes
val result = parser.parseFileContent(
filePath = "/test/large.org",
content = largeContent,
lastModified = TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
result.file.hash shouldHaveLength 64
}
it("should complete parsing in reasonable time") {
val parser = OrgFileParser()
val content = SampleOrgFiles.createLargeOrgContent(50)
val startTime = System.currentTimeMillis()
val result = parser.parseFileContent("/test.org", content, TestData.sampleTimestamp)
val endTime = System.currentTimeMillis()
result.shouldBeInstanceOf<ParseResult.Success>()
// Should complete in under 5 seconds even for large files
(endTime - startTime) shouldBeLessThanOrEqualTo 5000
}
}
describe("Parse result types") {
it("should return Success for valid content") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
"/test.org",
SampleOrgFiles.SIMPLE_ORG_CONTENT,
TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
result.file.shouldNotBeNull()
result.nodes.shouldNotBeNull()
result.links.shouldNotBeNull()
result.tags.shouldNotBeNull()
result.aliases.shouldNotBeNull()
result.fileProperties.shouldNotBeNull()
result.nodeProperties.shouldNotBeNull()
}
it("should handle parsing failures gracefully") {
val parser = OrgFileParser()
// Test with content that might cause orgmode-kmp to fail.
// The string is built from three separate quote-escaped "<"
// fragments so this .org source never contains a noweb-looking
// open/close ref token that the tangle tool would otherwise try to
// expand. The runtime value matches the original problematic input:
// three less-than signs, " invalid org syntax ", three greater-than.
val problematicContent = "<" + "<" + "< invalid org syntax >>>"
val result = parser.parseFileContent(
"/test.org",
problematicContent,
TestData.sampleTimestamp
)
when (result) {
is ParseResult.Success -> {
// If parsing succeeds, that's fine too
result.file.path shouldBe "/test.org"
}
is ParseResult.ParseError -> {
result.path shouldBe "/test.org"
result.error shouldContain "Failed to parse"
}
is ParseResult.FileNotFound -> {
throw AssertionError("Should not get FileNotFound for content parsing")
}
}
}
}Hash Calculation
The parser hashes every file's content with SHA-256 via calculateHash so the indexer can skip unchanged files. The HashUtils =expect=/=actual= pair is the parser's first call, so it lives here. The platform impls are tiny enough to tangle inline rather than via a separate assembly block.
private fun calculateHash(content: String): String {
return HashUtils.sha256(content)
}HashUtils (commonMain)
package computer.whatthefuck.arcology.utils
expect object HashUtils {
fun sha256(input: String): String
}It uses java.security.MessageDigest in both environments but commonMain doesn't see the java.* namespace; if we ever built a linux native or iOS Roam solution these would have different implementations, but they are the same in the pure-JVM and Android implementations.
package computer.whatthefuck.arcology.utils
import java.security.MessageDigest
actual object HashUtils {
actual fun sha256(input: String): String {
return MessageDigest.getInstance("SHA-256")
.digest(input.toByteArray())
.joinToString("") { "%02x".format(it) }
}
}package computer.whatthefuck.arcology.utils
import java.security.MessageDigest
actual object HashUtils {
actual fun sha256(input: String): String {
return MessageDigest.getInstance("SHA-256")
.digest(input.toByteArray())
.joinToString("") { "%02x".format(it) }
}
}HashUtilsTest
package computer.whatthefuck.arcology.utils
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.shouldBe
import io.kotest.matchers.shouldNotBe
import io.kotest.matchers.string.shouldHaveLength
import io.kotest.matchers.string.shouldMatch
import io.kotest.property.Arb
import io.kotest.property.arbitrary.string
import io.kotest.property.checkAll
class HashUtilsTest : DescribeSpec({
describe("HashUtils.sha256") {
it("should generate consistent hash for same input") {
val input = "test content"
val hash1 = HashUtils.sha256(input)
val hash2 = HashUtils.sha256(input)
hash1 shouldBe hash2
}
it("should generate different hashes for different inputs") {
val input1 = "content 1"
val input2 = "content 2"
val hash1 = HashUtils.sha256(input1)
val hash2 = HashUtils.sha256(input2)
hash1 shouldNotBe hash2
}
it("should generate 64-character hex string") {
val input = "test"
val hash = HashUtils.sha256(input)
hash shouldHaveLength 64
hash shouldMatch "^[0-9a-f]{64}$".toRegex()
}
it("should handle empty string") {
val hash = HashUtils.sha256("")
hash shouldHaveLength 64
// SHA-256 of empty string
hash shouldBe "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
it("should handle unicode content") {
val input = "Hello ไธ็ ๐ รฉmojis"
val hash1 = HashUtils.sha256(input)
val hash2 = HashUtils.sha256(input)
hash1 shouldBe hash2
hash1 shouldHaveLength 64
}
it("should handle very long content") {
val longContent = "x".repeat(100000)
val hash = HashUtils.sha256(longContent)
hash shouldHaveLength 64
hash shouldMatch "^[0-9a-f]{64}$".toRegex()
}
it("should handle newlines and whitespace") {
val content1 = "line1\nline2\n"
val content2 = "line1\r\nline2\r\n"
val hash1 = HashUtils.sha256(content1)
val hash2 = HashUtils.sha256(content2)
hash1 shouldNotBe hash2 // Different line endings should produce different hashes
hash1 shouldHaveLength 64
hash2 shouldHaveLength 64
}
it("should handle org-mode content") {
val orgContent = """
* Heading
:PROPERTIES:
:ID: test-id
:END:
Some content with [[links]].
""".trimIndent()
val hash = HashUtils.sha256(orgContent)
hash shouldHaveLength 64
hash shouldMatch "^[0-9a-f]{64}$".toRegex()
}
it("should detect content changes") {
val originalContent = """
* Original Heading
:PROPERTIES:
:ID: test-id
:END:
Original content.
""".trimIndent()
val modifiedContent = """
* Modified Heading
:PROPERTIES:
:ID: test-id
:END:
Modified content.
""".trimIndent()
val originalHash = HashUtils.sha256(originalContent)
val modifiedHash = HashUtils.sha256(modifiedContent)
originalHash shouldNotBe modifiedHash
}
it("should handle property-based testing") {
checkAll(50, Arb.string()) { input: String ->
val hash = HashUtils.sha256(input)
// Basic properties all hashes should have
hash shouldHaveLength 64
hash shouldMatch "^[0-9a-f]{64}$".toRegex()
// Consistency check
HashUtils.sha256(input) shouldBe hash
}
}
it("should handle special characters and symbols") {
val input = "!@#$%^&*()[]{}|;':\",./<>?`~"
val hash = HashUtils.sha256(input)
hash shouldHaveLength 64
hash shouldMatch "^[0-9a-f]{64}$".toRegex()
}
it("should be deterministic across calls") {
val input = "deterministic test"
val hashes = (1..10).map { HashUtils.sha256(input) }
hashes.forEach { hash ->
hash shouldBe hashes.first()
}
}
it("should handle binary-like content") {
val binaryLikeContent = (0..255).map { it.toChar() }.joinToString("")
val hash = HashUtils.sha256(binaryLikeContent)
hash shouldHaveLength 64
hash shouldMatch "^[0-9a-f]{64}$".toRegex()
}
it("should handle very similar inputs differently") {
val input1 = "very similar content"
val input2 = "very similar content " // Extra space
val hash1 = HashUtils.sha256(input1)
val hash2 = HashUtils.sha256(input2)
hash1 shouldNotBe hash2
}
}})Parser hash & unicode tests
OrgFileParserTest hash-determinism and unicode-handling describes, concatenated into parser-test-orgfileparser.
describe("Hash calculation") {
it("should generate consistent hashes for same content") {
val parser = OrgFileParser()
val content = SampleOrgFiles.SIMPLE_ORG_CONTENT
val result1 = parser.parseFileContent("/test.org", content, TestData.sampleTimestamp)
val result2 = parser.parseFileContent("/test.org", content, TestData.sampleTimestamp)
result1.shouldBeInstanceOf<ParseResult.Success>()
result2.shouldBeInstanceOf<ParseResult.Success>()
result1.file.hash shouldBe result2.file.hash
}
it("should generate different hashes for different content") {
val parser = OrgFileParser()
val result1 = parser.parseFileContent(
"/test1.org",
SampleOrgFiles.SIMPLE_ORG_CONTENT,
TestData.sampleTimestamp
)
val result2 = parser.parseFileContent(
"/test2.org",
SampleOrgFiles.COMPLEX_ORG_CONTENT,
TestData.sampleTimestamp
)
result1.shouldBeInstanceOf<ParseResult.Success>()
result2.shouldBeInstanceOf<ParseResult.Success>()
result1.file.hash shouldNotBe result2.file.hash // Different hashes for different content
}
it("should detect content changes via hash") {
val parser = OrgFileParser()
val originalContent = """* Original
:PROPERTIES:
:ID: original-id
:END:
Some content here.
"""
val modifiedContent = """* Modified
:PROPERTIES:
:ID: modified-id
:END:
Different content here.
"""
val result1 = parser.parseFileContent("/test.org", originalContent, TestData.sampleTimestamp)
val result2 = parser.parseFileContent("/test.org", modifiedContent, TestData.sampleTimestamp)
result1.shouldBeInstanceOf<ParseResult.Success>()
result2.shouldBeInstanceOf<ParseResult.Success>()
result1.file.hash shouldNotBe result2.file.hash // Should be different
}
}
describe("Unicode and special characters") {
it("should handle unicode content") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
filePath = "/test/unicode.org",
content = SampleOrgFiles.UNICODE_ORG_CONTENT,
lastModified = TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
result.file.path shouldBe "/test/unicode.org"
result.file.hash shouldHaveLength 64
}
it("should preserve unicode in file content hash") {
val parser = OrgFileParser()
val unicodeContent = """* ไธญๆๆ ้ข with รฉmojis ๐
:PROPERTIES:
:ID: unicode-id
:END:
Some unicode content here.
"""
val asciiContent = """* ASCII title
:PROPERTIES:
:ID: ascii-id
:END:
Some ASCII content here.
"""
val result1 = parser.parseFileContent("/test1.org", unicodeContent, TestData.sampleTimestamp)
val result2 = parser.parseFileContent("/test2.org", asciiContent, TestData.sampleTimestamp)
result1.shouldBeInstanceOf<ParseResult.Success>()
result2.shouldBeInstanceOf<ParseResult.Success>()
result1.file.hash shouldNotBe result2.file.hash // Should be different
}
}OrgFileParserPropertyTest uses FunSpec with a parser / timestamp scaffold; its context blocks are interleaved by feature. The scaffold opens here (its first context in document order, "Property: Hash calculation is deterministic", appears just below), and its closing }) is placed in Performance & Stability after the last context.
package computer.whatthefuck.arcology.parser
import computer.whatthefuck.arcology.fixtures.TestData
import computer.whatthefuck.arcology.generators.OrgContentGenerators
import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.shouldBe
import io.kotest.matchers.types.shouldBeInstanceOf
import io.kotest.property.Arb
import io.kotest.property.arbitrary.*
import io.kotest.property.forAll
class OrgFileParserPropertyTest : FunSpec({
val parser = OrgFileParser()
val timestamp = TestData.sampleTimestamp
context("Property: Hash calculation is deterministic") {
test("same content always produces the same hash") {
forAll(Arb.string(10..300)) { content ->
val result1 = parser.parseFileContent("/test.org", content, timestamp)
val result2 = parser.parseFileContent("/test.org", content, timestamp)
if (result1 is ParseResult.Success && result2 is ParseResult.Success) {
result1.file.hash == result2.file.hash
} else true
}
}
test("different content produces different hashes") {
forAll(
Arb.string(10..100, Codepoint.alphanumeric()),
Arb.string(10..100, Codepoint.alphanumeric())
) { content1, content2 ->
if (content1 == content2) true
else {
val result1 = parser.parseFileContent("/a.org", content1, timestamp)
val result2 = parser.parseFileContent("/b.org", content2, timestamp)
if (result1 is ParseResult.Success && result2 is ParseResult.Success) {
result1.file.hash != result2.file.hash
} else true
}
}
}
}Document-Level Extraction
extractDataFromDocument is the top-level extraction coordinator. It sets up accumulators for all domain types, extracts file-level metadata (title, :ID: property, #+FILETAGS), creates a level-0 node if the file has an ID property in its preamble, and then iterates all OrgSection children to extract heading-level nodes.
File-level helper functions handle preamble extraction:
extractFileLevelIdโ reads:ID:from document preamble propertiesextractFileLevelTitleโ reads#+TITLE:directive or falls back to filenameextractFileLevelPropertiesโ reads all preamble propertiesextractFileNameAsTitleโ strips path and extension from filenameextractFilePropertiesโ accumulatesFilePropertyrows from the preamble
private fun extractDataFromDocument(document: OrgDocument, filePath: String): ExtractedData {
val nodes = mutableListOf<OrgNode>()
val links = mutableListOf<OrgLink>()
val tags = mutableListOf<OrgTag>()
val aliases = mutableListOf<OrgAlias>()
val refs = mutableListOf<OrgRef>()
val fileProperties = mutableListOf<FileProperty>()
val nodeProperties = mutableListOf<NodeProperty>()
val nodeContents = mutableMapOf<String, String>()
var fileTitle: String? = null
var position = 0
// Extract filetags - these inherit to ALL nodes in the file
val fileTags: List<String> = document.preamble.filetags?.tags ?: emptyList()
// Check for file-level properties and ID (level 0 node)
val fileLevelId = extractFileLevelId(document, filePath)
var fileLevelNodeCreated = false
// Extract file-level properties
extractFileProperties(document, filePath, fileProperties)
// If there's a file-level ID, create a level 0 node
if (fileLevelId != null) {
fileTitle = extractFileLevelTitle(document, filePath)
val fileLevelProperties = extractFileLevelProperties(document)
val fileLevelDrawers = extractDrawersFromChunks(document.preface.body)
val fileNode = OrgNode(
id = fileLevelId,
file = filePath,
level = 0, // File level
position = position++,
todo = null,
priority = null,
scheduled = null,
deadline = null,
title = fileTitle ?: extractFileNameAsTitle(filePath),
properties = fileLevelProperties,
outlinePath = emptyList(),
drawers = fileLevelDrawers,
parentNodeId = null
)
nodes.add(fileNode)
fileLevelNodeCreated = true
// Add file-level properties to node properties
fileLevelProperties.forEach { (key, value) ->
nodeProperties.add(NodeProperty(fileLevelId, key, value))
// Handle ROAM_REFS at file level
if (key.equals("ROAM_REFS", ignoreCase = true)) {
parseRoamRefs(value).forEach { refUrl ->
refs.add(OrgRef(fileLevelId, refUrl, "http"))
}
}
}
// Extract content from preface for file-level node
val prefaceContent = extractContentFromChunks(document.preface.body)
if (prefaceContent.isNotBlank()) {
nodeContents[fileLevelId] = prefaceContent
}
// Add filetags to file-level node
fileTags.forEach { tag ->
tags.add(OrgTag(fileLevelId, tag))
}
}
// Extract links from preface (content before first heading)
if (fileLevelId != null) {
val prefaceLinks = extractLinksFromChunks(document.preface.body, fileLevelId, position)
links.addAll(prefaceLinks)
position += prefaceLinks.size
}
// Process all sections to extract nodes (these will be level 1+)
document.content.forEach { section ->
val extractedNodes = extractNodesFromSection(section, filePath, position, emptyList(), fileLevelId, fileTags)
nodes.addAll(extractedNodes.nodes)
links.addAll(extractedNodes.links)
tags.addAll(extractedNodes.tags)
aliases.addAll(extractedNodes.aliases)
refs.addAll(extractedNodes.refs)
nodeProperties.addAll(extractedNodes.properties)
nodeContents.putAll(extractedNodes.nodeContents)
position = extractedNodes.lastPosition
}
return ExtractedData(
fileTitle = fileTitle,
nodes = nodes,
links = links,
tags = tags,
aliases = aliases,
refs = refs,
fileProperties = fileProperties,
nodeProperties = nodeProperties,
nodeContents = nodeContents
)
}
private fun extractFileProperties(document: OrgDocument, filePath: String, fileProperties: MutableList<FileProperty>) {
document.preamble.properties?.map?.forEach { (key, orgLine) ->
val value = orgLine.items.filterIsInstance<OrgInlineElem.Text>()
.joinToString("") { it.text }
.trim()
.takeIf { it.isNotEmpty() }
if (value != null) {
fileProperties.add(FileProperty(file = filePath, key = key, value = value))
}
}
}
private fun extractFileLevelId(document: OrgDocument, filePath: String): String? {
return document.preamble.properties?.map?.get("ID")?.let { idLine ->
idLine.items.filterIsInstance<OrgInlineElem.Text>()
.joinToString("") { it.text }
.trim()
.takeIf { it.isNotEmpty() }
}
}
private fun extractFileLevelTitle(document: OrgDocument, filePath: String): String? {
val titleFromPreamble = document.preamble.title.items
.filterIsInstance<OrgInlineElem.Text>()
.joinToString("") { it.text }
.trim()
.takeIf { it.isNotEmpty() }
return titleFromPreamble ?: extractFileNameAsTitle(filePath)
}
private fun extractFileLevelProperties(document: OrgDocument): Map<String, String> {
return document.preamble.properties?.map?.mapValues { (_, orgLine) ->
orgLine.items.filterIsInstance<OrgInlineElem.Text>()
.joinToString("") { it.text }
.trim()
} ?: emptyMap()
}
private fun extractFileNameAsTitle(filePath: String): String {
return filePath.substringAfterLast('/').substringBeforeLast('.')
}File-level node creation, #+FILETAGS inheritance to all nodes, one-way tag attribution (a child heading's tags stay off the ancestor node), and content roll-up for child headings without IDs are exercised by the OrgFileParserEdgeCaseTest and OrgFileParserNestingTest scaffolds.
package computer.whatthefuck.arcology.parser
import computer.whatthefuck.arcology.fixtures.TestData
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.collections.shouldContainExactly
import io.kotest.matchers.collections.shouldHaveSize
import io.kotest.matchers.shouldBe
import io.kotest.matchers.collections.shouldContain
import io.kotest.matchers.types.shouldBeInstanceOf
class OrgFileParserEdgeCaseTest : DescribeSpec({
val parser = OrgFileParser()
val timestamp = TestData.sampleTimestamppackage computer.whatthefuck.arcology.parser
import computer.whatthefuck.arcology.fixtures.TestData
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.collections.shouldContainExactly
import io.kotest.matchers.collections.shouldHaveSize
import io.kotest.matchers.shouldBe
import io.kotest.matchers.types.shouldBeInstanceOf
class OrgFileParserNestingTest : DescribeSpec({
val parser = OrgFileParser()
val timestamp = TestData.sampleTimestamp
context("Property: File-level nodes") {
test("file-level ID creates a level-0 node") {
forAll(OrgContentGenerators.validId) { id ->
val content = """
:PROPERTIES:
:ID: $id
:END:
#+TITLE: File Level Test
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes.any { it.id == id && it.level == 0 }
}
}
}
describe("Filetag inheritance") {
it("should inherit filetags to all nodes") {
val content = """#+FILETAGS: :project:important:
,* Node A
:PROPERTIES:
:ID: filetag-a-id
:END:
Content for node A.
,* Node B
:PROPERTIES:
:ID: filetag-b-id
:END:
Content for node B.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
// Both nodes should have inherited filetags
val tagsA = result.tags.filter { it.nodeId == "filetag-a-id" }.map { it.tag }
val tagsB = result.tags.filter { it.nodeId == "filetag-b-id" }.map { it.tag }
tagsA.toSet().containsAll(setOf("project", "important")) shouldBe true
tagsB.toSet().containsAll(setOf("project", "important")) shouldBe true
}
}
describe("Content rollup for headings without IDs") {
it("should roll up content to parent node when child has no ID") {
val content = """
* Parent
:PROPERTIES:
:ID: parent-id
:END:
Parent content.
** Child Without ID
Child content should roll up to parent.
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes shouldHaveSize 1
result.nodes.first().id shouldBe "parent-id"
// The child content should be in the parent's node content
val parentContent = result.nodeContents["parent-id"]
parentContent?.contains("Parent content") shouldBe true
parentContent?.contains("Child content") shouldBe true
}
}
describe("Tag attribution for headings without IDs") {
it("should not roll up child heading tags to the parent node") {
val content = """
,#+FILETAGS: :project:
,* Parent
:PROPERTIES:
:ID: parent-id
:END:
Parent content.
,** Child Without ID :childtag:
:PROPERTIES:
:ROAM_TAGS: roamtag
:END:
Child content.
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
val parentTags = result.tags.filter { it.nodeId == "parent-id" }.map { it.tag }
parentTags.shouldContainExactly("project")
result.tags.none { it.tag == "childtag" } shouldBe true
result.tags.none { it.tag == "roamtag" } shouldBe true
}
it("should attach own tags and inherited filetags to ID-bearing children") {
val content = """
,#+FILETAGS: :project:
,* Parent
:PROPERTIES:
:ID: parent-id
:END:
Parent content.
,** Child With ID :childtag:
:PROPERTIES:
:ID: child-id
:END:
Child content.
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
val childTags = result.tags.filter { it.nodeId == "child-id" }.map { it.tag }
childTags.toSet() shouldBe setOf("childtag", "project")
val parentTags = result.tags.filter { it.nodeId == "parent-id" }.map { it.tag }
parentTags.toSet() shouldBe setOf("project")
}
}
describe("File-level node with nested children") {
it("should create file-level node and child nodes") {
val content = """#+TITLE: File With Children
:PROPERTIES:
:ID: file-node-id
:END:
File-level content.
,* Child 1
:PROPERTIES:
:ID: child-1-id
:END:
Child 1 content.
,** Grandchild
:PROPERTIES:
:ID: grandchild-id
:END:
Grandchild content.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
// File-level properties don't create a node, only headings do
result.nodes shouldHaveSize 2
val childNode = result.nodes.find { it.id == "child-1-id" }
childNode?.level shouldBe 1
val grandchild = result.nodes.find { it.id == "grandchild-id" }
grandchild?.level shouldBe 2
}
}Node Metadata Extraction
Four small functions for extracting the building blocks of an OrgNode:
extractNodeIdโ reads:ID:from the heading's properties block, joiningOrgInlineElem.Textitems into a plain stringextractTitleโ extracts plain text from the heading title usingplainText()to strip org markupextractPropertiesโ iterates the heading's:PROPERTIES:block and converts eachOrgLineto a key-value stringformatDTStampโ converts orgmode-kmp'sOrgInlineElem.DTStampto its date string representation for storage
private fun extractNodeId(heading: OrgHeading): String? {
// Extract ID from the heading's properties block
return heading.properties?.map?.get("ID")?.let { idLine ->
// Extract text content from the OrgLine
idLine.items.filterIsInstance<OrgInlineElem.Text>()
.joinToString("") { it.text }
.trim()
.takeIf { it.isNotEmpty() }
}
}
private fun extractTitle(heading: OrgHeading): String {
// Extract plain text from the heading title, removing org markup
return heading.title.plainText().trim()
}
private fun extractProperties(heading: OrgHeading): Map<String, String> {
// Extract all properties from the heading's properties block
return heading.properties?.map?.mapValues { (_, orgLine) ->
// Extract text content from each OrgLine value
orgLine.items.filterIsInstance<OrgInlineElem.Text>()
.joinToString("") { it.text }
.trim()
} ?: emptyMap()
}
private fun formatDTStamp(dtStamp: OrgInlineElem.DTStamp): String {
// Convert DTStamp back to org-mode timestamp format
return dtStamp.date.toString()
}
context("Property: ID extraction is consistent") {
test("nodes with IDs are always extractable") {
forAll(OrgContentGenerators.validId) { id ->
val content = """* Test Heading
:PROPERTIES:
:ID: $id
:END:
Some content here.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes.any { it.id == id }
}
}
test("headings without ID produce no node") {
forAll(OrgContentGenerators.headingTitle) { title ->
val content = "* $title\n\nSome body text."
val result = parser.parseFileContent("/test.org", content, timestamp)
if (result is ParseResult.Success) {
result.nodes.isEmpty()
} else true
}
}
}The OrgFileParserTest "Current implementation" describe covers the still-stubbed corners of node/property/tag extraction.
describe("Current implementation (stubbed functionality)") {
it("should handle stubbed node extraction") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
"/test.org",
SampleOrgFiles.SIMPLE_ORG_CONTENT,
TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
// Currently returns empty lists due to stubbed extractNodeId
// When ID extraction is implemented, this test should be updated
}
it("should handle stubbed property extraction") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
"/test.org",
SampleOrgFiles.PROPERTIES_HEAVY_ORG_CONTENT,
TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
// Currently returns empty properties due to stubbed extraction
// When property extraction is implemented, this should find properties
}
it("should extract tags from headings") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
"/test.org",
SampleOrgFiles.TAGS_AND_LINKS_ORG_CONTENT,
TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
// Tag extraction is now implemented
result.tags shouldHaveSize 7
}
}Tag Extraction
Tags come from three sources: the heading's tag line (:tag1:tag2:), the ROAM_TAGS property, and inherited #+FILETAGS. All three attach to the heading's own node (nodeId) โ tags never roll up to the nearest ID-bearing ancestor, so an ID-less child heading's tags are simply not indexed. fileTags is threaded down through the recursive walk so every ID-bearing node โ file-level or heading โ picks them up; inheritance is one-way, downward only.
context("Property: Tag extraction") {
test("inline heading tags are extracted for nodes with IDs") {
forAll(OrgContentGenerators.validId, OrgContentGenerators.tag) { id, tag ->
val content = """* Heading :$tag:
:PROPERTIES:
:ID: $id
:END:
Some content here.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.tags.any { it.tag == tag && it.nodeId == id }
}
}
}ROAM_REFS & ROAM_ALIASES Parsing
Parses the ROAM_REFS and ROAM_ALIASES property values. Each can contain multiple quoted URLs/aliases (format: "url1" "url2") or space-separated values, mixed freely. The shared quote-aware tokenizer lives in computer.whatthefuck.arcology.utils (parseRoamRefs / parseRoamAliases); the parser imports it in the preamble and calls it from the recursive walker, so no private helper is needed in OrgFileParser itself.
// ROAM_REFS / ROAM_ALIASES parsing is provided by
// computer.whatthefuck.arcology.utils (parseRoamRefs, parseRoamAliases).RoamPropertyParsing.kt (shared util)
Tiny single-file module; tangled inline.
package computer.whatthefuck.arcology.utils
/**
* Matches one ROAM_ALIASES / ROAM_REFS token: either a double-quoted string
* (group 1 = inner contents) or an unquoted whitespace-delimited token (group 2).
*/
private val TOKEN_PATTERN = """"([^"]+)"|(\S+)""".toRegex()
/**
* Parse a ROAM_ALIASES / ROAM_REFS property value into individual tokens.
*
* Handles three forms, including mixed input:
* - All quoted: `"alias one" "alias two"` -> ["alias one", "alias two"]
* - All unquoted: `alias1 alias2` -> ["alias1", "alias2"]
* - Mixed: `"Multi word alias" alias` -> ["Multi word alias", "alias"]
*
* Each match contributes the quoted contents (group 1) when present, or the
* bare token (group 2) otherwise. Blank results are filtered out.
*/
fun parseRoamAliases(value: String): List<String> = parseTokens(value)
fun parseRoamRefs(value: String): List<String> = parseTokens(value)
private fun parseTokens(value: String): List<String> =
TOKEN_PATTERN.findAll(value)
.map { match -> match.groupValues[1].ifBlank { match.groupValues[2] } }
.filter { it.isNotBlank() }
.toList()RoamPropertyParsingTest
package computer.whatthefuck.arcology.utils
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.collections.shouldBeEmpty
import io.kotest.matchers.collections.shouldContainExactly
import io.kotest.matchers.collections.shouldHaveSize
class RoamPropertyParsingTest : DescribeSpec({
describe("parseRoamAliases") {
it("should parse a single unquoted alias") {
parseRoamAliases("my-alias") shouldContainExactly listOf("my-alias")
}
it("should parse multiple unquoted aliases") {
parseRoamAliases("alias1 alias2 alias3") shouldContainExactly
listOf("alias1", "alias2", "alias3")
}
it("should parse multiple quoted aliases") {
parseRoamAliases("\"alias one\" \"alias two\"") shouldContainExactly
listOf("alias one", "alias two")
}
it("should parse a single multi-word quoted alias") {
parseRoamAliases("\"Multi word alias\"") shouldContainExactly listOf("Multi word alias")
}
it("should parse mixed quoted and unquoted aliases") {
parseRoamAliases("\"Multi word alias\" alias") shouldContainExactly
listOf("Multi word alias", "alias")
}
it("should return empty for blank input") {
parseRoamAliases("").shouldBeEmpty()
}
it("should return empty for whitespace-only input") {
parseRoamAliases(" \t ").shouldBeEmpty()
}
it("should collapse repeated whitespace in unquoted input") {
parseRoamAliases("a b\tc") shouldContainExactly listOf("a", "b", "c")
}
}
describe("parseRoamRefs") {
it("should parse quoted URLs") {
parseRoamRefs("\"https://example.com\" \"https://other.com\"") shouldContainExactly
listOf("https://example.com", "https://other.com")
}
it("should parse unquoted URLs") {
parseRoamRefs("https://example.com https://other.com") shouldContainExactly
listOf("https://example.com", "https://other.com")
}
it("should parse a single quoted ref") {
val result = parseRoamRefs("\"https://example.com\"")
result shouldHaveSize 1
result shouldContainExactly listOf("https://example.com")
}
it("should return empty for blank input") {
parseRoamRefs("").shouldBeEmpty()
}
}})Parser ROAM_REFS / ROAM_ALIASES edge cases
describe("ROAM_REFS edge cases") {
it("should parse multiple quoted ROAM_REFS") {
val content = """* Node with Refs
:PROPERTIES:
:ID: refs-node-id
:ROAM_REFS: "https://example.com/page1" "https://example.com/page2"
:END:
Some content here.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.refs shouldHaveSize 2
}
it("should parse single unquoted ROAM_REFS") {
val content = """* Node with Ref
:PROPERTIES:
:ID: single-ref-id
:ROAM_REFS: https://example.com/page
:END:
Some content here.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.refs shouldHaveSize 1
}
}
describe("ROAM_ALIASES extraction") {
it("should extract a single unquoted alias from ROAM_ALIASES property") {
val content = """* Aliased Node
:PROPERTIES:
:ID: aliased-id
:ROAM_ALIASES: my-alias
:END:
Some content here.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.aliases shouldHaveSize 1
result.aliases.map { it.alias } shouldContain "my-alias"
}
it("should extract multiple unquoted aliases from ROAM_ALIASES property") {
val content = """* Aliased Node
:PROPERTIES:
:ID: aliased-id
:ROAM_ALIASES: alias1 alias2 alias3
:END:
Some content here.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.aliases shouldHaveSize 3
result.aliases.map { it.alias } shouldContainExactly listOf("alias1", "alias2", "alias3")
}
it("should extract quoted multi-word aliases from ROAM_ALIASES property") {
val content = """* Aliased Node
:PROPERTIES:
:ID: aliased-id
:ROAM_ALIASES: "Multi word alias" "Another one"
:END:
Some content here.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.aliases shouldHaveSize 2
result.aliases.map { it.alias } shouldContainExactly listOf("Multi word alias", "Another one")
}
it("should extract mixed quoted and unquoted aliases from ROAM_ALIASES property") {
val content = """* Aliased Node
:PROPERTIES:
:ID: aliased-id
:ROAM_ALIASES: "Multi word alias" alias
:END:
Some content here.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.aliases shouldHaveSize 2
result.aliases.map { it.alias } shouldContainExactly listOf("Multi word alias", "alias")
}
}Recursive Section Walking
The heart of the parser, constructs a list of OrgNodes and other metadata. extractNodesFromSection is the public-facing wrapper; extractNodesFromSectionInternal does the recursive walk.
The algorithm: extract the heading's :ID:; if present, build an OrgNode with level TODO priority planning title properties drawers; otherwise roll content up to parentNodeId. Extract properties (โ aliases/refs), links, and FTS content from the section body โ these roll up to targetNodeId โ while tags (heading line + ROAM_TAGS + inherited filetags) attach to the heading's own node only, then recurse into child OrgSection chunks passing nodeId ?: parentNodeId down as the new parent.
The currentPosition counter advances past each section's extracted links before recursing into child sections. Without this, sibling subsections (whose links roll up to the same ancestor node) all start at the same position and their OrgLink rows collide on the links table primary key (pos, from_node), silently clobbering each other on insert. The file-preface path in extractData threads the counter the same way.
An org-roam node can contain multiple headings but only the first node has an :ID: property. When the parser encounters child headings without IDs, their content (plain text, links) rolls up to the nearest ancestor node that has an ID. Tag inheritance is one-way: #+FILETAGS flow down to every ID-bearing node in the file, but a child heading's tags never roll up โ a :noexport: on a sub-heading elides that subtree at render time without unpublishing the whole page.
private fun extractNodesFromSection(
section: OrgSection,
filePath: String,
startPosition: Int,
parentPath: List<String>,
parentNodeId: String? = null,
fileTags: List<String> = emptyList()
): ExtractedNodes {
return extractNodesFromSectionInternal(section, filePath, startPosition, parentPath, parentNodeId, fileTags)
}
private fun extractNodesFromSectionInternal(
section: OrgSection,
filePath: String,
startPosition: Int,
parentPath: List<String>,
parentNodeId: String? = null,
fileTags: List<String> = emptyList()
): ExtractedNodes {
val nodes = mutableListOf<OrgNode>()
val links = mutableListOf<OrgLink>()
val tags = mutableListOf<OrgTag>()
val aliases = mutableListOf<OrgAlias>()
val refs = mutableListOf<OrgRef>()
val properties = mutableListOf<NodeProperty>()
val nodeContents = mutableMapOf<String, String>()
var currentPosition = startPosition
val nodeId = extractNodeId(section.heading)
val outlinePath = parentPath + extractTitle(section.heading)
// Determine which node ID to use for content extraction
val targetNodeId = nodeId ?: parentNodeId
if (nodeId != null) {
// This heading has an ID - create a node for it
val node = OrgNode(
id = nodeId,
file = filePath,
level = section.heading.level.level,
position = currentPosition++,
todo = section.heading.todoState?.text,
priority = section.heading.priority?.priority?.toString(),
scheduled = section.heading.planningInfo?.scheduled?.let { dtStamp ->
formatDTStamp(dtStamp)
},
deadline = section.heading.planningInfo?.deadline?.let { dtStamp ->
formatDTStamp(dtStamp)
},
title = extractTitle(section.heading),
properties = extractProperties(section.heading),
outlinePath = outlinePath,
drawers = extractDrawersFromSection(section),
parentNodeId = parentNodeId
)
nodes.add(node)
}
// Extract content from this heading regardless of whether it has an ID
// If it doesn't have an ID, content rolls up to the parent node
if (targetNodeId != null) {
// Tags attach to the heading's own node only (one-way inheritance):
// filetags flow down to every ID-bearing node, but heading tags and
// ROAM_TAGS never roll up to the nearest ID-bearing ancestor.
if (nodeId != null) {
// Extract tags from heading
section.heading.tags?.tags?.forEach { tag ->
tags.add(OrgTag(nodeId, tag))
}
// Extract additional tags from ROAM_TAGS property
extractProperties(section.heading)["ROAM_TAGS"]?.let { roamTags ->
// Split space-separated tags
roamTags.split("\\s+".toRegex()).filter { it.isNotBlank() }.forEach { tag ->
tags.add(OrgTag(nodeId, tag))
}
}
// Add inherited filetags to this node
fileTags.forEach { tag ->
tags.add(OrgTag(nodeId, tag))
}
}
// Extract properties as separate entries (only if this heading has its own ID)
if (nodeId != null) {
extractProperties(section.heading).forEach { (key, value) ->
properties.add(NodeProperty(nodeId, key, value))
// Handle special properties like aliases
if (key.equals("ROAM_ALIASES", ignoreCase = true)) {
parseRoamAliases(value).forEach { alias ->
aliases.add(OrgAlias(nodeId, alias))
}
}
// Handle ROAM_REFS - can have multiple quoted URLs
if (key.equals("ROAM_REFS", ignoreCase = true)) {
parseRoamRefs(value).forEach { refUrl ->
refs.add(OrgRef(nodeId, refUrl, "http"))
}
}
}
}
// Extract links from section content
val extractedLinks = extractLinksFromSection(section, targetNodeId, currentPosition)
links.addAll(extractedLinks)
currentPosition += extractedLinks.size
// Extract content from section body for FTS indexing
// Only non-section chunks (paragraphs, lists, blocks, etc.)
val nonSectionChunks = section.body.filter { it !is OrgSection }
val sectionContent = extractContentFromChunks(nonSectionChunks)
if (sectionContent.isNotBlank()) {
// Append to existing content if this node already has content from child sections
val existingContent = nodeContents[targetNodeId] ?: ""
nodeContents[targetNodeId] = if (existingContent.isNotBlank()) {
"$existingContent $sectionContent"
} else {
sectionContent
}
}
}
// Process child sections from body
// Pass down the current nodeId if this section has one, otherwise pass the parent
val childParentNodeId = nodeId ?: parentNodeId
section.body.filterIsInstance<OrgSection>().forEach { childSection ->
val childResults = extractNodesFromSectionInternal(
childSection,
filePath,
currentPosition,
outlinePath,
childParentNodeId,
fileTags // Pass filetags to children for inheritance
)
nodes.addAll(childResults.nodes)
links.addAll(childResults.links)
tags.addAll(childResults.tags)
aliases.addAll(childResults.aliases)
refs.addAll(childResults.refs)
properties.addAll(childResults.properties)
// Merge child node contents (child content should be rolled up to parent if no ID)
childResults.nodeContents.forEach { (childNodeId, childContent) ->
val existing = nodeContents[childNodeId] ?: ""
nodeContents[childNodeId] = if (existing.isNotBlank()) {
"$existing $childContent"
} else {
childContent
}
}
currentPosition = childResults.lastPosition
}
return ExtractedNodes(nodes, links, tags, aliases, refs, properties, currentPosition, nodeContents)
}Links extracted from id-less sibling subsections roll up to the nearest ancestor node, so their OrgLink positions must stay unique โ otherwise the (pos, from_node) primary key of the links table silently drops all but the last of them on insert.
describe("Sibling section link positions") {
it("should not collide link positions across id-less sibling subsections") {
val content = """* Top
:PROPERTIES:
:ID: top-node-id
:END:
,** Sub A
- [[id:target-a][A target]]
,** Sub B
- [[id:target-b][B target]]
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
val idLinks = result.links.filter { it.type == "id" }
idLinks shouldHaveSize 2
idLinks.map { it.toNode }.toSet() shouldBe setOf("target-a", "target-b")
idLinks.map { it.position }.distinct() shouldHaveSize 2
idLinks.forEach { it.fromNode shouldBe "top-node-id" }
}
}
context("Property: Outline paths") {
test("node outline path length equals heading level") {
forAll(OrgContentGenerators.validId, Arb.int(1..5)) { id, level ->
val content = (1..level).joinToString("\n") { l ->
val heading = "${"*".repeat(l)} Level $l Heading"
if (l == level) {
"$heading\n:PROPERTIES:\n:ID: $id\n:END:\n\nContent at level $l"
} else {
"$heading\n\nContent at level $l"
}
}
val result = parser.parseFileContent("/test.org", content, timestamp)
if (result is ParseResult.Success) {
val node = result.nodes.find { it.id == id }
node != null && node.outlinePath.size == level
} else true
}
}
}
describe("Deep heading nesting") {
it("should handle 10 levels of nesting") {
val content = (1..10).joinToString("\n") { level ->
"${"*".repeat(level)} Level $level\n:PROPERTIES:\n:ID: level-$level-id\n:END:\n\nContent at level $level"
}
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes shouldHaveSize 10
}
it("should handle complex hierarchy with mixed levels") {
val content = """* Project
:PROPERTIES:
:ID: proj-id
:END:
Project content.
,** Phase 1
:PROPERTIES:
:ID: phase1-id
:END:
Phase 1 content.
,*** Task 1.1
:PROPERTIES:
:ID: task-1-1-id
:END:
Task 1.1 content.
,*** Task 1.2
:PROPERTIES:
:ID: task-1-2-id
:END:
Task 1.2 content.
,** Phase 2
:PROPERTIES:
:ID: phase2-id
:END:
Phase 2 content.
,*** Task 2.1
:PROPERTIES:
:ID: task-2-1-id
:END:
Task 2.1 content.
,* Notes
:PROPERTIES:
:ID: notes-id
:END:
Notes content.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes shouldHaveSize 7
// Verify outline paths
result.nodes.find { it.id == "task-1-1-id" }?.outlinePath shouldBe
listOf("Project", "Phase 1", "Task 1.1")
result.nodes.find { it.id == "task-2-1-id" }?.outlinePath shouldBe
listOf("Project", "Phase 2", "Task 2.1")
result.nodes.find { it.id == "notes-id" }?.outlinePath shouldBe
listOf("Notes")
}
}Link Extraction
There are two functions for recursively extracting links from org chunks and inline elements.
Both recursively walk inline elements (including formatted spans like Bold, Italic, etc.) to find nested OrgInlineElem.Link instances. id links extract the target node ID; file links and other types set toNode to null.
private fun extractLinksFromSection(section: OrgSection, fromNodeId: String, startPosition: Int): List<OrgLink> {
val links = mutableListOf<OrgLink>()
var position = startPosition
fun extractLinksFromInlineElems(elems: List<OrgInlineElem>) {
elems.forEach { elem ->
when (elem) {
is OrgInlineElem.Link -> {
val linkType = elem.type ?: "file"
val targetNodeId = when (linkType) {
"id" -> elem.target
"file" -> null
else -> null
}
val properties = mutableMapOf<String, String>()
elem.title?.let { title ->
properties["description"] = formatInlineElemsToPlaintext(title)
}
if (elem.target.isNotEmpty()) {
properties["target"] = elem.target
}
links.add(OrgLink(
position = position++,
fromNode = fromNodeId,
toNode = targetNodeId,
type = linkType,
properties = properties
))
}
is OrgInlineElem.Bold -> extractLinksFromInlineElems(elem.content)
is OrgInlineElem.Italic -> extractLinksFromInlineElems(elem.content)
is OrgInlineElem.Underline -> extractLinksFromInlineElems(elem.content)
is OrgInlineElem.StrikeThrough -> extractLinksFromInlineElems(elem.content)
is OrgInlineElem.Verbatim -> extractLinksFromInlineElems(elem.content)
is OrgInlineElem.Code -> extractLinksFromInlineElems(elem.content)
else -> { }
}
}
}
fun extractLinksFromChunk(chunk: OrgChunk) {
when (chunk) {
is OrgSection -> { }
is OrgChunk.OrgParagraph -> extractLinksFromInlineElems(chunk.items)
is OrgBlock.OrgQuoteBlock -> chunk.body.forEach { extractLinksFromChunk(it) }
is OrgBlock.OrgPageIntroBlock -> chunk.body.forEach { extractLinksFromChunk(it) }
is OrgBlock.OrgEditsBlock -> chunk.body.forEach { extractLinksFromChunk(it) }
is OrgBlock.OrgAsideBlock -> chunk.body.forEach { extractLinksFromChunk(it) }
is OrgList.OrgUnorderedList -> chunk.items.forEach { item -> item.content.forEach { extractLinksFromChunk(it) } }
is OrgList.OrgOrderedList -> chunk.items.forEach { item -> item.content.forEach { extractLinksFromChunk(it) } }
else -> { }
}
}
extractLinksFromInlineElems(section.heading.title.items)
section.body.forEach { chunk -> extractLinksFromChunk(chunk) }
return links
}
private fun extractLinksFromChunks(chunks: List<OrgChunk>, fromNodeId: String, startPosition: Int): List<OrgLink> {
val links = mutableListOf<OrgLink>()
var position = startPosition
fun extractLinksFromInlineElems(elems: List<OrgInlineElem>) {
elems.forEach { elem ->
when (elem) {
is OrgInlineElem.Link -> {
val linkType = elem.type ?: "file"
val targetNodeId = when (linkType) {
"id" -> elem.target
"file" -> null
else -> null
}
val properties = mutableMapOf<String, String>()
elem.title?.let { title ->
properties["description"] = formatInlineElemsToPlaintext(title)
}
if (elem.target.isNotEmpty()) {
properties["target"] = elem.target
}
links.add(OrgLink(
position = position++,
fromNode = fromNodeId,
toNode = targetNodeId,
type = linkType,
properties = properties
))
}
is OrgInlineElem.Bold -> extractLinksFromInlineElems(elem.content)
is OrgInlineElem.Italic -> extractLinksFromInlineElems(elem.content)
is OrgInlineElem.Underline -> extractLinksFromInlineElems(elem.content)
is OrgInlineElem.StrikeThrough -> extractLinksFromInlineElems(elem.content)
is OrgInlineElem.Verbatim -> extractLinksFromInlineElems(elem.content)
is OrgInlineElem.Code -> extractLinksFromInlineElems(elem.content)
else -> { }
}
}
}
fun extractLinksFromChunk(chunk: OrgChunk) {
when (chunk) {
is OrgSection -> { }
is OrgChunk.OrgParagraph -> extractLinksFromInlineElems(chunk.items)
is OrgBlock.OrgQuoteBlock -> chunk.body.forEach { extractLinksFromChunk(it) }
is OrgBlock.OrgPageIntroBlock -> chunk.body.forEach { extractLinksFromChunk(it) }
is OrgBlock.OrgEditsBlock -> chunk.body.forEach { extractLinksFromChunk(it) }
is OrgBlock.OrgAsideBlock -> chunk.body.forEach { extractLinksFromChunk(it) }
is OrgList.OrgUnorderedList -> chunk.items.forEach { item -> item.content.forEach { extractLinksFromChunk(it) } }
is OrgList.OrgOrderedList -> chunk.items.forEach { item -> item.content.forEach { extractLinksFromChunk(it) } }
else -> { }
}
}
chunks.forEach { extractLinksFromChunk(it) }
return links
}context("Property: Link extraction preserves invariants") {
test("ID links always extract the target node ID") {
forAll(OrgContentGenerators.validId, OrgContentGenerators.validId) { fromId, toId ->
val content = """
,* Source Node
:PROPERTIES:
:ID: $fromId
:END:
Link: [[id:$toId][Target]]
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.links.any { it.type == "id" && it.toNode == toId }
}
}
test("file links never have a toNode") {
forAll(OrgContentGenerators.validId, Arb.string(3..20, Codepoint.alphanumeric())) { nodeId, filename ->
val content = """
,* Node
:PROPERTIES:
:ID: $nodeId
:END:
[[file:$filename.org][Link]]
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.links.filter { it.type == "file" }.all { it.toNode == null }
}
}
}describe("Link parsing") {
it("should extract ID links from org content") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
"/test.org",
SampleOrgFiles.TAGS_AND_LINKS_ORG_CONTENT,
TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
// Should find links between nodes
result.links.shouldNotBeNull()
// Look for the ID link from main-topic-id to other-topic-id
val idLinks = result.links.filter { it.type == "id" }
idLinks shouldHaveSize 2 // One from main to other, one back reference
val mainToOther = idLinks.find {
it.fromNode == "main-topic-id" && it.toNode == "other-topic-id"
}
mainToOther.shouldNotBeNull()
mainToOther.properties["description"] shouldBe "Other Topic"
}
it("should extract external links") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
"/test.org",
SampleOrgFiles.COMPLEX_ORG_CONTENT,
TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
// Should find external HTTPS links
val httpsLinks = result.links.filter { it.type == "https" }
httpsLinks.shouldHaveSize(1)
val exampleLink = httpsLinks.first()
exampleLink.toNode.shouldBe(null) // External links don't have target nodes
exampleLink.properties["target"] shouldBe "//example.com"
exampleLink.properties["description"] shouldBe "Example Website"
}
it("should extract file links") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
"/test.org",
SampleOrgFiles.TAGS_AND_LINKS_ORG_CONTENT,
TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
val fileLinks = result.links.filter { it.type == "file" }
fileLinks.shouldHaveSize(1)
val fileLink = fileLinks.first()
fileLink.toNode.shouldBe(null) // File links don't have target nodes
fileLink.properties["target"] shouldBe "other.org"
fileLink.properties["description"] shouldBe "Other File"
}
it("should handle multiple link types in same content") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
"/test.org",
SampleOrgFiles.TAGS_AND_LINKS_ORG_CONTENT,
TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
// Should extract all different link types
val linkTypes = result.links.map { it.type }.toSet()
linkTypes shouldBe setOf("id", "https", "file")
// Should have correct total count
result.links shouldHaveSize 4 // 2 id links, 1 https, 1 file
}
}describe("Link edge cases") {
it("should handle links with special characters in description") {
val content = """
,* Links
:PROPERTIES:
:ID: link-special-id
:END:
[[id:target-id][Link with <special> & "characters"]]
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should handle links inside bold text") {
val content = """
,* Bold Links
:PROPERTIES:
:ID: bold-link-id
:END:
,*Bold text with [[id:target-id][link inside]]*
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
// Links inside bold should still be extracted
val idLinks = result.links.filter { it.type == "id" }
idLinks shouldHaveSize 1
}
it("should handle links inside italic text") {
val content = """
,* Italic Links
:PROPERTIES:
:ID: italic-link-id
:END:
/Italic with [[id:target-id][link]]/
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.links.filter { it.type == "id" } shouldHaveSize 1
}
it("should handle multiple links in one paragraph") {
val content = """
,* Multi Links
:PROPERTIES:
:ID: multi-link-id
:END:
See [[id:target-1][First]] and [[id:target-2][Second]] and [[https://example.com][External]].
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.links shouldHaveSize 3
}
}describe("Links across nesting levels") {
it("should extract links from deeply nested sections") {
val content = """
,* Parent
:PROPERTIES:
:ID: parent-id
:END:
,** Child
:PROPERTIES:
:ID: child-id
:END:
,*** Deep Child
:PROPERTIES:
:ID: deep-child-id
:END:
See [[id:parent-id][parent]] for context.
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
val deepLinks = result.links.filter { it.fromNode == "deep-child-id" }
deepLinks shouldHaveSize 1
deepLinks.first().toNode shouldBe "parent-id"
}
}Text Content Extraction for FTS
extractContentFromChunks walks all OrgChunk types that carry displayable content and produces a plain-text string for FTS indexing. Covers paragraphs, lists, quote blocks, source blocks, and example blocks.
private fun extractContentFromChunks(chunks: List<OrgChunk>): String {
val content = StringBuilder()
fun extractTextFromInlineElems(elems: List<OrgInlineElem>): String {
return elems.joinToString("") { elem ->
when (elem) {
is OrgInlineElem.Text -> elem.text
is OrgInlineElem.Bold -> extractTextFromInlineElems(elem.content)
is OrgInlineElem.Italic -> extractTextFromInlineElems(elem.content)
is OrgInlineElem.Underline -> extractTextFromInlineElems(elem.content)
is OrgInlineElem.StrikeThrough -> extractTextFromInlineElems(elem.content)
is OrgInlineElem.Verbatim -> extractTextFromInlineElems(elem.content)
is OrgInlineElem.Code -> extractTextFromInlineElems(elem.content)
is OrgInlineElem.Link -> elem.title?.let { formatInlineElemsToPlaintext(it) } ?: elem.target
else -> ""
}
}
}
fun extractFromChunk(chunk: OrgChunk) {
when (chunk) {
is OrgChunk.OrgParagraph -> {
val text = extractTextFromInlineElems(chunk.items)
if (text.isNotBlank()) {
content.append(text).append(" ")
}
}
is OrgList.OrgUnorderedList -> {
chunk.items.forEach { item -> item.content.forEach { extractFromChunk(it) } }
}
is OrgList.OrgOrderedList -> {
chunk.items.forEach { item -> item.content.forEach { extractFromChunk(it) } }
}
is OrgBlock.OrgQuoteBlock -> chunk.body.forEach { extractFromChunk(it) }
is OrgBlock.OrgSourceBlock -> {
if (chunk.body.isNotBlank()) content.append(chunk.body).append(" ")
}
is OrgBlock.OrgExampleBlock -> {
if (chunk.text.isNotBlank()) content.append(chunk.text).append(" ")
}
is OrgBlock.OrgPageIntroBlock -> chunk.body.forEach { extractFromChunk(it) }
is OrgBlock.OrgEditsBlock -> chunk.body.forEach { extractFromChunk(it) }
is OrgBlock.OrgAsideBlock -> chunk.body.forEach { extractFromChunk(it) }
is OrgSection -> { }
else -> { }
}
}
chunks.forEach { extractFromChunk(it) }
return content.toString().trim()
}
describe("Tag attribution against the file-level node") {
it("should not roll up ID-less child heading tags to the file-level node") {
val content = """
:PROPERTIES:
:ID: file-id
:END:
#+FILETAGS: :Project:
#+TITLE: File Level
Some file content.
,* Parent With ID
:PROPERTIES:
:ID: parent-id
:END:
Parent content.
,** To-do :noexport:
Task list content.
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
val fileTags = result.tags.filter { it.nodeId == "file-id" }.map { it.tag }
fileTags.shouldContainExactly("Project")
val parentTags = result.tags.filter { it.nodeId == "parent-id" }.map { it.tag }
parentTags.shouldContainExactly("Project")
// The noexport tag is attached to no node at all: the heading has no ID
// and tags never roll up to an ancestor.
result.tags.none { it.tag == "noexport" } shouldBe true
}
}
describe("Nested list handling") {
it("should parse headings containing deeply nested lists") {
val content = """
* Task List
:PROPERTIES:
:ID: task-list-id
:END:
- Item 1
- Sub-item 1.1
- Sub-sub-item 1.1.1
- Item 2
1. Numbered sub-item
2. Another numbered
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes shouldHaveSize 1
}
it("should extract content from nested lists for FTS") {
val content = """
* Shopping
:PROPERTIES:
:ID: shopping-id
:END:
- Groceries
- Apples
- Bananas
- Hardware
- Screws
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
val nodeContent = result.nodeContents["shopping-id"]
nodeContent?.contains("Apples") shouldBe true
nodeContent?.contains("Bananas") shouldBe true
}
}})
describe("Source block handling") {
it("should parse headings with source blocks") {
val content = """
* Code Section
:PROPERTIES:
:ID: code-section-id
:END:
#+BEGIN_SRC python
def hello():
print("Hello, World!")
#+END_SRC
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should parse multiple source blocks with different languages") {
val content = """
* Multi Language
:PROPERTIES:
:ID: multi-lang-id
:END:
#+BEGIN_SRC kotlin
fun main() = println("Kotlin")
#+END_SRC
#+BEGIN_SRC python
print("Python")
#+END_SRC
#+BEGIN_SRC sql
SELECT * FROM nodes;
#+END_SRC
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
}Drawer Extraction
Three functions that collect LOGBOOK and REVIEW_DATA drawer chunks from a section or a list of chunks, grouped by drawer name. Recursively walks into wrapper chunks (quote blocks, lists) to find nested drawers.
private fun extractDrawersFromSection(section: OrgSection): Map<String, List<OrgChunk>> {
val drawers = mutableMapOf<String, MutableList<OrgChunk>>()
section.body.forEach { chunk -> collectDrawerChunks(chunk, drawers) }
return drawers
}
private fun extractDrawersFromChunks(chunks: List<OrgChunk>): Map<String, List<OrgChunk>> {
val drawers = mutableMapOf<String, MutableList<OrgChunk>>()
chunks.forEach { chunk -> collectDrawerChunks(chunk, drawers) }
return drawers
}
private fun collectDrawerChunks(chunk: OrgChunk, drawers: MutableMap<String, MutableList<OrgChunk>>) {
when (chunk) {
is OrgChunk.OrgLogbookDrawer -> drawers.getOrPut("LOGBOOK") { mutableListOf() }.add(chunk)
is OrgChunk.OrgReviewDataDrawer -> drawers.getOrPut("REVIEW_DATA") { mutableListOf() }.add(chunk)
is OrgSection -> { }
is OrgBlock.OrgQuoteBlock -> chunk.body.forEach { collectDrawerChunks(it, drawers) }
is OrgBlock.OrgPageIntroBlock -> chunk.body.forEach { collectDrawerChunks(it, drawers) }
is OrgBlock.OrgEditsBlock -> chunk.body.forEach { collectDrawerChunks(it, drawers) }
is OrgBlock.OrgAsideBlock -> chunk.body.forEach { collectDrawerChunks(it, drawers) }
is OrgList.OrgUnorderedList -> chunk.items.forEach { item -> item.content.forEach { collectDrawerChunks(it, drawers) } }
is OrgList.OrgOrderedList -> chunk.items.forEach { item -> item.content.forEach { collectDrawerChunks(it, drawers) } }
else -> { }
}
}Property Drawer, Encoding & Timestamp Edge Cases
A bucket for OrgFileParserEdgeCaseTest describe blocks that exercise cross-cutting robustness rather than a single parser method: encoding quirks (BOM, CRLF, CJK/Arabic/emoji), property-drawer malformations, planning-timestamp variants, and content-only files with no headings. =OrgFileParserTest='s "File metadata" describe โ which checks path accessTime modificationTime / title plumbing โ lives here too, and the OrgFileParserTest closing }) lands here (its last describe in document order).
describe("Encoding edge cases") {
it("should handle UTF-8 BOM") {
val bom = "\uFEFF"
val content = bom + """
,* Test Heading
:PROPERTIES:
:ID: bom-test-id
:END:
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should handle CRLF line endings") {
val content = "* Heading\r\n:PROPERTIES:\r\n:ID: crlf-id\r\n:END:\r\n\r\nContent"
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should handle mixed CRLF and LF line endings") {
val content = "* Heading\r\n:PROPERTIES:\n:ID: mixed-id\r\n:END:\n\nContent"
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should handle CJK characters in headings") {
val content = """
,* ไธญๆๆ ้ข
:PROPERTIES:
:ID: cjk-heading-id
:END:
ไธญๆๅ
ๅฎนๅๆฅๆฌ่ชใใญในใ
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes shouldHaveSize 1
}
it("should handle Arabic text") {
val content = """
,* ุนููุงู ุนุฑุจู
:PROPERTIES:
:ID: arabic-heading-id
:END:
ู
ุญุชูู ุนุฑุจู
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should handle emoji-heavy content") {
val content = """
,* ๐ Emoji Heading ๐
:PROPERTIES:
:ID: emoji-heading-id
:END:
Content with ๐ emojis ๐ everywhere ๐ป
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
}
describe("Property drawer edge cases") {
it("should handle property values with special characters") {
val content = """* Test
:PROPERTIES:
:ID: special-props-id
:CUSTOM: Value with quotes and apostrophes
:URL: https://example.com/path
:END:
Some content here.
"""
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes shouldHaveSize 1
}
it("should handle missing END in properties drawer") {
val content = """
* Broken Properties
:PROPERTIES:
:ID: broken-id
:MISSING_END_TAG: value
* Another Heading
:PROPERTIES:
:ID: another-id
:END:
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
// Should handle gracefully - either parse what it can or error
(result is ParseResult.Success || result is ParseResult.ParseError) shouldBe true
}
it("should handle empty property values") {
val content = """
* Test
:PROPERTIES:
:ID: empty-val-id
:EMPTY_PROP:
:END:
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should handle many properties (14+)") {
val props = (1..20).joinToString("\n") { ":PROP_$it: value_$it" }
val content = """
* Many Props
:PROPERTIES:
:ID: many-props-id
$props
:END:
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
}
describe("Timestamp edge cases") {
it("should handle timestamps with repeaters") {
val content = """
* Recurring Task
:PROPERTIES:
:ID: recurring-id
:END:
SCHEDULED: <2024-01-01 Mon +1w>
DEADLINE: <2024-02-01 Thu .+1m>
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should handle inactive timestamps in body") {
val content = """
* Note
:PROPERTIES:
:ID: inactive-ts-id
:END:
Created: [2024-01-01 Mon 10:30]
Updated: [2024-06-15 Sat]
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should handle CLOSED timestamp") {
val content = """
* DONE Closed Task
:PROPERTIES:
:ID: closed-id
:END:
CLOSED: [2024-01-10 Wed 15:30]
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
}
describe("Content-only files (no headings)") {
it("should handle a file with only preamble content") {
val content = """
#+TITLE: Just a Title
#+AUTHOR: Test
This file has no headings, just preamble content.
Another paragraph.
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes shouldHaveSize 0
}
it("should handle file-level ID with no headings") {
val content = """
:PROPERTIES:
:ID: file-only-id
:END:
#+TITLE: File-Level Only
Content belongs to the file-level node.
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes shouldHaveSize 1
result.nodes.first().level shouldBe 0
}
}})
describe("File metadata") {
it("should set correct file metadata") {
val parser = OrgFileParser()
val filePath = "/specific/path/to/file.org"
val timestamp = TestData.sampleTimestamp
val result = parser.parseFileContent(
filePath = filePath,
content = SampleOrgFiles.SIMPLE_ORG_CONTENT,
lastModified = timestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
result.file.path shouldBe filePath
result.file.accessTime shouldBe timestamp
result.file.modificationTime shouldBe timestamp
}
it("should extract file title when available") {
val parser = OrgFileParser()
val result = parser.parseFileContent(
filePath = "/test/titled.org",
content = SampleOrgFiles.COMPLEX_ORG_CONTENT, // Contains #+TITLE:
lastModified = TestData.sampleTimestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
// Currently stubbed, but when implemented should extract title
// result.file.title shouldBe "Complex Org File"
}
}})Table Parsing
Org-mode table parsing: simple tables, tables without separator rows, Unicode tables, formula rows (#+TBLFM:), multi-table fixtures, tables adjacent to other content, single-column tables, and wide tables. Self-contained DescribeSpec โ prelude, single describe, and end all live here.
package computer.whatthefuck.arcology.parser
import computer.whatthefuck.arcology.fixtures.SampleOrgFiles
import computer.whatthefuck.arcology.fixtures.TestData
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.collections.shouldHaveSize
import io.kotest.matchers.shouldBe
import io.kotest.matchers.types.shouldBeInstanceOf
class OrgFileParserTableTest : DescribeSpec({
val parser = OrgFileParser()
val timestamp = TestData.sampleTimestamp
describe("Table parsing") {
it("should parse simple table without error") {
val content = """
* Heading with Table
:PROPERTIES:
:ID: table-test-id
:END:
| Col1 | Col2 | Col3 |
|------+------+------|
| A | B | C |
| 1 | 2 | 3 |
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes shouldHaveSize 1
}
it("should parse table with no separator row") {
val content = """
* No Separator
:PROPERTIES:
:ID: no-sep-table-id
:END:
| A | B | C |
| 1 | 2 | 3 |
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should parse table with Unicode characters") {
val content = """
* Unicode Table
:PROPERTIES:
:ID: unicode-table-id
:END:
| Name | Symbol |
|--------+--------|
| Omega | ฮฉ |
| Alpha | ฮฑ |
| Check | โ |
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should parse table with formula row") {
val content = """
* Spreadsheet
:PROPERTIES:
:ID: spreadsheet-id
:END:
| Item | Price |
|-------+-------|
| Apple | 1.00 |
| Total | 1.00 |
#+TBLFM: @3${'$'}2=vsum(@2${'$'}2..@2${'$'}2)
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should parse fixture with multiple tables") {
val result = parser.parseFileContent(
"/test.org",
SampleOrgFiles.TABLE_HEAVY_ORG_CONTENT,
timestamp
)
result.shouldBeInstanceOf<ParseResult.Success>()
// Should have nodes for each table section
result.nodes.size shouldBe 4 // parent + 3 sub-headings
}
it("should parse table adjacent to other content") {
val content = """
* Mixed Content
:PROPERTIES:
:ID: mixed-content-id
:END:
Some text before the table.
| A | B |
|---+---|
| 1 | 2 |
Some text after the table.
- List item after table
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should parse single-column table") {
val content = """
* Single Column
:PROPERTIES:
:ID: single-col-id
:END:
| Items |
|---------|
| Apple |
| Banana |
| Cherry |
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
it("should parse wide table with many columns") {
val header = (1..10).joinToString(" | ") { "Col$it" }
val sep = (1..10).joinToString("+") { "------" }
val row = (1..10).joinToString(" | ") { "val$it" }
val content = """
* Wide Table
:PROPERTIES:
:ID: wide-table-id
:END:
| $header |
|$sep|
| $row |
""".trimIndent()
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
}
}})Performance & Stability
The parser must never crash on arbitrary input. The property-based "never crashes" context lands here, along with the dedicated benchmark suite. The OrgFileParserPropertyTest closing }) lands here too โ its last context in document order is "Property: Parser never crashes".
context("Property: Parser never crashes") {
test("on arbitrary heading content") {
forAll(OrgContentGenerators.orgHeading) { heading ->
val content = heading.render()
val result = parser.parseFileContent("/test.org", content, timestamp)
result is ParseResult.Success || result is ParseResult.ParseError
}
}
test("on arbitrary string input (fuzzing)") {
forAll(100, Arb.string(0..500)) { content ->
val result = try {
parser.parseFileContent("/test.org", content, timestamp)
} catch (_: OutOfMemoryError) {
null
} catch (_: StackOverflowError) {
null
}
// Should not OOM or stack overflow; either parse or error gracefully
result != null
}
}
test("on generated org documents") {
forAll(20, OrgContentGenerators.orgDocument(5)) { content ->
val result = parser.parseFileContent("/test.org", content, timestamp)
result is ParseResult.Success || result is ParseResult.ParseError
}
}
test("on deeply nested headings up to 20 levels") {
forAll(Arb.int(1..20)) { depth ->
val content = (1..depth).joinToString("\n") { level ->
"${"*".repeat(level)} Heading at level $level\n:PROPERTIES:\n:ID: level-$level-id\n:END:\n\nContent at level $level"
}
val result = parser.parseFileContent("/test.org", content, timestamp)
result is ParseResult.Success
}
}
test("on headings with random property drawers") {
forAll(
OrgContentGenerators.headingTitle,
Arb.list(
Arb.pair(OrgContentGenerators.propertyKey, OrgContentGenerators.propertyValue),
0..10
)
) { title, properties ->
val propsBlock = if (properties.isNotEmpty()) {
val lines = properties.joinToString("\n") { (k, v) -> ":$k: $v" }
"\n:PROPERTIES:\n$lines\n:END:"
} else ""
val content = "* $title$propsBlock"
val result = parser.parseFileContent("/test.org", content, timestamp)
result is ParseResult.Success || result is ParseResult.ParseError
}
}
}})package computer.whatthefuck.arcology.parser
import computer.whatthefuck.arcology.fixtures.SampleOrgFiles
import computer.whatthefuck.arcology.fixtures.TestData
import io.kotest.core.spec.style.DescribeSpec
import io.kotest.matchers.collections.shouldHaveSize
import io.kotest.matchers.comparables.shouldBeLessThan
import io.kotest.matchers.shouldBe
import io.kotest.matchers.types.shouldBeInstanceOf
import kotlin.time.Duration.Companion.seconds
import kotlin.time.measureTime
class OrgFileParserPerformanceTest : DescribeSpec({
val parser = OrgFileParser()
val timestamp = TestData.sampleTimestamp
describe("Performance benchmarks") {
it("should parse 500-node stress test file in under 10 seconds") {
val content = SampleOrgFiles.createStressTestOrgContent(nodeCount = 500)
val duration = measureTime {
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
// All 500 heading nodes + 1 file-level node
result.nodes.size shouldBe 501
}
println("500-node parse time: $duration")
duration shouldBeLessThan 10.seconds
}
it("should parse 1000-node large content in under 15 seconds") {
val content = SampleOrgFiles.createLargeOrgContent(1000)
val duration = measureTime {
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.nodes shouldHaveSize 1000
}
println("1000-node parse time: $duration")
duration shouldBeLessThan 15.seconds
}
it("should handle files with many links efficiently") {
val content = SampleOrgFiles.createStressTestOrgContent(
nodeCount = 200,
linksPerNode = 10
)
val duration = measureTime {
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
// Should have extracted many links
result.links.size shouldBe result.links.size // just verify it completes
}
println("200-node 10-links-each parse time: $duration")
duration shouldBeLessThan 15.seconds
}
it("should parse real-world complexity fixture quickly") {
val content = SampleOrgFiles.REAL_WORLD_COMPLEXITY_ORG_CONTENT
val duration = measureTime {
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
// Verify it extracted a reasonable number of nodes
result.nodes.size shouldBe 11 // file-level + 10 heading nodes with IDs
}
println("Real-world complexity parse time: $duration")
duration shouldBeLessThan 2.seconds
}
it("should produce consistent results on repeated parsing") {
val content = SampleOrgFiles.createStressTestOrgContent(nodeCount = 100)
val result1 = parser.parseFileContent("/test.org", content, timestamp)
val result2 = parser.parseFileContent("/test.org", content, timestamp)
result1.shouldBeInstanceOf<ParseResult.Success>()
result2.shouldBeInstanceOf<ParseResult.Success>()
result1.nodes.size shouldBe result2.nodes.size
result1.links.size shouldBe result2.links.size
result1.tags.size shouldBe result2.tags.size
result1.file.hash shouldBe result2.file.hash
}
}
describe("Stress test content correctness") {
it("should extract all node IDs from stress test") {
val nodeCount = 50
val content = SampleOrgFiles.createStressTestOrgContent(nodeCount = nodeCount)
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
// Check that all expected IDs are present
for (i in 1..nodeCount) {
val expectedId = "stress-node-$i-id"
result.nodes.any { it.id == expectedId } shouldBe true
}
}
it("should extract file-level node from stress test") {
val content = SampleOrgFiles.createStressTestOrgContent(nodeCount = 10)
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
val fileNode = result.nodes.find { it.id == "stress-test-file-id" }
fileNode?.level shouldBe 0
}
it("should extract tags from stress test nodes") {
val content = SampleOrgFiles.createStressTestOrgContent(nodeCount = 20)
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
// Each node in the stress test has 2 tags
result.tags.size shouldBe result.tags.size // verify non-empty
result.tags.isNotEmpty() shouldBe true
}
it("should extract links from stress test nodes") {
val content = SampleOrgFiles.createStressTestOrgContent(
nodeCount = 20,
linksPerNode = 2
)
val result = parser.parseFileContent("/test.org", content, timestamp)
result.shouldBeInstanceOf<ParseResult.Success>()
result.links.isNotEmpty() shouldBe true
result.links.all { it.type == "id" } shouldBe true
}
}})Result Types
Three types that live in the same file as OrgFileParser: ExtractedData (internal accumulator), ExtractedNodes (per-section accumulator), and ParseResult (sealed class with Success, FileNotFound, ParseError variants).
ParseResult.Success carries the full parsed OrgDocument (nullable โ null for the blank-content early-return path) rather than tearing it into separate sections and keywordLines fields. Consumers that previously read result.sections now read result.document?.content; consumers that read result.keywordLines now read result.document?.keywordLines. Exposing the whole OrgDocument also gives downstream consumers (notably the web publishing stack) direct access to document.preamble and document.preface, which were previously inaccessible from ParseResult.
data class ExtractedData(
val fileTitle: String?,
val nodes: List<OrgNode>,
val links: List<OrgLink>,
val tags: List<OrgTag>,
val aliases: List<OrgAlias>,
val refs: List<OrgRef>,
val fileProperties: List<FileProperty>,
val nodeProperties: List<NodeProperty>,
val nodeContents: Map<String, String> = emptyMap()
)
data class ExtractedNodes(
val nodes: List<OrgNode>,
val links: List<OrgLink>,
val tags: List<OrgTag>,
val aliases: List<OrgAlias>,
val refs: List<OrgRef>,
val properties: List<NodeProperty>,
val lastPosition: Int,
val nodeContents: Map<String, String> = emptyMap()
)
sealed class ParseResult {
data class Success(
val file: OrgFile,
val nodes: List<OrgNode>,
val links: List<OrgLink>,
val tags: List<OrgTag>,
val aliases: List<OrgAlias>,
val refs: List<OrgRef>,
val fileProperties: List<FileProperty>,
val nodeProperties: List<NodeProperty>,
val nodeContents: Map<String, String> = emptyMap(),
val document: xyz.lepisma.orgmode.OrgDocument? = null,
val content: String = ""
) : ParseResult()
data class FileNotFound(val path: String) : ParseResult()
data class ParseError(val path: String, val error: String) : ParseResult()
}Tangle Targets
Assembly blocks. The four tiny source files (HashUtils.kt, HashUtils.jvm.kt, HashUtils.android.kt, RoamPropertyParsing.kt) tangle inline from their own `:tangle` directives under The Parser Class, so they don't appear here. Each test file is assembled from its prelude, the concatenation of all its :noweb-ref =describe=/=context= blocks (in document order), and its end.
OrgFileParser.kt
<<parser-preamble>>
<<parser-entry-point>>
<<parser-document-extraction>>
<<parser-calculate-hash>>
<<parser-section-walking>>
<<parser-node-extraction>>
<<parser-tag-ref-extraction>>
<<parser-link-extraction>>
<<parser-content-extraction>>
<<parser-drawer-extraction>>
<<parser-closing>>
<<parser-models>>OrgFileParserTest.kt
<<parser-test-orgfileparser-prelude>>
<<parser-test-orgfileparser>>
<<parser-test-orgfileparser-end>>OrgFileParserPropertyTest.kt
<<parser-test-property-prelude>>
<<parser-test-property>>
<<parser-test-property-end>>OrgFileParserEdgeCaseTest.kt
<<parser-test-edgecase-prelude>>
<<parser-test-edgecase>>
<<parser-test-edgecase-end>>OrgFileParserNestingTest.kt
<<parser-test-nesting-prelude>>
<<parser-test-nesting>>
<<parser-test-nesting-end>>OrgFileParserTableTest.kt
<<parser-test-table-prelude>>
<<parser-test-table>>
<<parser-test-table-end>>OrgFileParserPerformanceTest.kt
<<parser-test-performance-prelude>>
<<parser-test-performance>>
<<parser-test-performance-end>>HashUtilsTest.kt
<<hash-test-prelude>>
<<hash-test>>
<<hash-test-end>>RoamPropertyParsingTest.kt
<<roam-pp-test-prelude>>
<<roam-pp-test>>
<<roam-pp-test-end>>Related Modules
Data Models and Persistence Layer โ the domain types the parser produces (OrgNode, OrgLink, etc.)
The Indexer Pipeline โ calls
OrgFileParser.parseFileContent()for every discovered fileThe Indexer Platform Layer โ the FlowFileIndexerFactory wires the parser into the indexer
orgmode-kmpโ the upstream parsing library providingOrgLexer,parseWithDetails, and AST types