The core library that powers the arcology tangle and arcology detangle commands. This is a standalone Kotlin module in commonMain with no platform-specific dependencies — it compiles for JVM, Android, and native targets. The engine reads org-mode files parsed by orgmode-kmp, extracts named source blocks, resolves noweb references, wraps output with delimiter comments, and provides the reverse operation (detangle) to map delimiter-bounded regions back into named blocks.

Introduction

The literate programming and reverse engineering workflows needs two directions: tangle (extract source blocks from org files to their compile targets) and detangle (reverse changes in source files back into org block bodies). The core library implements both as pure functions over strings — it doesn't touch the filesystem, leaving I/O to the CLI layer.

The tangle pipeline: parse org content → collect named blocks → filter to blocks with :tangle → resolve noweb references → wrap with delimiter comments → group by output path. The detangle pipeline: parse source content for delimiter comments → match regions to org blocks → classify as matched/created/stale → use fallback anchor extractors when no delimiters found.

Design Decisions

Pure string-in/string-out design.

The library accepts String inputs (org content, source content, file paths) and returns data classes. This keeps the library testable without filesystem mocks, portable across platforms (commonMain), and simple to reason about. File I/O lives in the CLI layer (TangleCommand, DetangleCommand).

Delimiter comments as the round-trip mechanism.

Each tangled block is wrapped with comments like // file:roam/indexer.org::myBlock at the start (using org-mode link syntax with double brackets) and // myBlock ends here at the end. The detangler can precisely extract block bodies by finding these delimiters — no fuzzy diffs, no context-sensitive parsing. The delimiter format uses org-mode link syntax so Emacs users can navigate from tangled sources back to the org file.

Iterative noweb resolution with cycle detection.

Noweb references (<<ref>>) can be nested (A references B, B references C). The resolver iterates up to 50 times, expanding one reference per iteration. A visited set tracks resolved names; if a ref appears in its own resolution chain, a CircularNowebRef warning is emitted and the cycle is broken by removing the reference.

Fallback anchor extraction for pre-tangle files.

Source files written before the tangle system existed don't have delimiter comments. For these, the detangler uses language-specific anchor extractors that find declaration-level patterns (Kotlin class/fun/val/var, Nix attribute sets, Python def/class, SQL CREATE/ALTER, Elisp defun/defvar). These are weak matches — the result has isFallback: true and requires developer review.

Block name resolution from multiple sources.

Blocks get their name from the first match in priority order: #+name: foo on the preceding line, :noweb-ref foo in the block's header args, or an auto-generated name from the heading's :ID: property or title with a block index suffix (e.g., heading-id/1). This mirrors org-babel's behavior.

Implementation

Models

These are the data classes passed between the library and the CLI.

TangleResult — output of a tangle operation

A map from resolved file path (repo-root-relative) to the concatenated content, plus any warnings encountered during noweb resolution.

kotlin#+name: tangle-models:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/TangleModels.kt

package computer.whatthefuck.arroyo

data class TangleResult(
    val files: Map<String, String>,
    val warnings: List<TangleWarning>,
    val executableFiles: Set<String> = emptySet()
)

sealed class TangleWarning {
    data class MissingNowebRef(val name: String, val inBlockName: String) : TangleWarning()
    data class CircularNowebRef(val cycle: List<String>) : TangleWarning()
    data class UnresolvedTableRef(val tableName: String, val inBlockName: String) : TangleWarning()
    data class EvalError(val blockName: String, val message: String) : TangleWarning()
    data class MissingEvalContext(val blockName: String, val language: String) : TangleWarning()
}

class TangleEvalException(val blockName: String, message: String, cause: Throwable) :
    Exception("Eval failed in '$blockName': $message", cause)

data class ModuleMeta(
    val modulePath: String,
    val file: String,
    val title: String?,
    val headingId: String?
)

data class EmacsModuleMeta(
    val file: String,
    val moduleFile: String,
    val title: String?,
    val headingId: String?
)

data class FlakeInput(
    val name: String,
    val url: String
)

data class FlakeModuleRef(
    val modulePath: String
)

data class AgeRecipient(
    val recipient: String,
    val file: String,
    val title: String?,
    val nodeId: String?,
    val role: String
)

ArroyoTangleContext — eval host interface

The interface that generator scripts interact with at tangle time. Defined in commonMain so the tangle engine can reference it without platform dependencies. The LuaJ implementation (ArroyoScriptHost) lives in jvmMain.

tableFor is the Sprint 2 bridge: it returns the table data for a named table in the currently-tangling org file. The other methods query the Arroyo database for CCE metadata.

evalBlock executes a Lua script with varBindings (table data keyed by variable name) and stringBindings (scalar values keyed by variable name) available as Lua globals, and returns the captured output string. Throws on eval failure — the caller wraps it in an EvalError warning and continues.

kotlin#+name: arroyo-tangle-context:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/ArroyoTangleContext.kt

package computer.whatthefuck.arroyo

interface ArroyoTangleContext {
    fun nixosModules(role: String? = null): List<String>
    fun nixosModulesMeta(role: String? = null): List<ModuleMeta>
    fun homeModules(role: String? = null): List<String>
    fun homeModulesMeta(role: String? = null): List<ModuleMeta>
    fun emacsSnippets(): List<String>
    fun emacsInit(): String
    fun emacsModulesMeta(): List<EmacsModuleMeta>
    fun epkgOverrides(): List<String>
    fun fileTitleFor(filePath: String): String?
    fun tableFor(name: String): List<List<String>>
    fun evalBlock(script: String, varBindings: Map<String, List<List<String>>>, stringBindings: Map<String, String>): String
    fun flakeInputs(orgDir: String): List<FlakeInput>
    fun nixosRoleModules(role: String, orgDir: String): List<FlakeModuleRef>
    fun homeRoleModules(role: String, orgDir: String): List<FlakeModuleRef>
    fun systemOverlays(role: String? = null): List<String>
    fun inputs(role: String? = null): List<String>
    fun outputs(): List<String>
    fun ageRecipients(role: String? = null): List<AgeRecipient>
}

DetangleResult — output of a detangle operation

Three lists: matched (existing org blocks with updated body content), created (new delimiter regions that need org blocks), and stale (org blocks with no matching source region). The isFallback flag signals that anchor extractors were used instead of delimiters.

kotlin#+name: detangle-models:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/DetangleModels.kt

package computer.whatthefuck.arroyo

data class DetangleResult(
    val matched: List<BlockUpdate>,
    val created: List<BlockToCreate>,
    val stale: List<BlockStale>,
    val isFallback: Boolean = false
)

data class BlockUpdate(
    val blockName: String,
    val newContent: String
)

data class BlockToCreate(
    val suggestedName: String,
    val content: String,
    val insertAfterName: String?
)

data class BlockStale(
    val blockName: String,
    val reason: String
)

Org Parse Utilities — collecting named blocks and tables

The bridge between orgmode-kmp's parsed AST and the tangle engine. collectNamedBlocks walks the document tree — headings, quotes, lists, asides, edits boxes, page intro blocks — and collects every OrgSourceBlock with a resolved name. Names come from #+name: keyword lines, :noweb-ref header args, or auto-generated from heading IDs.

collectNamedTables walks the same tree, collecting every OrgTable that is preceded by a #+NAME: keyword. Tables without names are ignored. The table is converted to List<List<String>> via toNestedList(), which calls plainText() on each cell to strip org link syntax and inline markup.

tangleCommentFor generates the language-specific delimiter comments. For example, Kotlin gets // file:org::name, Python gets # file:org::name, Emacs Lisp gets ;; file:org::name — all using org-mode link syntax wrapped in double brackets. The comment prefix is chosen by language mapping via commentStyleFor, which returns a CommentStyle(prefix, suffix) pair to support languages with non-symmetric comment delimiters like HTML (~~), CSS (~~), and Jinja (~~). commentPrefixFor remains as a thin wrapper for backward compatibility.

defaultShebangFor returns a sensible #! line for scripting languages (bash, python, ruby, etc.) so that :shebang t (or :shebang yes) on a block inserts the appropriate interpreter without requiring the user to specify the full path. Languages without an obvious default return null and require an explicit :shebang string.

parseVarHeaderArg splits an org-babel :var varname=tablename argument into its component parts. This is used by the tangle engine to bind table data to source block variables during noweb resolution.

kotlin#+name: org-parse-utils:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/OrgParseUtils.kt

package computer.whatthefuck.arroyo

import xyz.lepisma.orgmode.OrgBlock
import xyz.lepisma.orgmode.OrgChunk
import xyz.lepisma.orgmode.OrgDocument
import xyz.lepisma.orgmode.OrgInlineElem
import xyz.lepisma.orgmode.OrgList
import xyz.lepisma.orgmode.OrgSection
import xyz.lepisma.orgmode.lexer.Token
import xyz.lepisma.orgmode.plainText

data class NamedBlock(
    val name: String,
    val headingId: String?,
    val headingTitle: String?,
    val blockIndex: Int,
    val block: OrgBlock.OrgSourceBlock
)

data class NamedTable(
    val name: String,
    val rows: List<List<String>>
)

fun collectNamedBlocks(document: OrgDocument): List<NamedBlock> {
    val blocks = mutableListOf<NamedBlock>()
    addBlocksFromChunks(document.preface.body, headingId = null, headingTitle = null, headingHeaderArgs = emptyMap(), blocks)
    document.content.forEach { section ->
        collectBlocksFromSection(section, blocks)
    }
    return blocks
}

fun collectTangleBlocks(document: OrgDocument): List<NamedBlock> {
    return collectNamedBlocks(document).filter {
        it.block.headerArgs["tangle"] != null
    }
}

fun collectNamedTables(document: OrgDocument): List<NamedTable> {
    val tables = mutableListOf<NamedTable>()
    collectTablesFromChunks(document.preface.body, tables)
    document.content.forEach { section ->
        collectTablesFromSection(section, tables)
    }
    return tables
}

internal fun collectBlocksFromSection(section: OrgSection, blocks: MutableList<NamedBlock>) {
    val headingId = extractHeadingId(section)
    val headingTitle = section.heading.title.plainText().trim()
    val headingHeaderArgs = extractHeadingHeaderArgs(section)
    addBlocksFromChunks(section.body, headingId, headingTitle, headingHeaderArgs, blocks)
}

internal fun collectTablesFromSection(section: OrgSection, tables: MutableList<NamedTable>) {
    collectTablesFromChunks(section.body, tables)
}

internal fun collectTablesFromChunks(chunks: List<OrgChunk>, tables: MutableList<NamedTable>) {
    var pendingName: String? = null

    for (chunk in chunks) {
        when (chunk) {
            is OrgChunk.OrgKeywordLine -> {
                if (chunk.keyword == "NAME") {
                    pendingName = chunk.value
                }
            }
            is OrgChunk.OrgTable -> {
                if (pendingName != null) {
                    tables.add(NamedTable(name = pendingName!!, rows = chunk.toNestedList()))
                    pendingName = null
                }
            }
            is OrgBlock.OrgSourceBlock -> {
                pendingName = null
            }
            is OrgSection -> collectTablesFromSection(chunk, tables)
            is OrgBlock.OrgQuoteBlock -> collectTablesFromChunks(chunk.body, tables)
            is OrgBlock.OrgPageIntroBlock -> collectTablesFromChunks(chunk.body, tables)
            is OrgBlock.OrgEditsBlock -> collectTablesFromChunks(chunk.body, tables)
            is OrgBlock.OrgAsideBlock -> collectTablesFromChunks(chunk.body, tables)
            is OrgList.OrgUnorderedList -> {
                chunk.items.forEach { item ->
                    collectTablesFromChunks(item.content, tables)
                }
            }
            is OrgList.OrgOrderedList -> {
                chunk.items.forEach { item ->
                    collectTablesFromChunks(item.content, tables)
                }
            }
            else -> { }
        }
    }
}

fun OrgChunk.OrgTable.toNestedList(): List<List<String>> {
    val result = mutableListOf<List<String>>()
    for (subtable in subtables) {
        for (row in subtable) {
            result.add(row.cells.map { it.plainText().trim() })
        }
    }
    return result
}

internal fun parseVarHeaderArg(value: String): Pair<String, String>? {
    val eq = value.indexOf('=')
    if (eq < 0) return null
    return Pair(value.substring(0, eq).trim(), value.substring(eq + 1).trim())
}

internal fun isQuotedString(value: String): Boolean {
    return value.length >= 2 && value.startsWith("\"") && value.endsWith("\"")
}

internal fun unquote(value: String): String {
    return if (isQuotedString(value)) value.substring(1, value.length - 1) else value
}

data class TableRef(val fileName: String?, val tableName: String)

internal fun parseTableRef(rawName: String): TableRef {
    val colon = rawName.lastIndexOf(':')
    if (colon > 0 && rawName.substring(0, colon).contains(".")) {
        return TableRef(rawName.substring(0, colon), rawName.substring(colon + 1))
    }
    return TableRef(null, rawName)
}

internal fun extractHeadingId(section: OrgSection): String? {
    return section.heading.properties?.map?.get("ID")?.let { idLine ->
        idLine.items.filterIsInstance<OrgInlineElem.Text>()
            .joinToString("") { it.text }
            .trim()
            .takeIf { it.isNotEmpty() }
    }
}

internal fun addBlocksFromChunks(
    chunks: List<OrgChunk>,
    headingId: String?,
    headingTitle: String?,
    headingHeaderArgs: Map<String, String>,
    blocks: MutableList<NamedBlock>
) {
    var pendingName: String? = null
    var blockIndex = 0

    for (chunk in chunks) {
        when (chunk) {
            is OrgChunk.OrgKeywordLine -> {
                if (chunk.keyword == "NAME") {
                    pendingName = chunk.value
                }
            }
            is OrgBlock.OrgSourceBlock -> {
                val name = resolveBlockName(chunk, pendingName, headingId, headingTitle, ++blockIndex)
                val mergedBlock = chunk.mergeHeaderArgs(headingHeaderArgs)
                blocks.add(
                    NamedBlock(
                        name = name,
                        headingId = headingId,
                        headingTitle = headingTitle,
                        blockIndex = blockIndex,
                        block = mergedBlock
                    )
                )
                pendingName = null
            }
            is OrgSection -> {
                collectBlocksFromSection(chunk, blocks)
            }
            is OrgBlock.OrgQuoteBlock -> {
                addBlocksFromChunks(chunk.body, headingId, headingTitle, headingHeaderArgs, blocks)
            }
            is OrgBlock.OrgPageIntroBlock -> {
                addBlocksFromChunks(chunk.body, headingId, headingTitle, headingHeaderArgs, blocks)
            }
            is OrgBlock.OrgEditsBlock -> {
                addBlocksFromChunks(chunk.body, headingId, headingTitle, headingHeaderArgs, blocks)
            }
            is OrgBlock.OrgAsideBlock -> {
                addBlocksFromChunks(chunk.body, headingId, headingTitle, headingHeaderArgs, blocks)
            }
            is OrgList.OrgUnorderedList -> {
                chunk.items.forEach { item ->
                    addBlocksFromChunks(item.content, headingId, headingTitle, headingHeaderArgs, blocks)
                }
            }
            is OrgList.OrgOrderedList -> {
                chunk.items.forEach { item ->
                    addBlocksFromChunks(item.content, headingId, headingTitle, headingHeaderArgs, blocks)
                }
            }
            else -> { }
        }
    }
}

internal fun resolveBlockName(
    block: OrgBlock.OrgSourceBlock,
    pendingName: String?,
    headingId: String?,
    headingTitle: String?,
    blockIndex: Int
): String {
    if (pendingName != null) return pendingName
    block.name?.let { return it }
    val prefix = when {
        headingId != null -> headingId
        headingTitle != null -> headingTitle.replace("""\s+""".toRegex(), "-").lowercase()
        else -> "preamble"
    }
    return "$prefix/$blockIndex"
}

data class CommentStyle(val prefix: String, val suffix: String = "")

fun tangleCommentFor(language: String, orgFilePath: String, name: String, headingId: String?): Pair<String, String> {
    val style = commentStyleFor(language)
    val headingLink = if (headingId != null) {
        "${style.prefix} [[id:$headingId]]${style.suffix}\n"
    } else {
        ""
    }
    val start = "${style.prefix} [[file:$orgFilePath::$name][$name]]${style.suffix}\n$headingLink"
    val end = "${style.prefix} $name ends here${style.suffix}"
    return Pair(start, end)
}

fun commentPrefixFor(language: String): String = commentStyleFor(language).prefix

fun defaultShebangFor(language: String): String? {
    return when (language.lowercase()) {
        "bash", "sh", "shell", "zsh", "fish" -> "#!/usr/bin/env ${language.lowercase()}"
        "python", "py" -> "#!/usr/bin/env python3"
        "ruby", "rb" -> "#!/usr/bin/env ruby"
        "perl", "pl" -> "#!/usr/bin/env perl"
        "makefile", "make" -> "#!/usr/bin/make -f"
        else -> null
    }
}

fun commentStyleFor(language: String): CommentStyle {
    return when (language.lowercase()) {
        "kotlin", "java", "c", "cpp", "c++", "rust", "rs", "go", "javascript", "js",
        "typescript", "ts", "scala", "swift", "dart", "groovy", "kts" -> CommentStyle("//")
        "makefile", "make", "nix", "bash", "sh", "shell", "zsh", "fish", "python", "py",
        "ruby", "rb", "perl", "pl", "yaml", "yml", "toml", "ini", "dockerfile", "gitignore",
        "conf", "properties", "r", "terraform", "tf", "hcl", "jinja", "jinja2" -> CommentStyle("#")
        "emacs-lisp", "elisp", "lisp", "clojure", "clj", "scheme", "racket" -> CommentStyle(";;")
        "sql", "haskell", "hs", "ada", "gnat" -> CommentStyle("--")
        "html", "htm", "xml", "svg", "markdown", "md" -> CommentStyle("<!--", " -->")
        "css", "scss", "less", "sass" -> CommentStyle("/*", " */")
        "lua" -> CommentStyle("--")
        "vim", "viml" -> CommentStyle("\"")
        "jinja-html", "html+jinja", "htmldjango", "django-html" -> CommentStyle("{#", " #}")
        else -> CommentStyle("//")
    }
}

internal fun extractHeadingHeaderArgs(section: OrgSection): Map<String, String> {
    val headerArgsProp = section.heading.properties?.map?.get("header-args")
        ?: section.heading.properties?.map?.get("HEADER-ARGS")
    return if (headerArgsProp != null) {
        parseHeaderArgsPropertyValue(headerArgsProp.plainText())
    } else {
        emptyMap()
    }
}

internal fun parseHeaderArgsPropertyValue(value: String): Map<String, String> {
    val result = mutableMapOf<String, String>()
    val parts = value.split("\\s+".toRegex()).filter { it.isNotEmpty() }
    var i = 0
    while (i < parts.size) {
        val tok = parts[i]
        if (tok.startsWith(":") && tok.length > 1) {
            val key = tok.removePrefix(":")
            val v = if (i + 1 < parts.size && !parts[i + 1].startsWith(":") && !parts[i + 1].startsWith("-")) {
                parts[++i]
            } else {
                ""
            }
            result[key] = v
        }
        i++
    }
    return result
}

internal fun OrgBlock.OrgSourceBlock.mergeHeaderArgs(headingHeaderArgs: Map<String, String>): OrgBlock.OrgSourceBlock {
    if (headingHeaderArgs.isEmpty()) return this
    val merged = headingHeaderArgs.toMutableMap()
    for ((k, v) in this.headerArgs) {
        merged[k] = v
    }
    return OrgBlock.OrgSourceBlock(
        language = this.language,
        switches = this.switches,
        headerArgs = merged,
        body = this.body,
        name = this.name ?: merged["noweb-ref"],
        tokens = this.tokens
    )
}

OrgTangle — noweb resolution and file assembly

The main tangle engine. tangle() accepts org content and a file path, parses the org document, collects named blocks and named tables, resolves noweb references, wraps with delimiter comments, and groups results by output path. It returns a TangleResult with a map of file paths to content plus any warnings.

Noweb resolution (resolveNoweb) is the heart of the engine. It scans for <<ref>> and <<ref(arg)>> patterns and replaces them with the referenced block body. The regex captures optional argument parentheses: <<gen-pkg-overlays(rix-pkgs)>> extracts both the ref name and the argument. The argument is validated against the tableBlocks map — an unresolved table arg emits an UnresolvedTableRef warning but the block body is still expanded. The resolver recurses into replaced content to handle nested references. A visited set prevents infinite loops on circular references. Missing references emit a MissingNowebRef warning and remove the placeholder. The loop runs up to 50 iterations.

For blocks with :eval arroyo, the Lua script is evaluated during noweb resolution. When <<gen-pkg-overlays(rix-pkgs)>> references a block with :eval arroyo and :var tbl=python-pkgs, the engine passes the arg's table data (rix-pkgs) to the eval context instead of the default (python-pkgs). The evaluated output is embedded at the call site. If no evalContext is available, a MissingEvalContext warning is emitted and the raw block body is used.

:var values and noweb args that are quoted strings (e.g. :var role="endpoint" or <<gen("endpoint")>>) are passed as string bindings rather than resolved as table names. The isQuotedString / unquote utilities in OrgParseUtils detect and strip the surrounding double-quotes.

For blocks with both :tangle and :eval arroyo, evaluation happens in the main tangle loop before noweb resolution. The default :var binding is used, the result replaces the block body, and wrapping proceeds normally.

:var tbl=tableName header args on source blocks are parsed by parseVarHeaderArg into (varName, tableName) pairs and stored in a varBindings map keyed by block name. During parameterized noweb resolution, the arg overrides the default table name — the resolver validates the arg table instead of the default.

Delimiter-comment wrapping is controlled by the :comments header arg, following org-babel semantics. The default wraps each tangled block with start/end delimiters (~name~ / name ends here) in the language's comment syntax. :comments none (or :comments no) skips wrapping entirely — useful for config files, JSON, and markdown with YAML frontmatter where delimiter comments would corrupt the output. The :results header arg is not a tangle knob; it governs how eval output is embedded in the org buffer and is ignored by the tangle engine.

Shebang support

The :shebang header arg makes a tangled file executable. :shebang "/path/to/interpreter" prepends that line to the file content and marks the file for chmod +x in the CLI layer. :shebang t (or :shebang yes) uses defaultShebangFor(language) to pick a sensible interpreter for scripting languages (bash, python, ruby, etc.); languages with no obvious default emit no shebang and the file is not marked executable. When multiple blocks target the same file, only the first block's shebang is applied — the engine guards against duplicate #! lines.

The TangleResult carries an executableFiles: Set<String> so the CLI commands (TangleCommand, FloodCommand) can setExecutable(true, true) after writing, without re-parsing the file content.

Path resolution (resolveTanglePath) normalizes ../ segments from the org file's directory. If an org file at roam/indexer.org has :tangle ../src/File.kt, the result is src/File.kt (compute roam/ + ../src/File.kt, normalize).

Cross-file table references (:var tbl=other.org:tableName) are resolved by an optional fileReader callback parameter. When a table lookup fails in the local tableBlocks map, the engine parses the raw name with parseTableRef: if it contains a . followed by :, it splits into (fileName, tableName), calls fileReader(fileName) to get the external org content, parses it with OrgLexer, and collects named tables. External tables are cached so each file is parsed at most once per tangle session.

kotlin#+name: org-tangle:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/OrgTangle.kt

package computer.whatthefuck.arroyo

import xyz.lepisma.orgmode.OrgParseResult
import xyz.lepisma.orgmode.lexer.OrgLexer
import xyz.lepisma.orgmode.parseWithDetails

class OrgTangle {
    internal data class EvalBindings(
        val tableBindings: Map<String, List<List<String>>>,
        val stringBindings: Map<String, String>
    )

    fun tangle(orgContent: String, orgFilePath: String, evalContext: ArroyoTangleContext? = null, fileReader: ((String) -> String?)? = null): TangleResult {
        if (orgContent.isBlank()) {
            return TangleResult(files = emptyMap(), warnings = emptyList())
        }

        val tokens = try {
            OrgLexer(orgContent).tokenize()
        } catch (e: Exception) {
            return TangleResult(files = emptyMap(), warnings = emptyList())
        }

        val parseResult = parseWithDetails(tokens)
        if (parseResult !is OrgParseResult.Success) {
            return TangleResult(files = emptyMap(), warnings = emptyList())
        }

        val document = parseResult.document
        val allBlocks = collectNamedBlocks(document)
        val blockMap = allBlocks.associateBy { it.name }
        val tangleBlocks = allBlocks.filter {
            val t = it.block.headerArgs["tangle"]
            t != null && t != "no"
        }

        val nowebBlocks = allBlocks.groupBy { it.name }.mapValues { (_, bs) -> bs.joinToString("\n") { it.block.body } }
        val tableBlocks = collectNamedTables(document).associate { it.name to it.rows }
        val varBindings = allBlocks.mapNotNull { block ->
            val varArg = block.block.headerArgs["var"] ?: return@mapNotNull null
            val parsed = parseVarHeaderArg(varArg) ?: return@mapNotNull null
            block.name to parsed
        }.toMap()
        val warnings = mutableListOf<TangleWarning>()
        val fileContents = mutableMapOf<String, MutableList<String>>()
        val executableFiles = mutableSetOf<String>()

        val externalTableCache = mutableMapOf<String, Map<String, List<List<String>>>>()
        val resolveTable: (String) -> List<List<String>>? = { rawName ->
            val local = tableBlocks[rawName]
            if (local != null) {
                local
            } else {
                val ref = parseTableRef(rawName)
                if (ref.fileName == null) {
                    null
                } else {
                    val fileTables = externalTableCache.getOrPut(ref.fileName) {
                        val content = try {
                            fileReader?.invoke(ref.fileName)
                        } catch (e: Exception) { null }
                        if (content == null || content.isEmpty()) return@getOrPut emptyMap()
                        val fileTokens = try {
                            if (content.isEmpty()) emptyList()
                            else OrgLexer(content).tokenize()
                        } catch (e: Exception) { emptyList() }
                        if (fileTokens.isEmpty()) return@getOrPut emptyMap()
                        val fileParse = parseWithDetails(fileTokens)
                        if (fileParse is OrgParseResult.Success) {
                            collectNamedTables(fileParse.document).associate { it.name to it.rows }
                        } else emptyMap()
                    }
                    fileTables[ref.tableName]
                }
            }
        }

        for (namedBlock in tangleBlocks) {
            val block = namedBlock.block
            val eval = block.headerArgs["eval"]
            val body = if (eval == "arroyo") {
                evaluateBlock(namedBlock, evalContext, resolveTable, varBindings, warnings)
            } else {
                block.body
            }
            val noweb = block.headerArgs["noweb"]
            val shouldResolve = noweb != null && noweb != "no"
            val resolvedBody = if (shouldResolve) {
                resolveNoweb(body, nowebBlocks, blockMap, resolveTable, varBindings, evalContext, namedBlock.name, setOf(), warnings)
            } else {
                body
            }
            val uncommad = uncommaBody(resolvedBody)
            var wrapped = wrapWithComments(uncommad, namedBlock, orgFilePath)

            val tanglePath = block.headerArgs["tangle"] ?: continue
            val resolvedPath = resolveTanglePath(orgFilePath, tanglePath)

            val shebang = block.headerArgs["shebang"]
            if (shebang != null && shebang != "no") {
                val normalizedShebang = if (shebang == "t" || shebang == "yes") {
                    defaultShebangFor(namedBlock.block.language)
                } else {
                    unquote(shebang)
                }
                if (normalizedShebang != null && !fileContents[resolvedPath].orEmpty().any { it.startsWith("#!") }) {
                    wrapped = "$normalizedShebang\n$wrapped"
                    if (!executableFiles.contains(resolvedPath)) {
                        executableFiles.add(resolvedPath)
                    }
                }
            }

            fileContents.getOrPut(resolvedPath) { mutableListOf() }.add(wrapped)
        }

        val files = fileContents.mapValues { (_, parts) -> parts.joinToString("\n") }

        return TangleResult(files = files, warnings = warnings, executableFiles = executableFiles)
    }

    internal fun evaluateBlock(
        namedBlock: NamedBlock,
        evalContext: ArroyoTangleContext?,
        resolveTable: (String) -> List<List<String>>?,
        varBindings: Map<String, Pair<String, String>>,
        warnings: MutableList<TangleWarning>
    ): String {
        val varArg = varBindings[namedBlock.name]
        val bindings = if (varArg != null) {
            val (varName, rawValue) = varArg
            if (isQuotedString(rawValue)) {
                EvalBindings(emptyMap(), mapOf(varName to unquote(rawValue)))
            } else {
                val tableData = resolveTable(rawValue)
                if (tableData != null) {
                    EvalBindings(mapOf(varName to tableData), emptyMap())
                } else {
                    warnings.add(TangleWarning.UnresolvedTableRef(rawValue, namedBlock.name))
                    EvalBindings(emptyMap(), emptyMap())
                }
            }
        } else {
            EvalBindings(emptyMap(), emptyMap())
        }
        if (evalContext == null) {
            if (varArg != null && bindings.tableBindings.isEmpty() && bindings.stringBindings.isEmpty()) {
                // warning already emitted above
            }
            warnings.add(TangleWarning.MissingEvalContext(namedBlock.name, namedBlock.block.language))
            return namedBlock.block.body
        }
        return try {
            evalContext.evalBlock(namedBlock.block.body, bindings.tableBindings, bindings.stringBindings)
        } catch (e: Exception) {
            warnings.add(TangleWarning.EvalError(namedBlock.name, e.message ?: e.toString()))
            throw TangleEvalException(namedBlock.name, e.message ?: e.toString(), e)
        }
    }

    /**
     ,* Strip org-escape commas from source block body lines.
     ,* Lines starting with optional whitespace followed by ',' and then '*', '#', or ','
     ,* are un-escaped by removing the leading comma. This is the org-mode convention for
     ,* escaping lines that would otherwise be interpreted as structural syntax.
     ,*/
    internal fun uncommaBody(body: String): String {
        return body.lines().joinToString("\n") { line ->
            val trimmed = line.trimStart()
            if (trimmed.startsWith(",") && (trimmed.startsWith(",*") || trimmed.startsWith(",#") || trimmed.startsWith(",,"))) {
                val indent = line.substring(0, line.length - trimmed.length)
                indent + trimmed.removePrefix(",")
            } else {
                line
            }
        }
    }

    /**
     ,* When a noweb ref appears on a line with a prefix (e.g. "# <<ref>>"),
     ,* and the replacement is multi-line, propagate the prefix to all
     ,* subsequent lines so the indentation/comment structure is preserved.
     ,*/
    private fun applyLinePrefix(body: String, match: MatchResult, replacement: String): String {
        val matchStart = match.range.first
        val lineStart = body.lastIndexOf('\n', matchStart - 1) + 1
        val prefix = body.substring(lineStart, matchStart)
        if (prefix.isEmpty() || !replacement.contains('\n')) {
            return replacement
        }
        val lines = replacement.split("\n")
        return lines.mapIndexed { i, line -> if (i == 0) line else "$prefix$line" }.joinToString("\n")
    }

    internal fun resolveNoweb(
        body: String,
        nowebBlocks: Map<String, String>,
        blockMap: Map<String, NamedBlock>,
        resolveTable: (String) -> List<List<String>>?,
        varBindings: Map<String, Pair<String, String>>,
        evalContext: ArroyoTangleContext?,
        contextName: String,
        visited: Set<String>,
        warnings: MutableList<TangleWarning>
    ): String {
        val nowebPattern = Regex("<<([^(>]+)(?:\\(([^)]*)\\))?>>")
        var result = body
        var changed = true
        var iterations = 0
        val maxIterations = 50

        while (changed && iterations < maxIterations) {
            changed = false
            iterations++
            val match = nowebPattern.find(result)
            if (match != null) {
                val refName = match.groupValues[1]
                val arg = match.groupValues.getOrNull(2)?.takeIf { it.isNotEmpty() }
                if (refName in visited) {
                    warnings.add(TangleWarning.CircularNowebRef(visited.toList() + refName))
                    return result.replace(match.value, "")
                }

                if (arg != null && !isQuotedString(arg) && resolveTable(arg) == null) {
                    warnings.add(TangleWarning.UnresolvedTableRef(arg, contextName))
                }

                val replacement = nowebBlocks[refName]
                if (replacement != null) {
                    val refBlock = blockMap[refName]
                    val isEvalBlock = refBlock?.block?.headerArgs?.get("eval") == "arroyo"
                    val resolved = if (isEvalBlock && evalContext != null) {
                        val bindings = resolveVarBindingsForEval(refBlock!!, arg, resolveTable, varBindings, warnings)
                        evalBlockForRef(refBlock, bindings, evalContext, warnings)
                    } else if (isEvalBlock) {
                        warnings.add(TangleWarning.MissingEvalContext(refName, refBlock!!.block.language))
                        replacement
                    } else {
                        resolveNoweb(replacement, nowebBlocks, blockMap, resolveTable, varBindings, evalContext, refName, visited + refName, warnings)
                    }
                    val prefixed = applyLinePrefix(result, match, resolved)
                    result = result.replaceFirst(match.value, prefixed)
                    changed = true
                } else {
                    warnings.add(TangleWarning.MissingNowebRef(refName, contextName))
                    result = result.replaceFirst(match.value, "")
                    changed = true
                }
            }
        }

        return result
    }

    internal fun resolveVarBindingsForEval(
        refBlock: NamedBlock,
        arg: String?,
        resolveTable: (String) -> List<List<String>>?,
        varBindings: Map<String, Pair<String, String>>,
        warnings: MutableList<TangleWarning>
    ): EvalBindings {
        val defaultVar = varBindings[refBlock.name]
        val rawValue = arg ?: defaultVar?.second ?: return EvalBindings(emptyMap(), emptyMap())
        if (isQuotedString(rawValue)) {
            val varName = defaultVar?.first ?: return EvalBindings(emptyMap(), emptyMap())
            return EvalBindings(emptyMap(), mapOf(varName to unquote(rawValue)))
        }
        val tableData = resolveTable(rawValue)
        if (arg != null && tableData == null) {
            warnings.add(TangleWarning.UnresolvedTableRef(arg, refBlock.name))
            if (defaultVar != null) {
                val fallback = resolveTable(defaultVar.second)
                if (fallback != null) return EvalBindings(mapOf(defaultVar.first to fallback), emptyMap())
            }
            return EvalBindings(emptyMap(), emptyMap())
        }
        val varName = defaultVar?.first ?: return EvalBindings(emptyMap(), emptyMap())
        return if (tableData != null) EvalBindings(mapOf(varName to tableData), emptyMap()) else EvalBindings(emptyMap(), emptyMap())
    }

    internal fun evalBlockForRef(
        refBlock: NamedBlock,
        bindings: EvalBindings,
        evalContext: ArroyoTangleContext,
        warnings: MutableList<TangleWarning>
    ): String {
        return try {
            evalContext.evalBlock(refBlock.block.body, bindings.tableBindings, bindings.stringBindings)
        } catch (e: Exception) {
            warnings.add(TangleWarning.EvalError(refBlock.name, e.message ?: e.toString()))
            throw TangleEvalException(refBlock.name, e.message ?: e.toString(), e)
        }
    }

    internal fun wrapWithComments(body: String, namedBlock: NamedBlock, orgFilePath: String): String {
        val comments = namedBlock.block.headerArgs["comments"]
        if (comments == "none" || comments == "no") {
            return body
        }
        val language = namedBlock.block.language
        val (startComment, endComment) = tangleCommentFor(language, orgFilePath, namedBlock.name, namedBlock.headingId)
        return "$startComment\n$body\n$endComment"
    }

    internal fun resolveTanglePath(orgFilePath: String, tanglePath: String): String {
        val expanded = if (tanglePath.startsWith("~")) {
            System.getProperty("user.home") + tanglePath.removePrefix("~")
        } else {
            tanglePath
        }
        if (expanded.startsWith("/")) return expanded
        val orgDir = orgFilePath.substringBeforeLast('/', "").ifEmpty { "." }
        val combined = if (orgDir == ".") {
            expanded
        } else {
            "$orgDir/$expanded"
        }
        val segments = mutableListOf<String>()
        for (segment in combined.split('/')) {
            when (segment) {
                "", "." -> { }
                ".." -> if (segments.isNotEmpty()) segments.removeLast()
                else -> segments.add(segment)
            }
        }
        return segments.joinToString("/")
    }
}

OrgDetangle — reverse tangling

The reverse of OrgTangle. Given a source file's content, the org file's content, and the tangle target path, it matches delimiter-bounded regions in the source back to named blocks in the org file.

The primary mode (delimitedDetangle) extracts regions using regex patterns that match the start/end delimiter comments. Each region is paired with its corresponding org block by name. Unmatched regions become created suggestions; org blocks without regions become stale warnings.

The fallback mode (fallbackDetangle) activates when no delimiters are found. It runs all language-specific anchor extractors through the AnchorRegistry, collects STRONG anchors (declaration-level patterns), and slices the source content between anchors into suggested blocks. The preamble (content before the first anchor) becomes a separate BlockToCreate with insertAfterName = null.

kotlin#+name: org-detangle:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/OrgDetangle.kt

package computer.whatthefuck.arroyo

import xyz.lepisma.orgmode.OrgParseResult
import xyz.lepisma.orgmode.lexer.OrgLexer
import xyz.lepisma.orgmode.parseWithDetails

class OrgDetangle(
    private val anchorRegistry: AnchorRegistry = AnchorRegistry.default()
) {
    private val startDelimiterPattern = Regex(
        """^\s*(?://|#|--|;;|<!--|/\*|\{#)\s*\[\[file:([^:]+)::([^\]]+)\]\[([^\]]*)\]\]"""
    )
    private val endDelimiterPattern = Regex(
        """^\s*(?://|#|--|;;|<!--|/\*|\{#)\s*(\S+)\s+ends\s+here\s*(?:-->|\*/| #\})?\s*$"""
    )

    fun detangle(
        sourceContent: String,
        orgContent: String,
        tangleTarget: String
    ): DetangleResult {
        val tokens = try {
            OrgLexer(orgContent).tokenize()
        } catch (e: Exception) {
            return DetangleResult(emptyList(), emptyList(), emptyList())
        }

        val parseResult = parseWithDetails(tokens)
        if (parseResult !is OrgParseResult.Success) {
            return DetangleResult(emptyList(), emptyList(), emptyList())
        }

        val allBlocks = collectNamedBlocks(parseResult.document)
        val orgBlocks = allBlocks.filter { it.block.headerArgs["tangle"] == tangleTarget }

        val delimRegions = extractDelimiterRegions(sourceContent)

        if (delimRegions.isNotEmpty()) {
            return delimitedDetangle(delimRegions, orgBlocks, sourceContent)
        }

        return fallbackDetangle(sourceContent, orgBlocks)
    }

    private data class DelimiterRegion(
        val name: String,
        val content: String,
        val startIndex: Int,
        val endIndex: Int
    )

    private fun extractDelimiterRegions(source: String): List<DelimiterRegion> {
        val regions = mutableListOf<DelimiterRegion>()
        val lines = source.lines()
        var i = 0
        var lineIdx = 0

        while (lineIdx < lines.size) {
            val startMatch = startDelimiterPattern.find(lines[lineIdx])
            if (startMatch != null) {
                val name = startMatch.groupValues[2]
                val startLine = lineIdx
                var contentStartIdx = i + lines[lineIdx].length + 1
                lineIdx++

                var endLine = -1
                while (lineIdx < lines.size) {
                    val endMatch = endDelimiterPattern.find(lines[lineIdx])
                    if (endMatch != null && endMatch.groupValues[1] == name) {
                        endLine = lineIdx
                        break
                    }
                    lineIdx++
                }

                if (endLine >= 0) {
                    val endPos = computePosition(lines, endLine)
                    val contentEndIdx = computePosition(lines, endLine)
                    val bodyLines = lines.subList(startLine + 1, endLine)
                    val body = bodyLines.joinToString("\n")
                    regions.add(DelimiterRegion(name, body.trim(), contentStartIdx, contentEndIdx))
                }
            } else {
                i += lines[lineIdx].length + 1
                lineIdx++
            }
        }

        return regions
    }

    private fun computePosition(lines: List<String>, targetLine: Int): Int {
        var pos = 0
        for (j in 0 until targetLine) {
            pos += lines[j].length + 1
        }
        return pos
    }

    private fun delimitedDetangle(
        regions: List<DelimiterRegion>,
        orgBlocks: List<NamedBlock>,
        sourceContent: String
    ): DetangleResult {
        val matched = mutableListOf<BlockUpdate>()
        val created = mutableListOf<BlockToCreate>()
        val stale = mutableListOf<BlockStale>()

        val orgBlockNames = orgBlocks.associate { it.name to it }

        for (region in regions) {
            if (orgBlockNames.containsKey(region.name)) {
                matched.add(BlockUpdate(region.name, region.content))
            } else {
                created.add(BlockToCreate(region.name, region.content, null))
            }
        }

        for (orgBlock in orgBlocks) {
            val found = regions.any { it.name == orgBlock.name }
            if (!found) {
                stale.add(BlockStale(orgBlock.name, "Anchor not found in source"))
            }
        }

        return DetangleResult(matched, created, stale)
    }

    private fun fallbackDetangle(
        sourceContent: String,
        orgBlocks: List<NamedBlock>
    ): DetangleResult {
        val anchors = anchorRegistry.extractAnchors(sourceContent)
        val strongAnchors = anchors.filter { it.strength == computer.whatthefuck.arroyo.anchor.Strength.STRONG }

        if (strongAnchors.isEmpty()) {
            return DetangleResult(
                matched = emptyList(),
                created = emptyList(),
                stale = orgBlocks.map { BlockStale(it.name, "No anchors found in source, fallback failed") },
                isFallback = true
            )
        }

        val created = strongAnchors.mapIndexed { idx, anchor ->
            val name = anchor.name
            val nextPos = if (idx + 1 < strongAnchors.size) strongAnchors[idx + 1].position else sourceContent.length
            val chunk = sourceContent.substring(anchor.position, nextPos).trim()
            BlockToCreate(
                suggestedName = name,
                content = chunk,
                insertAfterName = if (idx > 0) strongAnchors[idx - 1].name else null
            )
        }

        val preambleAnchor = strongAnchors.firstOrNull()
        val preambleToCreate = if (preambleAnchor != null && preambleAnchor.position > 0) {
            val preamble = sourceContent.substring(0, preambleAnchor.position).trim()
            if (preamble.isNotBlank()) {
                listOf(
                    BlockToCreate(
                        suggestedName = "preamble",
                        content = preamble,
                        insertAfterName = null
                    )
                )
            } else {
                emptyList()
            }
        } else {
            emptyList()
        }

        val matched = orgBlocks.mapNotNull { orgBlock ->
            anchors.find { it.name == orgBlock.name }?.let { _ ->
                val idx = strongAnchors.indexOfFirst { it.name == orgBlock.name }
                if (idx >= 0) {
                    val nextPos = if (idx + 1 < strongAnchors.size) strongAnchors[idx + 1].position else sourceContent.length
                    val chunk = sourceContent.substring(strongAnchors[idx].position, nextPos).trim()
                    BlockUpdate(orgBlock.name, chunk)
                } else {
                    null
                }
            }
        }

        val stale = orgBlocks.filter { orgBlock ->
            anchors.none { it.name == orgBlock.name }
        }.map { BlockStale(it.name, "Anchor not found in source (fallback mode)") }

        return DetangleResult(
            matched = matched,
            created = preambleToCreate + created,
            stale = stale,
            isFallback = true
        )
    }
}

Anchor Registry — fallback anchor extraction

When a source file has no delimiter comments (it was created before the tangle system existed), the detangler falls back to language-specific anchor extractors. The AnchorRegistry aggregates results from all extractors and sorts by position. The default registry includes extractors for Kotlin, Nix, Python, SQL, and Emacs Lisp.

kotlin#+name: anchor-registry:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/AnchorRegistry.kt

package computer.whatthefuck.arroyo

import computer.whatthefuck.arroyo.anchor.Anchor
import computer.whatthefuck.arroyo.anchor.AnchorExtractor
import computer.whatthefuck.arroyo.anchor.KotlinAnchorExtractor
import computer.whatthefuck.arroyo.anchor.NixAnchorExtractor
import computer.whatthefuck.arroyo.anchor.PythonAnchorExtractor
import computer.whatthefuck.arroyo.anchor.SqlAnchorExtractor
import computer.whatthefuck.arroyo.anchor.ElispAnchorExtractor
import computer.whatthefuck.arroyo.anchor.Strength

class AnchorRegistry(
    private val extractors: List<AnchorExtractor>
) {
    fun extractAnchors(code: String): List<Anchor> {
        return extractors.flatMap { it.extract(code) }
            .sortedBy { it.position }
    }

    companion object {
        fun default(): AnchorRegistry {
            return AnchorRegistry(
                listOf(
                    KotlinAnchorExtractor(),
                    NixAnchorExtractor(),
                    PythonAnchorExtractor(),
                    SqlAnchorExtractor(),
                    ElispAnchorExtractor()
                )
            )
        }
    }
}

Language-Specific Anchor Extractors

Each extractor implements AnchorExtractor, which returns a list of Anchor objects (name, position in source, strength). STRONG anchors correspond to declaration-level syntax (function definitions, class declarations, attribute assignments). WEAK anchors are secondary markers like comments or structural keywords.

Anchor interface

kotlin#+name: anchor-interface:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/anchor/AnchorExtractor.kt

package computer.whatthefuck.arroyo.anchor

interface AnchorExtractor {
    fun extract(code: String): List<Anchor>
}

data class Anchor(
    val name: String,
    val position: Int,
    val strength: Strength
)

enum class Strength { STRONG, WEAK }

Kotlin extractor

Matches class, object, interface, fun, val, var, typealias declarations (with optional visibility and modifier keywords). Weak anchors on comment lines.

kotlin#+name: kotlin-anchor-extractor:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/anchor/KotlinAnchorExtractor.kt

package computer.whatthefuck.arroyo.anchor

class KotlinAnchorExtractor : AnchorExtractor {
    private val strongPattern = Regex(
        """^\s*(public\s+|private\s+|internal\s+|protected\s+|suspend\s+|operator\s+|inline\s+|tailrec\s+)*"""
            + """(data\s+|sealed\s+|abstract\s+|open\s+)*"""
            + """(class\s+|object\s+|interface\s+|enum\s+class\s+|fun\s+|val\s+|var\s+|typealias\s+|\w+\s*@)"""
    )
    private val weakPattern = Regex("""^\s*//\s*\w""")

    override fun extract(code: String): List<Anchor> {
        val anchors = mutableListOf<Anchor>()
        var pos = 0
        code.lines().forEach { line ->
            val match = strongPattern.find(line)
            if (match != null) {
                val name = extractName(line)
                if (name != null) {
                    anchors.add(Anchor(name, pos, Strength.STRONG))
                }
            } else if (weakPattern.containsMatchIn(line)) {
                val name = line.trim().removePrefix("//").trim()
                if (name.isNotEmpty()) {
                    anchors.add(Anchor(name, pos, Strength.WEAK))
                }
            }
            pos += line.length + 1
        }
        return anchors
    }

    private fun extractName(line: String): String? {
        val trimmed = line.trim()
        val declPattern = Regex(
            """(?:class|object|interface|fun|val|var|typealias|enum\s+class)\s+([A-Za-z_]\w*)"""
        )
        return declPattern.find(trimmed)?.groupValues?.getOrNull(1)?.takeIf { it.isNotEmpty() }
    }
}

Nix extractor

Matches Nix attribute set assignments (attr = value) as strong anchors. let blocks and bare { openings as weak anchors.

kotlin#+name: nix-anchor-extractor:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/anchor/NixAnchorExtractor.kt

package computer.whatthefuck.arroyo.anchor

class NixAnchorExtractor : AnchorExtractor {
    private val attrPattern = Regex("""^(\s*[A-Za-z_]\w*)\s*=\s*""")
    private val letPattern = Regex("""^\s*let\s*$""")
    private val setPattern = Regex("""^\s*\{""")

    override fun extract(code: String): List<Anchor> {
        val anchors = mutableListOf<Anchor>()
        var pos = 0
        code.lines().forEach { line ->
            val attrMatch = attrPattern.find(line)
            if (attrMatch != null) {
                val name = attrMatch.groupValues[1].trim()
                anchors.add(Anchor(name, pos, Strength.STRONG))
            } else if (letPattern.containsMatchIn(line)) {
                anchors.add(Anchor("let", pos, Strength.WEAK))
            } else if (setPattern.containsMatchIn(line) && line.trimStart().startsWith("{")) {
                anchors.add(Anchor("set", pos, Strength.WEAK))
            }
            pos += line.length + 1
        }
        return anchors
    }
}

Python extractor

Matches def and class declarations (including async def) as strong anchors.

kotlin#+name: python-anchor-extractor:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/anchor/PythonAnchorExtractor.kt

package computer.whatthefuck.arroyo.anchor

class PythonAnchorExtractor : AnchorExtractor {
    private val defPattern = Regex("""^\s*(async\s+)?(def|class)\s+(\w+)""")

    override fun extract(code: String): List<Anchor> {
        val anchors = mutableListOf<Anchor>()
        var pos = 0
        code.lines().forEach { line ->
            val match = defPattern.find(line)
            if (match != null) {
                val name = match.groupValues[3]
                anchors.add(Anchor(name, pos, Strength.STRONG))
            }
            pos += line.length + 1
        }
        return anchors
    }
}

SQL extractor

Matches CREATE TABLE/INDEX/VIEW, ALTER TABLE, DROP TABLE/INDEX/VIEW as strong anchors. Also matches named query references (queryName:).

kotlin#+name: sql-anchor-extractor:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/anchor/SqlAnchorExtractor.kt

package computer.whatthefuck.arroyo.anchor

class SqlAnchorExtractor : AnchorExtractor {
    private val strongPattern = Regex(
        """^\s*(CREATE\s+(TABLE|INDEX|VIEW)\s+|ALTER\s+TABLE\s+|DROP\s+(TABLE|INDEX|VIEW)\s+)\s*(\w+)""",
        RegexOption.IGNORE_CASE
    )
    private val namedQueryPattern = Regex("""^([A-Za-z_]\w*)\s*:\s*$""")

    override fun extract(code: String): List<Anchor> {
        val anchors = mutableListOf<Anchor>()
        var pos = 0
        code.lines().forEach { line ->
            val match = strongPattern.find(line)
            if (match != null) {
                val name = match.groupValues.getOrNull(4) ?: match.groupValues[0].trim()
                anchors.add(Anchor(name, pos, Strength.STRONG))
            } else {
                val queryMatch = namedQueryPattern.find(line.trim())
                if (queryMatch != null) {
                    val name = queryMatch.groupValues[1]
                    anchors.add(Anchor(name, pos, Strength.STRONG))
                }
            }
            pos += line.length + 1
        }
        return anchors
    }
}

Emacs Lisp extractor

Matches defun, defvar, defcustom, defmacro, provide, define-minor-mode, define-derived-mode as strong anchors.

kotlin#+name: elisp-anchor-extractor:tangle ../src/commonMain/kotlin/computer/whatthefuck/arroyo/anchor/ElispAnchorExtractor.kt

package computer.whatthefuck.arroyo.anchor

class ElispAnchorExtractor : AnchorExtractor {
    private val defPattern = Regex("""^\(\s*(defun|defvar|defcustom|defmacro|provide|define-minor-mode|define-derived-mode)\s+'?([\w-]+)""")

    override fun extract(code: String): List<Anchor> {
        val anchors = mutableListOf<Anchor>()
        var pos = 0
        code.lines().forEach { line ->
            val match = defPattern.find(line)
            if (match != null) {
                val name = match.groupValues[2].trim('\'', ')')
                anchors.add(Anchor(name, pos, Strength.STRONG))
            }
            pos += line.length + 1
        }
        return anchors
    }
}

CLI Commands

The arcology CLI registers tangle and detangle subcommands (in roam/cli.org) that wrap this library with filesystem I/O, argument parsing, and output formatting. Both commands now live in the computer.whatthefuck.arroyo package alongside the library they invoke.

Design Decisions

CLI layer is deliberately thin.

The TangleCommand and DetangleCommand classes are thin wrappers — they handle argument parsing, file I/O, and output formatting. All tangle/detangle logic lives in the core library (OrgTangle, OrgDetangle). This separation keeps the core testable with pure string inputs (no filesystem mocks needed) and the CLI testable as an integration layer.

JSON output for detangle (not just printed text).

The detangle command outputs structured JSON (DetangleCliOutput) with matched, created, and stale sections plus an isFallback flag. This is consumed by the detangle agent (in .opencode/agents/detangle.md), which parses the JSON and applies changes back to the org file.

`--dry-run` for tangle.

The ~-n~/~--dry-run~ flag prints what would be written without touching the filesystem — useful for previewing the effect of noweb resolution and path normalization before committing to changes.

Implementation

TangleCommand — CLI entry point for tangling

Reads the org file, computes its path relative to the repo root, invokes OrgTangle.tangle() with an optional eval context, iterates over results, and writes each output file (or prints the path in dry-run mode). Warnings (missing/circular noweb refs, eval errors) are printed to stderr.

When the --db <path> option is provided, an ArroyoTangleContext backed by LuaJ is constructed so that :eval arroyo blocks are evaluated. Without --db, eval blocks emit MissingEvalContext warnings and are tangled as raw text.

Cross-file table references (:var tbl=other.org:tableName) are resolved relative to the org file's parent directory using a fileReader lambda that reads external org files from disk.

A :eval arroyo block whose Lua script throws is a hard error, not a recoverable warning. tangle() raises TangleEvalException; the command catches it, prints the error to stderr, and exits non-zero immediately. No files from the failing org file are written — embedding broken Lua source (or nothing) in the tangled output is worse than failing loudly. Files from earlier org files in the same invocation already on disk stay on disk.

kotlin#+name: tangle-command:tangle ../src/jvmMain/kotlin/computer/whatthefuck/arroyo/TangleCommand.kt

package computer.whatthefuck.arroyo

import com.github.ajalt.clikt.core.CliktCommand
import com.github.ajalt.clikt.parameters.arguments.argument
import com.github.ajalt.clikt.parameters.arguments.multiple
import com.github.ajalt.clikt.parameters.options.default
import com.github.ajalt.clikt.parameters.options.flag
import com.github.ajalt.clikt.parameters.options.option
import com.github.ajalt.clikt.parameters.types.path
import java.nio.file.Path
import kotlin.io.path.readText
import kotlin.io.path.writeText
import kotlin.io.path.createDirectories

class TangleCommand : CliktCommand(name = "tangle", help = "Tangle org files into source code") {
    private val orgFiles by argument(help = "Path(s) to .org file(s) to tangle").path(mustExist = true, canBeDir = false).multiple(required = true)
    private val repoRoot by option("--repo-root", help = "Repository root directory for output paths")
        .path(canBeDir = true)
        .default(Path.of("."))
    private val dryRun by option("--dry-run", "-n", help = "Show what would be written without writing").flag(default = false)
    private val dbPath by option("--db", help = "Database path for eval context").default("arcology.db")

    override fun run() {
        var totalFiles = 0
        var totalOrgFiles = 0
        for (orgFile in orgFiles) {
            val orgContent = orgFile.readText()
            val relativeOrgPath = repoRoot.relativize(orgFile).toString()

            val tangle = OrgTangle()
            val evalContext: ArroyoTangleContext? = if (java.io.File(dbPath).exists()) {
                val database = computer.whatthefuck.arcology.database.DatabaseFactory.createDatabase(dbPath)
                val repo = ArroyoRepositoryImpl(database)
                ArroyoScriptHost(repo, repoRoot.toString())
            } else {
                null
            }
            val fileReader: (String) -> String? = { fileName ->
                val base = orgFile.parent ?: repoRoot
                val resolved = base.resolve(fileName)
                if (resolved.toFile().exists()) resolved.readText() else null
            }
            val result = try {
                tangle.tangle(orgContent, relativeOrgPath, evalContext, fileReader)
            } catch (e: TangleEvalException) {
                echo("ERROR: ${e.message}", err = true)
                echo("Aborting tangle of $relativeOrgPath; no files written for this org file.", err = true)
                kotlin.system.exitProcess(1)
            }

            if (result.warnings.isNotEmpty()) {
                result.warnings.forEach { w ->
                    when (w) {
                        is TangleWarning.MissingNowebRef ->
                            echo("WARNING: Missing noweb ref '${w.name}' referenced in '${w.inBlockName}'", err = true)
                        is TangleWarning.CircularNowebRef ->
                            echo("WARNING: Circular noweb ref: ${w.cycle.joinToString(" -> ")}", err = true)
                        is TangleWarning.UnresolvedTableRef ->
                            echo("WARNING: Unresolved table arg '${w.tableName}' in block '${w.inBlockName}'", err = true)
                        is TangleWarning.EvalError ->
                            echo("WARNING: Eval error in '${w.blockName}': ${w.message}", err = true)
                        is TangleWarning.MissingEvalContext ->
                            echo("WARNING: No eval context for '${w.blockName}' language=${w.language}", err = true)
                    }
                }
            }

            if (result.files.isEmpty()) {
                echo("No tangle targets found in $relativeOrgPath")
                continue
            }

            for ((relativePath, content) in result.files) {
                val targetFile = repoRoot.resolve(relativePath)
                if (dryRun) {
                    echo("Would write: $targetFile")
                    if (relativePath in result.executableFiles) echo("Would chmod +x: $targetFile")
                } else {
                    targetFile.parent.createDirectories()
                    targetFile.writeText(content)
                    if (relativePath in result.executableFiles) {
                        try {
                            targetFile.toFile().setExecutable(true, true)
                        } catch (e: Exception) {
                            echo("WARNING: Failed to chmod +x $targetFile: ${e.message}", err = true)
                        }
                    }
                }
            }
            totalFiles += result.files.size
            totalOrgFiles++
        }

        echo("Tangled $totalFiles file(s) from $totalOrgFiles org file(s)")
    }
}

DetangleCommand — CLI entry point for detangling

Reads both the source file and org file, computes relative paths, and invokes OrgDetangle.detangle(). The result is converted to DetangleCliOutput (three serializable lists plus isFallback) and printed as pretty-printed JSON to stdout. The --tangle-target option overrides the source path for matching against the org file's :tangle directives.

kotlin#+name: detangle-command:tangle ../src/jvmMain/kotlin/computer/whatthefuck/arroyo/DetangleCommand.kt

package computer.whatthefuck.arroyo

import com.github.ajalt.clikt.core.CliktCommand
import com.github.ajalt.clikt.parameters.arguments.argument
import com.github.ajalt.clikt.parameters.options.default
import com.github.ajalt.clikt.parameters.options.option
import com.github.ajalt.clikt.parameters.types.path
import kotlinx.serialization.Serializable
import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
import java.nio.file.Path
import kotlin.io.path.readText

@Serializable
data class DetangleCliOutput(
    val matched: List<MatchedBlock>,
    val created: List<CreatedBlock>,
    val stale: List<StaleBlock>,
    val isFallback: Boolean
)

@Serializable
data class MatchedBlock(
    val blockName: String,
    val newContent: String
)

@Serializable
data class CreatedBlock(
    val suggestedName: String,
    val content: String,
    val insertAfterName: String?
)

@Serializable
data class StaleBlock(
    val blockName: String,
    val reason: String
)

class DetangleCommand : CliktCommand(name = "detangle", help = "Detangle source code back into an org file") {
    private val sourceFile by argument(help = "Path to source file to detangle").path(mustExist = true, canBeDir = false)
    private val orgFile by argument(help = "Path to .org file to detangle into").path(mustExist = true, canBeDir = false)
    private val tangleTarget by option("--tangle-target", help = "Override the tangle target path").default("")

    override fun run() {
        val repoRoot = Path.of(".")
        val sourceContent = sourceFile.readText()
        val orgContent = orgFile.readText()
        val sourcePath = repoRoot.relativize(sourceFile).toString()

        val target = tangleTarget.ifEmpty { sourcePath }

        val detangle = OrgDetangle()
        val result = detangle.detangle(sourceContent, orgContent, target)

        val output = DetangleCliOutput(
            matched = result.matched.map { MatchedBlock(it.blockName, it.newContent) },
            created = result.created.map { CreatedBlock(it.suggestedName, it.content, it.insertAfterName) },
            stale = result.stale.map { StaleBlock(it.blockName, it.reason) },
            isFallback = result.isFallback
        )

        val json = Json { prettyPrint = true }
        echo(json.encodeToString(output))
    }
}

Tests: Anchor Extractors

Unit tests verifying each language-specific anchor extractor. Tests cover Kotlin (class/fun/val/var with visibility modifiers), SQL (CREATE TABLE/INDEX, named queries), Nix (attribute assignments), Python (def/class/async def), Emacs Lisp (defun/defvar/defcustom/provide), and edge cases like skipping comments and blank lines.

kotlin#+name: arroyo-core-anchor-extractor-test:tangle ../src/commonTest/kotlin/computer/whatthefuck/arroyo/AnchorExtractorTest.kt

package computer.whatthefuck.arroyo

import computer.whatthefuck.arroyo.anchor.*
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue

class AnchorExtractorTest {

    @Test
    fun `extracts Kotlin class function val anchors`() {
        val code = """
package foo

class MyClass {
    fun doStuff() { }
    val x: Int = 1

    private suspend fun compute(): String = ""
}
        """.trimIndent()

        val extractor = KotlinAnchorExtractor()
        val anchors = extractor.extract(code)
        val names = anchors.map { it.name }
        assertTrue("MyClass" in names)
        assertTrue("doStuff" in names)
        assertTrue("x" in names)
        assertTrue("compute" in names)
    }

    @Test
    fun `extracts SQL CREATE TABLE and named query anchors`() {
        val code = """
CREATE TABLE nodes (
    id TEXT PRIMARY KEY
);

selectAllNodes:
SELECT * FROM nodes;

CREATE INDEX idx_nodes_file ON nodes(file);
        """.trimIndent()

        val extractor = SqlAnchorExtractor()
        val anchors = extractor.extract(code)
        val names = anchors.map { it.name }
        assertTrue("nodes" in names)
        assertTrue("selectAllNodes" in names)
        assertTrue("idx_nodes_file" in names)
    }

    @Test
    fun `extracts Nix attribute and let anchors`() {
        val code = """
{ config, lib, pkgs, ... }:

let
  myThing = "hello";
in

myPackage = callPackage ./default.nix { };
        """.trimIndent()

        val extractor = NixAnchorExtractor()
        val anchors = extractor.extract(code)
        val names = anchors.map { it.name }
        assertTrue("myPackage" in names)
    }

    @Test
    fun `extracts Python def and class anchors`() {
        val code = """
class Parser:
    def parse(self, input: str) -> str:
        pass

async def fetch_data(url: str):
    pass
        """.trimIndent()

        val extractor = PythonAnchorExtractor()
        val anchors = extractor.extract(code)
        val names = anchors.map { it.name }
        assertTrue("Parser" in names)
        assertTrue("parse" in names)
        assertTrue("fetch_data" in names)
    }

    @Test
    fun `extracts Elisp defun and defvar anchors`() {
        val code = """
(defun my-function (arg)
  "Documentation."

(defvar my-variable nil
  "A variable.")

(provide 'my-package)
        """.trimIndent()

        val extractor = ElispAnchorExtractor()
        val anchors = extractor.extract(code)
        val names = anchors.map { it.name }
        assertTrue("my-function" in names)
        assertTrue("my-variable" in names)
        assertTrue("my-package" in names)
    }

    @Test
    fun `skips comments and blank lines`() {
        val code = """
// this is a comment

class Actual {
    // another comment
    fun real() {}
}
        """.trimIndent()

        val extractor = KotlinAnchorExtractor()
        val anchors = extractor.extract(code)
        val strongNames = anchors.filter { it.strength == Strength.STRONG }.map { it.name }
        assertEquals(listOf("Actual", "real"), strongNames)
    }
}

arroyo/tools.org — the OpenCode agent definitions that drive the workflow
arroyo/tests.org — tests for the tangle/detangle library and CLI commands

Arcology Engine

Arroyo Core — Tangle/Detangle Engine + CLI Commands

Introduction

Design Decisions

Pure string-in/string-out design.

Delimiter comments as the round-trip mechanism.

Iterative noweb resolution with cycle detection.

Fallback anchor extraction for pre-tangle files.

Block name resolution from multiple sources.

Implementation

Models

TangleResult — output of a tangle operation

ArroyoTangleContext — eval host interface

DetangleResult — output of a detangle operation

Org Parse Utilities — collecting named blocks and tables

OrgTangle — noweb resolution and file assembly

Shebang support

OrgDetangle — reverse tangling

Anchor Registry — fallback anchor extraction

Language-Specific Anchor Extractors

Anchor interface

Kotlin extractor

Nix extractor

Python extractor

SQL extractor

Emacs Lisp extractor

CLI Commands

Design Decisions

CLI layer is deliberately thin.

JSON output for detangle (not just printed text).

`--dry-run` for tangle.

Implementation

TangleCommand — CLI entry point for tangling

DetangleCommand — CLI entry point for detangling

Tests: Anchor Extractors

Introduction

Design Decisions

Pure string-in/string-out design.

Delimiter comments as the round-trip mechanism.

Iterative noweb resolution with cycle detection.

Fallback anchor extraction for pre-tangle files.

Block name resolution from multiple sources.

Implementation

Models

TangleResult — output of a tangle operation

ArroyoTangleContext — eval host interface

DetangleResult — output of a detangle operation

Org Parse Utilities — collecting named blocks and tables

OrgTangle — noweb resolution and file assembly

Shebang support

OrgDetangle — reverse tangling

Anchor Registry — fallback anchor extraction

Language-Specific Anchor Extractors

Anchor interface

Kotlin extractor

Nix extractor

Python extractor

SQL extractor

Emacs Lisp extractor

CLI Commands

Design Decisions

CLI layer is deliberately thin.

JSON output for detangle (not just printed text).

--dry-run for tangle.

Implementation

TangleCommand — CLI entry point for tangling

DetangleCommand — CLI entry point for detangling

Tests: Anchor Extractors

Related Modules

`--dry-run` for tangle.