Arcology Engine

Arcology Project Configuration

Contents

,#+AUTO_TANGLE: vars:org-babel-default-header-args

This file contains user-configuration. Some sections are generated from org-mode tables if you're meant to be editing or extending them. This is a cool feature of org-babel, where you can use tables as data for code which can output more code or even org headings or links. We use this to generate configuration.

The Arcology's Site List

These can be customized by the user:

title key css file link color
The Lion's Rear lionsrear lionsrear.css #cfdcc2
The Arcology Garden garden garden.css #fcf6ed
The Complete Computer cce cce.css #dcdcec
The Arcology Site Engine arcology arcology.css #ccdfff
The Chance Encounter teasite lionsrear.css #cffc72
Local Dev Environment localhost arcology.css #075192
key domain
lionsrear thelionsrear.com
lionsrear rix.si
garden arcology.garden
garden whatthefuck.computer
cce cce.whatthefuck.computer
cce cce.rix.si
arcology engine.arcology.garden
localhost 127.0.0.1
localhost 127.0.0.1:29543
localhost 127.0.0.1:8000
localhost localhost
localhost localhost:29543
localhost localhost:8000
localhost virtuous-cassette
localhost virtuous-cassette:8000
lionsrear v2.thelionsrear.com
garden v2.arcology.garden
teasite thechanceencounter.com
cce cce2.whatthefuck.computer
arcology engine2.arcology.garden

The Code

Oh we doing some code gen in here, this is just gonna generate some JSON for now joining these two tables lul.

emacs-lisp#+name: elisp-join-tables:var sites=sites-config:results none
(json-encode
 (-map (lambda (siterow)
         (pcase-let* ((`(,title ,key ,css ,link) siterow))
           `((title . ,title)
             (key . ,key)
             (css_file . ,(format "arcology/css/%s" css))
             (link_color . ,link)
             (domains . ,(-map #'cadr
                            (-filter (pcase-lambda (`(,dkey ,domain))
                                       (equal key dkey))
                                     domains))))))
       sites))

That generates JSON that goes in to arcology/settings/sites.json for the Arcology Seed Command to use:

json:tangle arcology/settings/sites.json:noweb yes:comments none
<<elisp-join-tables()>>
emacs-lisp#+name: get_allowed_hosts:var domains=domains-config:results none
(thread-last domains
             (-map (pcase-lambda (`(,site ,domain))
                     domain))
             (s-join ","))

NEXT How to re-seed these if you change the configuration?

for now i just blow away the database but gosh!

Per-site CSS

These are generated from a table, too, because I have brainworms. Note that they're basically the same, but you can customize it further to your satisfaction.

Site alert primary secondary success warning white gray2 gray3 gray4 black
default-colors #cc6960 #707231 #ebbe7b #67b4f8 #7e5c41 #fcf6ed #f6e5cb #baad9b #82796c #211f1c
garden #cc6960 #707231 #ebbe7b #67b4f8 #7e5c41 #fcf6ed #f6e5cb #baad9b #82796c #211f1c
lionsrear #cc6960 #707231 #ebbe7b #67b4f8 #7e5c41 #cfdcc2 #87af87 #a0aa96 #82796c #211f1c
teasite #cc6960 #707231 #ebbe7b #67b4f8 #7e5c41 #cfdcc2 #87af87 #a0aa96 #82796c #211f1c
cce #cc6960 #707231 #ebbe7b #67b4f8 #7e5c41 #fcfcfc #dcdcec #cacada #808090 #211f1c
arcology #cc6960 #707231 #ebbe7b #67b4f8 #7e5c41 #fcf6ed #f6e5cb #baad9b #82796c #211f1c

The Code

emacs-lisp#+name: mk-site-colors:var site="default-colors":results none
(pcase-let ((`((,name ,alert ,primary ,secondary ,success ,warning ,white ,gray2 ,gray3 , gray4 ,black))
             (-filter (lambda (row)
                        (equal (first row) site))
                      colors)))
  (s-join "\n"
          (list ":root {"
                (concat "  --alert: " alert ";")
                (concat "  --primary: " primary ";")
                (concat "  --secondary: " secondary ";")
                (concat "  --success: " success ";")
                (concat "  --warning: " warning ";")
                (concat "  --white: " white ";")
                (concat "  --light-gray: " gray2 ";")
                (concat "  --medium-gray: " gray3 ";")
                (concat "  --dark-gray: " gray4 ";")
                (concat "  --black: " black ";")
                "}")))
css:tangle arcology/static/arcology/css/default-colors.css:noweb yes
<<mk-site-colors(site-colors, "default-colors")>>
css:tangle arcology/static/arcology/css/cce.css:noweb yes
<<mk-site-colors(site-colors, "cce")>>
css:tangle arcology/static/arcology/css/lionsrear.css:noweb yes
<<mk-site-colors(site-colors, "lionsrear")>>
css:tangle arcology/static/arcology/css/garden.css:noweb yes
<<mk-site-colors(site-colors, "garden")>>
css:tangle arcology/static/arcology/css/arcology.css:noweb yes
<<mk-site-colors(site-colors, "arcology")>>

Deploy manifests for Arroyo NixOS Generator or your own manifests

The Arroyo NixOS Generator will integrate this or you can do this yourself. just import the nixos module above, and then configure it as below:

nix:tangle ~/nix/nixos/arcology2.nix:noweb yes
{ inputs, systems, pkgs, ... }:

let
  arcology = inputs.arcology.packages.${pkgs.stdenv.hostPlatform.system}.default;
in {
  imports = [ ./arcology2-module.nix ];

  fileSystems."/media/org" = {
    device = "/home/rrix/org";
    options = ["bind"];
    fsType = "none";
    depends = ["/home" "/media"];
  };

  services.arcology-ng = {
    enable = true;
    packages.arcology = arcology;
    domains = pkgs.lib.splitString "," "<<get_allowed_hosts()>>";
    orgDir = "/media/org";
    folderId = "p1kld-oxnwd";
    dataDir = "/srv/arcology";
    logLevel = "INFO";
    ollamaHost = "windows:11434";
  };
}

The NixOS module which defines services.arcology-ng is in Deploying the Arcology.

NEXT the package import needs to be much better than this.

Deploy manifests for a Dynamic Home Manager Configuration of LocalAPI Deployment

This deploys A Localhost API for the Arcology to any Home Manager user using systemd User Units. See LocalAPI Deployment Options for the list of options that can be changed and how to use it:

nix:tangle ~/nix/snippets/arcology-inputs.nix
arcology = {
  url = "path:/home/rrix/org/arcology-django";
  inputs.nixpkgs.follows = "nixpkgs";
};
nix:tangle ~/nix/hm/arcology-localapi.nix
{ osConfig, inputs, system, pkgs, ... }: 

let
  hostname = osConfig.networking.hostName or "default";
  arcology = inputs.arcology.packages.${system}.default; 
in {
  imports = [ ./arcology-localapi-mod.nix ];

  services.arcology2 = {
    enable = true;
    packages.arcology = arcology;
    folderId = pkgs.lib.mkDefault "p1kld-oxnwd";
    logLevel = "INFO";
    environmentFile = "/home/rrix/sync/private-files/.arcology-env-${hostname}";
    ollamaHost = "windows:11434";

    address = "0.0.0.0";
  };
}

NEXT address the callPackages here...

Service Configuration

these are mostly defaults, a bit of this should be made user-configurable but a lot of this is baked in to how the project works.

Django settings for arcology project.

Generated by 'django-admin startproject' using Django 3.2.22.

For more information on this file, see https://docs.djangoproject.com/en/3.2/topics/settings/

For the full list of settings and their values, see https://docs.djangoproject.com/en/3.2/ref/settings/

Quick-start development settings - unsuitable for production See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/

python:tangle arcology/settings/__init__.py
from pathlib import Path
import os

from .generators import *

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent.parent
DATABASE_PATH = pathlib.Path(os.getenv("ARCOLOGY_DB_PATH", pathlib.Path(os.getcwd()) / "db.sqlite3")).expanduser()
WRITABLE_DB_PATH = pathlib.Path(os.getenv("ARCOLOGY_DB_WRITE_PATH", pathlib.Path(os.getcwd()) / "writable.sqlite3")).expanduser()

Environment Variables

ARCOLOGY_DJANGO_SECRET environment variable will be set in the NixOS deployment manifests. this is used for, uhh, security things

python:tangle arcology/settings/__init__.py
SECRET_KEY = os.getenv("ARCOLOGY_DJANGO_SECRET", "devsecret")

Setting =ARCOLOGY_ENVIRONMENT=production= in the NixOS deployment manifests will modify a bunch of behavior, probably

python:tangle arcology/settings/__init__.py
ARCOLOGY_ENVIRONMENT = os.getenv("ARCOLOGY_ENVIRONMENT", "dev")
DEBUG = (ARCOLOGY_ENVIRONMENT != "production")

The ARCOLOGY_SYNCTHING_KEY environment variable will be set to the Syncthing API key, each deployment will need to specify its own, i'll re-think this when it comes time to deploy this to many places.

python:tangle arcology/settings/__init__.py
SYNCTHING_KEY = os.getenv("ARCOLOGY_SYNCTHING_KEY")

The ARCOLOGY_CACHE_PATH is set to a path that multi-process django can use to cache processed HTML and Atom between processes.

python:tangle arcology/settings/__init__.py
BASE_CACHE_PATH = pathlib.Path(os.environ.get("ARCOLOGY_CACHE_PATH", '~/.cache/arcology2')).expanduser()
CACHES = {
    'default': {
        'BACKEND': 'django_prometheus.cache.backends.filebased.FileBasedCache',
        'LOCATION': BASE_CACHE_PATH,
    },
}
python:tangle arcology/settings/__init__.py
LOCALAPI_BEARER_TOKEN = os.environ.get("ARCOLOGY_LOCALAPI_BEARER_TOKEN", "changeme!")

Used by The Arcology's Web Server to store web-optimized attachments.

python:tangle arcology/settings/__init__.py
MEDIA_ROOT = BASE_CACHE_PATH.joinpath("media")
MEDIA_URL = "media/"
ATTACHMENT_DIR = BASE_CACHE_PATH.joinpath("attachments")

NEXT Hostname configuration from arcology.model.Site, eventually

When I have the sites organized in an org-mode table, i'll reapproach the hostname list, and probably before then when i want to test domain-based routing.

python:tangle arcology/settings/__init__.py:noweb yes
ALLOWED_HOSTS = "<<get_allowed_hosts()>>".split(',')

The Arcology is Modular

basically, each org file in this repository, and maybe one or two of your own, are all interoperable and build on top of the base data models of the org docs.

"roam" Arcology Roam Models
"arcology" The Arcology's Data Models and Web Server
"generators" The Arroyo Generators
"syncthonk" Arcology watchsync Command
"sitemap" The Arcology's Site Maps and Discovery Mechanisms
"reservations" The Arcology Reservation System
python:tangle arcology/settings/__init__.py:noweb yes
# Application definition
INSTALLED_APPS = [
    <<gen-config-list(arcology_apps)>>
    "django_htmx",
    "django_prometheus",
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
]
if ARCOLOGY_ENVIRONMENT != "production":
  INSTALLED_APPS = ["localapi"] + INSTALLED_APPS

Internationalization

I'll have to do this at some point. what does an internationalized arcology look like, how does one ship translations of this work?

python:tangle arcology/settings/__init__.py:noweb yes
# Internationalization
# https://docs.djangoproject.com/en/3.2/topics/i18n/

LANGUAGE_CODE = "en-us"
TIME_ZONE = "America/Los_Angeles"
USE_I18N = True
USE_L10N = True
USE_TZ = True

Logging Configuration

python:tangle arcology/settings/__init__.py:noweb yes
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
        },
    },
    "loggers": {
        "django": {
            "handlers": ["console"],
            "level": os.getenv("DJANGO_LOG_LEVEL", "INFO"),
            "propagate": False,
        },
        "arcology.cache_decorator": { # left as an example to change later.
            "handlers": ["console"],
            "level": "DEBUG",
        }
    },
    "root": {
        "handlers": ["console"],
        "level": os.getenv("ARCOLOGY_LOG_LEVEL", "INFO"),
    },
}

Generator Configuration

NEXT directory configuration

python:tangle arcology/settings/generators.py:noweb yes
import pathlib
import os

ARCOLOGY_BASE_DIR = pathlib.Path(os.environ.get("ARCOLOGY_BASE_DIR", "~/org")).expanduser()
ARCOLOGY_EMACS_SNIPPETS_DIR = str(pathlib.Path("~/org/cce/").expanduser())
ARROYO_BASE_DIR = str(pathlib.Path("~/arroyo-nix").expanduser())

Defining extension modules

It should be feasible to add new data types and models and generators to the Arcology system to fit a user's needs. The defaults are specified here.

These models are only persisted if the page is going to be published and are used by the core The Arcology Web Server to set up the page and render backlinks and tag pages and whatnot.

roam.models.Heading Arcology roam.models.Heading
roam.models.Reference Arcology roam.models.Reference
roam.models.Tag Arcology roam.models.Tag
roam.models.HeadingProperty Arcology roam.models.HeadingProperty
roam.models.Link Arcology roam.models.Link
roam.models.Alias Arcology roam.models.Alias
arcology.models.Page The Arcology's Data Models's Page entity
arcology.models.Feed The Arcology's Data Models's Feed entity, probably move to its own module some day...
arcology.models.FeedEntry
arcology.models.Attachment
reservations.models.Event The Arcology Reservation System
reservations.models.EventSlot The Arcology Reservation System

The Arroyo Generators are run regardless of whether a file is published. These are a dictionary so that the user can specify a module when invoking the Arcology generate Command.

emacs generators.models.EmacsSnippet
epkgs generators.models.EmacsEpkg
nixos generators.models.NixosModule
home-manager generators.models.HomeManagerModule

The Code

python:tangle arcology/settings/generators.py:noweb yes
ARCOLOGY_EXTRACTORS = [
  <<gen-config-list(arcology_extractors)>>
]

ARROYO_EXTRACTORS = {
  <<gen-config-dict(arroyo_extractors)>>
}

Enumerating System Roles

I would like to not need these enumerated here, it seems like it should be easier to add a new role than that. But for now here they are so that the DB seed works better.

server the Wobserver Configuration with Arroyo Nixos $
endpoint My NixOS configuration
edge The Wobserver's Edge Server
droid CCE in Nix On Droid
dev-vm WSL NixOS Box
settop NixOS Set Top Box
waterboy waterboy icebox'd data collection platform
python:tangle arcology/settings/generators.py:noweb yes
ARROYO_ROLES = [
  <<gen-config-list(arroyo-roles)>>
]

Roam Allowed Keywords

These tables will be used to create an allowlist of =#+KEYWORD='s to store in the database.

These are used to provide system features or power the various generators:

ARCOLOGY_FEED Add a route that will generate an Atom feed for this page
ARCOLOGY_KEY Add a route that will generate an HTML page for this page
ARCOLOGY_PAGE_TEMPLATE Arcology Page HTML Template override
ARCOLOGY_ALLOW_CRAWL Whether the Arcology page will be marked as allowed in robots.txt
ARCOLOGY_TOOT_VISIBILITY This is set to an Mastadan API visibility mode like "public" or "unlisted"
ARCOLOGY_RESERVATIONS Instruct The Arcology Reservation System to parse the doc for Events and have a form embedded.
ARROYO_EMACS_MODULE Instruct The Arroyo Generators to include an Emacs snippet in the init.
ARROYO_HOME_MODULE Instruct The Arroyo Generators to import the referenced home-manager module
ARROYO_NIXOS_MODULE Instruct The Arroyo Generators to import the referenced nixos module
ARROYO_HOME_EPKGS Instruct The Arroyo Generators to make a custom Emacs-Lisp package available to Arroyo Emacs
ARROYO_MODULE_WANTS Define a dependency relationship with the referenced org-roam document (this doc depends on that)
ARROYO_MODULE_WANTED Define a dependent relationship from the referenced org-roam document (that doc depends on this)
ARROYO_SYSTEM_ROLE Constrain the Nix expressions on the page to only load in this role (can be specified repeatedly)
ARROYO_SYSTEM_EXCLUDE Prevent the Nix expressions on the page from loading in the listed role (can be specified repeatedly)
ARROYO_DIRENV_DIR Load direnv from a different directory than the loaded buffer's directory. Useful for Org Babel.
ARROYO_TANGLE_THIS When arroyo-flood is called, these files will be tangled.
AUTHOR Used to populate the author in RSS feeds and page metadata

The value of ARROYO_TANGLE_THIS doesn't matter for now, but it'd be nice if set to roles to tangle for, space-separated.

These are used in some of my custom reports:

BIRTHDAY Used in People pages.
LOCATION Used in People pages.
PRONOUNS Used in People pages.
AGE_OF_MONEY Used in financial review of Monthly Review
JPMORGAN Used in financial review of Monthly Review
SAVINGS Used in financial review of Monthly Review
SPENT_THIS_MONTH Used in financial review of Monthly Review

The Code

python:tangle arcology/settings/generators.py:noweb yes
ROAM_ALLOWED_KEYWORDS = [
  <<gen-config-list(roam_allowed_keywords)>>
  <<gen-config-list(my_roam_allowed_keywords)>>
# used for deprecated universal-aggregator feed generation
    "ARROYO_FEEDS",
    "ARROYO_FEED_CATEGORY",
    "ARROYO_FEED_URL",
# unused
    "CCE_HOME_MODULE",
    "ARROYO_EMACS_TANGLED",
    "ARROYO_MODULE_DEP",
]

Ignored Roam Tags

Headings with these tags will not be exported.

NOEXPORT
noexport
Private

The Code

python:tangle arcology/settings/generators.py:noweb yes
IGNORED_ROAM_TAGS = [
  <<gen-config-list(ignored_roam_tags)>>
]

Look don't worry about the rest of these

python:tangle arcology/settings/__init__.py:noweb yes
MIDDLEWARE = [
    "django_prometheus.middleware.PrometheusBeforeMiddleware",
    "django_htmx.middleware.HtmxMiddleware",
    "django.middleware.security.SecurityMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.middleware.common.CommonMiddleware",
    "django.middleware.csrf.CsrfViewMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "django.contrib.messages.middleware.MessageMiddleware",
    "django.middleware.clickjacking.XFrameOptionsMiddleware",
    "django_prometheus.middleware.PrometheusAfterMiddleware",
]

ROOT_URLCONF = "arcology.urls"

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
            ],
        },
    },
]

WSGI_APPLICATION = "arcology.wsgi.application"
ASGI_APPLICATION = "arcology.asgi.application"

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": [("127.0.0.1", 6379)],
            "group_expiry": 3600,
        },
    },
}

# Database
# https://docs.djangoproject.com/en/3.2/ref/settings/#databases

DATABASES = {
    "default": {
        "ENGINE": "django_prometheus.db.backends.sqlite3",
        "NAME": DATABASE_PATH,
    },
    "writable": {
        # "ENGINE": "django_sqlite_vec.conn",
        "ENGINE": "django_prometheus.db.backends.sqlite3",
        "NAME": WRITABLE_DB_PATH,
    }
}

# important to separate things which come from docs from the web so that the docs can be safely re-parsed
# XXX make this configurable
class WriteRouter:
    route_app_labels = {"reservations"}
    route_model_names = {"reservations_pendingregistration", "pendingreservation",
                         }

    def db_for_read(self, model, **hints):
        if model._meta.app_label in self.route_app_labels:
            return "writable"
        return None

    def db_for_write(self, model, **hints):
        if model._meta.app_label in self.route_app_labels:
            return "writable"
        return None

    def allow_relation(self, obj1, obj2, **hints):
        if (
            obj1._meta.app_label in self.route_app_labels
            or obj2._meta.app_label in self.route_app_labels
        ):
            return True
        return None

    def allow_migrate(self, db, app_label, model_name=None, **hints):
        if app_label in self.route_app_labels:
            return db == "writable"
        return None

DATABASE_ROUTERS = [ "arcology.settings.WriteRouter" ]

# Password validation
# https://docs.djangoproject.com/en/3.2/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        "NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator",
    },
    {
        "NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",
    },
    {
        "NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",
    },
    {
        "NAME": "django.contrib.auth.password_validation.NumericPasswordValidator",
    },
]

# to ensure that nginx frontend proxy_pass is still considered secure
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

CSRF_TRUSTED_ORIGINS = list(map(lambda it: f"https://{it}", "<<get_allowed_hosts()>>".split(',')))
python:tangle arcology/settings/__init__.py:noweb yes
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.2/howto/static-files/

STATIC_URL = "/static/"

STORAGES = {
    "default": {
        "BACKEND": "django.core.files.storage.FileSystemStorage",
    },
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
    },
}
STATICFILES_DIRS = [
    BASE_DIR / "arcology/static",
    BASE_DIR / "sitemap/static",
]

STATIC_ROOT = os.getenv("ARCOLOGY_STATIC_ROOT", "/var/www/arcology")

# Default primary key field type
# https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field

DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"

Helpers

emacs-lisp#+name: gen-config-list:var tbl=roam_allowed_keywords
(mapconcat (lambda (row)
             (format "\"%s\", # %s\n" (first row) (second row)))
           tbl)
emacs-lisp#+name: gen-config-dict:var tbl=arroyo_extractors
(mapconcat (lambda (row)
             (format "\"%s\": \"%s\",\n" (first row) (second row)))
           tbl)