heartwood every commit a ring

the field learns to mirror itself in the clouds

753dc63a by Isaac Bythewood · 13 days ago

modified CLAUDE.md
@@ -28,8 +28,8 @@ There are no tests, linters, or build steps in this repo — it is pure configur## Architecture- **`dotfiles/`** — Terminal and editor config (bash, git, tmux, neovim). Neovim config is Lua-based with a custom statusline. These get COPYed into the container at build time.- **`containers/webdev/`** — Ubuntu 24.04 dev container running PostgreSQL 16 + Redis via supervisord. Includes Node 22, Python 3 (pip + uv), Bun, Yarn (via corepack — still available for legacy repos, though all current projects use Bun), Docker-in-Docker, and standard dev tools (neovim, tmux, git, rsync, htop, nmap, etc.).- **`hosts/alpine/`** — Production server setup: Caddy reverse proxy (auto HTTPS), Docker Compose for services, Borg backups, UFW firewall, push-to-deploy via git hooks.- **`containers/webdev/`** — Ubuntu 24.04 dev container running PostgreSQL 16 + Redis via supervisord. Includes Node 22, Python 3 (pip + uv), Bun, Docker-in-Docker, and standard dev tools (neovim, tmux, git, rsync, htop, nmap, unzip, etc.).- **`hosts/alpine/`** — Production server setup: Caddy reverse proxy (auto HTTPS), Docker Compose for services, restic backups to Backblaze B2, UFW firewall, push-to-deploy via git hooks.## Deployed Projects
@@ -49,7 +49,7 @@ Update ports, repos, or flags by editing `projects.conf` and re-running the rele## How Deployment Works`quickstart.sh` reads `projects.conf` and generates one bare repo under `/srv/git/<name>.git/` per project with a post-receive hook. Pushing to that remote triggers: `git pull`, `docker compose up --build --detach`, optional `manage.py migrate`, and a `borg` prune on hosts configured for backup. The Caddyfile proxies each project's subdomain to its bound port on `127.0.0.1`.`quickstart.sh` reads `projects.conf` and generates one bare repo under `/srv/git/<name>.git/` per project with a post-receive hook. Pushing to that remote triggers: `git pull`, `docker compose up --build --detach`, optional `manage.py migrate`, and `docker system prune`. The Caddyfile proxies each project's subdomain to its bound port on `127.0.0.1`.## Conventions
modified README.md
@@ -59,11 +59,13 @@ docker build --tag overshard/webdev:latest -f containers/webdev/Dockerfile .docker volume create --name bythewood-codedocker volume create --name bythewood-claudedocker volume create --name bythewood-sshdocker volume create --name bythewood-resticdocker run --detach --restart unless-stopped --name bythewood-webdev \    --volume bythewood-code:/home/dev/code \    --volume bythewood-claude:/home/dev/.claude \    --volume bythewood-ssh:/home/dev/.ssh \    --volume bythewood-restic:/home/dev/.restic \    --volume /var/run/docker.sock:/var/run/docker.sock \    -p 8000:8000 \    overshard/webdev:latest
@@ -84,9 +86,9 @@ Baked into the container at build time via COPY — no bootstrap script needed.## The hostAlpine Linux. Firewall, backups, and quiet daily maintenance. The Caddyfile,port assignments, and post-receive hooks are all generated from `projects.conf`so the server can be rebuilt from this repo alone.Alpine Linux. Firewall, daily restic backups to Backblaze B2, and quiet dailymaintenance. The Caddyfile, port assignments, and post-receive hooks are allgenerated from `projects.conf` so the server can be rebuilt from this repo alone.Provision a fresh server:
@@ -102,6 +104,56 @@ cd ~/codesh taproot/hosts/alpine/srv/bootstrap.sh```## BackupsBoth the webdev container and the alpine host back up to a single Backblaze B2bucket (`overshard-backups`) using restic, one repo per host:| Host | Repository | Paths backed up ||---|---|---|| webdev | `b2:overshard-backups:webdev` | `~/.claude`, `~/code`, `~/.ssh` || alpine | `b2:overshard-backups:alpine` | `/srv/git`, `/srv/docker`, `/srv/data` |Each host has its own application key (scoped to the bucket) and its ownrestic password — both stored in 1Password. Retention is 7 daily / 4 weekly /6 monthly snapshots, pruned after each backup.Place credentials before backups will run:```sh# webdev (paste from 1Password, then Ctrl-D)docker exec -i bythewood-webdev tee /home/dev/.restic/password > /dev/nulldocker exec -i bythewood-webdev tee /home/dev/.restic/b2-env > /dev/nulldocker exec bythewood-webdev chmod 600 /home/dev/.restic/password /home/dev/.restic/b2-env# alpinessh root@server "cat > /root/.restic/password && chmod 600 /root/.restic/password"ssh root@server "cat > /root/.restic/b2-env && chmod 600 /root/.restic/b2-env"````b2-env` contents (both hosts):```shexport B2_ACCOUNT_ID="<keyID>"export B2_ACCOUNT_KEY="<applicationKey>"```Run a backup manually:```shdocker exec -it bythewood-webdev /home/dev/backup.sh   # webdev (manual)ssh root@server /etc/periodic/daily/restic-autobackup  # alpine (also runs daily)```Restore the latest snapshot. Existing data is moved aside to`~/before-restore-<UTC-ISO>/` (webdev) or `/root/before-restore-<UTC-ISO>/srv/`(alpine) before restic writes the snapshot back:```shdocker exec -it bythewood-webdev /home/dev/restore.shssh root@server /root/restore.sh```## Philosophy- Keep defaults until they fail you.
modified containers/webdev/Dockerfile
@@ -11,12 +11,14 @@#     docker volume create --name bythewood-code#     docker volume create --name bythewood-claude#     docker volume create --name bythewood-ssh#     docker volume create --name bythewood-restic## Start container:#     docker run --detach --restart unless-stopped --name bythewood-webdev \#         --volume bythewood-code:/home/dev/code \#         --volume bythewood-claude:/home/dev/.claude \#         --volume bythewood-ssh:/home/dev/.ssh \#         --volume bythewood-restic:/home/dev/.restic \#         --volume /var/run/docker.sock:/var/run/docker.sock \#         -p 8000:8000 \#         overshard/webdev:latest
@@ -30,15 +32,28 @@# Connect:#     docker exec -it bythewood-webdev tmux## I use volumes for code, claude, and ssh to make rebuilds of the container# easy without losing project files, claude auth/memory, or SSH keys. I have# scripts setup on my hosts to rebuild images, delete old containers, and# start the new containers when I make updates.# Restic credentials (first time only) — pull from 1Password:#     docker exec -i bythewood-webdev tee /home/dev/.restic/password > /dev/null  # paste, Ctrl-D#     docker exec -i bythewood-webdev tee /home/dev/.restic/b2-env > /dev/null    # paste, Ctrl-D#     docker exec bythewood-webdev chmod 600 /home/dev/.restic/password /home/dev/.restic/b2-env## b2-env contents:#     export B2_ACCOUNT_ID="<keyID from 1Password>"#     export B2_ACCOUNT_KEY="<applicationKey from 1Password>"## Backup and restore:#     docker exec -it bythewood-webdev /home/dev/backup.sh#     docker exec -it bythewood-webdev /home/dev/restore.sh## I use volumes for code, claude, ssh, and restic to make rebuilds of the# container easy without losing project files, claude auth/memory, SSH keys,# or restic credentials. I have scripts setup on my hosts to rebuild images,# delete old containers, and start the new containers when I make updates.## NOTE: Fresh volumes will inherit correct dev:dev ownership from the image.# Existing volumes retain their permissions. If you hit root:root ownership# issues on existing volumes, fix with:#     docker exec -it bythewood-webdev sudo chown -R dev:dev /home/dev/.claude /home/dev/code#     docker exec -it bythewood-webdev sudo chown -R dev:dev /home/dev/.claude /home/dev/code /home/dev/.restic## Docker is included in this build so I can test docker stuff inside my# containers. Note that to get docker to function in a docker container you
@@ -61,7 +76,9 @@ ENV DEBIAN_FRONTEND=noninteractive \RUN apt-get update && \    apt-get install -y --no-install-recommends \        # Dev tools        curl git rsync neovim openssh-client tmux whois nmap \        curl git rsync neovim openssh-client tmux whois nmap unzip \        # Backups        restic \        # System essentials        tzdata ca-certificates sudo \        # Build tools
@@ -79,9 +96,6 @@ RUN apt-get update && \    curl -fsSL https://deb.nodesource.com/setup_22.x | bash - && \    apt-get install -y nodejs && \    curl -fsSL https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh && \    mkdir -p /root/.local/share/corepack/shims && \    corepack enable && \    corepack prepare yarn@stable --activate && \    curl -fsSL https://bun.sh/install | env BUN_INSTALL=/usr/local bash && \    rm -rf /var/lib/apt/lists/*
@@ -148,7 +162,8 @@ RUN chown -R dev:dev /home/dev/.bash_aliases /home/dev/.gitconfig /home/dev/.con#                    it up when launched from the home dir, not just from#                    inside ~/code/. Target resolves once the bythewood-code#                    volume mounts at runtime.RUN mkdir -p /home/dev/code /home/dev/.claude /home/dev/.ssh && \RUN mkdir -p /home/dev/code /home/dev/.claude /home/dev/.ssh /home/dev/.restic && \    chmod 700 /home/dev/.restic && \    ln -s /home/dev/.claude/.claude.json /home/dev/.claude.json && \    ln -s code/CLAUDE.md /home/dev/CLAUDE.md && \    printf '%s\n' \
@@ -165,7 +180,12 @@ RUN mkdir -p /home/dev/code /home/dev/.claude /home/dev/.ssh && \    > /home/dev/.ssh/config && \    chmod 700 /home/dev/.ssh && \    chmod 600 /home/dev/.ssh/config && \    chown -R dev:dev /home/dev/code /home/dev/.claude /home/dev/.ssh    chown -R dev:dev /home/dev/code /home/dev/.claude /home/dev/.ssh /home/dev/.resticCOPY containers/webdev/backup.sh /home/dev/backup.shCOPY containers/webdev/restore.sh /home/dev/restore.shRUN chmod +x /home/dev/backup.sh /home/dev/restore.sh && \    chown dev:dev /home/dev/backup.sh /home/dev/restore.shWORKDIR /home/devUSER dev
added containers/webdev/backup.sh
@@ -0,0 +1,44 @@#!/bin/sh## backup.sh## Run manually to back up this container to Backblaze B2 via restic.# Initializes the repo on first run. Prunes per the retention policy# (7 daily, 4 weekly, 6 monthly) after each successful backup.## Credentials live in ~/.restic/ (mounted from the bythewood-restic volume):#   ~/.restic/password   restic repo password (0600)#   ~/.restic/b2-env     exports B2_ACCOUNT_ID and B2_ACCOUNT_KEY (0600)#set -eu. "$HOME/.restic/b2-env"export RESTIC_REPOSITORY="b2:overshard-backups:webdev"export RESTIC_PASSWORD_FILE="$HOME/.restic/password"if ! restic cat config >/dev/null 2>&1; then    echo "Repository not initialized. Running restic init..."    restic initfirestic backup \    --verbose \    --exclude-caches \    --exclude='node_modules' \    --exclude='.next' \    --exclude='.venv' \    --exclude='__pycache__' \    --exclude='dist' \    --exclude='build' \    --exclude='.cache' \    --exclude='.vite' \    --exclude='*.pyc' \    "$HOME/.claude" \    "$HOME/code" \    "$HOME/.ssh"restic forget --prune \    --keep-daily   7 \    --keep-weekly  4 \    --keep-monthly 6
added containers/webdev/restore.sh
@@ -0,0 +1,39 @@#!/bin/sh## restore.sh## Restore this container from the latest snapshot in Backblaze B2.# Existing contents of ~/.claude, ~/code, and ~/.ssh are moved aside to# ~/before-restore-<UTC-ISO>/ first so nothing is lost.## Note: ~/.claude, ~/code, and ~/.ssh are docker volume mounts, so we move# their CONTENTS rather than the directories themselves.#set -eu. "$HOME/.restic/b2-env"export RESTIC_REPOSITORY="b2:overshard-backups:webdev"export RESTIC_PASSWORD_FILE="$HOME/.restic/password"ARCHIVE="$HOME/before-restore-$(date -u +%Y-%m-%dT%H-%M-%SZ)"echo "Moving existing data aside to $ARCHIVE"mkdir -p "$ARCHIVE/.claude" "$ARCHIVE/code" "$ARCHIVE/.ssh"for dir in .claude code .ssh; do    if [ -d "$HOME/$dir" ]; then        find "$HOME/$dir" -mindepth 1 -maxdepth 1 \            -exec mv {} "$ARCHIVE/$dir/" \;    fidoneecho "Restoring latest snapshot from $RESTIC_REPOSITORY"restic restore latest --target /echo ""echo "Restore complete. Previous data archived at:"echo "  $ARCHIVE"echo ""echo "Once you've verified everything looks right, you can remove the archive:"echo "  rm -rf $ARCHIVE"
deleted hosts/alpine/etc/periodic/daily/borg-autobackup
@@ -1,68 +0,0 @@#!/bin/sh# Based off of https://borgbackup.readthedocs.io/en/stable/quickstart.html#automating-backups# Init the repo with borg init -e none /srv/backup# Setting this, so the repo does not need to be given on the commandline:export BORG_REPO=/srv/backup# some helpers and error handling:info() { printf "\n%s %s\n\n" "$( date )" "$*" >&2; }trap 'echo $( date ) Backup interrupted >&2; exit 2' INT TERMinfo "Starting backup"# Backup the most important directories into an archive named after# the machine this script is currently running on:borg create                         \    --verbose                       \    --filter AME                    \    --list                          \    --stats                         \    --show-rc                       \    --compression lz4               \    --exclude-caches                \                                    \    ::'{now}'                       \    /srv/git                        \    /srv/docker                     \    /srv/data                       \    /etc/caddy                      \backup_exit=$?info "Pruning repository"# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly# archives of THIS machine.borg prune                          \    --list                          \    --show-rc                       \    --keep-daily    7               \    --keep-weekly   4               \    --keep-monthly  6               \prune_exit=$?# actually free repo disk space by compacting segmentsinfo "Compacting repository"borg compactcompact_exit=$?# use highest exit code as global exit codeglobal_exit=$(( backup_exit > prune_exit ? backup_exit : prune_exit ))global_exit=$(( compact_exit > global_exit ? compact_exit : global_exit ))if [ ${global_exit} -eq 0 ]; then    info "Backup, Prune, and Compact finished successfully"elif [ ${global_exit} -eq 1 ]; then    info "Backup, Prune, and/or Compact finished with warnings"else    info "Backup, Prune, and/or Compact finished with errors"fiexit ${global_exit}
added hosts/alpine/etc/periodic/daily/restic-autobackup
@@ -0,0 +1,61 @@#!/bin/sh## Daily backup of /srv to Backblaze B2 via restic.# Initializes the repo on first run. Prunes per the retention policy# (7 daily, 4 weekly, 6 monthly) after each successful backup.## Credentials:#   /root/.restic/password   restic repo password (0600 root)#   /root/.restic/b2-env     exports B2_ACCOUNT_ID and B2_ACCOUNT_KEY (0600 root)#. /root/.restic/b2-envexport RESTIC_REPOSITORY="b2:overshard-backups:alpine"export RESTIC_PASSWORD_FILE="/root/.restic/password"info() { printf "\n%s %s\n\n" "$(date)" "$*" >&2; }trap 'echo $(date) Backup interrupted >&2; exit 2' INT TERMif ! restic cat config >/dev/null 2>&1; then    info "Repository not initialized. Running restic init..."    restic initfiinfo "Starting backup"restic backup \    --verbose \    --exclude-caches \    --exclude='node_modules' \    --exclude='.next' \    --exclude='.venv' \    --exclude='__pycache__' \    --exclude='dist' \    --exclude='build' \    --exclude='.cache' \    --exclude='.vite' \    --exclude='*.pyc' \    /srv/git \    /srv/docker \    /srv/databackup_exit=$?info "Pruning repository"restic forget --prune \    --keep-daily   7 \    --keep-weekly  4 \    --keep-monthly 6prune_exit=$?global_exit=$(( backup_exit > prune_exit ? backup_exit : prune_exit ))if [ ${global_exit} -eq 0 ]; then    info "Backup and prune finished successfully"else    info "Backup and/or prune finished with errors (exit ${global_exit})"fiexit ${global_exit}
modified hosts/alpine/quickstart.sh
@@ -22,7 +22,7 @@ apk add \    ip6tables \    iptables \    ufw \    borgbackup \    restic \    docker \    docker-compose \    caddy
@@ -38,16 +38,21 @@ ufw --force enable# Install configuration filescp etc/apk/repositories /etc/apk/repositories && chmod 644 /etc/apk/repositoriescp etc/periodic/daily/apk-autoupgrade /etc/periodic/daily/apk-autoupgrade && chmod 700 /etc/periodic/daily/apk-autoupgradecp etc/periodic/daily/borg-autobackup /etc/periodic/daily/borg-autobackup && chmod 700 /etc/periodic/daily/borg-autobackupcp etc/periodic/daily/restic-autobackup /etc/periodic/daily/restic-autobackup && chmod 700 /etc/periodic/daily/restic-autobackupcp etc/docker/daemon.json /etc/docker/daemon.json && chmod 644 /etc/docker/daemon.jsoncp etc/caddy/Caddyfile /etc/caddy/Caddyfile && chmod 644 /etc/caddy/Caddyfilecp root/server-health-check.sh /root/server-health-check.sh && chmod 700 /root/server-health-check.shcp root/restore.sh /root/restore.sh && chmod 700 /root/restore.sh# Create directory structuremkdir -p /srv/git /srv/docker /srv/data /srv/backupmkdir -p /srv/git /srv/docker /srv/datamkdir -p /root/.restic && chmod 700 /root/.restic# Initialize borg backup repositoryborg init -e none /srv/backupecho ""echo "** Place restic credentials before backups will run: **"echo "   /root/.restic/password   restic repo password (chmod 600)"echo "   /root/.restic/b2-env     exports B2_ACCOUNT_ID and B2_ACCOUNT_KEY (chmod 600)"echo ""# Provision each project from projects.confwhile IFS='|' read -r name port repo branch has_data has_migrate; do
added hosts/alpine/root/restore.sh
@@ -0,0 +1,47 @@#!/bin/sh## restore.sh## Restore /srv from the latest snapshot in Backblaze B2. Existing /srv is# moved aside to /root/before-restore-<UTC-ISO>/srv/ first so nothing is lost.## Stops docker before moving /srv (running containers bind-mount /srv/docker/*)# and starts it again after restore completes.## After restore, you'll need to bring projects up manually, e.g.:#     for d in /srv/docker/*; do (cd "$d" && docker compose up --build -d); done#set -eu. /root/.restic/b2-envexport RESTIC_REPOSITORY="b2:overshard-backups:alpine"export RESTIC_PASSWORD_FILE="/root/.restic/password"ARCHIVE="/root/before-restore-$(date -u +%Y-%m-%dT%H-%M-%SZ)"echo "Stopping docker..."rc-service docker stop || trueecho "Moving existing /srv aside to $ARCHIVE/srv"mkdir -p "$ARCHIVE"if [ -d /srv ]; then    mv /srv "$ARCHIVE/srv"fimkdir -p /srvecho "Restoring latest snapshot from $RESTIC_REPOSITORY"restic restore latest --target /echo "Starting docker..."rc-service docker startecho ""echo "Restore complete. Previous /srv archived at:"echo "  $ARCHIVE/srv"echo ""echo "Bring projects up with:"echo "  for d in /srv/docker/*; do (cd \"\$d\" && docker compose up --build -d); done"echo ""echo "Once you've verified everything looks right, you can remove the archive:"echo "  rm -rf $ARCHIVE"
modified hosts/alpine/root/server-health-check.sh
@@ -3,9 +3,12 @@echo -e "\napk upgrades ------------------------------------------------------------------"tail /var/log/apk-autoupgrade.logecho -e "\nborg backups ------------------------------------------------------------------"borg info /srv/backup/ | tail -n5 | head -n3borg list /srv/backupecho -e "\nrestic backups ----------------------------------------------------------------". /root/.restic/b2-envexport RESTIC_REPOSITORY="b2:overshard-backups:alpine"export RESTIC_PASSWORD_FILE="/root/.restic/password"restic stats latest 2>/dev/null | grep -E "Snapshot|Total File Count|Total Size"restic snapshots --compact 2>/dev/null | tail -n5echo -e "\nfree memory  ------------------------------------------------------------------"free -h | head -n2