diff options
| author | Lukasz Kasprzak <lukasz.kasprzak@pm.me> | 2026-03-19 17:26:34 +0100 |
|---|---|---|
| committer | Lukasz Kasprzak <lukasz.kasprzak@pm.me> | 2026-03-19 17:26:34 +0100 |
| commit | 0ad332db7e3ea4f0fce0f4db332193883ebec254 (patch) | |
| tree | 967e6ea58cd1109ca3216fb395281018bee16bd8 /straper | |
| parent | b1844d805a8f190af00faf3f6e5bed7d997ecaae (diff) | |
| download | bin-0ad332db7e3ea4f0fce0f4db332193883ebec254.tar.gz bin-0ad332db7e3ea4f0fce0f4db332193883ebec254.zip | |
modified sorter and added lint-db to compare it against the db
Diffstat (limited to 'straper')
| -rw-r--r-- | straper/.restore-configs.sh.swp | bin | 0 -> 53248 bytes | |||
| -rw-r--r-- | straper/README.md | 594 | ||||
| -rwxr-xr-x | straper/capture-full.sh | 980 | ||||
| -rwxr-xr-x[-rw-r--r--] | straper/common.sh | 0 | ||||
| -rwxr-xr-x[-rw-r--r--] | straper/doctor.sh | 0 | ||||
| -rwxr-xr-x[-rw-r--r--] | straper/install-base.sh | 0 | ||||
| -rwxr-xr-x | straper/lint-db.sh | 175 | ||||
| -rw-r--r-- | straper/logs/sanctum-rebuild-toolkist-2026-03-19T15:14:13.log | 325 | ||||
| -rwxr-xr-x[-rw-r--r--] | straper/restore-configs.sh | 321 |
9 files changed, 2282 insertions, 113 deletions
diff --git a/straper/.restore-configs.sh.swp b/straper/.restore-configs.sh.swp Binary files differnew file mode 100644 index 0000000..68b8429 --- /dev/null +++ b/straper/.restore-configs.sh.swp diff --git a/straper/README.md b/straper/README.md index a076f8f..b5df645 100644 --- a/straper/README.md +++ b/straper/README.md @@ -1,123 +1,579 @@ +Below is a rewritten `README.md` that matches the **actual suite**, the **actual workflow**, and the **validated lessons** from today. + +````markdown # sanctum / labunix rebuild toolkit -A modular, idempotent, Unix-style rebuild toolkit designed to reduce recovery time from weeks to hours. +A modular rebuild toolkit for restoring a Debian 13 system in a controlled order: + +1. **capture** +2. **lint** +3. **install** +4. **restore** +5. **doctor** + +The goal is to reduce rebuild time from weeks to hours while keeping risky state changes explicit, reversible, and testable. + +--- + +## What this suite is for + +This toolkit is for rebuilding a server such as `sanctum` onto: + +- a **lab VM** for safe testing +- a **hardware-like machine** for conservative migration +- a **replacement target** after real failure -## Design goals +It is designed to separate: -- Separate **install**, **restore**, and **verify**. -- Keep base connectivity stable before touching overlay networking. -- Make secrets and identities opt-in. -- Never abort the whole run because one restore item failed. -- Back up before overwrite and write a machine-readable report. -- Work on both **VM** and **real hardware**. +- **base OS/bootstrap** +- **captured configuration state** +- **service-by-service restore** +- **read-only validation** -## Files +This suite is intentionally **interactive by default** and does **not** assume that every category should be restored in one step. -- `lib/common.sh` — shared helpers -- `install-base.sh` — installs packages and base structure only -- `restore-configs.sh` — restores configs from `capture-full.sh` database +--- + +## Current toolkit components + +- `capture-full.sh` — captures the source machine into a rebuild DB +- `lint-db.sh` — checks the DB for structural problems before restore +- `install-base.sh` — installs packages and generic baseline only +- `restore-configs.sh` — restores selected categories from the DB - `doctor.sh` — read-only health and readiness checks +- `common.sh` — shared helpers and runtime/reporting functions + +--- -## Defaults +## Core design rules -- Safe by default -- No automatic identity transplant -- No blind `/etc/network` overwrite unless explicitly requested -- No immutable `/etc/resolv.conf` -- No automatic WireGuard activation unless explicitly requested +- Keep **capture**, **restore**, and **verify** separate. +- Prefer **fresh VMs or snapshots** when testing restore behavior. +- Restore **category-by-category**, not all at once. +- Do not let one failed item abort the whole restore run unless explicitly required. +- Back up before overwrite. +- Write machine-readable reports and state files. +- Keep **identity/state restores** deliberate. +- Do not touch risky network/DNS/firewall state early in a rebuild. +- Treat the DB as a contract: **capture must produce a clean tree before restore can be trusted**. + +--- ## Roles -- `lab` — VM / test clone, least invasive -- `hardware` — real machine, still conservative -- `replacement` — real disaster-recovery target, identities allowed when requested +### `lab` +Use for: +- VM validation +- dry runs +- low-risk service restore tests + +Behavior: +- conservative +- identity-heavy/private restores are skipped or limited +- network/DNS/firewall restore is intentionally deferred + +### `hardware` +Use for: +- real machine migration when you still want to stay conservative -## Recommended recovery order +Behavior: +- broader than `lab` +- still avoids blindly assuming full disaster-recovery replacement semantics -### 1. Base install +### `replacement` +Use for: +- real rebuild after failure +- full recovery target where identities and service state may need to be restored + +Behavior: +- allows more sensitive state restores when explicitly requested +- should be used only after DB quality and restore flow have already been validated + +--- + +## Canonical workflow + +## 1. On the source host: capture the DB + +Create a fresh rebuild DB from the live server: ```bash -sudo ./install-base.sh --role replacement --profile core --start-safe-services +sudo ~/.local/bin/straper/capture-full.sh +```` + +To capture into a different path: + +```bash +sudo CAPTURE_DIR=/tmp/sanctum-rebuild-clean ~/.local/bin/straper/capture-full.sh ``` -### 2. Restore essentials first +This creates a rebuild database with: + +* `db/public/` +* `db/secret/` + +### Important + +Capture should be run on the **real source machine**, not on the test VM. + +--- + +## 2. Lint the DB before trusting it + +Always lint the DB after capture and before restore: ```bash -sudo ./restore-configs.sh \ - --role replacement \ +sudo ~/.local/bin/straper/lint-db.sh --db-dir /tmp/sanctum-rebuild-clean +``` + +The linter is meant to catch DB-shape problems such as: + +* nested duplicate roots like `nginx/nginx` or `i2pd/i2pd` +* overlapping category captures +* captured backup directories like `*.bak.*` +* duplicate canonical files at two depths + +### Rule + +If lint fails, do **not** treat that DB as canonical for restore. + +--- + +## 3. Move toolkit and DB to the target machine + +Typical pattern: + +### Copy toolkit + +```bash +rsync -av --delete ~/.local/bin/straper/ lukasz@TARGET:~/.local/bin/straper/ +``` + +### Copy DB + +```bash +rsync -av /tmp/sanctum-rebuild-clean/db/ lukasz@TARGET:~/sanctum-rebuild/db/ +``` + +The target machine should then have: + +* toolkit in `~/.local/bin/straper/` +* DB in `~/sanctum-rebuild/db/` + +--- + +## 4. Install base system first + +On the target machine: + +```bash +cd ~/.local/bin/straper +sudo bash ./install-base.sh --role lab --profile core +``` + +For real rebuilds, use the appropriate role: + +```bash +sudo bash ./install-base.sh --role replacement --profile core +``` + +### What `install-base.sh` is for + +It prepares: + +* package set +* generic base services +* baseline directories and runtime assumptions + +It is **not** meant to transplant full server identity by itself. + +--- + +## 5. Restore configs category-by-category + +Do **not** restore everything blindly. + +Use `restore-configs.sh` in blocks, validating after each block. + +### Safe first block + +```bash +sudo bash ./restore-configs.sh \ + --db-dir /home/lukasz/sanctum-rebuild \ + --role lab \ --category system-basics \ --category users \ - --category ssh \ - --category apt-sources \ - --start-services + --category ssh ``` -### 3. Restore network/DNS/firewall carefully +### Service block ```bash -sudo ./restore-configs.sh \ +sudo bash ./restore-configs.sh \ + --db-dir /home/lukasz/sanctum-rebuild \ + --role lab \ + --category nginx \ + --category mariadb \ + --category postfix \ + --category prosody +``` + +### Additional validated categories + +```bash +sudo bash ./restore-configs.sh \ + --db-dir /home/lukasz/sanctum-rebuild \ + --role lab \ + --category monitoring \ + --category docker \ + --category tor \ + --category i2pd +``` + +### Risky categories to leave for later + +These should be handled only after the machine is already stable: + +* `network` +* `dns` +* `firewall` + +And for replacement scenarios, also: + +* SSH host keys +* `/var/lib/tor` +* `/var/lib/i2pd` + +--- + +## 6. Run doctor after each block + +Use `doctor.sh` after install and after meaningful restore steps: + +```bash +sudo bash ./doctor.sh --db-dir /home/lukasz/sanctum-rebuild --role lab +``` + +This gives a read-only summary of: + +* system readiness +* config validation +* service state +* warning profile +* current run state file + +--- + +## Validated restore order + +The currently validated rebuild order is: + +1. **capture on source host** +2. **lint the DB** +3. **move toolkit and DB to target** +4. **run `install-base.sh`** +5. **run `restore-configs.sh` category-by-category** +6. **run `doctor.sh`** +7. **only then move into risky network/identity categories** + +This is the canonical order to follow unless there is a strong reason to deviate. + +--- + +## Current restore categories + +`restore-configs.sh --list-categories` currently supports: + +* `system-basics` +* `users` +* `ssh` +* `network` +* `dns` +* `firewall` +* `nginx` +* `mariadb` +* `postfix` +* `prosody` +* `tor` +* `i2pd` +* `docker` +* `monitoring` + +--- + +## What has been validated in `lab` + +The following categories have been exercised successfully in `lab`: + +* `system-basics` +* `users` +* `ssh` +* `nginx` +* `mariadb` +* `postfix` +* `prosody` +* `docker` +* `tor` +* `i2pd` +* `monitoring` + +### Important caveat + +For tree-category retests on a **reused VM**, stale files may survive because tree restores currently behave like **overlay restores**, not strict replacements. + +That means: + +* use a **fresh VM/snapshot** when possible, or +* move the old destination tree aside before re-testing a tree category + +Examples of tree categories affected by this testing rule: + +* `nginx` +* `monitoring` +* `i2pd` +* `mariadb` + +This is a testing-method concern, not necessarily a DB defect. + +--- + +## Public vs secret DB tiers + +## `db/public` + +Safe, non-secret inventory and configs intended for rebuild structure and general restore use. + +## `db/secret` + +Sensitive material such as: + +* SSH keys +* TLS private keys +* service secrets +* database credentials +* Tor/I2P private state +* user shell configs that may contain tokens + +### Rule + +Do **not** commit `db/secret` to git. + +--- + +## Reports, backups, and state + +The suite writes runtime artifacts under: + +* `/var/log/labunix-rebuild/` +* `/var/lib/labunix-rebuild/` + +Common outputs include: + +* report TSV files +* doctor output +* state files +* per-run backups of overwritten targets + +### Typical state file + +* `/var/lib/labunix-rebuild/state-<RUN_ID>.env` + +### Typical report file + +* `/var/log/labunix-rebuild/report-<RUN_ID>.tsv` + +--- + +## Lab VM workflow + +Use this when validating on a fresh VM. + +### On source host + +```bash +sudo CAPTURE_DIR=/tmp/sanctum-rebuild-clean ~/.local/bin/straper/capture-full.sh +sudo ~/.local/bin/straper/lint-db.sh --db-dir /tmp/sanctum-rebuild-clean +``` + +### Copy to VM + +```bash +rsync -av ~/.local/bin/straper/ lukasz@VM:~/rebuild-test/sanctum-rebuild-toolkit/ +rsync -av /tmp/sanctum-rebuild-clean/db/ lukasz@VM:~/sanctum-rebuild/db/ +``` + +### On VM + +```bash +cd ~/rebuild-test/sanctum-rebuild-toolkit +sudo bash ./install-base.sh --role lab --profile core +``` + +Then restore in blocks and run `doctor.sh` after each block. + +--- + +## Hardware / replacement workflow + +Use this when preparing a real migration or real recovery target. + +### On source machine + +Capture and lint first: + +```bash +sudo ~/.local/bin/straper/capture-full.sh +sudo ~/.local/bin/straper/lint-db.sh --db-dir /srv/sanctum-rebuild +``` + +### On target + +Install base: + +```bash +sudo bash ./install-base.sh --role replacement --profile core +``` + +Restore safe categories first: + +```bash +sudo bash ./restore-configs.sh \ + --db-dir /srv/sanctum-rebuild \ --role replacement \ - --category network-base \ - --category dns \ - --category firewall \ - --network-mode source \ - --dns-mode chain \ - --start-services + --category system-basics \ + --category users \ + --category ssh ``` -### 4. Restore services +Then service categories: ```bash -sudo ./restore-configs.sh \ +sudo bash ./restore-configs.sh \ + --db-dir /srv/sanctum-rebuild \ --role replacement \ --category nginx \ --category mariadb \ --category postfix \ --category prosody \ --category docker \ - --restore-secrets \ - --start-services + --category monitoring ``` -### 5. Restore identities last +Only after the machine is stable should you move into: + +* `network` +* `dns` +* `firewall` +* identity/state-heavy restores + +Then run: ```bash -sudo ./restore-configs.sh \ - --role replacement \ - --category privacy \ - --category tls \ - --category identities \ - --restore-secrets \ - --restore-identities \ - --start-services +sudo bash ./doctor.sh --db-dir /srv/sanctum-rebuild --role replacement +``` + +--- + +## What this toolkit deliberately does not assume + +* that a dirty VM is a valid proof environment for repeated tree restores +* that network restore is safe early +* that secrets/identities should be transplanted automatically +* that the DB is trustworthy unless it passes lint +* that one giant all-at-once restore is the right recovery method + +--- + +## Known lessons from validation + +### 1. Metadata matters + +Restoring content is not enough. Sensitive files and trees also need: + +* owner +* group +* mode + +### 2. DB shape matters + +A structurally polluted DB can create duplicate nested restore trees even if restore logic is otherwise correct. + +### 3. Tree restores are not strict replacement + +Current behavior is overlay-style unless the destination is cleaned first. + +### 4. Testing method matters + +For tree-category retests: + +* use a fresh VM/snapshot, or +* move the old destination aside first + +--- + +## Current recommended stopping rule + +A rebuild step is “good enough to proceed” when: + +* DB passes `lint-db.sh` +* `install-base.sh` completes +* restore block completes without new obvious regressions +* `doctor.sh` baseline remains stable +* service-specific validation passes where relevant + +--- + +## Minimal quick-reference + +### Capture + +```bash +sudo ~/.local/bin/straper/capture-full.sh ``` -### 6. Verify +### Lint ```bash -sudo ./doctor.sh --role replacement --strict +sudo ~/.local/bin/straper/lint-db.sh --db-dir /srv/sanctum-rebuild ``` -## Reports and state +### Install base + +```bash +sudo bash ./install-base.sh --role lab --profile core +``` + +### Restore safe block + +```bash +sudo bash ./restore-configs.sh \ + --db-dir /home/lukasz/sanctum-rebuild \ + --role lab \ + --category system-basics \ + --category users \ + --category ssh +``` + +### Verify + +```bash +sudo bash ./doctor.sh --db-dir /home/lukasz/sanctum-rebuild --role lab +``` -Each run writes: +--- -- `/var/log/labunix-rebuild/report-<RUN_ID>.tsv` -- `/var/log/labunix-rebuild/doctor-<RUN_ID>.txt` -- `/var/lib/labunix-rebuild/state-<RUN_ID>.env` -- `/var/lib/labunix-rebuild/backups/<RUN_ID>/...` +## Status -## Notes +This suite is now suitable for: -- `install-base.sh` does **not** restore your server identity. -- `restore-configs.sh` is intentionally interactive unless `--yes` is used. -- `doctor.sh` is read-only. -- For network restore, use local console access when possible. +* structured lab rebuild testing +* staged migration preparation +* disciplined disaster-recovery rehearsal +It should still be used with: -## Latest patch notes +* staged restores +* DB linting +* post-step validation +* caution around risky network and identity layers -- v0.1.1 fixes early exit in `load_optional_config()` under `set -e` -- `install-base.sh` now has a bootstrap DNS fallback for broken `systemd-resolved` stub setups -- initial `apt-get update` is now validated and fails loudly instead of being reported as success -- locale handling no longer requires locale tools before base packages install them diff --git a/straper/capture-full.sh b/straper/capture-full.sh new file mode 100755 index 0000000..a8f0e3f --- /dev/null +++ b/straper/capture-full.sh @@ -0,0 +1,980 @@ +#!/usr/bin/env bash +# ============================================================================= +# capture-full.sh — sanctum full-state capture with encrypted secret database +# ============================================================================= +# Captures the complete state of sanctum into two tiers: +# +# db/public/ — safe, unencrypted inventory (same as capture-state.sh) +# db/secret/ — ALL secrets, root-only (chmod 700), never committed to git +# +# Together these two tiers contain everything needed to fully reproduce +# this machine on a fresh Debian 13 minimal install via bootstrap.sh. +# +# USAGE +# sudo ./capture-full.sh [--dry-run] [--help] +# +# OPTIONS +# --dry-run Show what would be captured without writing anything +# --help Show this message +# +# REQUIREMENTS +# - Must be run as root (sudo) +# - GPG must be installed: apt install gnupg +# - GPG_RECIPIENT must be set (key fingerprint or email), OR +# set it in /etc/labunix/capture.conf or ~/.config/labunix/capture.conf +# - jq must be installed: apt install jq +# - Output goes to CAPTURE_DIR (default: /srv/sanctum-rebuild) +# +# OUTPUT STRUCTURE +# /srv/sanctum-rebuild/ +# ├── capture-full.sh this script (self-referential copy) +# ├── bootstrap.sh reproduction script (generated separately) +# ├── db/ +# │ ├── public/ unencrypted inventory — git-safe +# │ │ ├── system/ +# │ │ ├── packages/ +# │ │ ├── services/ +# │ │ ├── docker/ +# │ │ ├── network/ +# │ │ ├── firewall/ +# │ │ └── ... +# │ ├── secret/ GPG-encrypted — NEVER commit to git +# │ │ ├── secrets.tar.gpg all secret material in one encrypted archive +# │ │ └── secrets.index unencrypted index of what is inside (no values) +# │ └── README.md +# └── README.md +# +# SECURITY MODEL +# - db/public/ : safe to push to private git repo +# - db/secret/ : local only, root-only permissions (chmod 700) +# protected by ZFS-on-LUKS disk encryption +# - The .gitignore in CAPTURE_DIR excludes db/secret/ entirely +# - bootstrap.sh reads db/public/ for structure and db/secret/ for values +# ============================================================================= + +set -euo pipefail + +# --------------------------------------------------------------------------- # +# CONFIGURATION +# --------------------------------------------------------------------------- # +CAPTURE_DIR="${CAPTURE_DIR:-/srv/sanctum-rebuild}" +PUBLIC_DIR="${CAPTURE_DIR}/db/public" +SECRET_DIR="${CAPTURE_DIR}/db/secret" +TIMESTAMP="$(date +"%Y-%m-%dT%H:%M:%S")" +HOSTNAME_SHORT="$(hostname -s)" +DRY_RUN=false +WARNINGS=() + +# Load config file if present +for conf in /etc/labunix/capture.conf "${HOME}/.config/labunix/capture.conf"; do + [[ -f "$conf" ]] && source "$conf" || true +done + +# --------------------------------------------------------------------------- # +# ARGUMENT PARSING +# --------------------------------------------------------------------------- # +for arg in "$@"; do + case "$arg" in + --dry-run) DRY_RUN=true ;; + --help) + grep '^#' "$0" | head -40 | sed 's/^# \?//' + exit 0 + ;; + *) echo "Unknown option: $arg" >&2; exit 1 ;; + esac +done + +# --------------------------------------------------------------------------- # +# HELPERS +# --------------------------------------------------------------------------- # +log() { echo " [$(date +%H:%M:%S)] $*"; } +warn() { echo " [$(date +%H:%M:%S)] WARN: $*" >&2; WARNINGS+=("$*"); } +section() { echo ""; echo "── $* ──"; } +die() { echo "ERROR: $*" >&2; exit 1; } + +# pub <outfile> <cmd...> +# Capture command output into public (unencrypted) tier. +pub() { + local out="${PUBLIC_DIR}/$1"; shift + if $DRY_RUN; then echo " [pub] $* → ${out#$PUBLIC_DIR/}"; return; fi + mkdir -p "$(dirname "$out")" + if ! "$@" > "$out" 2>&1; then + warn "command failed: $*" + echo "# ERROR: '$*' failed at ${TIMESTAMP}" > "$out" + fi +} + +# pub_copy <src> <dst_relative> +# Copy file/dir into public tier. +pub_copy() { + local src="$1" dst="${PUBLIC_DIR}/$2" + if $DRY_RUN; then echo " [pub] copy ${src} → $2"; return; fi + [[ -e "$src" ]] || return 0 + mkdir -p "$(dirname "$dst")" + cp -a "$src" "$dst" 2>/dev/null || warn "could not copy ${src}" +} + +# sec <dst_relative> <src_path> +# Copy secret material directly into the secret directory. +sec() { + local dst="${SECRET_DIR}/$1" src="$2" + if $DRY_RUN; then echo " [sec] ${src} → secret/$1"; return; fi + [[ -e "$src" ]] || return 0 + mkdir -p "$(dirname "$dst")" + cp -a "$src" "$dst" 2>/dev/null || warn "could not stage secret: ${src}" +} + +# sec_val <dst_relative> <value> +# Write a literal secret value as a file. +sec_val() { + local dst="${SECRET_DIR}/$1" + if $DRY_RUN; then echo " [sec] value → secret/$1"; return; fi + mkdir -p "$(dirname "$dst")" + printf '%s\n' "$2" > "$dst" +} + +# sec_cmd <dst_relative> <cmd...> +# Capture command output as a secret. +sec_cmd() { + local dst="${SECRET_DIR}/$1"; shift + if $DRY_RUN; then echo " [sec] command → secret/$1"; return; fi + mkdir -p "$(dirname "$dst")" + if ! "$@" > "$dst" 2>&1; then + warn "secret command failed: $*" + echo "# ERROR: '$*' failed" > "$dst" + fi +} + +# index_secret <path> <description> +# Add an entry to the secrets index (unencrypted — shows what exists, not values). +SECRET_INDEX_ENTRIES=() +index_secret() { + SECRET_INDEX_ENTRIES+=("$(printf '%-50s %s' "$1" "$2")") +} + +# --------------------------------------------------------------------------- # +# PRE-FLIGHT CHECKS +# --------------------------------------------------------------------------- # +[[ $EUID -eq 0 ]] || die "run with sudo" + +if ! $DRY_RUN; then + command -v jq &>/dev/null || die "jq not found — apt install jq" + command -v tar &>/dev/null || die "tar not found" + + mkdir -p "${PUBLIC_DIR}" "${SECRET_DIR}" + chmod 700 "${SECRET_DIR}" +fi + +echo "" +echo "╔══════════════════════════════════════════════════════╗" +echo "║ labunix · capture-full — full state + secret DB ║" +echo "╚══════════════════════════════════════════════════════╝" +echo "" +log "host : ${HOSTNAME_SHORT}" +log "timestamp : ${TIMESTAMP}" +log "public : ${PUBLIC_DIR}" +log "secret : ${SECRET_DIR}/ (root-only, chmod 700)" +$DRY_RUN && log "(DRY RUN — nothing will be written)" + +# =========================================================================== # +# PUBLIC TIER — safe inventory, no secret values +# =========================================================================== # + +# --------------------------------------------------------------------------- # +# P1. SYSTEM IDENTITY +# --------------------------------------------------------------------------- # +section "P1 system" + +pub "system/hostname.txt" hostname -f +pub "system/os-release.txt" cat /etc/os-release +pub "system/kernel.txt" uname -a +pub "system/lscpu.txt" lscpu +pub "system/memory.txt" free -h +pub "system/lsblk.txt" lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,UUID,LABEL +pub "system/df.txt" df -hT --exclude-type=tmpfs --exclude-type=devtmpfs +pub "system/locale.txt" locale +pub "system/timedatectl.txt" timedatectl +pub "system/uptime.txt" uptime +pub "system/dmesg-err.txt" dmesg --level=err,crit,alert,emerg + +pub_copy /etc/hostname "system/etc-hostname" +pub_copy /etc/hosts "system/etc-hosts" +pub_copy /etc/fstab "system/etc-fstab" +pub_copy /etc/crypttab "system/etc-crypttab" +pub_copy /etc/environment "system/etc-environment" +pub_copy /etc/locale.gen "system/locale.gen" +pub_copy /etc/timezone "system/timezone" + +log "done" + +# --------------------------------------------------------------------------- # +# P2. ZFS +# --------------------------------------------------------------------------- # +section "P2 zfs" + +if command -v zpool &>/dev/null; then + pub "zfs/zpool-status.txt" zpool status -v + pub "zfs/zpool-list.txt" zpool list -v + pub "zfs/zpool-history.txt" zpool history + pub "zfs/zfs-list.txt" zfs list -t all -o name,used,avail,refer,mountpoint,type + pub "zfs/zfs-properties.txt" zfs get all + pub "zfs/zfs-snapshots.txt" zfs list -t snapshot -o name,used,creation -s creation + pub_copy /etc/zfs "zfs/etc-zfs" + log "done" +else + warn "zpool not found — skipping ZFS" +fi + +# --------------------------------------------------------------------------- # +# P3. NETWORK +# --------------------------------------------------------------------------- # +section "P3 network" + +pub "network/ip-addr.txt" ip -s addr show +pub "network/ip-route.txt" ip route show table all +pub "network/ip-rule.txt" ip rule show +pub "network/ss-listening.txt" ss -tlnpue + +pub_copy /etc/network "network/etc-network" +pub_copy /etc/netplan "network/etc-netplan" +pub_copy /etc/systemd/network "network/systemd-network" +pub_copy /etc/nsswitch.conf "network/nsswitch.conf" +pub_copy /etc/hosts.allow "network/hosts.allow" +pub_copy /etc/hosts.deny "network/hosts.deny" + +log "done" + +# --------------------------------------------------------------------------- # +# P4. FIREWALL +# --------------------------------------------------------------------------- # +section "P4 firewall" + +pub "firewall/nft-ruleset.txt" nft list ruleset +pub "firewall/nft-ruleset.json" nft -j list ruleset + +pub_copy /etc/nftables.conf "firewall/nftables.conf" +pub_copy /etc/nftables.d "firewall/nftables.d" + +log "done" + +# --------------------------------------------------------------------------- # +# P5. WIREGUARD — public info only +# --------------------------------------------------------------------------- # +section "P5 wireguard" + +if command -v wg &>/dev/null; then + pub "wireguard/wg-show.txt" wg show + pub "wireguard/wg-interfaces.txt" ip link show type wireguard + + # Redacted public copy (PrivateKey + PresharedKey scrubbed) + if ! $DRY_RUN && [[ -d /etc/wireguard ]]; then + mkdir -p "${PUBLIC_DIR}/wireguard/etc-wireguard" + for f in /etc/wireguard/*.conf; do + [[ -f "$f" ]] || continue + base="$(basename "$f")" + sed \ + -e 's/^\(PrivateKey\s*=\s*\).*/\1<REDACTED>/' \ + -e 's/^\(PresharedKey\s*=\s*\).*/\1<REDACTED>/' \ + "$f" > "${PUBLIC_DIR}/wireguard/etc-wireguard/${base}" 2>/dev/null \ + || warn "could not sanitize ${f}" + done + log "public: PrivateKey + PresharedKey redacted" + fi + log "done" +else + warn "wg not found — skipping WireGuard" +fi + +# --------------------------------------------------------------------------- # +# P6. DNS +# --------------------------------------------------------------------------- # +section "P6 dns" + +if [[ -f /etc/dnsmasq.conf ]] || command -v dnsmasq &>/dev/null; then + command -v dnsmasq &>/dev/null && pub "dns/dnsmasq-version.txt" dnsmasq --version + pub_copy /etc/dnsmasq.conf "dns/dnsmasq.conf" + pub_copy /etc/dnsmasq.d "dns/dnsmasq.d" +fi + +if [[ -d /etc/unbound ]]; then + command -v unbound &>/dev/null && pub "dns/unbound-version.txt" unbound -V + pub_copy /etc/unbound "dns/etc-unbound" +fi + +pub_copy /etc/resolv.conf "dns/resolv.conf" + +log "done" + +# --------------------------------------------------------------------------- # +# P7. TLS — cert metadata only (no key material) +# --------------------------------------------------------------------------- # +section "P7 tls" + +if command -v certbot &>/dev/null; then + pub "tls/certbot-version.txt" certbot --version + pub "tls/certbot-certificates.txt" certbot certificates +fi + +pub_copy /etc/letsencrypt/renewal "tls/renewal-configs" +pub_copy /etc/letsencrypt/cli.ini "tls/cli.ini" + +# Cert expiry summary +if ! $DRY_RUN && [[ -d /etc/letsencrypt/live ]]; then + { + printf "# TLS certificate expiry — %s\n" "${TIMESTAMP}" + printf "# %-40s %-28s %-28s %s\n" "domain" "not_before" "not_after" "days_left" + echo "" + for cert in /etc/letsencrypt/live/*/fullchain.pem; do + [[ -f "$cert" ]] || continue + domain="$(basename "$(dirname "$cert")")" + not_before="$(openssl x509 -noout -startdate -in "$cert" 2>/dev/null | cut -d= -f2)" + not_after="$(openssl x509 -noout -enddate -in "$cert" 2>/dev/null | cut -d= -f2)" + expiry_epoch="$(date -d "$not_after" +%s 2>/dev/null || echo 0)" + days_left=$(( (expiry_epoch - $(date +%s)) / 86400 )) + printf "%-42s %-28s %-28s %d\n" \ + "$domain" "$not_before" "$not_after" "$days_left" + done + } > "${PUBLIC_DIR}/tls/cert-expiry.txt" +fi + +log "done" + +# --------------------------------------------------------------------------- # +# P8. NGINX — config (may reference secrets but config itself is public) +# --------------------------------------------------------------------------- # +section "P8 nginx" + +if command -v nginx &>/dev/null; then + pub "nginx/nginx-version.txt" nginx -v + pub "nginx/nginx-T.txt" nginx -T + pub "nginx/nginx-t.txt" nginx -t + pub_copy /etc/nginx "nginx/etc-nginx" + + if ! $DRY_RUN; then + # Remove auth files from the public copy + find "${PUBLIC_DIR}/nginx/etc-nginx" \ + \( -name "*.htpasswd" -o -name "htpasswd-*" -o -name ".htpasswd" -o -name "lufi.htpasswd" \) \ + -delete 2>/dev/null || true + + # Remove local backup trees from the public copy + find "${PUBLIC_DIR}/nginx/etc-nginx" \ + -maxdepth 1 -type d -name 'sites-available.bak.*' \ + -exec rm -rf {} + 2>/dev/null || true + + log "htpasswd files removed from public nginx copy" + log "nginx backup dirs removed from public copy" + fi + + log "done" +else + warn "nginx not found" +fi + +# --------------------------------------------------------------------------- # +# P9. DOCKER — sanitized inventory +# --------------------------------------------------------------------------- # +section "P9 docker" + +if command -v docker &>/dev/null; then + pub "docker/docker-version.txt" docker version + pub "docker/docker-info.txt" docker info + pub "docker/docker-ps.txt" docker ps -a \ + --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}" + pub "docker/docker-images.txt" docker images \ + --format "table {{.Repository}}\t{{.Tag}}\t{{.ID}}\t{{.Size}}\t{{.CreatedAt}}" + pub "docker/docker-volumes.txt" docker volume ls + pub "docker/docker-networks.txt" docker network ls \ + --format "table {{.Name}}\t{{.Driver}}\t{{.Scope}}\t{{.ID}}" + pub "docker/docker-compose-ls.txt" docker compose ls 2>/dev/null || true + pub_copy /etc/docker/daemon.json "docker/daemon.json" + + # Inspect with Env[] stripped + if ! $DRY_RUN && command -v jq &>/dev/null; then + mkdir -p "${PUBLIC_DIR}/docker/inspect" + docker ps -aq 2>/dev/null | while read -r cid; do + name="$(docker inspect --format '{{.Name}}' "$cid" \ + | tr '/' '_' | sed 's/^_//')" + docker inspect "$cid" 2>/dev/null \ + | jq '.[0] | del(.Config.Env, .HostConfig.Env)' \ + > "${PUBLIC_DIR}/docker/inspect/${name}.json" 2>/dev/null || true + done + log "inspect captured (Env[] stripped)" + fi + + # Compose files — redact secret-looking values + if ! $DRY_RUN; then + mkdir -p "${PUBLIC_DIR}/docker/compose-redacted" + find /srv /opt /home /root -maxdepth 3 \ + \( -name "docker-compose.yml" -o -name "docker-compose.yaml" \ + -o -name "compose.yml" -o -name "compose.yaml" \) \ + -not -path "*/sanctum-rebuild/*" \ + -not -path "*/labunix-infra/*" \ + -not -path "*/\.*" \ + 2>/dev/null \ + | while read -r f; do + rel="$(echo "$f" | sed 's|^/||; s|/|__|g')" + sed -E \ + -e 's/(PASSWORD|SECRET|TOKEN|API_KEY|APIKEY|AUTH_KEY|PRIVATE_KEY|DB_PASS|MYSQL_PASS|POSTGRES_PASS|REDIS_PASS|SMTP_PASS)[[:space:]]*=[[:space:]]*[^[:space:]#].*/\1=<REDACTED>/gI' \ + -e 's/(PASSWORD|SECRET|TOKEN|API_KEY|APIKEY|AUTH_KEY|PRIVATE_KEY|DB_PASS|MYSQL_PASS|POSTGRES_PASS|REDIS_PASS|SMTP_PASS):[[:space:]]*"[^"]*"/\1: "<REDACTED>"/gI' \ + "$f" > "${PUBLIC_DIR}/docker/compose-redacted/${rel}" 2>/dev/null || true + done + log "compose files collected (secrets redacted)" + fi + + log "done" +else + warn "docker not found" +fi + +# --------------------------------------------------------------------------- # +# P10. PACKAGES +# --------------------------------------------------------------------------- # +section "P10 packages" + +pub "packages/dpkg-list.txt" dpkg -l +pub "packages/dpkg-selections.txt" dpkg --get-selections +pub "packages/apt-mark-manual.txt" apt-mark showmanual +pub "packages/apt-mark-auto.txt" apt-mark showauto + +pub_copy /etc/apt/sources.list "packages/apt/sources.list" +pub_copy /etc/apt/sources.list.d "packages/apt/sources.list.d" +pub_copy /etc/apt/preferences "packages/apt/preferences" +pub_copy /etc/apt/preferences.d "packages/apt/preferences.d" +pub_copy /etc/apt/apt.conf.d "packages/apt/apt.conf.d" + +command -v pip3 &>/dev/null && pub "packages/pip3-list.txt" pip3 list --format=columns +command -v npm &>/dev/null && pub "packages/npm-global.txt" npm list -g --depth=0 +command -v cargo &>/dev/null && pub "packages/cargo-list.txt" cargo install --list +command -v gem &>/dev/null && pub "packages/gem-list.txt" gem list + +log "done" + +# --------------------------------------------------------------------------- # +# P11. SYSTEMD SERVICES +# --------------------------------------------------------------------------- # +section "P11 services" + +pub "services/units-enabled.txt" systemctl list-unit-files --state=enabled --no-pager +pub "services/units-running.txt" systemctl list-units --type=service --state=running --no-pager +pub "services/units-failed.txt" systemctl list-units --state=failed --no-pager +pub "services/timers.txt" systemctl list-timers --all --no-pager +pub "services/unit-files-all.txt" systemctl list-unit-files --no-pager + +if ! $DRY_RUN; then + mkdir -p "${PUBLIC_DIR}/services/custom-units" + for f in /etc/systemd/system/*.service \ + /etc/systemd/system/*.timer \ + /etc/systemd/system/*.socket \ + /etc/systemd/system/*.mount \ + /etc/systemd/system/*.target \ + /etc/systemd/system/*.path; do + [[ -f "$f" ]] || continue + cp "$f" "${PUBLIC_DIR}/services/custom-units/" 2>/dev/null || true + done + for d in /etc/systemd/system/*.d; do + [[ -d "$d" ]] || continue + cp -a "$d" "${PUBLIC_DIR}/services/custom-units/" 2>/dev/null || true + done + log "custom units + drop-ins copied" +fi + +pub_copy /etc/systemd/journald.conf "services/journald.conf" +pub_copy /etc/systemd/logind.conf "services/logind.conf" +pub_copy /etc/systemd/system.conf "services/system.conf" +pub_copy /etc/systemd/timesyncd.conf "services/timesyncd.conf" + +log "done" + +# --------------------------------------------------------------------------- # +# P12. CRON +# --------------------------------------------------------------------------- # +section "P12 cron" + +pub_copy /etc/crontab "cron/crontab" +pub_copy /etc/cron.d "cron/cron.d" +pub_copy /etc/cron.daily "cron/cron.daily" +pub_copy /etc/cron.weekly "cron/cron.weekly" +pub_copy /etc/cron.monthly "cron/cron.monthly" + +log "done" + +# --------------------------------------------------------------------------- # +# P13. USERS — sanitized (no hashes) +# --------------------------------------------------------------------------- # +section "P13 users" + +if ! $DRY_RUN; then + mkdir -p "${PUBLIC_DIR}/users" + awk -F: '{print $1":"$3":"$4":"$5":"$6":"$7}' /etc/passwd \ + > "${PUBLIC_DIR}/users/passwd-sanitized.txt" 2>/dev/null || true + awk -F: '{print $1":"$3}' /etc/group \ + > "${PUBLIC_DIR}/users/groups-sanitized.txt" 2>/dev/null || true + grep -E '(/bin/bash|/bin/zsh|/bin/sh|/usr/bin/zsh)$' /etc/passwd \ + | awk -F: '{print $1" home="$6" shell="$7}' \ + > "${PUBLIC_DIR}/users/login-users.txt" 2>/dev/null || true +fi + +pub_copy /etc/ssh/sshd_config "users/sshd_config" +pub_copy /etc/ssh/sshd_config.d "users/sshd_config.d" +pub_copy /etc/sudoers "users/sudoers" +pub_copy /etc/sudoers.d "users/sudoers.d" +pub_copy /etc/shells "users/shells" +pub_copy /etc/login.defs "users/login.defs" + +log "done" + +# --------------------------------------------------------------------------- # +# P14–P35: remaining public captures (configs without embedded secrets) +# --------------------------------------------------------------------------- # +section "P14-P35 service configs" + +# Kernel +pub "kernel/sysctl-all.txt" sysctl -a +pub "kernel/lsmod.txt" lsmod +pub_copy /etc/sysctl.conf "kernel/sysctl.conf" +pub_copy /etc/sysctl.d "kernel/sysctl.d" +pub_copy /etc/modules "kernel/modules" +pub_copy /etc/modules-load.d "kernel/modules-load.d" +pub_copy /etc/modprobe.d "kernel/modprobe.d" + +# i2pd (no key material) +if [[ -f /etc/i2pd/i2pd.conf ]]; then + pub_copy /etc/i2pd "i2pd/etc-i2pd" +fi + +# Tor (no private keys) +if [[ -f /etc/tor/torrc ]]; then + pub_copy /etc/tor/torrc "tor/torrc" + pub_copy /etc/tor/torrc.d "tor/torrc.d" + if ! $DRY_RUN && [[ -d /var/lib/tor ]]; then + mkdir -p "${PUBLIC_DIR}/tor/hidden-services" + find /var/lib/tor -name "hostname" 2>/dev/null \ + | while read -r hf; do + svc="$(basename "$(dirname "$hf")")" + cp "$hf" "${PUBLIC_DIR}/tor/hidden-services/${svc}.hostname" 2>/dev/null || true + done + fi +fi + +# Hugo +command -v hugo &>/dev/null && pub "hugo/hugo-version.txt" hugo version +for site_dir in /srv/www/labunix /srv/www/labunix.xyz; do + [[ -d "$site_dir" ]] || continue + pub_copy "${site_dir}/hugo.toml" "hugo/hugo.toml" + pub_copy "${site_dir}/config" "hugo/config-dir" + pub_copy "${site_dir}/go.mod" "hugo/go.mod" + break +done + +# Agate +if [[ -f /home/lukasz/apps/agate/agate ]]; then + pub "agate/agate-version.txt" /home/lukasz/apps/agate/agate --version + if ! $DRY_RUN && [[ -d /srv/gemini-capsule ]]; then + find /srv/gemini-capsule -not -path "*/.certificates*" \ + | sort > "${PUBLIC_DIR}/agate/capsule-structure.txt" 2>/dev/null || true + fi +fi + +# Pygopherd +pub_copy /etc/pygopherd "pygopherd/etc-pygopherd" + +# Agate service unit +pub_copy /etc/systemd/system/agate.service "agate/agate.service" + +# Postfix (main.cf is public, sasl_passwd goes to secret) +if [[ -d /etc/postfix ]]; then + pub_copy /etc/postfix/main.cf "postfix/main.cf" + pub_copy /etc/postfix/master.cf "postfix/master.cf" + pub "postfix/postconf.txt" postconf 2>/dev/null || true +fi + +# MariaDB (config only, no credentials) +pub_copy /etc/mysql "mariadb/etc-mysql" +if ! $DRY_RUN; then + mariadb --defaults-file=/etc/mysql/debian.cnf \ + -e "SHOW DATABASES;" 2>/dev/null \ + > "${PUBLIC_DIR}/mariadb/databases.txt" || true +fi + +# Alloy +pub_copy /etc/alloy "alloy/etc-alloy" +pub_copy /etc/default/alloy "alloy/etc-default-alloy" + +# Prometheus +pub_copy /etc/prometheus "prometheus/etc-prometheus" +pub_copy /etc/default/prometheus-node-exporter "prometheus/node-exporter-defaults" + +# Fail2ban +pub_copy /etc/fail2ban "fail2ban/etc-fail2ban" + +# Prosody +pub_copy /etc/prosody "prosody/etc-prosody" + +# Grafana (provisioning configs, no secrets) +pub_copy /etc/grafana "grafana/etc-grafana" + +# Loki + Promtail +pub_copy /etc/loki "loki/etc-loki" +pub_copy /etc/promtail "loki/etc-promtail" + +# Mumble +pub_copy /etc/mumble "mumble/etc-mumble" + +# AppArmor +command -v aa-status &>/dev/null \ + && pub "apparmor/aa-status.txt" aa-status 2>/dev/null || true +pub_copy /etc/apparmor.d "apparmor/apparmor.d" + +# Boot / initramfs — critical for LUKS+ZFS rebuild +pub_copy /etc/default/grub "boot/grub-default" +pub_copy /etc/grub.d "boot/grub.d" +pub_copy /etc/kernel "boot/etc-kernel" +pub_copy /etc/cryptsetup-initramfs "boot/cryptsetup-initramfs" +pub_copy /etc/initramfs-tools "boot/initramfs-tools" +pub_copy /etc/default "boot/etc-default" + +# Lufi +pub_copy /srv/lufi/cpanfile "lufi/cpanfile" +if ! $DRY_RUN && [[ -d /srv/lufi ]]; then + git -C /srv/lufi log --oneline -5 2>/dev/null \ + > "${PUBLIC_DIR}/lufi/git-log.txt" || true +fi + +# Environment +pub_copy /etc/profile "environment/profile" +pub_copy /etc/profile.d "environment/profile.d" +pub_copy /etc/zsh "environment/zsh" + +# Logging +pub_copy /etc/logrotate.conf "logging/logrotate.conf" +pub_copy /etc/logrotate.d "logging/logrotate.d" +pub_copy /etc/rsyslog.conf "logging/rsyslog.conf" +pub_copy /etc/rsyslog.d "logging/rsyslog.d" + +# Hardware +command -v lshw &>/dev/null && pub "hardware/lshw.txt" lshw -short +command -v lspci &>/dev/null && pub "hardware/lspci.txt" lspci -v +command -v lsusb &>/dev/null && pub "hardware/lsusb.txt" lsusb +pub "hardware/cpuinfo.txt" cat /proc/cpuinfo +pub "hardware/meminfo.txt" cat /proc/meminfo + +log "done" + +# =========================================================================== # +# SECRET TIER — full secrets, will be GPG-encrypted +# =========================================================================== # + +section "SECRET TIER" +log "Staging secret material..." + +# ── WireGuard — real private keys ─────────────────────────────────────────── +if [[ -d /etc/wireguard ]]; then + sec "wireguard/etc-wireguard" /etc/wireguard + index_secret "wireguard/etc-wireguard" "WireGuard interface configs with PrivateKey + PresharedKey" +fi + +# ── TLS private keys ───────────────────────────────────────────────────────── +if [[ -d /etc/letsencrypt ]]; then + sec "tls/letsencrypt" /etc/letsencrypt + index_secret "tls/letsencrypt" "Full Let's Encrypt dir including private keys" +fi + +# ── SSH host keys ──────────────────────────────────────────────────────────── +if [[ -d /etc/ssh ]]; then + sec "ssh/etc-ssh" /etc/ssh + index_secret "ssh/etc-ssh" "SSH host keys (id_* private key files)" +fi + +# ── User SSH keys + authorized_keys ───────────────────────────────────────── +if ! $DRY_RUN; then + while IFS=: read -r user _ _ _ _ homedir _; do + [[ -d "${homedir}/.ssh" ]] || continue + sec "ssh/user-${user}" "${homedir}/.ssh" + index_secret "ssh/user-${user}" "SSH keys and authorized_keys for ${user}" + done < /etc/passwd +fi + +# ── Docker compose files — full unredacted ────────────────────────────────── +if ! $DRY_RUN; then + mkdir -p "${SECRET_DIR}/docker/compose-full" + find /srv /opt /home /root -maxdepth 3 \ + \( -name "docker-compose.yml" -o -name "docker-compose.yaml" \ + -o -name "compose.yml" -o -name "compose.yaml" \ + -o -name ".env" -o -name "*.env" \) \ + -not -path "*/sanctum-rebuild/*" \ + -not -path "*/labunix-infra/*" \ + -not -path "*/\.*env.example" \ + 2>/dev/null \ + | while read -r f; do + rel="$(echo "$f" | sed 's|^/||; s|/|__|g')" + cp "$f" "${SECRET_DIR}/docker/compose-full/${rel}" 2>/dev/null || true + done + index_secret "docker/compose-full" "All compose files + .env files with real secrets" +fi + +# ── MariaDB credentials + dumps ────────────────────────────────────────────── +if [[ -f /etc/mysql/debian.cnf ]]; then + sec "mariadb/debian.cnf" /etc/mysql/debian.cnf + index_secret "mariadb/debian.cnf" "MariaDB debian-sys-maint credentials" +fi + +# Dump all database schemas (structure only, no data) — for rebuild reference +if ! $DRY_RUN && command -v mysqldump &>/dev/null; then + mkdir -p "${SECRET_DIR}/mariadb" + mariadb --defaults-file=/etc/mysql/debian.cnf \ + -e "SHOW DATABASES;" 2>/dev/null \ + | grep -v "^Database\|information_schema\|performance_schema\|sys" \ + | while read -r db; do + mysqldump --defaults-file=/etc/mysql/debian.cnf \ + --no-data --routines --triggers \ + "$db" 2>/dev/null \ + > "${SECRET_DIR}/mariadb/schema-${db}.sql" || true + done + index_secret "mariadb/schema-*.sql" "MariaDB schema dumps (structure only, no data)" +fi + +# ── Postfix SASL credentials ───────────────────────────────────────────────── +if [[ -f /etc/postfix/sasl_passwd ]]; then + sec "postfix/sasl_passwd" /etc/postfix/sasl_passwd + index_secret "postfix/sasl_passwd" "Postfix SASL relay credentials" +fi + +# ── GPG keys (export secret keyring) ──────────────────────────────────────── +if ! $DRY_RUN && command -v gpg &>/dev/null; then + mkdir -p "${SECRET_DIR}/gpg" + # Export all secret keys for all users with home dirs + while IFS=: read -r user _ _ _ _ homedir _; do + [[ -d "${homedir}/.gnupg" ]] || continue + # Run gpg as the actual user, not as root, so it can access their keyring + sudo -u "${user}" GNUPGHOME="${homedir}/.gnupg" gpg --batch --yes --export-secret-keys --armor > "${SECRET_DIR}/gpg/${user}-secret-keys.asc" 2>/dev/null || true + sudo -u "${user}" GNUPGHOME="${homedir}/.gnupg" gpg --batch --yes --export --armor > "${SECRET_DIR}/gpg/${user}-public-keys.asc" 2>/dev/null || true + sudo -u "${user}" GNUPGHOME="${homedir}/.gnupg" gpg --batch --yes --export-ownertrust > "${SECRET_DIR}/gpg/${user}-ownertrust.txt" 2>/dev/null || true + done < /etc/passwd + index_secret "gpg/" "GPG secret + public keyrings and ownertrust for all users" +fi + +# ── i2pd destination keys ──────────────────────────────────────────────────── +if [[ -d /var/lib/i2pd ]]; then + sec "i2pd/var-lib-i2pd" /var/lib/i2pd + index_secret "i2pd/var-lib-i2pd" "i2pd full data dir including destination private keys" +fi + +# ── Tor hidden service keys ────────────────────────────────────────────────── +if [[ -d /var/lib/tor ]]; then + sec "tor/var-lib-tor" /var/lib/tor + index_secret "tor/var-lib-tor" "Tor full data dir including hidden service private keys" +fi + +# ── Nginx htpasswd files ───────────────────────────────────────────────────── +if ! $DRY_RUN; then + mkdir -p "${SECRET_DIR}/nginx" + find /etc/nginx -name "*.htpasswd" -o -name "htpasswd-*" -o -name ".htpasswd" \ + 2>/dev/null \ + | while read -r f; do + cp "$f" "${SECRET_DIR}/nginx/" 2>/dev/null || true + done + index_secret "nginx/" "nginx htpasswd files (bcrypt password hashes)" +fi + +# ── User shell configs (may contain tokens/API keys) ──────────────────────── +for user_home in /root /home/lukasz; do + user="$(basename "$user_home")" + [[ "$user" == "root" ]] && user="root" + if ! $DRY_RUN; then + mkdir -p "${SECRET_DIR}/shell/${user}" + for f in .bashrc .zshrc .profile .bash_profile .zprofile \ + .netrc .ssh/config; do + [[ -f "${user_home}/${f}" ]] \ + && cp "${user_home}/${f}" \ + "${SECRET_DIR}/shell/${user}/$(basename "$f")" 2>/dev/null || true + done + fi + index_secret "shell/${user}" "Shell configs for ${user} (may contain tokens/API keys)" +done + +# ── Lufi app config (contains app secret + DB path) ───────────────────────── +if [[ -f /srv/lufi/lufi.conf ]]; then + sec "lufi/lufi.conf" /srv/lufi/lufi.conf + index_secret "lufi/lufi.conf" "Lufi app config (secret key, DB path, domain)" +fi + +# ── Grafana secrets ────────────────────────────────────────────────────────── +if [[ -f /etc/grafana/grafana.ini ]]; then + sec "grafana/grafana.ini" /etc/grafana/grafana.ini + index_secret "grafana/grafana.ini" "Grafana config (admin password, secret key)" +fi + +# ── Alloy config (may contain remote write credentials) ───────────────────── +if [[ -d /etc/alloy ]]; then + sec "alloy/etc-alloy" /etc/alloy + index_secret "alloy/etc-alloy" "Grafana Alloy config (may contain remote write tokens)" +fi + +# ── Borg backup passphrase / config ───────────────────────────────────────── +if [[ -d /etc/borg ]]; then + sec "borg/etc-borg" /etc/borg + index_secret "borg/etc-borg" "Borg backup config (repo paths, passphrase)" +fi +for f in /root/.config/borg/config /root/.borgmatic.yaml /etc/borgmatic.d; do + if [[ -e "$f" ]]; then + sec "borg/$(basename "$f")" "$f" + index_secret "borg/$(basename "$f")" "Borgmatic config" + fi +done + +# ── User crontabs (may contain tokens in commands) ────────────────────────── +if [[ -d /var/spool/cron/crontabs ]]; then + sec "cron/user-crontabs" /var/spool/cron/crontabs + index_secret "cron/user-crontabs" "User crontabs (may contain API keys in commands)" +fi + +# ── Prosody secrets (SSL certs + account data location) ───────────────────── +if [[ -d /etc/prosody ]]; then + sec "prosody/etc-prosody" /etc/prosody + index_secret "prosody/etc-prosody" "Full Prosody config including any embedded credentials" +fi + +# ── LUKS header backup ─────────────────────────────────────────────────────── +# Backing up the LUKS header is critical for recovery if the header gets corrupted +if ! $DRY_RUN && command -v cryptsetup &>/dev/null; then + mkdir -p "${SECRET_DIR}/luks" + # Find LUKS devices + lsblk -o NAME,TYPE,FSTYPE -J 2>/dev/null \ + | jq -r '.blockdevices[] | .. | objects | select(.fstype=="crypto_LUKS") | .name' \ + 2>/dev/null \ + | while read -r dev; do + rm -f "${SECRET_DIR}/luks/${dev}-header.bin" + cryptsetup luksHeaderBackup "/dev/${dev}" \ + --header-backup-file "${SECRET_DIR}/luks/${dev}-header.bin" \ + 2>/dev/null \ + && log "LUKS header backup: /dev/${dev}" \ + || warn "could not backup LUKS header for /dev/${dev}" + done + index_secret "luks/" "LUKS header backups — critical for encrypted volume recovery" +fi + +log "Secret staging complete" + +# =========================================================================== # +# ENCRYPT SECRET TIER +# =========================================================================== # + +section "finalise secrets" + +if ! $DRY_RUN; then + # Lock down permissions — secret dir readable only by root + chmod 700 "${SECRET_DIR}" + find "${SECRET_DIR}" -type f -exec chmod 600 {} \; + find "${SECRET_DIR}" -type d -exec chmod 700 {} \; + log "Secret dir locked: chmod 700 ${SECRET_DIR}" + + log "Secret files written to ${SECRET_DIR}/" +else + log "[dry] secret files would be written to ${SECRET_DIR}/ (chmod 700)" + log "[dry] staging area would be wiped after encryption" +fi + +# =========================================================================== # +# WRITE .gitignore +# =========================================================================== # + +if ! $DRY_RUN; then + cat > "${CAPTURE_DIR}/.gitignore" << 'GITIGNORE' +# Secret tier — NEVER commit (contains private keys, credentials, LUKS headers) +db/secret/ + +# Exception: the index is safe to commit (shows what exists, not values) +!db/secret/secrets.index +GITIGNORE +fi + +# =========================================================================== # +# WRITE db/README.md +# =========================================================================== # + +if ! $DRY_RUN; then + OS_NAME="$(grep PRETTY_NAME /etc/os-release | cut -d= -f2 | tr -d '"')" + cat > "${CAPTURE_DIR}/db/README.md" << EOF +# sanctum — full state database + +| | | +|---|---| +| **Host** | ${HOSTNAME_SHORT} | +| **OS** | ${OS_NAME} | +| **Kernel** | $(uname -r) | +| **Captured** | ${TIMESTAMP} | + +## Structure + +\`\`\` +db/ +├── public/ Safe inventory — no secret values, git-trackable +└── secret/ + ├── <category>/ Secret files in plain subdirectories + └── secrets.index Index of what is inside (no values) +\`\`\` + +## Decrypt secrets + +\`\`\`bash +# List contents without extracting + find /srv/sanctum-rebuild/db/secret -type f | sort + +# Extract to a directory +mkdir /tmp/secrets-out + sudo cat /srv/sanctum-rebuild/db/secret/wireguard/etc-wireguard/wg0.conf + +# Wipe after use +find /tmp/secrets-out -type f -exec shred -u {} \\; +rm -rf /tmp/secrets-out +\`\`\` + +## What is in the secret archive + +See \`db/secret/secrets.index\` for a full list. + +Key items: +- WireGuard private + preshared keys +- TLS private keys (Let's Encrypt) +- SSH host keys + user SSH keys +- GPG secret keyrings (all users) +- Docker .env files and full unredacted compose files +- MariaDB credentials + schema dumps +- i2pd destination private keys +- Tor hidden service private keys +- LUKS header backups +- Lufi app secret, Grafana credentials, Alloy tokens +- Postfix SASL credentials +- nginx htpasswd files + +## Warnings during capture +$(if [[ ${#WARNINGS[@]} -gt 0 ]]; then + for w in "${WARNINGS[@]}"; do echo "- ${w}"; done +else + echo "None" +fi) +EOF +fi + +# =========================================================================== # +# DONE +# =========================================================================== # + +echo "" +echo "╔══════════════════════════════════════════════════════╗" +if $DRY_RUN; then + echo "║ dry run complete — nothing written ║" +else + echo "║ capture-full complete ✓ ║" + echo "║ ║" + printf "║ public: %-42s║\n" "${PUBLIC_DIR}" + printf "║ secret: %-42s║\n" "${SECRET_DIR}/" + if [[ ${#WARNINGS[@]} -gt 0 ]]; then + printf "║ ⚠ %-47s║\n" "${#WARNINGS[@]} warning(s) — see db/README.md" + fi +fi +echo "╚══════════════════════════════════════════════════════╝" +echo "" +echo " Next step: bootstrap.sh will read this database to" +echo " reproduce this machine on a fresh Debian 13 install." +echo "" diff --git a/straper/common.sh b/straper/common.sh index 32f3581..32f3581 100644..100755 --- a/straper/common.sh +++ b/straper/common.sh diff --git a/straper/doctor.sh b/straper/doctor.sh index 6aed53e..6aed53e 100644..100755 --- a/straper/doctor.sh +++ b/straper/doctor.sh diff --git a/straper/install-base.sh b/straper/install-base.sh index e2a2d42..e2a2d42 100644..100755 --- a/straper/install-base.sh +++ b/straper/install-base.sh diff --git a/straper/lint-db.sh b/straper/lint-db.sh new file mode 100755 index 0000000..dc0de37 --- /dev/null +++ b/straper/lint-db.sh @@ -0,0 +1,175 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "$0")" && pwd)" +# shellcheck disable=SC1091 +source "${SCRIPT_DIR}/common.sh" + +SCRIPT_NAME="lint-db.sh" +ISSUES=0 + +usage() { + cat <<'USAGE' +Usage: sudo ./lint-db.sh [options] + +Purpose: + Non-destructively lint a rebuild DB for structural problems that can pollute + restore results. + +Checks: + - backup directories captured into DB (e.g. *.bak.*) + - nested duplicate service roots (e.g. nginx/nginx, i2pd/i2pd) + - overlapping category roots (e.g. etc-i2pd plus sibling tunnels.d) + - duplicate canonical files at two depths within the same category + +Options: + --db-dir PATH + --verbose + --help +USAGE +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --db-dir) DB_DIR="$2"; PUBLIC_DIR="${DB_DIR}/db/public"; SECRET_DIR="${DB_DIR}/db/secret"; shift ;; + --verbose) VERBOSE=true ;; + --help) usage; exit 0 ;; + *) die "unknown option: $1" ;; + esac + shift +done + +load_optional_config +ensure_runtime_dirs + +[[ -d "${PUBLIC_DIR}" ]] || die "public DB not found: ${PUBLIC_DIR}" + +say() { printf '%s\n' "$*"; } +warnx() { printf 'WARN | %s\n' "$*"; ISSUES=1; } +okx() { printf 'OK | %s\n' "$*"; } + +emit_section() { + printf '\n%s\n' "$1" +} + +find_matches() { + local base="$1" pattern="$2" + find "$base" -path "$pattern" -print 2>/dev/null | sort || true +} + +check_backup_dirs() { + local hits + hits="$(find "${PUBLIC_DIR}" "${SECRET_DIR}" \ + -type d -name '*.bak.*' -print 2>/dev/null | sort || true)" + + if [[ -n "${hits}" ]]; then + warnx "captured backup directories found" + printf '%s\n' "${hits}" + else + okx "no captured backup directories found" + fi +} + +check_nested_duplicate_roots() { + local name hits + local names=(nginx i2pd grafana loki prometheus alloy tor postfix prosody mysql docker) + + for name in "${names[@]}"; do + hits="$(find "${PUBLIC_DIR}" "${SECRET_DIR}" \ + -type d -path "*/${name}/${name}" -print 2>/dev/null | sort || true)" + if [[ -n "${hits}" ]]; then + warnx "nested duplicate root '${name}/${name}' found" + printf '%s\n' "${hits}" + fi + done +} + +check_overlapping_i2pd_capture() { + local has_root=false has_tunnels=false has_tunnels_conf=false + + [[ -d "${PUBLIC_DIR}/i2pd/etc-i2pd" ]] && has_root=true + [[ -e "${PUBLIC_DIR}/i2pd/tunnels.d" ]] && has_tunnels=true + [[ -e "${PUBLIC_DIR}/i2pd/tunnels.conf" ]] && has_tunnels_conf=true + + if [[ "${has_root}" == true && ( "${has_tunnels}" == true || "${has_tunnels_conf}" == true ) ]]; then + warnx "overlapping i2pd capture roots found (etc-i2pd plus sibling tunnels paths)" + [[ -d "${PUBLIC_DIR}/i2pd/etc-i2pd" ]] && printf '%s\n' "${PUBLIC_DIR}/i2pd/etc-i2pd" + [[ -e "${PUBLIC_DIR}/i2pd/tunnels.conf" ]] && printf '%s\n' "${PUBLIC_DIR}/i2pd/tunnels.conf" + [[ -e "${PUBLIC_DIR}/i2pd/tunnels.d" ]] && printf '%s\n' "${PUBLIC_DIR}/i2pd/tunnels.d" + else + okx "no overlapping i2pd capture roots found" + fi +} + +check_duplicate_canonical_files() { + emit_section "Duplicate canonical file checks" + + check_dup_pair() { + local a="$1" b="$2" label="$3" + local ha=false hb=false + [[ -e "${a}" ]] && ha=true + [[ -e "${b}" ]] && hb=true + + if [[ "${ha}" == true && "${hb}" == true ]]; then + warnx "${label} exists at two depths" + printf '%s\n%s\n' "${a}" "${b}" + else + okx "${label} not duplicated across checked depths" + fi + } + + check_dup_pair \ + "${PUBLIC_DIR}/prometheus/etc-prometheus/prometheus.yml" \ + "${PUBLIC_DIR}/prometheus/etc-prometheus/prometheus/prometheus.yml" \ + "prometheus.yml" + + check_dup_pair \ + "${PUBLIC_DIR}/loki/etc-loki/config.yml" \ + "${PUBLIC_DIR}/loki/etc-loki/loki/config.yml" \ + "loki config.yml" + + check_dup_pair \ + "${PUBLIC_DIR}/grafana/etc-grafana/grafana.ini" \ + "${PUBLIC_DIR}/grafana/etc-grafana/grafana/grafana.ini" \ + "grafana.ini" + + check_dup_pair \ + "${SECRET_DIR}/alloy/etc-alloy/config.alloy" \ + "${SECRET_DIR}/alloy/etc-alloy/alloy/config.alloy" \ + "alloy config.alloy" + + check_dup_pair \ + "${PUBLIC_DIR}/i2pd/etc-i2pd/i2pd.conf" \ + "${PUBLIC_DIR}/i2pd/etc-i2pd/i2pd/i2pd.conf" \ + "i2pd.conf" + + check_dup_pair \ + "${PUBLIC_DIR}/nginx/etc-nginx/nginx.conf" \ + "${PUBLIC_DIR}/nginx/etc-nginx/nginx/nginx.conf" \ + "nginx.conf" +} + +emit_section "labunix/sanctum rebuild DB lint" +say "db: ${DB_DIR}" +say "public: ${PUBLIC_DIR}" +say "secret: ${SECRET_DIR}" + +emit_section "Backup junk checks" +check_backup_dirs + +emit_section "Nested duplicate root checks" +check_nested_duplicate_roots + +emit_section "Overlapping capture-root checks" +check_overlapping_i2pd_capture + +check_duplicate_canonical_files + +emit_section "Summary" +if [[ ${ISSUES} -eq 0 ]]; then + okx "no DB shape issues detected" + exit 0 +else + warnx "DB shape issues detected" + exit 1 +fi diff --git a/straper/logs/sanctum-rebuild-toolkist-2026-03-19T15:14:13.log b/straper/logs/sanctum-rebuild-toolkist-2026-03-19T15:14:13.log new file mode 100644 index 0000000..502e03a --- /dev/null +++ b/straper/logs/sanctum-rebuild-toolkist-2026-03-19T15:14:13.log @@ -0,0 +1,325 @@ +# sanctum-rebuild-toolkit — lab validation log + +## Date + +* 2026-03-19 + +## Scope + +* Objective: continue category-by-category validation of the Debian 13 disaster-recovery toolkit in a KVM/libvirt lab VM, keeping host `~/.local/bin/straper` as canonical source of truth. +* Role under test: `lab` +* DB under test: `/home/lukasz/sanctum-rebuild` +* VM toolkit path: `~/rebuild-test/sanctum-rebuild-toolkit/` + +## Baseline assumptions + +* Edit only on host in `~/.local/bin/straper`, then sync to VM with `rsync`. +* Test low-risk categories first. +* Defer `network`, `dns`, `firewall`, `tor`, `i2pd`, `docker` category execution in lab. +* Treat some doctor warnings as acceptable in `lab` when they reflect intentionally disabled or absent services. + +## Categories validated in this session + +* `nginx` +* `mariadb` +* `postfix` +* `prosody` + +## Final lab baseline after this session + +* `doctor.sh` summary: `ok=20 warn=9 fail=0 manual=1` +* Validated as working in `lab`: + + * `system-basics` + * `ssh` + * `users` + * `nginx` + * `mariadb` + * `postfix` + * `prosody` + +## Problems found and fixes applied + +### 1. `nginx` restore imported production TLS/vhost state into `lab` + +#### Symptom + +* `sudo nginx -t` failed after restore. +* Failure was due to missing production certificate paths such as: + + * `/etc/letsencrypt/live/labunix.xyz/fullchain.pem` +* Restored tree also contained junk such as: + + * nested `/etc/nginx/nginx/...` + * `sites-available.bak.*` +* `sites-enabled` contained a full production vhost set, including TLS-dependent vhosts. + +#### Root cause + +* `restore_nginx()` restored the captured nginx tree wholesale into `/etc/nginx`. +* No `lab`-specific sanitization existed after restore. + +#### Fix applied + +* Patched `restore_nginx()` on host `restore-configs.sh`. +* In `lab` role, after restore it now: + + * removes `/etc/nginx/nginx` + * removes top-level `sites-available.bak.*` + * clears `sites-enabled` + * re-enables only `sites-available/default` if present + +#### Logic + +* In `lab`, restore enough nginx structure to validate service/config integrity, but do not enable production-facing TLS vhosts. +* Avoid fake cert hacks. +* Keep the change canonical in the host script, not as a VM-only workaround. + +#### Result + +* `sudo nginx -t` passed. +* `doctor.sh` no longer reported nginx validation failure. + +--- + +### 2. Shared restore logic preserved bad ownership/mode from snapshot DB + +#### Symptom + +* `postfix` restore left `/etc/postfix/main.cf` owned by `lukasz:lukasz`. +* `postfix check` warned: + + * `not owned by root: /etc/postfix/./main.cf` +* `prosody` restore left: + + * `/etc/prosody` = `0700 lukasz:lukasz` + * `/etc/prosody/prosody.cfg.lua` = `0600 lukasz:lukasz` +* `prosodyctl check` failed because config was not readable by the `prosody` user. + +#### Root cause + +* `copy_path()` preserves metadata from the source snapshot (`cp -a` / `rsync -a`). +* `restore_path()` only normalizes mode/ownership if explicit arguments are provided. +* `restore_path()` previously returned early when contents matched, which prevented metadata normalization from running on already-identical files/trees. +* Many categories used `maybe_restore()` without explicit mode/owner/group policy. + +#### Fix applied — shared helper + +* Patched `restore_path()` in host `common.sh`. +* New behavior: + + * if source and destination content are identical **and** no mode/owner/group is requested, return early as before + * if source and destination content are identical **but** metadata is requested, do **not** return early + * in metadata-only cases, skip copy and normalize metadata anyway + +#### Logic + +* Content equality must not block required metadata correction. +* This prevents repeated restores from silently leaving sensitive files with incorrect owner/group/mode. + +--- + +### 3. `postfix` category needed explicit file metadata policy + +#### Symptom + +* `/etc/postfix/main.cf` remained `lukasz:lukasz` after restore. + +#### Root cause + +* `restore_postfix()` used generic `maybe_restore()` for `main.cf` and `master.cf`. +* No explicit owner/group/mode policy was applied. + +#### Fix applied + +* Replaced `restore_postfix()` with an explicit restore function using `restore_path()` directly. +* Both files now restore with: + + * owner: `root` + * group: `root` + * mode: `0644` + +#### Logic + +* These are sensitive service config files and should not depend on DB-captured metadata. +* File-level normalization is the correct pattern for this category. + +#### Result + +* `/etc/postfix/main.cf` and `/etc/postfix/master.cf` now restore as `root:root`. +* `postfix check` no longer warns about `main.cf` ownership. +* Remaining warnings under `/var/spool/postfix/etc/*` were treated as runtime/chroot noise, not current restore-blocking defects. + +--- + +### 4. `prosody` category needed directory-tree metadata normalization + +#### Symptom + +* `prosodyctl check` reported config unreadable by `prosody` user. + +#### Root cause + +* `restore_prosody()` restored `/etc/prosody` as a tree without normalizing permissions/ownership afterward. +* Snapshot ownership from user-controlled files was preserved. + +#### Manual proof on VM + +* Manual correction used to confirm diagnosis: + + * `chown -R root:root /etc/prosody` + * `find /etc/prosody -type d -exec chmod 0755 {} +` + * `find /etc/prosody -type f -exec chmod 0644 {} +` + * `chmod 0640 /etc/prosody/certs/*` where applicable +* After manual normalization, `prosodyctl check` moved past the readability issue and only reported expected lab-environment DNS/cert/public-IP mismatches. + +#### Fix applied + +* Patched `restore_prosody()` on host `restore-configs.sh`. +* After restoring `/etc/prosody`, it now normalizes: + + * ownership: `root:root` + * directories: `0755` + * files: `0644` + * cert files (if present): `0640` + +#### Logic + +* Tree restores cannot use one blanket mode like `0644` on the root directory. +* Directory/file/cert normalization must be handled separately after restore. + +#### Result + +* `prosodyctl check` no longer reports unreadable config. +* Remaining findings are expected in `lab`: + + * missing `lua-unbound` + * NAT/public-IP mismatch + * production DNS mismatch + * missing production Let’s Encrypt private key paths + +## Commands and validation patterns used + +### Canonical workflow + +* Edit host canonical copy only: + + * `~/.local/bin/straper/restore-configs.sh` + * `~/.local/bin/straper/common.sh` +* Syntax check before sync: + + * `bash -n ~/.local/bin/straper/restore-configs.sh` + * `bash -n ~/.local/bin/straper/common.sh` +* Sync to VM with `rsync` +* Re-run only the affected category on VM +* Validate service-specific behavior, then re-run `doctor.sh` + +### Service validation used + +* nginx: + + * `sudo nginx -t` +* mariadb: + + * `sudo systemctl is-active mariadb` + * `sudo journalctl -u mariadb -n 20 --no-pager` + * `sudo mariadb -e 'SELECT VERSION();'` +* postfix: + + * `sudo postfix check` + * `sudo systemctl is-active postfix` + * `sudo journalctl -u postfix -n 20 --no-pager` + * `sudo ls -l /etc/postfix/main.cf /etc/postfix/master.cf` +* prosody: + + * `sudo prosodyctl check` + * `sudo systemctl is-active prosody` + * `sudo journalctl -u prosody -n 20 --no-pager` + * `sudo ls -ld /etc/prosody` + * `sudo ls -l /etc/prosody/prosody.cfg.lua` + * `sudo namei -l /etc/prosody/prosody.cfg.lua` +* doctor baseline: + + * `sudo bash ./doctor.sh --db-dir /home/lukasz/sanctum-rebuild --role lab` + +## Design lessons extracted + +### Invariant 1 + +* Restoring config content is not enough. +* Sensitive paths also require explicit metadata policy: + + * owner + * group + * mode + +### Invariant 2 + +* Tree restores and file restores need different strategies. +* Files can use explicit `restore_path(... mode owner group)`. +* Trees often need post-restore normalization with separate rules for directories and files. + +### Invariant 3 + +* `lab` must not blindly import production-facing state. +* Examples: + + * production TLS vhosts + * production cert paths + * public DNS assumptions + * external-address checks tied to real deployment + +### Invariant 4 + +* Shared helper behavior matters more than local category patches. +* Fixing `restore_path()` was high leverage because it affects repeated restores everywhere. + +## Remaining warning set accepted in `lab` + +* `virtualization detected: kvm` +* `unbound` root key missing +* inactive intentionally deferred services: + + * `dnsmasq` + * `unbound` + * `nftables` + * `tor` + * `i2pd` +* not installed: + + * `docker` + * `grafana-server` +* `MANUAL| current run state file ...` + +## Known deferred items + +* Do not yet test these restore categories in `lab`: + + * `network` + * `dns` + * `firewall` + * `tor` + * `i2pd` + * `docker` +* Postfix chroot mirror warnings under `/var/spool/postfix/etc/*` are deferred for later runtime-hygiene review. +* `doctor.sh` is not yet role-aware enough to downgrade all expected `lab` warnings. + +## Categories likely to need future metadata hardening review + +* `ssh` +* `tor` +* `i2pd` +* `monitoring` +* possibly `mariadb` tree permissions review +* eventually `dns` / `firewall` once testing scope expands beyond current lab-safe subset + +## Recommended next move after this checkpoint + +* Treat this as a stable milestone. +* Log or commit: + + * `restore-configs.sh` + * `common.sh` + * updated rationale for `lab` restore behavior +* Before expanding into riskier categories, perform a proactive audit of restore targets that still rely on generic `maybe_restore()` without explicit metadata normalization. + diff --git a/straper/restore-configs.sh b/straper/restore-configs.sh index db81487..3127cd8 100644..100755 --- a/straper/restore-configs.sh +++ b/straper/restore-configs.sh @@ -201,13 +201,60 @@ restore_ssh() { return 0 fi - maybe_restore "ssh" "sshd_config" "${pub_users}/sshd_config" "/etc/ssh/sshd_config" - maybe_restore "ssh" "sshd_config.d" "${pub_users}/sshd_config.d" "/etc/ssh/sshd_config.d" + if prompt_yes_no "Restore ssh/sshd_config -> /etc/ssh/sshd_config?" yes; then + restore_path "${SCRIPT_NAME}" "ssh" "sshd_config" \ + "${pub_users}/sshd_config" "/etc/ssh/sshd_config" \ + 0644 root root || true + else + report "${SCRIPT_NAME}" "ssh" "sshd_config" "restore" "skipped" "operator skipped" "" + fi + + if prompt_yes_no "Restore ssh/sshd_config.d -> /etc/ssh/sshd_config.d?" yes; then + restore_path "${SCRIPT_NAME}" "ssh" "sshd_config.d" \ + "${pub_users}/sshd_config.d" "/etc/ssh/sshd_config.d" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R root:root /etc/ssh/sshd_config.d\n' + printf '[dry] find /etc/ssh/sshd_config.d -type d -exec chmod 0755 {} +\n' + printf '[dry] find /etc/ssh/sshd_config.d -type f -exec chmod 0644 {} +\n' + else + if [[ -d /etc/ssh/sshd_config.d ]]; then + chown -R root:root /etc/ssh/sshd_config.d 2>/dev/null || true + find /etc/ssh/sshd_config.d -type d -exec chmod 0755 {} + 2>/dev/null || true + find /etc/ssh/sshd_config.d -type f -exec chmod 0644 {} + 2>/dev/null || true + fi + fi + + report "${SCRIPT_NAME}" "ssh" "sshd_config.d-metadata" "restore" "changed" \ + "normalized /etc/ssh/sshd_config.d ownership=root:root dirs=0755 files=0644" "" + else + report "${SCRIPT_NAME}" "ssh" "sshd_config.d" "restore" "skipped" "operator skipped" "" + fi if [[ "${ROLE}" == "replacement" || "${ROLE}" == "hardware" ]]; then if [[ -d "${sec_ssh}/etc-ssh" ]]; then if prompt_yes_no "Restore SSH host keys from secret DB?" no; then - restore_path "${SCRIPT_NAME}" "ssh" "host-keys" "${sec_ssh}/etc-ssh" "/etc/ssh" || true + restore_path "${SCRIPT_NAME}" "ssh" "host-keys" \ + "${sec_ssh}/etc-ssh" "/etc/ssh" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown root:root /etc/ssh\n' + printf '[dry] chmod 0755 /etc/ssh\n' + printf '[dry] find /etc/ssh -maxdepth 1 -type f -name '\''ssh_host_*_key'\'' -exec chown root:root {} +\n' + printf '[dry] find /etc/ssh -maxdepth 1 -type f -name '\''ssh_host_*_key'\'' -exec chmod 0600 {} +\n' + printf '[dry] find /etc/ssh -maxdepth 1 -type f -name '\''ssh_host_*_key.pub'\'' -exec chown root:root {} +\n' + printf '[dry] find /etc/ssh -maxdepth 1 -type f -name '\''ssh_host_*_key.pub'\'' -exec chmod 0644 {} +\n' + else + chown root:root /etc/ssh 2>/dev/null || true + chmod 0755 /etc/ssh 2>/dev/null || true + find /etc/ssh -maxdepth 1 -type f -name 'ssh_host_*_key' -exec chown root:root {} + 2>/dev/null || true + find /etc/ssh -maxdepth 1 -type f -name 'ssh_host_*_key' -exec chmod 0600 {} + 2>/dev/null || true + find /etc/ssh -maxdepth 1 -type f -name 'ssh_host_*_key.pub' -exec chown root:root {} + 2>/dev/null || true + find /etc/ssh -maxdepth 1 -type f -name 'ssh_host_*_key.pub' -exec chmod 0644 {} + 2>/dev/null || true + fi + + report "${SCRIPT_NAME}" "ssh" "host-keys-metadata" "restore" "changed" \ + "normalized SSH host key ownership=root:root private=0600 public=0644" "" else report "${SCRIPT_NAME}" "ssh" "host-keys" "restore" "skipped" "operator skipped host keys" "" fi @@ -218,12 +265,21 @@ restore_ssh() { if [[ -d "${sec_ssh}/user-lukasz" ]]; then if prompt_yes_no "Restore lukasz user SSH material from secret DB?" no; then - restore_path "${SCRIPT_NAME}" "ssh" "user-lukasz" "${sec_ssh}/user-lukasz" "/home/lukasz/.ssh" || true - if [[ ${DRY_RUN} == false ]]; then + restore_path "${SCRIPT_NAME}" "ssh" "user-lukasz" \ + "${sec_ssh}/user-lukasz" "/home/lukasz/.ssh" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R lukasz:lukasz /home/lukasz/.ssh\n' + printf '[dry] chmod 700 /home/lukasz/.ssh\n' + printf '[dry] find /home/lukasz/.ssh -maxdepth 1 -type f -exec chmod 600 {} +\n' + else chown -R lukasz:lukasz /home/lukasz/.ssh 2>/dev/null || true chmod 700 /home/lukasz/.ssh 2>/dev/null || true - find /home/lukasz/.ssh -maxdepth 1 -type f -exec chmod 600 {} \; 2>/dev/null || true + find /home/lukasz/.ssh -maxdepth 1 -type f -exec chmod 600 {} + 2>/dev/null || true fi + + report "${SCRIPT_NAME}" "ssh" "user-lukasz-metadata" "restore" "changed" \ + "normalized /home/lukasz/.ssh ownership=lukasz:lukasz dir=0700 files=0600" "" else report "${SCRIPT_NAME}" "ssh" "user-lukasz" "restore" "skipped" "operator skipped user ssh material" "" fi @@ -353,35 +409,25 @@ restore_mariadb() { return 0 fi - maybe_restore "mariadb" "etc-mysql" "${base}" "/etc/mysql" -} + if prompt_yes_no "Restore mariadb/etc-mysql -> /etc/mysql?" yes; then + restore_path "${SCRIPT_NAME}" "mariadb" "etc-mysql" "${base}" "/etc/mysql" || true -restore_postfix() { - local base="${PUBLIC_DIR}/postfix" - [[ -d "${base}" ]] || { - mark_manual "${SCRIPT_NAME}" "postfix" "db" "missing ${base}" - return 0 - } - - if ! prompt_yes_no "Restore category 'postfix'?" no; then - report "${SCRIPT_NAME}" "postfix" "category" "restore" "skipped" "operator skipped category" "" - return 0 - fi - - if prompt_yes_no "Restore postfix/main.cf -> /etc/postfix/main.cf?" yes; then - restore_path "${SCRIPT_NAME}" "postfix" "main.cf" \ - "${base}/main.cf" "/etc/postfix/main.cf" \ - 0644 root root || true - else - report "${SCRIPT_NAME}" "postfix" "main.cf" "restore" "skipped" "operator skipped" "" - fi + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R root:root /etc/mysql\n' + printf '[dry] find /etc/mysql -type d -exec chmod 0755 {} +\n' + printf '[dry] find /etc/mysql -type f -exec chmod 0644 {} +\n' + else + if [[ -d /etc/mysql ]]; then + chown -R root:root /etc/mysql 2>/dev/null || true + find /etc/mysql -type d -exec chmod 0755 {} + 2>/dev/null || true + find /etc/mysql -type f -exec chmod 0644 {} + 2>/dev/null || true + fi + fi - if prompt_yes_no "Restore postfix/master.cf -> /etc/postfix/master.cf?" yes; then - restore_path "${SCRIPT_NAME}" "postfix" "master.cf" \ - "${base}/master.cf" "/etc/postfix/master.cf" \ - 0644 root root || true + report "${SCRIPT_NAME}" "mariadb" "metadata" "restore" "changed" \ + "normalized /etc/mysql ownership=root:root dirs=0755 files=0644" "" else - report "${SCRIPT_NAME}" "postfix" "master.cf" "restore" "skipped" "operator skipped" "" + report "${SCRIPT_NAME}" "mariadb" "etc-mysql" "restore" "skipped" "operator skipped" "" fi } @@ -438,11 +484,58 @@ restore_tor() { return 0 fi - maybe_restore "tor" "torrc" "${pub_base}/torrc" "/etc/tor/torrc" - maybe_restore "tor" "torrc.d" "${pub_base}/torrc.d" "/etc/tor/torrc.d" + if prompt_yes_no "Restore tor/torrc -> /etc/tor/torrc?" yes; then + restore_path "${SCRIPT_NAME}" "tor" "torrc" \ + "${pub_base}/torrc" "/etc/tor/torrc" \ + 0644 root root || true + else + report "${SCRIPT_NAME}" "tor" "torrc" "restore" "skipped" "operator skipped" "" + fi + + if prompt_yes_no "Restore tor/torrc.d -> /etc/tor/torrc.d?" yes; then + restore_path "${SCRIPT_NAME}" "tor" "torrc.d" \ + "${pub_base}/torrc.d" "/etc/tor/torrc.d" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R root:root /etc/tor/torrc.d\n' + printf '[dry] find /etc/tor/torrc.d -type d -exec chmod 0755 {} +\n' + printf '[dry] find /etc/tor/torrc.d -type f -exec chmod 0644 {} +\n' + else + if [[ -d /etc/tor/torrc.d ]]; then + chown -R root:root /etc/tor/torrc.d 2>/dev/null || true + find /etc/tor/torrc.d -type d -exec chmod 0755 {} + 2>/dev/null || true + find /etc/tor/torrc.d -type f -exec chmod 0644 {} + 2>/dev/null || true + fi + fi + + report "${SCRIPT_NAME}" "tor" "torrc.d-metadata" "restore" "changed" \ + "normalized /etc/tor/torrc.d ownership=root:root dirs=0755 files=0644" "" + else + report "${SCRIPT_NAME}" "tor" "torrc.d" "restore" "skipped" "operator skipped" "" + fi if [[ "${ROLE}" == "replacement" && -d "${sec_base}/var-lib-tor" ]]; then - maybe_restore "tor" "var-lib-tor" "${sec_base}/var-lib-tor" "/var/lib/tor" + if prompt_yes_no "Restore tor/var-lib-tor -> /var/lib/tor?" no; then + restore_path "${SCRIPT_NAME}" "tor" "var-lib-tor" \ + "${sec_base}/var-lib-tor" "/var/lib/tor" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R debian-tor:debian-tor /var/lib/tor\n' + printf '[dry] find /var/lib/tor -type d -exec chmod 0700 {} +\n' + printf '[dry] find /var/lib/tor -type f -exec chmod 0600 {} +\n' + else + if [[ -d /var/lib/tor ]]; then + chown -R debian-tor:debian-tor /var/lib/tor 2>/dev/null || true + find /var/lib/tor -type d -exec chmod 0700 {} + 2>/dev/null || true + find /var/lib/tor -type f -exec chmod 0600 {} + 2>/dev/null || true + fi + fi + + report "${SCRIPT_NAME}" "tor" "var-lib-tor-metadata" "restore" "changed" \ + "normalized /var/lib/tor ownership=debian-tor:debian-tor dirs=0700 files=0600" "" + else + report "${SCRIPT_NAME}" "tor" "var-lib-tor" "restore" "skipped" "operator skipped" "" + fi else report "${SCRIPT_NAME}" "tor" "identity" "restore" "skipped" "tor private data not restored in this role" "" fi @@ -457,10 +550,50 @@ restore_i2pd() { return 0 fi - maybe_restore "i2pd" "etc-i2pd" "${pub_base}/etc-i2pd" "/etc/i2pd" + if prompt_yes_no "Restore i2pd/etc-i2pd -> /etc/i2pd?" yes; then + restore_path "${SCRIPT_NAME}" "i2pd" "etc-i2pd" \ + "${pub_base}/etc-i2pd" "/etc/i2pd" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R root:root /etc/i2pd\n' + printf '[dry] find /etc/i2pd -type d -exec chmod 0755 {} +\n' + printf '[dry] find /etc/i2pd -type f -exec chmod 0644 {} +\n' + else + if [[ -d /etc/i2pd ]]; then + chown -R root:root /etc/i2pd 2>/dev/null || true + find /etc/i2pd -type d -exec chmod 0755 {} + 2>/dev/null || true + find /etc/i2pd -type f -exec chmod 0644 {} + 2>/dev/null || true + fi + fi + + report "${SCRIPT_NAME}" "i2pd" "etc-i2pd-metadata" "restore" "changed" \ + "normalized /etc/i2pd ownership=root:root dirs=0755 files=0644" "" + else + report "${SCRIPT_NAME}" "i2pd" "etc-i2pd" "restore" "skipped" "operator skipped" "" + fi if [[ "${ROLE}" == "replacement" && -d "${sec_base}/var-lib-i2pd" ]]; then - maybe_restore "i2pd" "var-lib-i2pd" "${sec_base}/var-lib-i2pd" "/var/lib/i2pd" + if prompt_yes_no "Restore i2pd/var-lib-i2pd -> /var/lib/i2pd?" no; then + restore_path "${SCRIPT_NAME}" "i2pd" "var-lib-i2pd" \ + "${sec_base}/var-lib-i2pd" "/var/lib/i2pd" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R i2pd:i2pd /var/lib/i2pd\n' + printf '[dry] find /var/lib/i2pd -type d -exec chmod 0700 {} +\n' + printf '[dry] find /var/lib/i2pd -type f -exec chmod 0600 {} +\n' + else + if [[ -d /var/lib/i2pd ]]; then + chown -R i2pd:i2pd /var/lib/i2pd 2>/dev/null || true + find /var/lib/i2pd -type d -exec chmod 0700 {} + 2>/dev/null || true + find /var/lib/i2pd -type f -exec chmod 0600 {} + 2>/dev/null || true + fi + fi + + report "${SCRIPT_NAME}" "i2pd" "var-lib-i2pd-metadata" "restore" "changed" \ + "normalized /var/lib/i2pd ownership=i2pd:i2pd dirs=0700 files=0600" "" + else + report "${SCRIPT_NAME}" "i2pd" "var-lib-i2pd" "restore" "skipped" "operator skipped" "" + fi else report "${SCRIPT_NAME}" "i2pd" "identity" "restore" "skipped" "i2pd private data not restored in this role" "" fi @@ -475,10 +608,17 @@ restore_docker() { return 0 fi - maybe_restore "docker" "daemon.json" "${pub_base}/daemon.json" "/etc/docker/daemon.json" + if prompt_yes_no "Restore docker/daemon.json -> /etc/docker/daemon.json?" yes; then + restore_path "${SCRIPT_NAME}" "docker" "daemon.json" \ + "${pub_base}/daemon.json" "/etc/docker/daemon.json" \ + 0644 root root || true + else + report "${SCRIPT_NAME}" "docker" "daemon.json" "restore" "skipped" "operator skipped" "" + fi if [[ -d "${sec_base}/compose-full" ]]; then - report "${SCRIPT_NAME}" "docker" "compose-full" "manual" "manual" "compose files available in secret DB; restore manually per stack" "" + report "${SCRIPT_NAME}" "docker" "compose-full" "manual" "manual" \ + "compose files available in secret DB; restore manually per stack" "" fi } @@ -488,11 +628,104 @@ restore_monitoring() { return 0 fi - [[ -d "${PUBLIC_DIR}/prometheus/etc-prometheus" ]] && maybe_restore "monitoring" "prometheus" "${PUBLIC_DIR}/prometheus/etc-prometheus" "/etc/prometheus" - [[ -f "${PUBLIC_DIR}/prometheus/node-exporter-defaults" ]] && maybe_restore "monitoring" "prometheus-node-exporter" "${PUBLIC_DIR}/prometheus/node-exporter-defaults" "/etc/default/prometheus-node-exporter" - [[ -d "${PUBLIC_DIR}/loki/etc-loki" ]] && maybe_restore "monitoring" "loki" "${PUBLIC_DIR}/loki/etc-loki" "/etc/loki" - [[ -d "${PUBLIC_DIR}/grafana/etc-grafana" ]] && maybe_restore "monitoring" "grafana" "${PUBLIC_DIR}/grafana/etc-grafana" "/etc/grafana" - [[ -d "${SECRET_DIR}/alloy/etc-alloy" ]] && maybe_restore "monitoring" "alloy" "${SECRET_DIR}/alloy/etc-alloy" "/etc/alloy" + if [[ -d "${PUBLIC_DIR}/prometheus/etc-prometheus" ]]; then + if prompt_yes_no "Restore monitoring/prometheus -> /etc/prometheus?" yes; then + restore_path "${SCRIPT_NAME}" "monitoring" "prometheus" \ + "${PUBLIC_DIR}/prometheus/etc-prometheus" "/etc/prometheus" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R root:root /etc/prometheus\n' + printf '[dry] find /etc/prometheus -type d -exec chmod 0755 {} +\n' + printf '[dry] find /etc/prometheus -type f -exec chmod 0644 {} +\n' + else + [[ -d /etc/prometheus ]] && chown -R root:root /etc/prometheus 2>/dev/null || true + [[ -d /etc/prometheus ]] && find /etc/prometheus -type d -exec chmod 0755 {} + 2>/dev/null || true + [[ -d /etc/prometheus ]] && find /etc/prometheus -type f -exec chmod 0644 {} + 2>/dev/null || true + fi + + report "${SCRIPT_NAME}" "monitoring" "prometheus-metadata" "restore" "changed" \ + "normalized /etc/prometheus ownership=root:root dirs=0755 files=0644" "" + else + report "${SCRIPT_NAME}" "monitoring" "prometheus" "restore" "skipped" "operator skipped" "" + fi + fi + + if [[ -f "${PUBLIC_DIR}/prometheus/node-exporter-defaults" ]]; then + if prompt_yes_no "Restore monitoring/prometheus-node-exporter -> /etc/default/prometheus-node-exporter?" yes; then + restore_path "${SCRIPT_NAME}" "monitoring" "prometheus-node-exporter" \ + "${PUBLIC_DIR}/prometheus/node-exporter-defaults" \ + "/etc/default/prometheus-node-exporter" \ + 0644 root root || true + else + report "${SCRIPT_NAME}" "monitoring" "prometheus-node-exporter" "restore" "skipped" "operator skipped" "" + fi + fi + + if [[ -d "${PUBLIC_DIR}/loki/etc-loki" ]]; then + if prompt_yes_no "Restore monitoring/loki -> /etc/loki?" yes; then + restore_path "${SCRIPT_NAME}" "monitoring" "loki" \ + "${PUBLIC_DIR}/loki/etc-loki" "/etc/loki" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R root:root /etc/loki\n' + printf '[dry] find /etc/loki -type d -exec chmod 0755 {} +\n' + printf '[dry] find /etc/loki -type f -exec chmod 0644 {} +\n' + else + [[ -d /etc/loki ]] && chown -R root:root /etc/loki 2>/dev/null || true + [[ -d /etc/loki ]] && find /etc/loki -type d -exec chmod 0755 {} + 2>/dev/null || true + [[ -d /etc/loki ]] && find /etc/loki -type f -exec chmod 0644 {} + 2>/dev/null || true + fi + + report "${SCRIPT_NAME}" "monitoring" "loki-metadata" "restore" "changed" \ + "normalized /etc/loki ownership=root:root dirs=0755 files=0644" "" + else + report "${SCRIPT_NAME}" "monitoring" "loki" "restore" "skipped" "operator skipped" "" + fi + fi + + if [[ -d "${PUBLIC_DIR}/grafana/etc-grafana" ]]; then + if prompt_yes_no "Restore monitoring/grafana -> /etc/grafana?" yes; then + restore_path "${SCRIPT_NAME}" "monitoring" "grafana" \ + "${PUBLIC_DIR}/grafana/etc-grafana" "/etc/grafana" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R root:root /etc/grafana\n' + printf '[dry] find /etc/grafana -type d -exec chmod 0755 {} +\n' + printf '[dry] find /etc/grafana -type f -exec chmod 0644 {} +\n' + else + [[ -d /etc/grafana ]] && chown -R root:root /etc/grafana 2>/dev/null || true + [[ -d /etc/grafana ]] && find /etc/grafana -type d -exec chmod 0755 {} + 2>/dev/null || true + [[ -d /etc/grafana ]] && find /etc/grafana -type f -exec chmod 0644 {} + 2>/dev/null || true + fi + + report "${SCRIPT_NAME}" "monitoring" "grafana-metadata" "restore" "changed" \ + "normalized /etc/grafana ownership=root:root dirs=0755 files=0644" "" + else + report "${SCRIPT_NAME}" "monitoring" "grafana" "restore" "skipped" "operator skipped" "" + fi + fi + + if [[ -d "${SECRET_DIR}/alloy/etc-alloy" ]]; then + if prompt_yes_no "Restore monitoring/alloy -> /etc/alloy?" yes; then + restore_path "${SCRIPT_NAME}" "monitoring" "alloy" \ + "${SECRET_DIR}/alloy/etc-alloy" "/etc/alloy" || true + + if [[ ${DRY_RUN} == true ]]; then + printf '[dry] chown -R root:root /etc/alloy\n' + printf '[dry] find /etc/alloy -type d -exec chmod 0755 {} +\n' + printf '[dry] find /etc/alloy -type f -exec chmod 0644 {} +\n' + else + [[ -d /etc/alloy ]] && chown -R root:root /etc/alloy 2>/dev/null || true + [[ -d /etc/alloy ]] && find /etc/alloy -type d -exec chmod 0755 {} + 2>/dev/null || true + [[ -d /etc/alloy ]] && find /etc/alloy -type f -exec chmod 0644 {} + 2>/dev/null || true + fi + + report "${SCRIPT_NAME}" "monitoring" "alloy-metadata" "restore" "changed" \ + "normalized /etc/alloy ownership=root:root dirs=0755 files=0644" "" + else + report "${SCRIPT_NAME}" "monitoring" "alloy" "restore" "skipped" "operator skipped" "" + fi + fi } for category in "${CATEGORIES[@]}"; do |
