diff options
Diffstat (limited to 'straper/logs')
| -rw-r--r-- | straper/logs/sanctum-rebuild-toolkist-2026-03-19T15:14:13.log | 325 |
1 files changed, 325 insertions, 0 deletions
diff --git a/straper/logs/sanctum-rebuild-toolkist-2026-03-19T15:14:13.log b/straper/logs/sanctum-rebuild-toolkist-2026-03-19T15:14:13.log new file mode 100644 index 0000000..502e03a --- /dev/null +++ b/straper/logs/sanctum-rebuild-toolkist-2026-03-19T15:14:13.log @@ -0,0 +1,325 @@ +# sanctum-rebuild-toolkit — lab validation log + +## Date + +* 2026-03-19 + +## Scope + +* Objective: continue category-by-category validation of the Debian 13 disaster-recovery toolkit in a KVM/libvirt lab VM, keeping host `~/.local/bin/straper` as canonical source of truth. +* Role under test: `lab` +* DB under test: `/home/lukasz/sanctum-rebuild` +* VM toolkit path: `~/rebuild-test/sanctum-rebuild-toolkit/` + +## Baseline assumptions + +* Edit only on host in `~/.local/bin/straper`, then sync to VM with `rsync`. +* Test low-risk categories first. +* Defer `network`, `dns`, `firewall`, `tor`, `i2pd`, `docker` category execution in lab. +* Treat some doctor warnings as acceptable in `lab` when they reflect intentionally disabled or absent services. + +## Categories validated in this session + +* `nginx` +* `mariadb` +* `postfix` +* `prosody` + +## Final lab baseline after this session + +* `doctor.sh` summary: `ok=20 warn=9 fail=0 manual=1` +* Validated as working in `lab`: + + * `system-basics` + * `ssh` + * `users` + * `nginx` + * `mariadb` + * `postfix` + * `prosody` + +## Problems found and fixes applied + +### 1. `nginx` restore imported production TLS/vhost state into `lab` + +#### Symptom + +* `sudo nginx -t` failed after restore. +* Failure was due to missing production certificate paths such as: + + * `/etc/letsencrypt/live/labunix.xyz/fullchain.pem` +* Restored tree also contained junk such as: + + * nested `/etc/nginx/nginx/...` + * `sites-available.bak.*` +* `sites-enabled` contained a full production vhost set, including TLS-dependent vhosts. + +#### Root cause + +* `restore_nginx()` restored the captured nginx tree wholesale into `/etc/nginx`. +* No `lab`-specific sanitization existed after restore. + +#### Fix applied + +* Patched `restore_nginx()` on host `restore-configs.sh`. +* In `lab` role, after restore it now: + + * removes `/etc/nginx/nginx` + * removes top-level `sites-available.bak.*` + * clears `sites-enabled` + * re-enables only `sites-available/default` if present + +#### Logic + +* In `lab`, restore enough nginx structure to validate service/config integrity, but do not enable production-facing TLS vhosts. +* Avoid fake cert hacks. +* Keep the change canonical in the host script, not as a VM-only workaround. + +#### Result + +* `sudo nginx -t` passed. +* `doctor.sh` no longer reported nginx validation failure. + +--- + +### 2. Shared restore logic preserved bad ownership/mode from snapshot DB + +#### Symptom + +* `postfix` restore left `/etc/postfix/main.cf` owned by `lukasz:lukasz`. +* `postfix check` warned: + + * `not owned by root: /etc/postfix/./main.cf` +* `prosody` restore left: + + * `/etc/prosody` = `0700 lukasz:lukasz` + * `/etc/prosody/prosody.cfg.lua` = `0600 lukasz:lukasz` +* `prosodyctl check` failed because config was not readable by the `prosody` user. + +#### Root cause + +* `copy_path()` preserves metadata from the source snapshot (`cp -a` / `rsync -a`). +* `restore_path()` only normalizes mode/ownership if explicit arguments are provided. +* `restore_path()` previously returned early when contents matched, which prevented metadata normalization from running on already-identical files/trees. +* Many categories used `maybe_restore()` without explicit mode/owner/group policy. + +#### Fix applied — shared helper + +* Patched `restore_path()` in host `common.sh`. +* New behavior: + + * if source and destination content are identical **and** no mode/owner/group is requested, return early as before + * if source and destination content are identical **but** metadata is requested, do **not** return early + * in metadata-only cases, skip copy and normalize metadata anyway + +#### Logic + +* Content equality must not block required metadata correction. +* This prevents repeated restores from silently leaving sensitive files with incorrect owner/group/mode. + +--- + +### 3. `postfix` category needed explicit file metadata policy + +#### Symptom + +* `/etc/postfix/main.cf` remained `lukasz:lukasz` after restore. + +#### Root cause + +* `restore_postfix()` used generic `maybe_restore()` for `main.cf` and `master.cf`. +* No explicit owner/group/mode policy was applied. + +#### Fix applied + +* Replaced `restore_postfix()` with an explicit restore function using `restore_path()` directly. +* Both files now restore with: + + * owner: `root` + * group: `root` + * mode: `0644` + +#### Logic + +* These are sensitive service config files and should not depend on DB-captured metadata. +* File-level normalization is the correct pattern for this category. + +#### Result + +* `/etc/postfix/main.cf` and `/etc/postfix/master.cf` now restore as `root:root`. +* `postfix check` no longer warns about `main.cf` ownership. +* Remaining warnings under `/var/spool/postfix/etc/*` were treated as runtime/chroot noise, not current restore-blocking defects. + +--- + +### 4. `prosody` category needed directory-tree metadata normalization + +#### Symptom + +* `prosodyctl check` reported config unreadable by `prosody` user. + +#### Root cause + +* `restore_prosody()` restored `/etc/prosody` as a tree without normalizing permissions/ownership afterward. +* Snapshot ownership from user-controlled files was preserved. + +#### Manual proof on VM + +* Manual correction used to confirm diagnosis: + + * `chown -R root:root /etc/prosody` + * `find /etc/prosody -type d -exec chmod 0755 {} +` + * `find /etc/prosody -type f -exec chmod 0644 {} +` + * `chmod 0640 /etc/prosody/certs/*` where applicable +* After manual normalization, `prosodyctl check` moved past the readability issue and only reported expected lab-environment DNS/cert/public-IP mismatches. + +#### Fix applied + +* Patched `restore_prosody()` on host `restore-configs.sh`. +* After restoring `/etc/prosody`, it now normalizes: + + * ownership: `root:root` + * directories: `0755` + * files: `0644` + * cert files (if present): `0640` + +#### Logic + +* Tree restores cannot use one blanket mode like `0644` on the root directory. +* Directory/file/cert normalization must be handled separately after restore. + +#### Result + +* `prosodyctl check` no longer reports unreadable config. +* Remaining findings are expected in `lab`: + + * missing `lua-unbound` + * NAT/public-IP mismatch + * production DNS mismatch + * missing production Let’s Encrypt private key paths + +## Commands and validation patterns used + +### Canonical workflow + +* Edit host canonical copy only: + + * `~/.local/bin/straper/restore-configs.sh` + * `~/.local/bin/straper/common.sh` +* Syntax check before sync: + + * `bash -n ~/.local/bin/straper/restore-configs.sh` + * `bash -n ~/.local/bin/straper/common.sh` +* Sync to VM with `rsync` +* Re-run only the affected category on VM +* Validate service-specific behavior, then re-run `doctor.sh` + +### Service validation used + +* nginx: + + * `sudo nginx -t` +* mariadb: + + * `sudo systemctl is-active mariadb` + * `sudo journalctl -u mariadb -n 20 --no-pager` + * `sudo mariadb -e 'SELECT VERSION();'` +* postfix: + + * `sudo postfix check` + * `sudo systemctl is-active postfix` + * `sudo journalctl -u postfix -n 20 --no-pager` + * `sudo ls -l /etc/postfix/main.cf /etc/postfix/master.cf` +* prosody: + + * `sudo prosodyctl check` + * `sudo systemctl is-active prosody` + * `sudo journalctl -u prosody -n 20 --no-pager` + * `sudo ls -ld /etc/prosody` + * `sudo ls -l /etc/prosody/prosody.cfg.lua` + * `sudo namei -l /etc/prosody/prosody.cfg.lua` +* doctor baseline: + + * `sudo bash ./doctor.sh --db-dir /home/lukasz/sanctum-rebuild --role lab` + +## Design lessons extracted + +### Invariant 1 + +* Restoring config content is not enough. +* Sensitive paths also require explicit metadata policy: + + * owner + * group + * mode + +### Invariant 2 + +* Tree restores and file restores need different strategies. +* Files can use explicit `restore_path(... mode owner group)`. +* Trees often need post-restore normalization with separate rules for directories and files. + +### Invariant 3 + +* `lab` must not blindly import production-facing state. +* Examples: + + * production TLS vhosts + * production cert paths + * public DNS assumptions + * external-address checks tied to real deployment + +### Invariant 4 + +* Shared helper behavior matters more than local category patches. +* Fixing `restore_path()` was high leverage because it affects repeated restores everywhere. + +## Remaining warning set accepted in `lab` + +* `virtualization detected: kvm` +* `unbound` root key missing +* inactive intentionally deferred services: + + * `dnsmasq` + * `unbound` + * `nftables` + * `tor` + * `i2pd` +* not installed: + + * `docker` + * `grafana-server` +* `MANUAL| current run state file ...` + +## Known deferred items + +* Do not yet test these restore categories in `lab`: + + * `network` + * `dns` + * `firewall` + * `tor` + * `i2pd` + * `docker` +* Postfix chroot mirror warnings under `/var/spool/postfix/etc/*` are deferred for later runtime-hygiene review. +* `doctor.sh` is not yet role-aware enough to downgrade all expected `lab` warnings. + +## Categories likely to need future metadata hardening review + +* `ssh` +* `tor` +* `i2pd` +* `monitoring` +* possibly `mariadb` tree permissions review +* eventually `dns` / `firewall` once testing scope expands beyond current lab-safe subset + +## Recommended next move after this checkpoint + +* Treat this as a stable milestone. +* Log or commit: + + * `restore-configs.sh` + * `common.sh` + * updated rationale for `lab` restore behavior +* Before expanding into riskier categories, perform a proactive audit of restore targets that still rely on generic `maybe_restore()` without explicit metadata normalization. + |