1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
|
# sanctum / labunix rebuild toolkit
A modular disaster-recovery toolkit for rebuilding a Debian 13 server (`sanctum`) in a controlled, validated order.
```
capture → lint → install → restore → doctor
```
Rebuild time: weeks → hours. All risky changes are explicit, reversible, and tested.
---
## Toolkit layout
```
~/.local/bin/straper/
├── capture-full.sh Capture live server state into a rebuild DB
├── lint-db.sh Validate DB shape before trusting it for restore
├── install-base.sh Install packages and baseline — no identity transplant
├── restore-configs.sh Restore config categories from DB, one at a time
├── doctor.sh Read-only health and readiness checks
├── rebuild.conf.example Optional config overrides
└── lib/
└── common.sh Shared helpers, restore_path(), report(), etc.
```
Runtime artifacts go to `/var/log/labunix-rebuild/` and `/var/lib/labunix-rebuild/`.
---
## Roles
| Role | Use for | Behavior |
|------|---------|----------|
| `lab` | VM testing, dry runs | Skips DNS/firewall/network, no identity restores |
| `hardware` | Real machine migration | Full adaptive restore, services started |
| `replacement` | True disaster recovery | All restores including private identities |
---
## Canonical workflow
### Step 1 — Capture (on source host)
```bash
sudo ~/.local/bin/straper/capture-full.sh
# or to a specific path:
sudo CAPTURE_DIR=/tmp/sanctum-rebuild-clean ~/.local/bin/straper/capture-full.sh
```
Produces `db/public/` (safe, git-tracked) and `db/secret/` (root-only, never committed).
### Step 2 — Lint
Always lint before restore. Do not proceed if lint fails.
```bash
sudo ~/.local/bin/straper/lint-db.sh --db-dir /tmp/sanctum-rebuild-clean
```
Catches: nested duplicate roots, overlapping captures, backup junk, duplicate canonical files.
### Step 3 — Transfer to target
```bash
rsync -av ~/.local/bin/straper/ lukasz@TARGET:~/rebuild-test/sanctum-rebuild-toolkit/
rsync -av /tmp/sanctum-rebuild-clean/db/ lukasz@TARGET:~/sanctum-rebuild/db/
```
### Step 4 — Install base
```bash
cd ~/rebuild-test/sanctum-rebuild-toolkit
sudo bash ./install-base.sh --role lab --profile core
```
Installs packages, sets up baseline, stops conflicting services. Does **not** restore any config.
### Step 5 — Restore categories
Restore in blocks. Run `doctor.sh` after each block.
**Safe first block:**
```bash
sudo bash ./restore-configs.sh \
--db-dir ~/sanctum-rebuild \
--role lab \
--category system-basics \
--category users \
--category ssh
```
**Service block:**
```bash
sudo bash ./restore-configs.sh \
--db-dir ~/sanctum-rebuild \
--role lab \
--category nginx \
--category mariadb \
--category postfix \
--category prosody \
--category docker \
--category monitoring \
--category tor \
--category i2pd
```
**Risky block** (use `--role hardware` or `--role replacement`, run last):
```bash
sudo bash ./restore-configs.sh \
--db-dir ~/sanctum-rebuild \
--role hardware \
--category dns \
--category firewall \
--category network
```
### Step 6 — Verify
```bash
sudo bash ./doctor.sh --db-dir ~/sanctum-rebuild --role hardware
```
---
## Smart adaptive restore
`dns`, `firewall`, and `network` are not naive file copies — they adapt to the target machine.
### DNS (`restore_dns`)
1. Scans live interfaces, IPs, WireGuard ifaces, Docker bridges, port 53 owner
2. Adapts `dnsmasq.conf` and all `dnsmasq.d/*` — strips non-live IPs from `listen-address`, comments out `interface=` lines for absent interfaces
3. Classifies each file: `safe` | `dormant-wireguard` | `dormant-docker` | `dormant-hardware`
4. Fetches `root.hints` from internic.net if referenced by unbound config
5. Disables DNSSEC validator for bootstrap (re-enable manually once stable)
6. Disables `systemd-resolved` stub listener if it would conflict with dnsmasq
7. Starts unbound → dnsmasq in correct order, verifies each is listening
8. Writes `/etc/resolv.conf` **only after** resolution is confirmed — falls back to `1.1.1.1` if not
9. Prints precise manual task list
WireGuard-dependent and Docker-dependent rules are preserved as dormant comments — they activate automatically when those interfaces come up.
### Firewall (`restore_firewall`)
1. Scans live interfaces
2. Adapts `nftables.conf` — comments out rules referencing absent interfaces
3. Classifies: `wg*` → dormant-wireguard, `enx*/eth*` → dormant-hardware-nic, `br-*/docker0` → dormant-docker (collapsed to one task)
4. Validates with `nft -c` before loading
5. Loads with `nft -f`, enables `nftables.service`
### Network (`restore_network`)
Restores `/etc/network`, `systemd-network`, `nsswitch.conf`, `hosts.allow/deny`. Skipped in `lab` role.
---
## Validated restore order
```
system-basics → users → ssh
nginx → mariadb → postfix → prosody → docker → monitoring → tor → i2pd
dns → firewall → network
```
DNS must come before firewall. Network last. Run `doctor.sh` after each block.
---
## Validated doctor baseline (hardware role)
```
Summary: ok=22 warn=7 fail=0 manual=1
```
Expected warnings in lab: `kvm` detected, `nftables` inactive before first boot, `docker`/`grafana` not installed, `tor`/`i2pd` not started, DNS chain intentional.
---
## Restore categories
| Category | Risk | Notes |
|----------|------|-------|
| `system-basics` | low | hostname, hosts, locale, timezone |
| `users` | low | sudoers with ownership fix, shells, login.defs |
| `ssh` | low | sshd_config; host keys in hardware/replacement only |
| `nginx` | low | lab: default site only, no production vhosts |
| `mariadb` | low | /etc/mysql tree with normalization |
| `postfix` | low | main.cf, master.cf with explicit root:root ownership |
| `prosody` | low | tree + cert permission normalization |
| `docker` | low | daemon.json; compose files are manual |
| `monitoring` | low | prometheus, loki, grafana, alloy trees |
| `tor` | low | torrc; /var/lib/tor only in replacement role |
| `i2pd` | low | /etc/i2pd tree; /var/lib/i2pd only in replacement role |
| `dns` | **risky** | smart adaptive — see above |
| `firewall` | **risky** | smart adaptive — see above |
| `network` | **risky** | skipped in lab |
---
## Known lessons
**Metadata matters.** Content restore alone is not enough — owner, group, and mode must be normalized. `restore_path()` handles this.
**DB shape matters.** A polluted DB produces bad restores even with correct logic. Always lint first.
**Tree restores are overlay-style.** On a reused VM, stale files survive. Use a fresh VM or clean the destination before retesting.
**resolv.conf is sacred.** Never write it until a local resolver is confirmed listening.
**DNSSEC on Debian 13.** `unbound-anchor` is not shipped. Disable the validator module for initial bring-up; re-enable after the system is stable.
**Service start order.** unbound must be listening before dnsmasq starts — dnsmasq validates its `server=` upstream at startup and exits with `INVALIDARGUMENT` if it can't reach it.
**nftables interface references.** Rules referencing absent interfaces must be commented out or nftables refuses to load entirely.
---
## Public vs secret DB tiers
`db/public/` — unencrypted, git-tracked. Safe for remote backup.
`db/secret/` — root-only (`chmod 700`), never committed. Protected at rest by ZFS-on-LUKS. Contains SSH keys, TLS private keys, WireGuard keys, LUKS headers, service secrets.
---
## Quick reference
```bash
# Capture
sudo ~/.local/bin/straper/capture-full.sh
# Lint
sudo ~/.local/bin/straper/lint-db.sh --db-dir /srv/sanctum-rebuild
# Install
sudo bash ./install-base.sh --role hardware --profile core
# Restore (example)
sudo bash ./restore-configs.sh \
--db-dir ~/sanctum-rebuild \
--role hardware \
--category dns
# Verify
sudo bash ./doctor.sh --db-dir ~/sanctum-rebuild --role hardware
# List categories
sudo bash ./restore-configs.sh --list-categories
```
---
## Status
All restore categories validated including risky (`dns`, `firewall`, `network`).
Suitable for: structured lab testing, staged hardware migration, disciplined disaster-recovery rehearsal.
|
