Final configuration

docs/FAILURES.md

@@ -6,7 +6,7 @@ Compiler environment.
 | | |
 |---|---|
 | Scope | All instances. Staging entries are marked `srv-b`. |
-| Updated | 2026-08-17, after work order 002 |
+| Updated | 2026-08-17, after work order 003 |
 | Rule | Append only. Never edit an entry except to add a `Resolution` line. |
 | Numbering | Sequential, never reused. See §0 on the renumbering. |
 
@@ -526,6 +526,111 @@ would break that. The question is which clients may ask, not where it may send.
 **Consequence:** A correction that fixes the observed failure may widen an
 adjacent boundary. Record what a trust change grants, not only what it repairs.
 
+**Project-local correction (2026-08-17, WO-003):** Complete. A persistent
+`srv-b` FORWARD rule drops TCP 25, 465 and 587 from `10.20.0.0/24` to
+`10.110.0.0/22`. Reachability was proven **before** the change, both containers
+are blocked after it, ordinary internet egress is intact, host-originated mail
+still delivers end to end, and the rule survived a host reboot.
+
+**Estate half: still open.** Whether `wg-pk` should narrow `mynetworks` from
+`10.110.0.0/22` to the explicit hosts that legitimately originate mail is a
+CIVICVS decision. No change was made to `wg-pk`, `mx1` or `kane-il.us`.
+
+F-025 is therefore **partially corrected**, not closed. The two halves are kept
+in one entry rather than split, because the exposure is a single chain and
+splitting it would let one half be closed while the other is forgotten.
+
+---
+
+### F-026 — containers were not configured to start with the host
+`srv-b`, CT 100, CT 101. Work order 003.
+
+**Observed:** After the WO-003 persistence reboot — the first true **host**
+reboot since the containers were built — both were stopped:
+
+```
+pct status 100  ->  status: stopped
+pct status 101  ->  status: stopped
+```
+
+**Cause:** **Proven.** Neither container configuration contained `onboot: 1`, so
+Proxmox had no instruction to start either guest. `pve-guests.service` was
+enabled, active, and its `startall` task completed successfully — proving the
+startup machinery worked and simply had nothing to do.
+**Correction:** `pct set 100 -onboot 1` and `pct set 101 -onboot 1`. The
+containers were deliberately left stopped so the correction could be proven by
+reboot rather than masked by a manual start. After a second host reboot both
+returned automatically and host plus both guests reported `running`.
+**Consequence:** **Specification defect.** `ENVIRONMENT.md` never required
+`onboot: 1`, and every earlier reboot in this project was `pct reboot` — which
+restarts a guest without ever exercising host-boot autostart. A container that
+survives `pct reboot` is not thereby proven to survive a host reboot. Gate 1
+must assert guest state after a **host** reboot specifically.
+
+---
+
+### F-027 — a verification test conflated tool failure with the tested condition
+`srv-b`. Work order 003.
+
+**Observed:** WO-003 B3 and B4 used this pattern to test SMTP reachability from
+a container:
+
+```bash
+pct exec $c -- timeout 5 bash -c '...' \
+  && echo "REACHABLE — WRONG" || echo "blocked — correct"
+```
+
+With the containers stopped (F-026) it produced:
+
+```
+-- CT 100: container '100' not running!
+blocked — correct
+```
+
+**Cause:** **Proven.** The shell `||` branch catches *any* non-zero exit from
+`pct exec`, including failures of `pct exec` itself. The test could not
+distinguish "the firewall blocked the connection" from "the command never ran."
+**Correction:** Assert guest state before interpreting any network result:
+
+```bash
+if [ "$(pct status "$c" | awk '{print $2}')" != "running" ]; then
+    echo "CT $c NOT RUNNING — TEST INVALID"
+elif pct exec "$c" -- timeout 5 bash -c '...'; then
+    echo "REACHABLE — WRONG"
+else
+    echo "blocked — correct"
+fi
+```
+
+**Consequence:** **A test whose failure mode is indistinguishable from success
+is worse than no test**, because it manufactures confidence. This one printed a
+pass while the thing under test did not exist. Only a separate health assertion
+caught it, and that was ordering luck rather than design. Every negative
+assertion — "X is unreachable", "Y is absent", "Z is refused" — must first
+establish that the test could have observed the positive case.
+
+Applies retroactively: the isolation proofs in F-018 used the same shape. They
+happened to be run against containers that were up, and the result was
+independently corroborated, but the pattern was unsafe there too.
+
+---
+
+### F-028 — three command defects in work order 003
+`srv-b`. Work order 003. Minor, grouped.
+
+**Observed and corrected in place by the operator:**
+
+| Defect | Cause | Correction |
+|---|---|---|
+| Serial numbers absent from A1 output | `grep -E "Serial Number"` is case-sensitive; SCSI output says `Serial number:` | `grep -iE` with a case-insensitive alternation on product, model, serial and SMART support |
+| `journalctl -u smartd -b` returned `-- No entries --` | `smartd.service` is an alias; the real unit is `smartmontools.service` and owns the journal | Query `smartmontools` |
+| A3 `sed` would have produced a malformed directive | Prefix substitution left the trailing `-M exec ...` in place and duplicated `-m root` | Replace the whole `DEFAULT` line, in both A3 and A4 |
+
+**Consequence:** Match on the canonical unit name, not a convenience alias.
+Prefer whole-line replacement over prefix patching in configuration edits. Use
+case-insensitive matching when parsing tool output whose field names vary by
+device class.
+
 ---
 
 ## Open, not closed
@@ -539,6 +644,9 @@ adjacent boundary. Record what a trust change grants, not only what it repairs.
 | F-022 | **Open** — cause unproven, no correction applied. |
 | F-023 | **Closed** 2026-08-17. Cause proven at `mx1`; delivery proven twice. |
 | F-024 | **Closed** 2026-08-17. Specification corrected. |
-| F-025 | **Open** — correction pending operator decision. |
+| F-025 | **Partially corrected** 2026-08-17. Project-local half closed and reboot-proven; **estate half open**, awaiting a CIVICVS decision on `wg-pk`. |
+| F-026 | **Corrected** 2026-08-17. Reboot persistence proven. |
+| F-027 | **Corrected** 2026-08-17. Applies retroactively to F-018's proofs. |
+| F-028 | **Corrected** 2026-08-17. |
 
 Everything else is closed with a proven cause and a proven correction.