diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index b5e8165..5e8eb2d 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -4,9 +4,9 @@ Specification for a Mechanical Compiler instance. | | | |---|---| -| Revision | 5 (2026-08-16) | +| Revision | 5.1 (2026-08-17) | | Supersedes | Revisions 1 through 4 | -| Basis | Revision 4, reconciled against the proven staging build on `srv-b` | +| Basis | Revision 4, reconciled against the proven staging build, then work order 002 | | Scope | **Host-agnostic.** Applies to any instance. | | Instance state | `STAGING-STATE.md`, and later `PRODUCTION-STATE.md` | | Failure evidence | `FAILURES.md` | @@ -623,19 +623,45 @@ Nothing else. ## 12. Mail and monitoring -**DEFERRED — end-to-end alerting is not accepted.** See F-023. +**REQ** — **SMTP 250 from a relay and an empty local queue prove handoff, not +delivery.** Alerting acceptance requires demonstrated end-to-end receipt, twice, +with full headers captured. This is the whole content of F-023: a message can be +accepted by every hop and still be rejected at the last one. -**REQ when resumed** — **SMTP 250 from a relay and an empty local queue prove -handoff, not delivery.** Alerting acceptance requires demonstrated end-to-end -receipt. Until then, mail, disk monitoring and any mail-dependent backup -alerting are unaccepted. +**REQ** — Diagnose relay problems with **RCPT-only probes, over every address +family, before and after any change**, with no message body sent. A `554` +before and a `250` after proves the change caused the fix rather than +coinciding with it. + +**REQ (F-024)** — On a Proxmox host, inspect mail logs with +`journalctl -u postfix@-`. There is no `rsyslog` and no `/var/log/mail.log`. +Automation that greps a logfile path finds nothing silently, which is worse than +failing. **REQ** — Disk monitoring must be configured **explicitly per device** where the controller does not present members to automatic scanning. An active `smartmontools.service` monitoring zero devices is not monitoring, and is easy -to mistake for success. +to mistake for success. Acceptance is a delivered test alert, not an active +unit. ---- +### 12.1 Downstream infrastructure is out of bounds + +**REQ** — An instance may configure its own relay client. It may not alter the +mail infrastructure it relays through. Where a downstream change is genuinely +required, it is escalated with evidence, not made locally. + +Two standing constraints apply to this estate: + +| Host | Constraint | +|---|---| +| The final mailbox host | Runs YunoHost. Its mail configuration must not be altered — which is the reason a separate relay exists. | +| The relay MTA | Uses DANE. Its TLS and certificate configuration must not be broken. Client-trust changes such as `mynetworks` do not interact with DANE; certificate changes do. | + +**REQ (F-025)** — Record what a trust change **grants**, not only what it +repairs. `permit_mynetworks` is destination-agnostic: authorising a client to +fix one rejected recipient authorises it for every destination. Where an +instance's containers have no reason to originate mail, block SMTP egress at the +instance boundary rather than relying on downstream policy to refuse it. ## 13. Instance parameters @@ -702,10 +728,12 @@ scaffolding survives a reboot. **PROVEN (F-021)** — the reboot is not optional. Connectivity can be perfect while boot is degraded. -### Gate 2 — Deferred operational services +### Gate 2 — Operational services -Mail delivered **end to end and received**. Disk monitoring reporting a non-zero -device count. Not gated on gate 1. +Mail delivered **end to end and received twice**, with full headers captured, an +empty queue and no deferred entries. Disk monitoring reporting a **non-zero +device count** and a delivered test alert. Container SMTP egress blocked where +the containers have no reason to originate mail. Not gated on gate 1. ### Gate 3 — Application runtime diff --git a/docs/FAILURES.md b/docs/FAILURES.md index 4df101b..acb4d4b 100644 --- a/docs/FAILURES.md +++ b/docs/FAILURES.md @@ -6,7 +6,7 @@ Compiler environment. | | | |---|---| | Scope | All instances. Staging entries are marked `srv-b`. | -| Updated | 2026-08-16, after staging acceptance | +| Updated | 2026-08-17, after work order 002 | | Rule | Append only. Never edit an entry except to add a `Resolution` line. | | Numbering | Sequential, never reused. See §0 on the renumbering. | @@ -445,6 +445,87 @@ divergence between `/var/spool/postfix` copies and their host originals, including `/etc/hosts` and NSS libraries — a known cause of resolution failure inside the chroot, and adjacent enough to this failure to be checked first. +**Resolution (2026-08-17):** Cause proven, three hops downstream of the origin. +`mx1` runs `permit_mynetworks, permit_auth_destination, reject` on both relay +and recipient restrictions. `wg-pk` connected from public addresses absent from +`mx1`'s `mynetworks`, and `kane-il.us` is not an authorised destination there, +so `RCPT TO ` was rejected `554 5.7.1 Access denied` over +both IPv4 and IPv6. Added only `198.58.111.109/32` and +`[2600:3c00::f03c:92ff:fe42:43d7]/128` to `mx1`'s `mynetworks` and reloaded. + +Four hypotheses were tested and ruled out with evidence before any change: the +`srv-b` alias (`postalias -q root` resolved correctly and the journal showed +`orig_to=` forwarded), sender-domain rejection (`wg-pk` rewrites the +sender to `postmaster@diagnostics.kane-il.us` and `MAIL FROM` was accepted), +address-family asymmetry (both families failed identically), and routing on +`wg-pk` (it connected and received the rejection). + +Method worth reusing: **RCPT-only probes before and after the change, over both +address families, with no message body.** `554` before, `250` after. That +proves the change caused the fix rather than coinciding with it — the standard +F-012 was written to enforce. Two messages then delivered end to end with full +headers captured, empty queue, no deferred entries. **Closed.** + +--- + +### F-024 — work order specified a log file that does not exist +`srv-b`. Work order 002. + +**Observed:** WORK ORDER 002 instructed `grep /var/log/mail.log`. The file does +not exist on `srv-b`. +**Cause:** **Proven.** Proxmox VE ships without `rsyslog`. Postfix logs to +journald only. +**Correction:** The operator used `journalctl -u postfix@-` and completed the +diagnosis. No configuration was changed; installing `rsyslog` to satisfy a +document would have been the wrong direction. +**Consequence:** **Specification defect.** All log inspection on a Proxmox host +uses `journalctl`, not files under `/var/log`. Corrected in `ENVIRONMENT.md` +revision 5. Automation that greps a logfile path will silently find nothing, +which is worse than failing. + +--- + +### F-025 — the F-023 correction granted wider relay than required +`mx1`, `wg-pk`. Estate scope. + +**Observed:** Adding `wg-pk`'s two public addresses to `mx1`'s `mynetworks` +authorises them under `permit_mynetworks`, which appears in **both** +`smtpd_relay_restrictions` and `smtpd_recipient_restrictions`. That grants relay +to **any** destination, not only to `kane-il.us`. + +`wg-pk` in turn carries `mynetworks = 10.110.0.0/22` with +`permit_mynetworks permit_sasl_authenticated defer_unauth_destination`. The +resulting chain: + +``` +any peer on 10.110.0.0/22 (including CT 100 and CT 101, which arrive + as 10.110.0.12 through the srv-b masquerade) + -> wg-pk permit_mynetworks + -> mx1 permit_mynetworks + -> any destination on the internet, as kane-il.us infrastructure +``` + +Before the correction `mx1` rejected at the final hop, so the path was closed by +accident rather than by policy. + +**Cause:** **Proven.** `permit_mynetworks` is destination-agnostic by design. +**Correction:** **Pending — operator decision.** Two independent parts: + +- *Ours:* block SMTP egress from the container network at `srv-b`. The + containers have no reason to originate mail; if they ever should, that is a + deliberate decision rather than something inherited from a masquerade. Local, + precise, touches no estate policy. +- *Estate:* whether `wg-pk`'s `mynetworks` should be explicit `/32` entries for + the hosts that legitimately originate mail rather than the whole tunnel range. + +**Constraint:** `wg-pk` must continue to relay to arbitrary external +destinations — that is its purpose, since Hubzilla registration and notification +mail depends on it and the ISP blocks port 25. Restricting it by *recipient* +would break that. The question is which clients may ask, not where it may send. + +**Consequence:** A correction that fixes the observed failure may widen an +adjacent boundary. Record what a trust change grants, not only what it repairs. + --- ## Open, not closed @@ -456,6 +537,8 @@ inside the chroot, and adjacent enough to this failure to be checked first. | F-019 | **Corrected** 2026-08-16. Reboot persistence proven. | | F-021 | **Corrected** 2026-08-16. Reboot persistence proven. | | F-022 | **Open** — cause unproven, no correction applied. | -| F-023 | **Deferred** — downstream mail failure, cause unproven. Hypothesis recorded. | +| F-023 | **Closed** 2026-08-17. Cause proven at `mx1`; delivery proven twice. | +| F-024 | **Closed** 2026-08-17. Specification corrected. | +| F-025 | **Open** — correction pending operator decision. | Everything else is closed with a proven cause and a proven correction. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 2da85b3..c41604a 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -4,7 +4,7 @@ What the Mechanical Compiler is for, and the order in which it gets built. | | | |---|---| -| Updated | 2026-08-16 | +| Updated | 2026-08-17 | | Companions | `ENVIRONMENT.md`, `STAGING-STATE.md`, `FAILURES.md` | --- @@ -127,6 +127,8 @@ a new call into the same machinery, not a new machine. explicit disk monitoring are deferred; backup strategy is postponed by operator decision; application deployment remains blocked on application code. See `STAGING-STATE.md`. +- **Mail alerting accepted, 2026-08-17.** Delivered end to end and received + twice. `smartd` is now unblocked. One open exposure is recorded as F-025. ### Next — the port @@ -246,3 +248,9 @@ on whether it makes distributed manufacturing capacity legible. 8. **Handoff is not delivery.** An upstream acceptance code proves the message left, not that it arrived (F-023). The same distinction applies wherever a subsystem reports success on behalf of something downstream. +9. **Record what a change grants, not only what it repairs.** A trust rule + added to fix one rejection may authorise far more than the case that + prompted it (F-025). +10. **Shared infrastructure is not ours to reconfigure.** An instance may + configure its own client side; changes further down the chain are escalated + with evidence. diff --git a/docs/STAGING-STATE.md b/docs/STAGING-STATE.md index 448b10c..9fa1396 100644 --- a/docs/STAGING-STATE.md +++ b/docs/STAGING-STATE.md @@ -4,7 +4,7 @@ Live state of the Mechanical Compiler staging instance on `srv-b`. | | | |---|---| -| Updated | 2026-08-16, formal infrastructure acceptance | +| Updated | 2026-08-17, after work order 002 (mail) | | Instance | Staging / development | | Specification | `ENVIRONMENT.md` revision 5 | | Failure log | `FAILURES.md` | @@ -36,8 +36,8 @@ and must not be represented as either hidden failures or completed work: | Subsystem | Status | |---|---| -| Mail alert delivery | Partially configured, characterised, **not accepted end to end** | -| `smartd` monitoring | Deferred with mail alerting | +| Mail alert delivery | **Accepted 2026-08-17.** Delivered end to end, twice, headers captured. | +| `smartd` monitoring | Deferred. Now unblocked — mail works. | | Backup infrastructure | **Postponed by operator decision** — strategy may change | --- @@ -148,24 +148,43 @@ MECHCOMP_ARTIFACT_RETENTION_DAYS=30 MECHCOMP_SECRET_KEY= ``` -### Mail, as currently configured +### Mail — accepted 2026-08-17 ``` -relay endpoint 10.110.0.1:25 over wg0 -banner wg-pk.diagnostics.kane-il.us, STARTTLS offered -STARTTLS cert self-signed, CN = wg-pk -authentication unauthenticated accepted from 10.110.0.12 -ports 465 / 587 unavailable; public 198.58.111.109 exposes no SMTP on this path -srv-b relayhost [10.110.0.1]:25 -smtp_tls_security_level may -smtp_sasl_auth_enable no +srv-b relayhost [10.110.0.1]:25 over wg0 +root alias sandor@kane-il.us (postalias verified) inet_interfaces loopback-only -root alias sandor@kane-il.us -local handoff succeeds, relay returns SMTP 250, queue empties -FINAL DELIVERY NOT ACCEPTED — operator received a delivery-failure message -status deferred, cause unproven downstream (F-023) +delivery path srv-b -> wg-pk -> mx1 -> kane-il.us +status DELIVERED end to end, twice, full headers captured + queue empty, no bounced or deferred entries ``` +F-023 was proven to be a relay-trust mismatch three hops downstream: `mx1` +rejected `RCPT TO` with `554 5.7.1 Access denied` because `wg-pk`'s public +addresses were absent from its `mynetworks`. Corrected by adding only those two +addresses. + +**Standing constraints on this path — do not violate:** + +| Host | Constraint | +|---|---| +| `kane-il.us` | Runs YunoHost. **Its mail configuration must not be altered.** This is why `mx1` exists. | +| `mx1.diagnostics.kane-il.us` | Uses DANE. **Its TLS and certificate configuration must not be broken.** The `mynetworks` change is inbound client trust and does not interact with DANE. | +| `wg-pk` | Must continue relaying to arbitrary external destinations. Hubzilla registration and notification mail depends on it, and the ISP blocks port 25. Do not restrict it by recipient. | + +**Fragility to know about:** delivery now depends on Linode not reassigning +`198.58.111.109` or `2600:3c00::f03c:92ff:fe42:43d7`. If either changes, mail +stops silently and the cause is in `mx1`'s `mynetworks`. + +**Open exposure — see F-025.** The correction authorises `wg-pk` under +`permit_mynetworks`, which grants relay to any destination. Combined with +`wg-pk`'s own `mynetworks = 10.110.0.0/22`, every tunnel peer — including both +containers, which arrive as `10.110.0.12` through the host masquerade — can +originate mail as `kane-il.us` infrastructure. Correction pending decision. + +**Logging note (F-024):** `/var/log/mail.log` does not exist. Proxmox ships +without `rsyslog`; Postfix logs to journald. Use `journalctl -u postfix@-`. + ### Placeholder backend — staging scaffold, not application code ``` @@ -212,6 +231,7 @@ Each line was demonstrated by command output, not inferred. - [x] LAN to Proxmox console `10.0.0.12:8006` returns 200, unaffected - [x] NAT and FORWARD rules present live **and** persisted, in correct order - [x] Host: `running`, zero failed units +- [x] **Mail delivered end to end from `root` on `srv-b`, twice, headers captured** ### CT 100 @@ -258,15 +278,19 @@ Nothing. The infrastructure boundary is accepted. Recorded here so they are visually distinct from failures and from forgotten work. None is a defect. -- [ ] **Mail end-to-end delivery** (F-023). Handoff to `wg-pk` works; final - delivery does not. Leading untested hypothesis: envelope sender - `root@srv-b.dev.infra` rejected on sender-domain verification, since - `dev.infra` does not resolve publicly. Candidate remedies `myorigin` or - `smtp_generic_maps`. Check the `postfix check` chroot divergence first. +- [ ] **SMTP egress block for the container network** (F-025). The containers + have no reason to originate mail. A `FORWARD` DROP on ports 25/465/587 + from the service network to `10.110.0.0/22` is local, precise, and touches + no estate policy. `srv-b`'s own alerts are unaffected — they originate + locally and take OUTPUT, never FORWARD. +- [ ] **`wg-pk` `mynetworks` scope** (F-025). Estate decision: whether every + tunnel peer should originate mail as `kane-il.us` infrastructure, or only + the hosts that legitimately do. Not this project's to change unilaterally. - [ ] **`smartd` explicit four-member configuration.** The package is installed and the service is active, but `DEVICESCAN` currently monitors **zero devices**; the P410i members are visible only through explicit - `-d cciss,N`. Active is not the same as monitoring. Deferred with mail. + `-d cciss,N`. Active is not the same as monitoring. **Now unblocked** — + mail is accepted, so `-M test` gives a real end-to-end acceptance. - [ ] **Backup infrastructure, entirely.** Postponed 2026-08-16 because the strategy may change: `vzdump` job, archive sizing, retention, free-space guard, host-side pull, `mechcomp-backup`, backup alerting, gold media, @@ -341,11 +365,12 @@ The Shapely port gates all of the above. | # | Question | Blocks | |---|---|---| -| 1 | Why does delivery fail downstream of `wg-pk`? | mail, `smartd`, backup alerting | -| 2 | What is the backup strategy? | all backup work | -| 3 | Where is the 3+ TB USB disk attached? | gold redundancy step | +| 1 | Should container SMTP egress be blocked at `srv-b`? | F-025, ours to fix | +| 2 | Should `wg-pk` `mynetworks` narrow to explicit hosts? | F-025, estate decision | +| 3 | What is the backup strategy? | all backup work | +| 4 | Where is the 3+ TB USB disk attached? | gold redundancy step | -Question 1 is answered by diagnosis. Questions 2 and 3 need CIVICVS. +All four need CIVICVS. --- @@ -361,3 +386,5 @@ Question 1 is answered by diagnosis. Questions 2 and 3 need CIVICVS. | Docker storage driver | `overlay2` / `systemd`. No fallback needed. | | LAN workstation access | Not required. WireGuard through `srv-b`. | | `MECHCOMP_BIND` semantics | Scalar, service address only. Loopback not bound. | +| Why did delivery fail downstream? | `mx1` relay trust. Proven and corrected (F-023). | +| Where does Postfix log on this host? | journald. No `rsyslog`, no `/var/log/mail.log` (F-024). |