Updated

docs/STAGING-STATE.md

@@ -4,7 +4,7 @@ Live state of the Mechanical Compiler staging instance on `srv-b`.
 
 | | |
 |---|---|
-| Updated | 2026-08-15, after network isolation |
+| Updated | 2026-08-16, formal infrastructure acceptance |
 | Instance | Staging / development |
 | Specification | `ENVIRONMENT.md` revision 5 |
 | Failure log | `FAILURES.md` |
@@ -15,24 +15,36 @@ Live state of the Mechanical Compiler staging instance on `srv-b`.
 ## 0. How to use this file
 
 This is the authoritative record of what is true on `srv-b`. Where it and
-`ENVIRONMENT.md` disagree, **this file wins for facts** and the specification
-is defective and must be corrected.
+`ENVIRONMENT.md` disagree, **this file wins for facts** and the specification is
+defective and must be corrected.
 
-Completed and remaining work are in the same document deliberately. They are
-two halves of one boundary; separating them guarantees they drift.
+Completed and remaining work are in the same document deliberately. They are two
+halves of one boundary; separating them guarantees they drift.
 
 Before any command: read this file, confirm the immediately relevant live state
-with a read-only command, then issue one command group. If it fails, record it
-in `FAILURES.md` before changing anything else.
+with a read-only command, then issue one command group. If it fails, record it in
+`FAILURES.md` before changing anything else.
 
-Production will get its own `PRODUCTION-STATE.md`. The specification is shared;
-the state is not.
+Production gets its own `PRODUCTION-STATE.md`. The specification is shared; the
+state is not.
+
+### Acceptance boundary, 2026-08-16
+
+**Staging infrastructure is accepted.** That claim is narrower than "the
+application is deployed," and deliberately so. Three subsystems sit outside it
+and must not be represented as either hidden failures or completed work:
+
+| Subsystem | Status |
+|---|---|
+| Mail alert delivery | Partially configured, characterised, **not accepted end to end** |
+| `smartd` monitoring | Deferred with mail alerting |
+| Backup infrastructure | **Postponed by operator decision** — strategy may change |
 
 ---
 
 ## 1. Instance values
 
-These are the `srv-b` bindings for the parameters in `ENVIRONMENT.md`.
+The `srv-b` bindings for the parameters in `ENVIRONMENT.md`.
 
 ### Host
 
@@ -40,13 +52,16 @@ These are the `srv-b` bindings for the parameters in `ENVIRONMENT.md`.
 hostname          srv-b  /  srv-b.dev.infra
 platform          Proxmox VE 8.4.0, Debian 12, kernel 6.8.12-9-pve
 hardware          HP ProLiant DL360 G7, 2 x Xeon X5650, 24 threads, 31 GiB
-storage           P410i, 4 x EG0146FAWHU, RAID 1+0, all SMART OK
+storage           P410i, 4 x EG0146FAWHU, RAID 1+0, all members SMART OK
 local             directory /var/lib/vz, ~70 GiB free   iso,vztmpl,backup
 local-lvm         LVM-thin pve/data, 166.9 GiB          rootdir,images
 vmbr0             10.0.0.12/24 on enp3s0f0, gw 10.0.0.1   management, LAN
 vmbr1             10.20.0.1/24, bridge-ports none         service, portless
 wg0               10.110.0.12/32, peer wg-pk.civicus.us:51820, allowed 10.110.0.0/22
 resolver          75.75.75.75, search dev.infra
+ip_forward        1
+timezone          America/Chicago, NTP active, clock synchronised
+systemd           running, zero failed units
 spare NICs        enp3s0f1, enp4s0f0, enp4s0f1 — unconfigured, deliberately
 template          local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst
 ```
@@ -62,9 +77,11 @@ template          local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst
 | rootfs | 40 GiB `local-lvm` | 8 GiB `local-lvm` |
 | `mp0` | 60 GiB → `/var/lib/mechcomp`, `backup=0` | — |
 | `net0` | `eth0` on `vmbr1`, `10.20.0.10/24`, gw `10.20.0.1` | `eth0` on `vmbr1`, `10.20.0.11/24`, gw `10.20.0.1` |
-| Debian | 12.15 | 12.15 |
+| Debian | 12 bookworm, fully upgraded | 12 bookworm, fully upgraded |
+| systemd | running, zero failed units | running, zero failed units |
 
-**Neither container has a LAN interface.** See F-017.
+**Neither container has a LAN interface** (F-017). **Neither has a stale `eth1`
+stanza** (F-021) — verified persistent across reboot.
 
 ### Names and identity
 
@@ -73,16 +90,15 @@ mechanical-compiler.dev.infra   ->  10.20.0.11    (CT 101, the proxy)
 mechcomp.dev.infra              ->  10.20.0.10    PVE-generated
 mcproxy.dev.infra               ->  10.20.0.11    PVE-generated
 
-ADMIN_USER                      sandor
-                                uid 1000, groups sandor + mechcomp(996)
-                                shell /bin/bash, both containers
-authorized key                  SHA256:2pNffCscUUW5Wbs9uMepLEvMLKPqUV/7Lpk5tw7iWSY
-                                root@srv-b — bastion pattern, see §4
-service user                    mechcomp, uid 999, gid 996
-                                home /var/www/mechcomp, shell /usr/sbin/nologin
+ADMIN_USER      sandor, uid 1000, groups sandor + mechcomp(996)
+                shell /bin/bash, both containers
+authorized key  SHA256:2pNffCscUUW5Wbs9uMepLEvMLKPqUV/7Lpk5tw7iWSY
+                matches /root/.ssh/id_rsa.pub on srv-b — bastion, see section 4
+service user    mechcomp, uid 999, gid 996
+                home /var/www/mechcomp, shell /usr/sbin/nologin
 ```
 
-### Host NAT, as persisted in `/etc/iptables/rules.v4`
+### Host NAT, persisted in `/etc/iptables/rules.v4`
 
 ```
 nat POSTROUTING
@@ -96,7 +112,8 @@ filter FORWARD
   -s 10.20.0.0/24 -d 10.0.0.0/24            -j DROP         # LAN blocked
 ```
 
-Rule order is load-bearing. `RETURN` must precede both masquerades.
+Rule order is load-bearing. `RETURN` must precede both service-network
+masquerades. Live and persisted states match.
 
 ### TLS
 
@@ -104,15 +121,14 @@ Rule order is load-bearing. `RETURN` must precede both masquerades.
 CA        CN = Mechanical Compiler Staging CA   (locally generated)
 leaf      CN = mechanical-compiler.dev.infra
           SAN = DNS:mechanical-compiler.dev.infra
-validity  2026-08-16 -> 2028-11-18            <-- expires, nothing renews it
+validity  2026-08-16 -> 2028-11-18            <-- nothing renews this
 trusted   srv-b, CT 100, CT 101
-key mode  0600 on CT 101
+key       root:root 0600 /etc/ssl/mechcomp/server.key
 ```
 
-No Kane County Civic Infrastructure CA issuance path exists on `srv-b`: no
-trust anchor, no `step`, no `cfssl`, no EasyRSA. Proxmox's own CA was
-deliberately not reused. Replacing this leaf later is two file copies and a
-reload.
+No Kane County Civic Infrastructure CA issuance path exists on `srv-b`: no trust
+anchor, no `step`, no `cfssl`, no EasyRSA. Proxmox's own CA was deliberately not
+reused. Replacing this leaf later is two file copies and a reload.
 
 ### Application environment
 
@@ -120,7 +136,7 @@ reload.
 
 ```
 MECHCOMP_ENV=staging
-MECHCOMP_BIND=10.20.0.10
+MECHCOMP_BIND=10.20.0.10          # scalar; loopback is NOT bound
 MECHCOMP_PORT=8770
 MECHCOMP_BASE_URL=https://mechanical-compiler.dev.infra
 MECHCOMP_DATA_DIR=/var/lib/mechcomp
@@ -132,6 +148,36 @@ MECHCOMP_ARTIFACT_RETENTION_DAYS=30
 MECHCOMP_SECRET_KEY=<generated, not recorded>
 ```
 
+### Mail, as currently configured
+
+```
+relay endpoint    10.110.0.1:25 over wg0
+banner            wg-pk.diagnostics.kane-il.us, STARTTLS offered
+STARTTLS cert     self-signed, CN = wg-pk
+authentication    unauthenticated accepted from 10.110.0.12
+ports 465 / 587   unavailable; public 198.58.111.109 exposes no SMTP on this path
+srv-b relayhost   [10.110.0.1]:25
+smtp_tls_security_level  may
+smtp_sasl_auth_enable    no
+inet_interfaces   loopback-only
+root alias        sandor@kane-il.us
+local handoff     succeeds, relay returns SMTP 250, queue empties
+FINAL DELIVERY    NOT ACCEPTED — operator received a delivery-failure message
+status            deferred, cause unproven downstream (F-023)
+```
+
+### Placeholder backend — staging scaffold, not application code
+
+```
+/usr/local/libexec/mechcomp-placeholder.py
+/etc/systemd/system/mechcomp-placeholder.service
+runs as mechcomp:mechcomp, reads mechcomp.env, binds 10.20.0.10:8770
+Restart=on-failure, hardening set applied, enabled at multi-user.target
+```
+
+Exists solely to prove the proxy chain independently of the application. It is
+removed when the real service arrives.
+
 ### Rollback copies on disk
 
 ```
@@ -143,7 +189,9 @@ srv-b     /etc/network/interfaces.before-mechcomp
           /root/pct-100.before-svcnet
           /root/pct-101.before-svcnet
 CT 100    /etc/hosts.before-svcfqdn
+          /etc/network/interfaces pre-F-021 copy
 CT 101    /etc/hosts.before-svcfqdn
+          /etc/network/interfaces pre-F-021 copy
 ```
 
 ---
@@ -156,122 +204,136 @@ Each line was demonstrated by command output, not inferred.
 
 - [x] `vmbr1` created, active, `10.20.0.1/24`, portless
 - [x] Duplicate address detection run before each container creation
-- [x] Container → internet reachable (`deb.debian.org`, `gitea.barternetwork.us`, both 200)
-- [x] Container → WireGuard network reachable (`10.110.0.12`)
-- [x] **Container → LAN unreachable** — the isolation requirement
-- [x] Container → `srv-b` reachable at `10.0.0.12` (INPUT path, required)
-- [x] Container ↔ container reachable over `vmbr1`
-- [x] LAN → Proxmox console unaffected (`10.0.0.12:8006` → 200)
-- [x] Network configuration persisted, survives reboot
+- [x] Container to internet reachable (`deb.debian.org`, `gitea.barternetwork.us`)
+- [x] Container to WireGuard reachable (`10.110.0.1`)
+- [x] **Container to LAN gateway `10.0.0.1` = 100% packet loss** — the requirement
+- [x] Container to `srv-b` `10.0.0.12` reachable via INPUT path (required, not a leak)
+- [x] Container to container reachable over `vmbr1`
+- [x] LAN to Proxmox console `10.0.0.12:8006` returns 200, unaffected
+- [x] NAT and FORWARD rules present live **and** persisted, in correct order
+- [x] Host: `running`, zero failed units
 
 ### CT 100
 
-- [x] Debian 12.15, systemd running, zero failed units, no pending upgrades
-- [x] `en_US.UTF-8` generated and active
+- [x] Debian 12 bookworm, fully upgraded, zero pending
+- [x] `en_US.UTF-8` generated and active; `America/Chicago`
+- [x] **`running`, zero failed units — after reboot** (F-021 corrected)
 - [x] `/var/lib/mechcomp` is a real separate ext4 filesystem
-- [x] Service user `mechcomp`, home `/var/www/mechcomp`, writable
-- [x] Application directory layout created with correct ownership
-- [x] Base packages installed; OpenSCAD / Qt / X11 absent — verified `clean`
-- [x] Docker working, `overlay2` / `systemd`, no `fuse-overlayfs` needed
+- [x] Service user `mechcomp`, home `/var/www/mechcomp`, correct ownership
+- [x] All four data subdirectories present, `mechcomp:mechcomp`, `0750`
+- [x] `mechcomp.env` present, `root:mechcomp`, `0640`, secret generated
+- [x] Base packages installed; OpenSCAD, Qt and X11 absent
+- [x] Docker active, `overlay2` / `systemd`, no fallback needed
 - [x] Repository cloned at `e85c4f4e`, verified as the owning user
 - [x] Python 3.11.2 venv created, owned by `mechcomp`
-- [x] `mechcomp.env` written with generated secret
-- [x] `openssh-server` installed, enabled, active
-- [x] Application listener bound to service network only — LAN-side bind refused
+- [x] `openssh-server` enabled and active
+- [x] **`mechcomp-placeholder.service` enabled, active, reboot-persistent** (F-019)
+- [x] Listener on `10.20.0.10:8770` only — not `0.0.0.0`
 
 ### CT 101
 
-- [x] Debian 12.15, systemd running, zero failed units after `nesting=1`
-- [x] `en_US.UTF-8` generated and active
-- [x] nginx 1.22.1 installed, `nginx -t` passes
-- [x] Debian `default` site removed; only the project vhost is enabled
+- [x] Debian 12 bookworm, fully upgraded, zero pending
+- [x] `en_US.UTF-8` generated and active; `America/Chicago`
+- [x] **`running`, zero failed units — after reboot** (F-021 corrected)
+- [x] nginx 1.22.1, `nginx -t` passes, enabled and active
+- [x] Only the project vhost enabled; Debian `default` removed
+- [x] `listen 443 ssl default_server` on both address families
+- [x] HTTP to HTTPS 301 redirect
 - [x] Local CA created, leaf issued, trusted on all three hosts
-- [x] HTTP → HTTPS redirect, TLS termination, proxy to `10.20.0.10:8770`
-- [x] `openssh-server` installed, enabled, active
-- [x] **`X-Forwarded-Proto: https` observed at the backend** — needs re-proof, see §3
+- [x] **HTTPS end to end from `srv-b`, CT 100 and CT 101 without `-k`**
+- [x] **`X-Forwarded-Proto: https` observed at the backend** — re-proved after the
+      topology change and again after CT 100's reboot
+- [x] `openssh-server` enabled and active
 
 ---
 
 ## 3. Remaining — infrastructure
 
-In dependency order. Application-dependent work is in §5.
-
 ### Immediate
 
-- [ ] **`mechcomp-placeholder.service`** — recreate the backend as a supervised
-      unit. It died on the F-017 reboot and nothing currently listens on 8770.
-      Everything below that touches the proxy depends on this. See F-019.
-- [ ] **Re-prove `X-Forwarded-Proto`** end to end. The earlier proof was taken
-      when clients were on `10.0.0.x`; header values will now read `10.20.0.x`.
-      Re-establish rather than assume it survived the topology change.
-- [ ] **`default_server`** on the CT 101 vhost, both `listen 443` lines. Two
-      words. Prevents catch-all behaviour depending on file ordering once a
-      second server block exists. See F-012.
+Nothing. The infrastructure boundary is accepted.
 
-### Mail — blocks alerting
+### Deferred by operator decision
 
-- [ ] **Postfix relay.** Currently `inet_interfaces = loopback-only`,
-      `relayhost` empty, no `root:` alias. Alerts go to a mailbox nobody opens.
-- [ ] **`root:` alias** to a real destination.
-- [ ] Needs: address of the `wg-pk` relay, and whether it accepts
-      unauthenticated from `10.110.0.12`. **Open question for CIVICVS.**
+Recorded here so they are visually distinct from failures and from forgotten
+work. None is a defect.
 
-### Monitoring and backup
+- [ ] **Mail end-to-end delivery** (F-023). Handoff to `wg-pk` works; final
+      delivery does not. Leading untested hypothesis: envelope sender
+      `root@srv-b.dev.infra` rejected on sender-domain verification, since
+      `dev.infra` does not resolve publicly. Candidate remedies `myorigin` or
+      `smtp_generic_maps`. Check the `postfix check` chroot divergence first.
+- [ ] **`smartd` explicit four-member configuration.** The package is installed
+      and the service is active, but `DEVICESCAN` currently monitors **zero
+      devices**; the P410i members are visible only through explicit
+      `-d cciss,N`. Active is not the same as monitoring. Deferred with mail.
+- [ ] **Backup infrastructure, entirely.** Postponed 2026-08-16 because the
+      strategy may change: `vzdump` job, archive sizing, retention, free-space
+      guard, host-side pull, `mechcomp-backup`, backup alerting, gold media,
+      3+ TB redundancy.
 
-- [ ] **`smartd`** on `/dev/sda -d cciss,0` through `cciss,3`. Depends on mail.
-- [ ] **`vzdump` job** — both containers, `local`, snapshot, zstd, 02:30.
-      `mp0` already carries `backup=0`.
-- [ ] **First `vzdump` run — report the archive size.** Retention is a
-      placeholder `keep-last=3` until this number exists.
-- [ ] **Free-space guard** — refuse and alert below 20 GiB on `/var/lib/vz`.
-- [ ] `/var/lib/vz/mechcomp-app/` and the host-side pull script.
-- [ ] `mechcomp-backup --stdout` in CT 100. Structure can be built and tested
-      against the current data directories without application output.
+**One request standing against the postponement:** a single manual `vzdump` of
+both containers as a point-in-time snapshot before further change. It
+presupposes nothing about the eventual strategy and yields the archive size that
+has been an open question since revision 4. Four 2010-vintage disks currently
+hold the only instance with no monitoring, no alerting and no backup.
 
-### Gold media
+### Optional, recorded not scheduled
 
-- [ ] 32 GB USB stick: LUKS2, ext4, label `MC-GOLD-01`, `/mnt/gold` `noauto`.
-- [ ] `gold-archive.sh` with `SHA256SUMS` verification before unmount.
-- [ ] **Open:** where the 3+ TB USB disk is attached now that `annales` is out
-      of scope. Until known, stick-to-disk copying is a manual step.
+- [ ] Second `nameserver` line in container resolver configuration, as a
+      mitigation for F-022. One line, no daemon. Deliberately not applied on a
+      single unexplained event.
+- [ ] Staging certificate expires 2028-11-18 and nothing renews it.
 
 ---
 
 ## 4. Access model
 
-Confirmed by CIVICVS: no workstation access from the home LAN is required.
+Confirmed: no workstation access from the home LAN is required.
 
 ```
 internet -> WireGuard -> srv-b -> containers
 ```
 
-`srv-b` is the bastion. The authorized key is `root@srv-b`, which is
-consistent: root on the host can `pct enter` regardless, so SSH to the
-containers adds no privilege. It does mean **every path to a container runs
-through `srv-b`** — deliberate, not a limitation.
+`srv-b` is the bastion, confirmed by the authorized key matching
+`/root/.ssh/id_rsa.pub` on the host. Root on the host can `pct enter` regardless,
+so SSH adds no privilege — but **every path to a container runs through
+`srv-b`**, which is deliberate.
 
-If direct WireGuard-side access to the catalogue is wanted later, that is a
-route addition for `10.20.0.0/24` on the hub. Not now.
+Direct WireGuard-side access to the catalogue would be a route addition for
+`10.20.0.0/24` on the hub. Not now.
 
-No DHCP anywhere. Two containers with fixed addresses on a portless bridge is
-the entire address space; a DHCP server would add a daemon, a lease database
-and a failure mode in exchange for nothing.
+No DHCP anywhere. Two containers with fixed addresses on a portless bridge is the
+entire address space.
 
 ---
 
 ## 5. Blocked on application code
 
-None of this can be honestly completed while the repository is `LICENSE` and
-`README.md`. It is not provisioning work and should not be attempted as such.
+Repository state:
 
-- [ ] `requirements-base.txt` / `requirements-cad.txt` and dependency install
-- [ ] `mechcomp.service` and `mechcomp-worker.service`
+```
+HEAD        e85c4f4e5bab9a4f032230c99aa6784ace4c80e2
+top level   .git  LICENSE  README.md  venv
+git status  ?? venv/
+absent      requirements-base.txt, requirements-cad.txt, service units
+```
+
+None of the following can be honestly completed, and none may be fabricated by
+provisioning:
+
+- [ ] Dependency install from committed manifests
+- [ ] `mechcomp.service`, `mechcomp-worker.service`
 - [ ] Reference toolchain image `mechcomp/reference-toolchain:8.0.0`
-- [ ] Fixture reproduction against `ddd0f154…`
+- [ ] Fixture reproduction against `ddd0f154...`
 - [ ] Application-runtime acceptance
 - [ ] Replacing the placeholder with the real service
 
-The Shapely port is the architect's work item and gates all of the above.
+**Architect decision:** `venv/` belongs in `.gitignore` — it is a legitimate
+artifact inside `install_dir` per YunoHost convention, it simply should not be
+tracked. Applied with the first application commit.
+
+The Shapely port gates all of the above.
 
 ---
 
@@ -279,11 +341,11 @@ The Shapely port is the architect's work item and gates all of the above.
 
 | # | Question | Blocks |
 |---|---|---|
-| 1 | `wg-pk` relay address; unauthenticated from `10.110.0.12`? | mail, `smartd`, backup alerts |
-| 2 | Where is the 3+ TB USB disk attached? | gold redundancy step |
-| 3 | First `vzdump` archive size | final retention value |
+| 1 | Why does delivery fail downstream of `wg-pk`? | mail, `smartd`, backup alerting |
+| 2 | What is the backup strategy? | all backup work |
+| 3 | Where is the 3+ TB USB disk attached? | gold redundancy step |
 
-Question 3 is answered by doing. Questions 1 and 2 need CIVICVS.
+Question 1 is answered by diagnosis. Questions 2 and 3 need CIVICVS.
 
 ---
 
@@ -291,9 +353,11 @@ Question 3 is answered by doing. Questions 1 and 2 need CIVICVS.
 
 | Question | Answer |
 |---|---|
-| Kane County CA issuance path | None on `srv-b`. Local staging CA generated instead. |
+| Kane County CA issuance path | None on `srv-b`. Local staging CA generated. |
+| Relay address and authentication | `10.110.0.1:25` over `wg0`, unauthenticated from `10.110.0.12` accepted. |
 | `openssh-server` present | Yes, both containers. |
-| `ADMIN_USER` / key | `sandor`; `root@srv-b` key, bastion pattern. |
-| DHCP pool on `10.0.0.0/24` | **Not applicable.** Containers are no longer on that network. |
+| `ADMIN_USER` / key | `sandor`; `root@srv-b` key, bastion pattern confirmed. |
+| DHCP pool on `10.0.0.0/24` | **Not applicable.** Containers are not on that network. |
 | Docker storage driver | `overlay2` / `systemd`. No fallback needed. |
 | LAN workstation access | Not required. WireGuard through `srv-b`. |
+| `MECHCOMP_BIND` semantics | Scalar, service address only. Loopback not bound. |