From eac601ed99d678e7db3403abed7e9822e94b8d91 Mon Sep 17 00:00:00 2001 From: TheRON Date: Sun, 16 Aug 2026 12:01:20 -0400 Subject: [PATCH] Current state Live state of the Mechanical Compiler staging instance on `srv-b`. --- docs/STAGING-STATE.md | 299 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 299 insertions(+) create mode 100644 docs/STAGING-STATE.md diff --git a/docs/STAGING-STATE.md b/docs/STAGING-STATE.md new file mode 100644 index 0000000..194e95a --- /dev/null +++ b/docs/STAGING-STATE.md @@ -0,0 +1,299 @@ +# STAGING-STATE.md + +Live state of the Mechanical Compiler staging instance on `srv-b`. + +| | | +|---|---| +| Updated | 2026-08-15, after network isolation | +| Instance | Staging / development | +| Specification | `ENVIRONMENT.md` revision 5 | +| Failure log | `FAILURES.md` | +| Method | Manual, one command group at a time | + +--- + +## 0. How to use this file + +This is the authoritative record of what is true on `srv-b`. Where it and +`ENVIRONMENT.md` disagree, **this file wins for facts** and the specification +is defective and must be corrected. + +Completed and remaining work are in the same document deliberately. They are +two halves of one boundary; separating them guarantees they drift. + +Before any command: read this file, confirm the immediately relevant live state +with a read-only command, then issue one command group. If it fails, record it +in `FAILURES.md` before changing anything else. + +Production will get its own `PRODUCTION-STATE.md`. The specification is shared; +the state is not. + +--- + +## 1. Instance values + +These are the `srv-b` bindings for the parameters in `ENVIRONMENT.md`. + +### Host + +``` +hostname srv-b / srv-b.dev.infra +platform Proxmox VE 8.4.0, Debian 12, kernel 6.8.12-9-pve +hardware HP ProLiant DL360 G7, 2 x Xeon X5650, 24 threads, 31 GiB +storage P410i, 4 x EG0146FAWHU, RAID 1+0, all SMART OK +local directory /var/lib/vz, ~70 GiB free iso,vztmpl,backup +local-lvm LVM-thin pve/data, 166.9 GiB rootdir,images +vmbr0 10.0.0.12/24 on enp3s0f0, gw 10.0.0.1 management, LAN +vmbr1 10.20.0.1/24, bridge-ports none service, portless +wg0 10.110.0.12/32, peer wg-pk.civicus.us:51820, allowed 10.110.0.0/22 +resolver 75.75.75.75, search dev.infra +spare NICs enp3s0f1, enp4s0f0, enp4s0f1 — unconfigured, deliberately +template local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst +``` + +### Containers + +| | CT 100 | CT 101 | +|---|---|---| +| hostname | `mechcomp` | `mcproxy` | +| role | application, worker | reverse proxy, TLS | +| features | `nesting=1,keyctl=1` | `nesting=1` | +| cores / RAM / swap | 16 / 16384 / 4096 | 2 / 1024 / 512 | +| rootfs | 40 GiB `local-lvm` | 8 GiB `local-lvm` | +| `mp0` | 60 GiB → `/var/lib/mechcomp`, `backup=0` | — | +| `net0` | `eth0` on `vmbr1`, `10.20.0.10/24`, gw `10.20.0.1` | `eth0` on `vmbr1`, `10.20.0.11/24`, gw `10.20.0.1` | +| Debian | 12.15 | 12.15 | + +**Neither container has a LAN interface.** See F-017. + +### Names and identity + +``` +mechanical-compiler.dev.infra -> 10.20.0.11 (CT 101, the proxy) +mechcomp.dev.infra -> 10.20.0.10 PVE-generated +mcproxy.dev.infra -> 10.20.0.11 PVE-generated + +ADMIN_USER sandor + uid 1000, groups sandor + mechcomp(996) + shell /bin/bash, both containers +authorized key SHA256:2pNffCscUUW5Wbs9uMepLEvMLKPqUV/7Lpk5tw7iWSY + root@srv-b — bastion pattern, see §4 +service user mechcomp, uid 999, gid 996 + home /var/www/mechcomp, shell /usr/sbin/nologin +``` + +### Host NAT, as persisted in `/etc/iptables/rules.v4` + +``` +nat POSTROUTING + -s 10.20.0.0/24 -d 10.0.0.0/24 -j RETURN # LAN not translated + -s 10.0.0.0/24 -o wg0 -j MASQUERADE # pre-existing + -s 10.20.0.0/24 -o wg0 -j MASQUERADE # containers -> WireGuard + -s 10.20.0.0/24 -o vmbr0 -j MASQUERADE # containers -> internet + +filter FORWARD + -s 10.20.0.0/24 -d 10.20.0.0/24 -j ACCEPT # container <-> container + -s 10.20.0.0/24 -d 10.0.0.0/24 -j DROP # LAN blocked +``` + +Rule order is load-bearing. `RETURN` must precede both masquerades. + +### TLS + +``` +CA CN = Mechanical Compiler Staging CA (locally generated) +leaf CN = mechanical-compiler.dev.infra + SAN = DNS:mechanical-compiler.dev.infra +validity 2026-08-16 -> 2028-11-18 <-- expires, nothing renews it +trusted srv-b, CT 100, CT 101 +key mode 0600 on CT 101 +``` + +No Kane County Civic Infrastructure CA issuance path exists on `srv-b`: no +trust anchor, no `step`, no `cfssl`, no EasyRSA. Proxmox's own CA was +deliberately not reused. Replacing this leaf later is two file copies and a +reload. + +### Application environment + +`/etc/mechcomp/mechcomp.env`, `root:mechcomp`, `0640`: + +``` +MECHCOMP_ENV=staging +MECHCOMP_BIND=10.20.0.10 +MECHCOMP_PORT=8770 +MECHCOMP_BASE_URL=https://mechanical-compiler.dev.infra +MECHCOMP_DATA_DIR=/var/lib/mechcomp +MECHCOMP_LOG_LEVEL=info +MECHCOMP_DB_URL=sqlite:////var/lib/mechcomp/db/mechcomp.sqlite3 +MECHCOMP_WORKER_CONCURRENCY=4 +MECHCOMP_CAD_BACKEND=none +MECHCOMP_ARTIFACT_RETENTION_DAYS=30 +MECHCOMP_SECRET_KEY= +``` + +### Rollback copies on disk + +``` +srv-b /etc/network/interfaces.before-mechcomp + /etc/hosts.before-mechcomp + /etc/hosts.before-svcfqdn + /etc/iptables/rules.v4.before-svcnat + /etc/iptables/rules.v4.before-lanblock + /root/pct-100.before-svcnet + /root/pct-101.before-svcnet +CT 100 /etc/hosts.before-svcfqdn +CT 101 /etc/hosts.before-svcfqdn +``` + +--- + +## 2. Verified + +Each line was demonstrated by command output, not inferred. + +### Host and network + +- [x] `vmbr1` created, active, `10.20.0.1/24`, portless +- [x] Duplicate address detection run before each container creation +- [x] Container → internet reachable (`deb.debian.org`, `gitea.barternetwork.us`, both 200) +- [x] Container → WireGuard network reachable (`10.110.0.12`) +- [x] **Container → LAN unreachable** — the isolation requirement +- [x] Container → `srv-b` reachable at `10.0.0.12` (INPUT path, required) +- [x] Container ↔ container reachable over `vmbr1` +- [x] LAN → Proxmox console unaffected (`10.0.0.12:8006` → 200) +- [x] Network configuration persisted, survives reboot + +### CT 100 + +- [x] Debian 12.15, systemd running, zero failed units, no pending upgrades +- [x] `en_US.UTF-8` generated and active +- [x] `/var/lib/mechcomp` is a real separate ext4 filesystem +- [x] Service user `mechcomp`, home `/var/www/mechcomp`, writable +- [x] Application directory layout created with correct ownership +- [x] Base packages installed; OpenSCAD / Qt / X11 absent — verified `clean` +- [x] Docker working, `overlay2` / `systemd`, no `fuse-overlayfs` needed +- [x] Repository cloned at `e85c4f4e`, verified as the owning user +- [x] Python 3.11.2 venv created, owned by `mechcomp` +- [x] `mechcomp.env` written with generated secret +- [x] `openssh-server` installed, enabled, active +- [x] Application listener bound to service network only — LAN-side bind refused + +### CT 101 + +- [x] Debian 12.15, systemd running, zero failed units after `nesting=1` +- [x] `en_US.UTF-8` generated and active +- [x] nginx 1.22.1 installed, `nginx -t` passes +- [x] Debian `default` site removed; only the project vhost is enabled +- [x] Local CA created, leaf issued, trusted on all three hosts +- [x] HTTP → HTTPS redirect, TLS termination, proxy to `10.20.0.10:8770` +- [x] `openssh-server` installed, enabled, active +- [x] **`X-Forwarded-Proto: https` observed at the backend** — needs re-proof, see §3 + +--- + +## 3. Remaining — infrastructure + +In dependency order. Application-dependent work is in §5. + +### Immediate + +- [ ] **`mechcomp-placeholder.service`** — recreate the backend as a supervised + unit. It died on the F-017 reboot and nothing currently listens on 8770. + Everything below that touches the proxy depends on this. See F-019. +- [ ] **Re-prove `X-Forwarded-Proto`** end to end. The earlier proof was taken + when clients were on `10.0.0.x`; header values will now read `10.20.0.x`. + Re-establish rather than assume it survived the topology change. +- [ ] **`default_server`** on the CT 101 vhost, both `listen 443` lines. Two + words. Prevents catch-all behaviour depending on file ordering once a + second server block exists. See F-012. + +### Mail — blocks alerting + +- [ ] **Postfix relay.** Currently `inet_interfaces = loopback-only`, + `relayhost` empty, no `root:` alias. Alerts go to a mailbox nobody opens. +- [ ] **`root:` alias** to a real destination. +- [ ] Needs: address of the `wg-pk` relay, and whether it accepts + unauthenticated from `10.110.0.12`. **Open question for CIVICVS.** + +### Monitoring and backup + +- [ ] **`smartd`** on `/dev/sda -d cciss,0` through `cciss,3`. Depends on mail. +- [ ] **`vzdump` job** — both containers, `local`, snapshot, zstd, 02:30. + `mp0` already carries `backup=0`. +- [ ] **First `vzdump` run — report the archive size.** Retention is a + placeholder `keep-last=3` until this number exists. +- [ ] **Free-space guard** — refuse and alert below 20 GiB on `/var/lib/vz`. +- [ ] `/var/lib/vz/mechcomp-app/` and the host-side pull script. +- [ ] `mechcomp-backup --stdout` in CT 100. Structure can be built and tested + against the current data directories without application output. + +### Gold media + +- [ ] 32 GB USB stick: LUKS2, ext4, label `MC-GOLD-01`, `/mnt/gold` `noauto`. +- [ ] `gold-archive.sh` with `SHA256SUMS` verification before unmount. +- [ ] **Open:** where the 3+ TB USB disk is attached now that `annales` is out + of scope. Until known, stick-to-disk copying is a manual step. + +--- + +## 4. Access model + +Confirmed by CIVICVS: no workstation access from the home LAN is required. + +``` +internet -> WireGuard -> srv-b -> containers +``` + +`srv-b` is the bastion. The authorized key is `root@srv-b`, which is +consistent: root on the host can `pct enter` regardless, so SSH to the +containers adds no privilege. It does mean **every path to a container runs +through `srv-b`** — deliberate, not a limitation. + +If direct WireGuard-side access to the catalogue is wanted later, that is a +route addition for `10.20.0.0/24` on the hub. Not now. + +No DHCP anywhere. Two containers with fixed addresses on a portless bridge is +the entire address space; a DHCP server would add a daemon, a lease database +and a failure mode in exchange for nothing. + +--- + +## 5. Blocked on application code + +None of this can be honestly completed while the repository is `LICENSE` and +`README.md`. It is not provisioning work and should not be attempted as such. + +- [ ] `requirements-base.txt` / `requirements-cad.txt` and dependency install +- [ ] `mechcomp.service` and `mechcomp-worker.service` +- [ ] Reference toolchain image `mechcomp/reference-toolchain:8.0.0` +- [ ] Fixture reproduction against `ddd0f154…` +- [ ] Application-runtime acceptance +- [ ] Replacing the placeholder with the real service + +The Shapely port is the architect's work item and gates all of the above. + +--- + +## 6. Open questions + +| # | Question | Blocks | +|---|---|---| +| 1 | `wg-pk` relay address; unauthenticated from `10.110.0.12`? | mail, `smartd`, backup alerts | +| 2 | Where is the 3+ TB USB disk attached? | gold redundancy step | +| 3 | First `vzdump` archive size | final retention value | + +Question 3 is answered by doing. Questions 1 and 2 need CIVICVS. + +--- + +## 7. Closed questions + +| Question | Answer | +|---|---| +| Kane County CA issuance path | None on `srv-b`. Local staging CA generated instead. | +| `openssh-server` present | Yes, both containers. | +| `ADMIN_USER` / key | `sandor`; `root@srv-b` key, bastion pattern. | +| DHCP pool on `10.0.0.0/24` | **Not applicable.** Containers are no longer on that network. | +| Docker storage driver | `overlay2` / `systemd`. No fallback needed. | +| LAN workstation access | Not required. WireGuard through `srv-b`. |