# STAGING-STATE.md Live state of the Mechanical Compiler staging instance on `srv-b`. | | | |---|---| | Updated | 2026-08-15, after network isolation | | Instance | Staging / development | | Specification | `ENVIRONMENT.md` revision 5 | | Failure log | `FAILURES.md` | | Method | Manual, one command group at a time | --- ## 0. How to use this file This is the authoritative record of what is true on `srv-b`. Where it and `ENVIRONMENT.md` disagree, **this file wins for facts** and the specification is defective and must be corrected. Completed and remaining work are in the same document deliberately. They are two halves of one boundary; separating them guarantees they drift. Before any command: read this file, confirm the immediately relevant live state with a read-only command, then issue one command group. If it fails, record it in `FAILURES.md` before changing anything else. Production will get its own `PRODUCTION-STATE.md`. The specification is shared; the state is not. --- ## 1. Instance values These are the `srv-b` bindings for the parameters in `ENVIRONMENT.md`. ### Host ``` hostname srv-b / srv-b.dev.infra platform Proxmox VE 8.4.0, Debian 12, kernel 6.8.12-9-pve hardware HP ProLiant DL360 G7, 2 x Xeon X5650, 24 threads, 31 GiB storage P410i, 4 x EG0146FAWHU, RAID 1+0, all SMART OK local directory /var/lib/vz, ~70 GiB free iso,vztmpl,backup local-lvm LVM-thin pve/data, 166.9 GiB rootdir,images vmbr0 10.0.0.12/24 on enp3s0f0, gw 10.0.0.1 management, LAN vmbr1 10.20.0.1/24, bridge-ports none service, portless wg0 10.110.0.12/32, peer wg-pk.civicus.us:51820, allowed 10.110.0.0/22 resolver 75.75.75.75, search dev.infra spare NICs enp3s0f1, enp4s0f0, enp4s0f1 — unconfigured, deliberately template local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst ``` ### Containers | | CT 100 | CT 101 | |---|---|---| | hostname | `mechcomp` | `mcproxy` | | role | application, worker | reverse proxy, TLS | | features | `nesting=1,keyctl=1` | `nesting=1` | | cores / RAM / swap | 16 / 16384 / 4096 | 2 / 1024 / 512 | | rootfs | 40 GiB `local-lvm` | 8 GiB `local-lvm` | | `mp0` | 60 GiB → `/var/lib/mechcomp`, `backup=0` | — | | `net0` | `eth0` on `vmbr1`, `10.20.0.10/24`, gw `10.20.0.1` | `eth0` on `vmbr1`, `10.20.0.11/24`, gw `10.20.0.1` | | Debian | 12.15 | 12.15 | **Neither container has a LAN interface.** See F-017. ### Names and identity ``` mechanical-compiler.dev.infra -> 10.20.0.11 (CT 101, the proxy) mechcomp.dev.infra -> 10.20.0.10 PVE-generated mcproxy.dev.infra -> 10.20.0.11 PVE-generated ADMIN_USER sandor uid 1000, groups sandor + mechcomp(996) shell /bin/bash, both containers authorized key SHA256:2pNffCscUUW5Wbs9uMepLEvMLKPqUV/7Lpk5tw7iWSY root@srv-b — bastion pattern, see §4 service user mechcomp, uid 999, gid 996 home /var/www/mechcomp, shell /usr/sbin/nologin ``` ### Host NAT, as persisted in `/etc/iptables/rules.v4` ``` nat POSTROUTING -s 10.20.0.0/24 -d 10.0.0.0/24 -j RETURN # LAN not translated -s 10.0.0.0/24 -o wg0 -j MASQUERADE # pre-existing -s 10.20.0.0/24 -o wg0 -j MASQUERADE # containers -> WireGuard -s 10.20.0.0/24 -o vmbr0 -j MASQUERADE # containers -> internet filter FORWARD -s 10.20.0.0/24 -d 10.20.0.0/24 -j ACCEPT # container <-> container -s 10.20.0.0/24 -d 10.0.0.0/24 -j DROP # LAN blocked ``` Rule order is load-bearing. `RETURN` must precede both masquerades. ### TLS ``` CA CN = Mechanical Compiler Staging CA (locally generated) leaf CN = mechanical-compiler.dev.infra SAN = DNS:mechanical-compiler.dev.infra validity 2026-08-16 -> 2028-11-18 <-- expires, nothing renews it trusted srv-b, CT 100, CT 101 key mode 0600 on CT 101 ``` No Kane County Civic Infrastructure CA issuance path exists on `srv-b`: no trust anchor, no `step`, no `cfssl`, no EasyRSA. Proxmox's own CA was deliberately not reused. Replacing this leaf later is two file copies and a reload. ### Application environment `/etc/mechcomp/mechcomp.env`, `root:mechcomp`, `0640`: ``` MECHCOMP_ENV=staging MECHCOMP_BIND=10.20.0.10 MECHCOMP_PORT=8770 MECHCOMP_BASE_URL=https://mechanical-compiler.dev.infra MECHCOMP_DATA_DIR=/var/lib/mechcomp MECHCOMP_LOG_LEVEL=info MECHCOMP_DB_URL=sqlite:////var/lib/mechcomp/db/mechcomp.sqlite3 MECHCOMP_WORKER_CONCURRENCY=4 MECHCOMP_CAD_BACKEND=none MECHCOMP_ARTIFACT_RETENTION_DAYS=30 MECHCOMP_SECRET_KEY= ``` ### Rollback copies on disk ``` srv-b /etc/network/interfaces.before-mechcomp /etc/hosts.before-mechcomp /etc/hosts.before-svcfqdn /etc/iptables/rules.v4.before-svcnat /etc/iptables/rules.v4.before-lanblock /root/pct-100.before-svcnet /root/pct-101.before-svcnet CT 100 /etc/hosts.before-svcfqdn CT 101 /etc/hosts.before-svcfqdn ``` --- ## 2. Verified Each line was demonstrated by command output, not inferred. ### Host and network - [x] `vmbr1` created, active, `10.20.0.1/24`, portless - [x] Duplicate address detection run before each container creation - [x] Container → internet reachable (`deb.debian.org`, `gitea.barternetwork.us`, both 200) - [x] Container → WireGuard network reachable (`10.110.0.12`) - [x] **Container → LAN unreachable** — the isolation requirement - [x] Container → `srv-b` reachable at `10.0.0.12` (INPUT path, required) - [x] Container ↔ container reachable over `vmbr1` - [x] LAN → Proxmox console unaffected (`10.0.0.12:8006` → 200) - [x] Network configuration persisted, survives reboot ### CT 100 - [x] Debian 12.15, systemd running, zero failed units, no pending upgrades - [x] `en_US.UTF-8` generated and active - [x] `/var/lib/mechcomp` is a real separate ext4 filesystem - [x] Service user `mechcomp`, home `/var/www/mechcomp`, writable - [x] Application directory layout created with correct ownership - [x] Base packages installed; OpenSCAD / Qt / X11 absent — verified `clean` - [x] Docker working, `overlay2` / `systemd`, no `fuse-overlayfs` needed - [x] Repository cloned at `e85c4f4e`, verified as the owning user - [x] Python 3.11.2 venv created, owned by `mechcomp` - [x] `mechcomp.env` written with generated secret - [x] `openssh-server` installed, enabled, active - [x] Application listener bound to service network only — LAN-side bind refused ### CT 101 - [x] Debian 12.15, systemd running, zero failed units after `nesting=1` - [x] `en_US.UTF-8` generated and active - [x] nginx 1.22.1 installed, `nginx -t` passes - [x] Debian `default` site removed; only the project vhost is enabled - [x] Local CA created, leaf issued, trusted on all three hosts - [x] HTTP → HTTPS redirect, TLS termination, proxy to `10.20.0.10:8770` - [x] `openssh-server` installed, enabled, active - [x] **`X-Forwarded-Proto: https` observed at the backend** — needs re-proof, see §3 --- ## 3. Remaining — infrastructure In dependency order. Application-dependent work is in §5. ### Immediate - [ ] **`mechcomp-placeholder.service`** — recreate the backend as a supervised unit. It died on the F-017 reboot and nothing currently listens on 8770. Everything below that touches the proxy depends on this. See F-019. - [ ] **Re-prove `X-Forwarded-Proto`** end to end. The earlier proof was taken when clients were on `10.0.0.x`; header values will now read `10.20.0.x`. Re-establish rather than assume it survived the topology change. - [ ] **`default_server`** on the CT 101 vhost, both `listen 443` lines. Two words. Prevents catch-all behaviour depending on file ordering once a second server block exists. See F-012. ### Mail — blocks alerting - [ ] **Postfix relay.** Currently `inet_interfaces = loopback-only`, `relayhost` empty, no `root:` alias. Alerts go to a mailbox nobody opens. - [ ] **`root:` alias** to a real destination. - [ ] Needs: address of the `wg-pk` relay, and whether it accepts unauthenticated from `10.110.0.12`. **Open question for CIVICVS.** ### Monitoring and backup - [ ] **`smartd`** on `/dev/sda -d cciss,0` through `cciss,3`. Depends on mail. - [ ] **`vzdump` job** — both containers, `local`, snapshot, zstd, 02:30. `mp0` already carries `backup=0`. - [ ] **First `vzdump` run — report the archive size.** Retention is a placeholder `keep-last=3` until this number exists. - [ ] **Free-space guard** — refuse and alert below 20 GiB on `/var/lib/vz`. - [ ] `/var/lib/vz/mechcomp-app/` and the host-side pull script. - [ ] `mechcomp-backup --stdout` in CT 100. Structure can be built and tested against the current data directories without application output. ### Gold media - [ ] 32 GB USB stick: LUKS2, ext4, label `MC-GOLD-01`, `/mnt/gold` `noauto`. - [ ] `gold-archive.sh` with `SHA256SUMS` verification before unmount. - [ ] **Open:** where the 3+ TB USB disk is attached now that `annales` is out of scope. Until known, stick-to-disk copying is a manual step. --- ## 4. Access model Confirmed by CIVICVS: no workstation access from the home LAN is required. ``` internet -> WireGuard -> srv-b -> containers ``` `srv-b` is the bastion. The authorized key is `root@srv-b`, which is consistent: root on the host can `pct enter` regardless, so SSH to the containers adds no privilege. It does mean **every path to a container runs through `srv-b`** — deliberate, not a limitation. If direct WireGuard-side access to the catalogue is wanted later, that is a route addition for `10.20.0.0/24` on the hub. Not now. No DHCP anywhere. Two containers with fixed addresses on a portless bridge is the entire address space; a DHCP server would add a daemon, a lease database and a failure mode in exchange for nothing. --- ## 5. Blocked on application code None of this can be honestly completed while the repository is `LICENSE` and `README.md`. It is not provisioning work and should not be attempted as such. - [ ] `requirements-base.txt` / `requirements-cad.txt` and dependency install - [ ] `mechcomp.service` and `mechcomp-worker.service` - [ ] Reference toolchain image `mechcomp/reference-toolchain:8.0.0` - [ ] Fixture reproduction against `ddd0f154…` - [ ] Application-runtime acceptance - [ ] Replacing the placeholder with the real service The Shapely port is the architect's work item and gates all of the above. --- ## 6. Open questions | # | Question | Blocks | |---|---|---| | 1 | `wg-pk` relay address; unauthenticated from `10.110.0.12`? | mail, `smartd`, backup alerts | | 2 | Where is the 3+ TB USB disk attached? | gold redundancy step | | 3 | First `vzdump` archive size | final retention value | Question 3 is answered by doing. Questions 1 and 2 need CIVICVS. --- ## 7. Closed questions | Question | Answer | |---|---| | Kane County CA issuance path | None on `srv-b`. Local staging CA generated instead. | | `openssh-server` present | Yes, both containers. | | `ADMIN_USER` / key | `sandor`; `root@srv-b` key, bastion pattern. | | DHCP pool on `10.0.0.0/24` | **Not applicable.** Containers are no longer on that network. | | Docker storage driver | `overlay2` / `systemd`. No fallback needed. | | LAN workstation access | Not required. WireGuard through `srv-b`. |