diff --git a/docs/FAILURES.md b/docs/FAILURES.md new file mode 100644 index 0000000..e1f8894 --- /dev/null +++ b/docs/FAILURES.md @@ -0,0 +1,349 @@ +# FAILURES.md + +Append-only record of every failure encountered building the Mechanical +Compiler environment. + +| | | +|---|---| +| Scope | All instances. Staging entries are marked `srv-b`. | +| Rule | Append only. Never edit an entry except to add a `Resolution` line. | +| Numbering | Sequential, never reused. See §0 on the renumbering. | + +--- + +## 0. Why this file exists separately + +This is the primary input to provisioning automation. + +Every entry below is something a script written from `ENVIRONMENT.md` alone +would have got wrong. `F-003` and `F-004` in particular would have produced a +container that booted, reported success, and was quietly broken. The +specification cannot anticipate them; only contact with a host reveals them. + +When the production automation is written, it should be read **before** +`ENVIRONMENT.md`, not after. + +### Renumbering note, 2026-08-15 + +The staging checkpoint and the reconciliation checkpoint were written +independently and both allocated `F-008` and `F-009`. The reconciliation +entries have been renumbered to `F-015` and `F-016`. Original phase-1 numbering +is preserved because it is cited elsewhere. + +### Entry format + +``` +### F-nnn — one-line summary +Host / container. Phase. +**Observed:** what was seen, verbatim where possible. +**Cause:** proven cause, or explicitly "unproven". +**Correction:** the smallest change that fixed it. +**Consequence:** what the specification or automation must do differently. +``` + +A failure with no `Consequence` line is either not yet understood or not worth +recording. + +--- + +## Phase 1 — initial container build + +### F-001 — `arping` absent on the Proxmox host +`srv-b`. + +**Observed:** `command -v arping` → not installed. +**Cause:** Not part of the base PVE install. +**Correction:** `apt install iputils-arping`. +**Consequence:** Automation must install `iputils-arping` before duplicate +address detection, or the DAD check silently does not happen. + +--- + +### F-002 — invalid Proxmox inspection command +`srv-b`. + +**Observed:** `pvesm config local-lvm` returned CLI usage text. +**Cause:** Operator command error. Not a host fault. +**Correction:** Read `/etc/pve/storage.cfg` directly. +**Consequence:** Read storage configuration from the file, not from a +subcommand that does not exist. + +--- + +### F-003 — CT 101 booted with degraded systemd +CT 101. + +**Observed:** + +``` +status=226/NAMESPACE +Failed to set up mount namespacing: Permission denied +systemd-logind.service failed +systemd-networkd.service failed +systemd-timedated.service failed +systemd-networkd.socket failed +``` + +**Cause:** Debian 12 systemd requires mount namespacing that an unprivileged +LXC denies without `nesting`. +**Correction:** `features: nesting=1`. Deliberately tested alone rather than +copying `nesting=1,keyctl=1` from CT 100 — `keyctl=1` proved unnecessary. +**Consequence:** **Every Debian 12 unprivileged container needs +`nesting=1`**, including ones running nothing but nginx. `keyctl=1` is needed +only where Docker runs. Promoted into the specification. + +This is the clearest example of why the manual build was correct: the container +started, appeared healthy at a glance, and was broken in four services. + +--- + +### F-004 — locale configured but never generated +CT 100, CT 101. + +**Observed:** `LANG=en_US.UTF-8` set, but `locale -a` listed only `C`, +`C.utf8`, `POSIX`. `locale` emitted `Cannot set LC_*` warnings. +**Cause:** Setting `LANG` does not generate a locale. The container template +ships none. +**Correction:** install `locales`, enable `en_US.UTF-8 UTF-8`, `locale-gen`, +`update-locale`. +**Consequence:** Specification must state the generation step, not just the +value. Silent until something depends on collation or encoding. + +--- + +### F-005 — locale repair exposed a pending upgrade backlog +CT 100, CT 101. + +**Observed:** Installing `locales` pulled glibc-related packages and revealed +52 further pending upgrades. +**Cause:** Template was not current. +**Correction:** Full upgrade of each container, one at a time. Both reached +Debian 12.15, zero pending, no reboot required. +**Consequence:** Automation must `apt full-upgrade` immediately after first +boot, before anything else is installed. + +--- + +### F-006 — transient `curl` / `git` command-not-found +CT 100. + +**Observed:** An egress test returned `curl: command not found` and +`git: command not found`. Immediately afterward both packages were installed +and current, `PATH` normal, both executables present and working. +**Cause:** **Unproven.** Hypothesis only: the test ran during the F-005 +upgrade, while `dpkg` had the binaries briefly unlinked mid-transaction. +Adjacent in time, consistent with the symptom, not demonstrated. +**Correction:** None required. Repeat tests with absolute paths passed. +**Consequence:** Do not close. If it recurs, capture `dpkg` lock state at the +moment of failure. + +--- + +### F-007 — recursive `chown` failed on ext4 `lost+found` +CT 100. + +**Observed:** `chown: cannot read directory '/var/lib/mechcomp/lost+found': +Permission denied`. That directory is `nobody:nogroup`, mode `0700`. +**Cause:** `/var/lib/mechcomp` is a real filesystem root, not a directory. +`lost+found` is created by `mkfs` and is not ours to manage. +**Correction:** Own explicit application paths only. Leave `lost+found` alone. +**Consequence:** **Never `chown -R` a mount point root.** Enumerate the +directories the application actually uses. Promoted into the specification. + +--- + +### F-008 — Git "dubious ownership" +CT 100. + +**Observed:** `fatal: detected dubious ownership in repository at +'/var/www/mechcomp'` when verification ran as root. +**Cause:** Repository cloned as `mechcomp`, inspected as `root`. +**Correction:** Re-ran verification as the owning user. **No root +`safe.directory` exception was added** — the correct fix, since the exception +would have masked every future instance of the same mistake. +**Consequence:** All repository operations run as the service user. + +--- + +### F-009 — dependency manifests absent from the application repository +CT 100. + +**Observed:** `requirements-base.txt` and `requirements-cad.txt` do not exist; +the repository is `LICENSE` and `README.md` at commit `e85c4f4e`. +**Cause:** Application implementation has not started. Not a provisioning +failure. +**Correction:** None. Do not invent pins. +**Consequence:** Infrastructure and application acceptance are separately +gated. Infrastructure may complete without the application existing. + +--- + +## Phase 2 — architect corrections + +### F-010 — placeholder PID file rewrite failed to substitute +CT 100. + +**Observed:** An automated rewrite left the literal `$!` in the PID file. +**Cause:** Quoting error in the rewriting command. +**Correction:** Identified the real PID by inspection. +**Consequence:** Superseded by F-019 — the placeholder should be a systemd +unit and have no PID file at all. + +--- + +### F-011 — root could not overwrite a `mechcomp`-owned file in `/tmp` +CT 100. + +**Observed:** `Permission denied` writing a file owned by `mechcomp:mechcomp` +in `/tmp`, as root. +**Cause:** `fs.protected_regular = 2` with `/tmp` mode `1777`. Expected kernel +behaviour, not a fault. +**Correction:** Rewrote the file as the owning user. +**Consequence:** Do not use `/tmp` for state shared across users. Moot once +services run under systemd with `PrivateTmp=true`. + +--- + +### F-012 — first request after nginx reload returned the Debian welcome page +CT 101. + +**Observed:** `nginx -t` passed; the immediately following request returned the +615-byte Debian default page. All subsequent requests proxied correctly. +**Cause:** **Unproven.** The obvious candidate — Debian's `default` site still +enabled and acting as `default_server` — was tested and ruled out: +`sites-enabled` contains only the project vhost. Remaining hypothesis is a +graceful worker transition. Not demonstrated. +**Correction:** None required. +**Consequence:** Do not close. Independently, declare `default_server` +explicitly on the project vhost so that adding a second server block cannot +make catch-all behaviour depend on file ordering. + +--- + +### F-013 — TLS test validated against an IP literal +CT 101. + +**Observed:** `SSL: no alternative certificate subject name matches target +host name '10.0.0.21'`. +**Cause:** Test used an address; the certificate SAN contains a DNS name. +Correct behaviour by both curl and OpenSSL. +**Correction:** Test by hostname. +**Consequence:** Acceptance checks must use the service FQDN. An IP-literal +TLS test is always wrong unless the certificate carries an IP SAN. + +--- + +### F-014 — CT 101 had no `mechcomp` group +CT 101. + +**Observed:** Adding `sandor` to `mechcomp` failed; the group did not exist. +**Cause:** Specification said the admin joins the `mechcomp` group in *both* +containers, but the group is created as a side effect of creating the service +user, which exists only in CT 100. +**Correction:** Created the system group and added `sandor`. +**Consequence:** Specification defect. Either create the group explicitly where +it is required, or scope the group membership to CT 100. GID 996 now matches +across both containers, which is harmless and mildly useful. + +--- + +### F-015 — CT 101 verification lacked `curl` +CT 101. *(Renumbered from a duplicate `F-008`.)* + +**Observed:** `curl: command not found`. +**Cause:** Not in the base template; CT 101's package list did not include it. +**Correction:** Installed `curl` and `ca-certificates`. +**Consequence:** The proxy container needs its own minimal toolset. Do not +assume packages installed in CT 100 exist in CT 101. + +--- + +### F-016 — placeholder PID capture stored a literal `$!` +CT 100. *(Renumbered from a duplicate `F-009`.)* + +**Observed:** PID file contained `$!` rather than a number. The listener itself +was healthy. +**Cause:** Quoting error. +**Correction:** Identified PID `12102` by inspection. +**Consequence:** Superseded by F-019. + +--- + +## Phase 3 — network isolation + +### F-017 — containers were provisioned on the home LAN +`srv-b`, CT 100, CT 101. + +**Observed:** Both containers had `net0` on `vmbr0` at `10.0.0.20/24` and +`10.0.0.21/24`, gateway `10.0.0.1` — the ISP router's network. Anything on the +home LAN could reach them. +**Cause:** **Specification defect in `ENVIRONMENT.md` revision 4.** The +document placed containers on both bridges, giving each a management address on +the LAN, on the unstated assumption that a LAN workstation would browse +staging. That assumption was never confirmed and was wrong. +**Correction:** Both containers moved to `vmbr1` only, gateway `10.20.0.1`. +`net1` deleted. Host NAT extended for `10.20.0.0/24`. +**Consequence:** Containers are on the portless service bridge only. `srv-b` is +router and bastion; access arrives via WireGuard. Promoted into the +specification. + +--- + +### F-018 — removing the LAN interface did not remove LAN reachability +CT 100, CT 101. + +**Observed:** After F-017, both containers still reached `10.0.0.1` +successfully. +**Cause:** Removing an interface removes an address, not a route. +`ip_forward=1` plus the new `MASQUERADE -s 10.20.0.0/24 -o vmbr0` — added to +give the containers internet access — also gave them the LAN, translated to the +host's address. +**Correction:** A `RETURN` in `nat POSTROUTING` ahead of both masquerades, and +a `DROP` in `FORWARD`, both scoped `-s 10.20.0.0/24 -d 10.0.0.0/24`, plus an +explicit `ACCEPT` for container-to-container traffic. +**Consequence:** **Interface removal is not isolation.** Automation must assert +the negative — that the LAN is unreachable — not merely that the interface is +gone. Note that the containers still reach `srv-b` itself at `10.0.0.12`, +because a container addressing its gateway takes the INPUT path and never +enters FORWARD. That is required, not a leak. + +--- + +### F-019 — placeholder backend did not survive reboot +CT 100. + +**Observed:** After the F-017 reboot, nothing listened on `10.20.0.10:8770`. +**Cause:** The placeholder was a bare foreground process with a PID file. No +supervision. +**Correction:** Pending — to be recreated as `mechcomp-placeholder.service`. +**Consequence:** Anything a proof depends on must be supervised, or the proof +expires silently at the next reboot and the following session diagnoses a +proxy fault that does not exist. Applies to temporary scaffolding as much as to +real services. + +--- + +### F-020 — service FQDN mapping changed twice +`srv-b`, CT 100, CT 101. + +**Observed:** `mechanical-compiler.dev.infra` was mapped to `10.20.0.10`, then +corrected to `10.0.0.21`, then corrected again to `10.20.0.11`. +**Cause:** Three different states, each correct for its moment. `10.20.0.10` +was wrong — it pointed at the application rather than the proxy. `10.0.0.21` +was right while CT 101 had a LAN address and workstation access was assumed. +`10.20.0.11` became right once F-017 removed the LAN interface and access moved +to WireGuard through `srv-b`. +**Correction:** `10.20.0.11` on all three hosts. +**Consequence:** Not an error in either direction — a value that tracked a +topology decision. Recorded because the reasoning matters more than the value: +**the service FQDN names the proxy, and the proxy has exactly one address.** + +--- + +## Open, not closed + +| # | Status | +|---|---| +| F-006 | Cause unproven. Recurrence should capture `dpkg` lock state. | +| F-012 | Cause unproven. Leading candidate ruled out by inspection. | +| F-019 | Correction pending. |