Failure records
Append-only record of every failure encountered building the Mechanical Compiler environment.
This commit is contained in:
@@ -0,0 +1,349 @@
|
||||
# FAILURES.md
|
||||
|
||||
Append-only record of every failure encountered building the Mechanical
|
||||
Compiler environment.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Scope | All instances. Staging entries are marked `srv-b`. |
|
||||
| Rule | Append only. Never edit an entry except to add a `Resolution` line. |
|
||||
| Numbering | Sequential, never reused. See §0 on the renumbering. |
|
||||
|
||||
---
|
||||
|
||||
## 0. Why this file exists separately
|
||||
|
||||
This is the primary input to provisioning automation.
|
||||
|
||||
Every entry below is something a script written from `ENVIRONMENT.md` alone
|
||||
would have got wrong. `F-003` and `F-004` in particular would have produced a
|
||||
container that booted, reported success, and was quietly broken. The
|
||||
specification cannot anticipate them; only contact with a host reveals them.
|
||||
|
||||
When the production automation is written, it should be read **before**
|
||||
`ENVIRONMENT.md`, not after.
|
||||
|
||||
### Renumbering note, 2026-08-15
|
||||
|
||||
The staging checkpoint and the reconciliation checkpoint were written
|
||||
independently and both allocated `F-008` and `F-009`. The reconciliation
|
||||
entries have been renumbered to `F-015` and `F-016`. Original phase-1 numbering
|
||||
is preserved because it is cited elsewhere.
|
||||
|
||||
### Entry format
|
||||
|
||||
```
|
||||
### F-nnn — one-line summary
|
||||
Host / container. Phase.
|
||||
**Observed:** what was seen, verbatim where possible.
|
||||
**Cause:** proven cause, or explicitly "unproven".
|
||||
**Correction:** the smallest change that fixed it.
|
||||
**Consequence:** what the specification or automation must do differently.
|
||||
```
|
||||
|
||||
A failure with no `Consequence` line is either not yet understood or not worth
|
||||
recording.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — initial container build
|
||||
|
||||
### F-001 — `arping` absent on the Proxmox host
|
||||
`srv-b`.
|
||||
|
||||
**Observed:** `command -v arping` → not installed.
|
||||
**Cause:** Not part of the base PVE install.
|
||||
**Correction:** `apt install iputils-arping`.
|
||||
**Consequence:** Automation must install `iputils-arping` before duplicate
|
||||
address detection, or the DAD check silently does not happen.
|
||||
|
||||
---
|
||||
|
||||
### F-002 — invalid Proxmox inspection command
|
||||
`srv-b`.
|
||||
|
||||
**Observed:** `pvesm config local-lvm` returned CLI usage text.
|
||||
**Cause:** Operator command error. Not a host fault.
|
||||
**Correction:** Read `/etc/pve/storage.cfg` directly.
|
||||
**Consequence:** Read storage configuration from the file, not from a
|
||||
subcommand that does not exist.
|
||||
|
||||
---
|
||||
|
||||
### F-003 — CT 101 booted with degraded systemd
|
||||
CT 101.
|
||||
|
||||
**Observed:**
|
||||
|
||||
```
|
||||
status=226/NAMESPACE
|
||||
Failed to set up mount namespacing: Permission denied
|
||||
systemd-logind.service failed
|
||||
systemd-networkd.service failed
|
||||
systemd-timedated.service failed
|
||||
systemd-networkd.socket failed
|
||||
```
|
||||
|
||||
**Cause:** Debian 12 systemd requires mount namespacing that an unprivileged
|
||||
LXC denies without `nesting`.
|
||||
**Correction:** `features: nesting=1`. Deliberately tested alone rather than
|
||||
copying `nesting=1,keyctl=1` from CT 100 — `keyctl=1` proved unnecessary.
|
||||
**Consequence:** **Every Debian 12 unprivileged container needs
|
||||
`nesting=1`**, including ones running nothing but nginx. `keyctl=1` is needed
|
||||
only where Docker runs. Promoted into the specification.
|
||||
|
||||
This is the clearest example of why the manual build was correct: the container
|
||||
started, appeared healthy at a glance, and was broken in four services.
|
||||
|
||||
---
|
||||
|
||||
### F-004 — locale configured but never generated
|
||||
CT 100, CT 101.
|
||||
|
||||
**Observed:** `LANG=en_US.UTF-8` set, but `locale -a` listed only `C`,
|
||||
`C.utf8`, `POSIX`. `locale` emitted `Cannot set LC_*` warnings.
|
||||
**Cause:** Setting `LANG` does not generate a locale. The container template
|
||||
ships none.
|
||||
**Correction:** install `locales`, enable `en_US.UTF-8 UTF-8`, `locale-gen`,
|
||||
`update-locale`.
|
||||
**Consequence:** Specification must state the generation step, not just the
|
||||
value. Silent until something depends on collation or encoding.
|
||||
|
||||
---
|
||||
|
||||
### F-005 — locale repair exposed a pending upgrade backlog
|
||||
CT 100, CT 101.
|
||||
|
||||
**Observed:** Installing `locales` pulled glibc-related packages and revealed
|
||||
52 further pending upgrades.
|
||||
**Cause:** Template was not current.
|
||||
**Correction:** Full upgrade of each container, one at a time. Both reached
|
||||
Debian 12.15, zero pending, no reboot required.
|
||||
**Consequence:** Automation must `apt full-upgrade` immediately after first
|
||||
boot, before anything else is installed.
|
||||
|
||||
---
|
||||
|
||||
### F-006 — transient `curl` / `git` command-not-found
|
||||
CT 100.
|
||||
|
||||
**Observed:** An egress test returned `curl: command not found` and
|
||||
`git: command not found`. Immediately afterward both packages were installed
|
||||
and current, `PATH` normal, both executables present and working.
|
||||
**Cause:** **Unproven.** Hypothesis only: the test ran during the F-005
|
||||
upgrade, while `dpkg` had the binaries briefly unlinked mid-transaction.
|
||||
Adjacent in time, consistent with the symptom, not demonstrated.
|
||||
**Correction:** None required. Repeat tests with absolute paths passed.
|
||||
**Consequence:** Do not close. If it recurs, capture `dpkg` lock state at the
|
||||
moment of failure.
|
||||
|
||||
---
|
||||
|
||||
### F-007 — recursive `chown` failed on ext4 `lost+found`
|
||||
CT 100.
|
||||
|
||||
**Observed:** `chown: cannot read directory '/var/lib/mechcomp/lost+found':
|
||||
Permission denied`. That directory is `nobody:nogroup`, mode `0700`.
|
||||
**Cause:** `/var/lib/mechcomp` is a real filesystem root, not a directory.
|
||||
`lost+found` is created by `mkfs` and is not ours to manage.
|
||||
**Correction:** Own explicit application paths only. Leave `lost+found` alone.
|
||||
**Consequence:** **Never `chown -R` a mount point root.** Enumerate the
|
||||
directories the application actually uses. Promoted into the specification.
|
||||
|
||||
---
|
||||
|
||||
### F-008 — Git "dubious ownership"
|
||||
CT 100.
|
||||
|
||||
**Observed:** `fatal: detected dubious ownership in repository at
|
||||
'/var/www/mechcomp'` when verification ran as root.
|
||||
**Cause:** Repository cloned as `mechcomp`, inspected as `root`.
|
||||
**Correction:** Re-ran verification as the owning user. **No root
|
||||
`safe.directory` exception was added** — the correct fix, since the exception
|
||||
would have masked every future instance of the same mistake.
|
||||
**Consequence:** All repository operations run as the service user.
|
||||
|
||||
---
|
||||
|
||||
### F-009 — dependency manifests absent from the application repository
|
||||
CT 100.
|
||||
|
||||
**Observed:** `requirements-base.txt` and `requirements-cad.txt` do not exist;
|
||||
the repository is `LICENSE` and `README.md` at commit `e85c4f4e`.
|
||||
**Cause:** Application implementation has not started. Not a provisioning
|
||||
failure.
|
||||
**Correction:** None. Do not invent pins.
|
||||
**Consequence:** Infrastructure and application acceptance are separately
|
||||
gated. Infrastructure may complete without the application existing.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — architect corrections
|
||||
|
||||
### F-010 — placeholder PID file rewrite failed to substitute
|
||||
CT 100.
|
||||
|
||||
**Observed:** An automated rewrite left the literal `$!` in the PID file.
|
||||
**Cause:** Quoting error in the rewriting command.
|
||||
**Correction:** Identified the real PID by inspection.
|
||||
**Consequence:** Superseded by F-019 — the placeholder should be a systemd
|
||||
unit and have no PID file at all.
|
||||
|
||||
---
|
||||
|
||||
### F-011 — root could not overwrite a `mechcomp`-owned file in `/tmp`
|
||||
CT 100.
|
||||
|
||||
**Observed:** `Permission denied` writing a file owned by `mechcomp:mechcomp`
|
||||
in `/tmp`, as root.
|
||||
**Cause:** `fs.protected_regular = 2` with `/tmp` mode `1777`. Expected kernel
|
||||
behaviour, not a fault.
|
||||
**Correction:** Rewrote the file as the owning user.
|
||||
**Consequence:** Do not use `/tmp` for state shared across users. Moot once
|
||||
services run under systemd with `PrivateTmp=true`.
|
||||
|
||||
---
|
||||
|
||||
### F-012 — first request after nginx reload returned the Debian welcome page
|
||||
CT 101.
|
||||
|
||||
**Observed:** `nginx -t` passed; the immediately following request returned the
|
||||
615-byte Debian default page. All subsequent requests proxied correctly.
|
||||
**Cause:** **Unproven.** The obvious candidate — Debian's `default` site still
|
||||
enabled and acting as `default_server` — was tested and ruled out:
|
||||
`sites-enabled` contains only the project vhost. Remaining hypothesis is a
|
||||
graceful worker transition. Not demonstrated.
|
||||
**Correction:** None required.
|
||||
**Consequence:** Do not close. Independently, declare `default_server`
|
||||
explicitly on the project vhost so that adding a second server block cannot
|
||||
make catch-all behaviour depend on file ordering.
|
||||
|
||||
---
|
||||
|
||||
### F-013 — TLS test validated against an IP literal
|
||||
CT 101.
|
||||
|
||||
**Observed:** `SSL: no alternative certificate subject name matches target
|
||||
host name '10.0.0.21'`.
|
||||
**Cause:** Test used an address; the certificate SAN contains a DNS name.
|
||||
Correct behaviour by both curl and OpenSSL.
|
||||
**Correction:** Test by hostname.
|
||||
**Consequence:** Acceptance checks must use the service FQDN. An IP-literal
|
||||
TLS test is always wrong unless the certificate carries an IP SAN.
|
||||
|
||||
---
|
||||
|
||||
### F-014 — CT 101 had no `mechcomp` group
|
||||
CT 101.
|
||||
|
||||
**Observed:** Adding `sandor` to `mechcomp` failed; the group did not exist.
|
||||
**Cause:** Specification said the admin joins the `mechcomp` group in *both*
|
||||
containers, but the group is created as a side effect of creating the service
|
||||
user, which exists only in CT 100.
|
||||
**Correction:** Created the system group and added `sandor`.
|
||||
**Consequence:** Specification defect. Either create the group explicitly where
|
||||
it is required, or scope the group membership to CT 100. GID 996 now matches
|
||||
across both containers, which is harmless and mildly useful.
|
||||
|
||||
---
|
||||
|
||||
### F-015 — CT 101 verification lacked `curl`
|
||||
CT 101. *(Renumbered from a duplicate `F-008`.)*
|
||||
|
||||
**Observed:** `curl: command not found`.
|
||||
**Cause:** Not in the base template; CT 101's package list did not include it.
|
||||
**Correction:** Installed `curl` and `ca-certificates`.
|
||||
**Consequence:** The proxy container needs its own minimal toolset. Do not
|
||||
assume packages installed in CT 100 exist in CT 101.
|
||||
|
||||
---
|
||||
|
||||
### F-016 — placeholder PID capture stored a literal `$!`
|
||||
CT 100. *(Renumbered from a duplicate `F-009`.)*
|
||||
|
||||
**Observed:** PID file contained `$!` rather than a number. The listener itself
|
||||
was healthy.
|
||||
**Cause:** Quoting error.
|
||||
**Correction:** Identified PID `12102` by inspection.
|
||||
**Consequence:** Superseded by F-019.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — network isolation
|
||||
|
||||
### F-017 — containers were provisioned on the home LAN
|
||||
`srv-b`, CT 100, CT 101.
|
||||
|
||||
**Observed:** Both containers had `net0` on `vmbr0` at `10.0.0.20/24` and
|
||||
`10.0.0.21/24`, gateway `10.0.0.1` — the ISP router's network. Anything on the
|
||||
home LAN could reach them.
|
||||
**Cause:** **Specification defect in `ENVIRONMENT.md` revision 4.** The
|
||||
document placed containers on both bridges, giving each a management address on
|
||||
the LAN, on the unstated assumption that a LAN workstation would browse
|
||||
staging. That assumption was never confirmed and was wrong.
|
||||
**Correction:** Both containers moved to `vmbr1` only, gateway `10.20.0.1`.
|
||||
`net1` deleted. Host NAT extended for `10.20.0.0/24`.
|
||||
**Consequence:** Containers are on the portless service bridge only. `srv-b` is
|
||||
router and bastion; access arrives via WireGuard. Promoted into the
|
||||
specification.
|
||||
|
||||
---
|
||||
|
||||
### F-018 — removing the LAN interface did not remove LAN reachability
|
||||
CT 100, CT 101.
|
||||
|
||||
**Observed:** After F-017, both containers still reached `10.0.0.1`
|
||||
successfully.
|
||||
**Cause:** Removing an interface removes an address, not a route.
|
||||
`ip_forward=1` plus the new `MASQUERADE -s 10.20.0.0/24 -o vmbr0` — added to
|
||||
give the containers internet access — also gave them the LAN, translated to the
|
||||
host's address.
|
||||
**Correction:** A `RETURN` in `nat POSTROUTING` ahead of both masquerades, and
|
||||
a `DROP` in `FORWARD`, both scoped `-s 10.20.0.0/24 -d 10.0.0.0/24`, plus an
|
||||
explicit `ACCEPT` for container-to-container traffic.
|
||||
**Consequence:** **Interface removal is not isolation.** Automation must assert
|
||||
the negative — that the LAN is unreachable — not merely that the interface is
|
||||
gone. Note that the containers still reach `srv-b` itself at `10.0.0.12`,
|
||||
because a container addressing its gateway takes the INPUT path and never
|
||||
enters FORWARD. That is required, not a leak.
|
||||
|
||||
---
|
||||
|
||||
### F-019 — placeholder backend did not survive reboot
|
||||
CT 100.
|
||||
|
||||
**Observed:** After the F-017 reboot, nothing listened on `10.20.0.10:8770`.
|
||||
**Cause:** The placeholder was a bare foreground process with a PID file. No
|
||||
supervision.
|
||||
**Correction:** Pending — to be recreated as `mechcomp-placeholder.service`.
|
||||
**Consequence:** Anything a proof depends on must be supervised, or the proof
|
||||
expires silently at the next reboot and the following session diagnoses a
|
||||
proxy fault that does not exist. Applies to temporary scaffolding as much as to
|
||||
real services.
|
||||
|
||||
---
|
||||
|
||||
### F-020 — service FQDN mapping changed twice
|
||||
`srv-b`, CT 100, CT 101.
|
||||
|
||||
**Observed:** `mechanical-compiler.dev.infra` was mapped to `10.20.0.10`, then
|
||||
corrected to `10.0.0.21`, then corrected again to `10.20.0.11`.
|
||||
**Cause:** Three different states, each correct for its moment. `10.20.0.10`
|
||||
was wrong — it pointed at the application rather than the proxy. `10.0.0.21`
|
||||
was right while CT 101 had a LAN address and workstation access was assumed.
|
||||
`10.20.0.11` became right once F-017 removed the LAN interface and access moved
|
||||
to WireGuard through `srv-b`.
|
||||
**Correction:** `10.20.0.11` on all three hosts.
|
||||
**Consequence:** Not an error in either direction — a value that tracked a
|
||||
topology decision. Recorded because the reasoning matters more than the value:
|
||||
**the service FQDN names the proxy, and the proxy has exactly one address.**
|
||||
|
||||
---
|
||||
|
||||
## Open, not closed
|
||||
|
||||
| # | Status |
|
||||
|---|---|
|
||||
| F-006 | Cause unproven. Recurrence should capture `dpkg` lock state. |
|
||||
| F-012 | Cause unproven. Leading candidate ruled out by inspection. |
|
||||
| F-019 | Correction pending. |
|
||||
Reference in New Issue
Block a user