Files

otivm c0f5aebc44 Add infrastructure documentation and project setup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-04-25 10:48:33 +00:00

6.1 KiB

Raw Blame History

Container Provisioning — srv-a Ground Rules

Established: April 2026

This document records the correct procedure for provisioning new Debian 12 LXC containers on srv-a (dev Proxmox, 10.0.0.11). Follow this exactly. Every step was learned from direct experience — do not skip any of them.

1. Create the Container

pct create <VMID> local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
  --hostname <name> \
  --memory 512 \
  --cores 2 \
  --rootfs local-lvm:8 \
  --net0 name=eth0,bridge=vmbr0,gw=10.0.0.1,ip=10.0.0.<X>/24,type=veth \
  --nameserver 10.0.0.1 \
  --unprivileged 1 \
  --start 1

Check available template name first:

pveam list local

2. IP Address Allocation

IP	Container
10.0.0.10	apt-cache (CT 1104)
10.0.0.20	tessera-pipeline (CT 1101)
10.0.0.21	tessera-store (CT 1102)
10.0.0.22	tessera-dev (CT 1103)
10.0.0.23	otivm-dev (CT 1105)

Next available: 10.0.0.24

3. WireGuard IP Allocation

IP	Node
10.110.0.1	wg-pk (Linode hub)
10.110.0.2	shell.infra.civicus.us
10.110.0.4	fw (CVSTOS push node)
10.110.0.5	OVH node
10.110.0.6	corpusdb.infra.civicus.us
10.110.0.7–0.14	Home lab containers
10.110.0.15	OVH node
10.110.0.16	OVH node
10.110.0.17	mcp.civicus.us
10.110.0.18	otivm-dev

Next available: 10.110.0.19

4. Set Root Password

From srv-a (not from inside the container):

pct exec <VMID> -- passwd root

5. Fix Locale

⚠ Do NOT use update-locale — it fails in this environment.

echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen
echo "LANG=en_US.UTF-8" > /etc/default/locale
export LANG=en_US.UTF-8

6. Configure apt Proxy

The apt-cache container (10.0.0.10) runs apt-cacher-ng on port 3142. All new containers must be configured to use it.

⚠ Two lines required — HTTP through cache, HTTPS direct:

echo 'Acquire::http::Proxy "http://10.0.0.10:3142";' > /etc/apt/apt.conf.d/01proxy
echo 'Acquire::https::Proxy "DIRECT";' >> /etc/apt/apt.conf.d/01proxy

The HTTPS direct line is required because apt-cacher-ng cannot proxy HTTPS tunnels. Without it, any HTTPS apt source (e.g. nodesource) will fail with "403 CONNECT denied".

7. Install Base Packages

apt update && apt upgrade -y
apt install -y git python3 python3-venv python3-pip curl wget wireguard

8. Install Node.js v22

⚠ Do NOT use the nodesource setup script (curl ... | bash) — it creates conflicting source files. Use the manual method:

mkdir -p /etc/apt/keyrings
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | \
  gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] \
  https://deb.nodesource.com/node_22.x nodistro main" \
  > /etc/apt/sources.list.d/nodesource.list
apt update && apt install -y nodejs
node --version  # should be v22.x.x
npm --version

9. Install WireGuard

apt install -y wireguard
wg genkey | tee /etc/wireguard/private.key | wg pubkey > /etc/wireguard/public.key
chmod 600 /etc/wireguard/private.key
cat /etc/wireguard/public.key  # give this to wg-pk

Create /etc/wireguard/wg0.conf:

[Interface]
Address = 10.110.0.<X>/32
ListenPort = 51820
PrivateKey = <contents of /etc/wireguard/private.key>

[Peer]
PublicKey = 1+Wb++fjXNbY0joOvj4AZvJgF6b125YOPSFsmNqVo3I=
AllowedIPs = 10.110.0.0/22
Endpoint = 198.58.111.109:51820
PersistentKeepalive = 25

systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0
ping -c 3 10.110.0.1  # confirm mesh

On wg-pk — add peer (live, no restart needed):

wg set wg0 peer <pubkey> allowed-ips 10.110.0.<X>/32
# append to /etc/wireguard/wg0.conf as well

10. Create App User

useradd -m -s /bin/bash <appuser>
passwd <appuser>

11. Python Venv

Always as the app user, never as root:

su - <appuser>
python3 -m venv /home/<appuser>/venv
echo 'source /home/<appuser>/venv/bin/activate' >> /home/<appuser>/.bashrc

⚠ Never run Python outside the venv. This is an absolute rule.

12. Install PM2

As root:

npm install -g pm2

Set up systemd service under app user:

env PATH=$PATH:/usr/bin pm2 startup systemd -u <appuser> --hp /home/<appuser>

PM2 must always run as the app user. Never as root.

13. Install Claude Code

As the app user:

su - <appuser>
mkdir -p ~/.npm-global
npm config set prefix '~/.npm-global'
echo 'export PATH=~/.npm-global/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
npm install -g @anthropic-ai/claude-code
claude --version

Authenticate on first run:

cd ~/REPONAME
claude
# Select: Claude account with subscription
# Follow browser-based login

14. Install Webmin

As root:

curl -o webmin-setup-repo.sh https://raw.githubusercontent.com/webmin/webmin/master/webmin-setup-repo.sh
sh webmin-setup-repo.sh
apt install -y webmin
systemctl disable webmin  # do not autostart

Add aliases to /root/.bashrc:

echo "alias webmin-on='systemctl start webmin && echo Webmin started'" >> /root/.bashrc
echo "alias webmin-off='systemctl stop webmin && echo Webmin stopped'" >> /root/.bashrc
source /root/.bashrc

Webmin listens on port 10000. Start it only when needed.

15. SSL Certificate Rule

⚠ ABSOLUTE RULE for all nodes with Nginx:

Issue the SSL certificate FIRST with --nginx authenticator. Add any basic auth AFTER the certificate is issued. Never reverse this order — basic auth blocks the ACME challenge.

# 1. Create vhost with no auth
# 2. Dry run
certbot certonly --nginx -d <domain> --dry-run
# 3. Issue real certificate
certbot certonly --nginx -d <domain>
# 4. Add SSL stanzas to vhost
# 5. Add basic auth if needed
# 6. Reload nginx

16. Take Baseline Backup

Before deploying any code, take a backup:

vzdump <VMID> --storage local --compress zstd --mode stop

Download via Webmin → File Manager → /var/lib/vz/dump/ Store offline. This is the recovery point.

docs/provisioning.md — TheRON srv-a — April 2026

6.1 KiB Raw Blame History Unescape Escape