244 lines
8.7 KiB
Markdown
244 lines
8.7 KiB
Markdown
# Handover — TESSERA Dataset Assistant
|
||
### Date: 2026-04-27
|
||
### For: Incoming dataset assistant
|
||
### Read this completely before doing anything
|
||
|
||
---
|
||
|
||
## 0. Your role
|
||
|
||
You are the dataset assistant. You own the pipeline that populates
|
||
`data/otivm.sqlite3` with physical-world data from the USB drives.
|
||
You do not touch game code, frontend, backend, or PM2.
|
||
|
||
The game development assistant works in parallel. They own `src/`,
|
||
`server/`, and everything the player sees. You own:
|
||
- `pipeline/` — all extraction and promotion scripts
|
||
- `data/create_otivm_db.sql` — the schema source of truth
|
||
- `data/staging_otivm.sqlite3` — your working database (never in git)
|
||
- `docs/` — dataset and pipeline documentation
|
||
|
||
You do not write to `data/otivm.sqlite3` directly. You write to
|
||
`data/staging_otivm.sqlite3`, verify, then copy to production on
|
||
explicit project owner approval.
|
||
|
||
---
|
||
|
||
## 1. Read these files before doing anything
|
||
|
||
In order:
|
||
|
||
1. `CLAUDE.md` — workflow, three-shell model, ground rules
|
||
2. `docs/TESSERA-dataset-registry.md` — every dataset evaluated,
|
||
triage decisions, drive inventory, what is on drives and what is not
|
||
3. `docs/RFC-TESSERA-4.0-001.md` — the database schema contract
|
||
4. `docs/RFC-TESSERA-3.0-PALEO-001.md` — paleo epoch table spec
|
||
5. `docs/TESSERA-pipeline-registry.md` — history of the old batch
|
||
pipeline, what completed, what failed, and why
|
||
6. This file
|
||
|
||
---
|
||
|
||
## 2. Current database state — as of 2026-04-27
|
||
|
||
### `data/otivm.sqlite3` — production
|
||
- 12,005 H9 rows across five waypoints, all `status=2` (current)
|
||
- All H5s at `status=2` in `h5_coverage`
|
||
- `paleo_epochs` table populated with 9 epochs per RFC-TESSERA-3.0-PALEO-001
|
||
- H3 IDs stored as INTEGER (64-bit)
|
||
|
||
### Five waypoints
|
||
| City | H5 TEXT | H9 cells |
|
||
|---|---|---|
|
||
| Ostia | `851e805bfffffff` | 2401 |
|
||
| Capua | `851e8333fffffff` | 2401 |
|
||
| Brundisium | `851e8ba3fffffff` | 2401 |
|
||
| Carthago | `85386e23fffffff` | 2401 |
|
||
| Alexandria | `853f5ba7fffffff` | 2401 |
|
||
|
||
### Field status
|
||
| Field | Status | Source |
|
||
|---|---|---|
|
||
| `elev_cm` | ✅ Current | GEBCO 2025 |
|
||
| `terrain` | ✅ Current (modern only — see Section 4) | ESA WorldCover 2021 |
|
||
| `hydro` | ✅ Current | HydroSHEDS v1.1 |
|
||
| `geo_dep` | ✅ Current | USGS MRDS |
|
||
| `geo_flag` | ✅ Current | BGR IGME5000 |
|
||
| `occ_flag` | ⚠ Placeholder (0x00 everywhere) | Stage 06 not written |
|
||
|
||
### `data/staging_otivm.sqlite3`
|
||
Identical to production as of last session. Always reset from
|
||
production before starting a new pipeline run:
|
||
```
|
||
cp data/otivm.sqlite3 data/staging_otivm.sqlite3
|
||
```
|
||
|
||
---
|
||
|
||
## 3. USB drives — what is present
|
||
|
||
Both drives mounted read-only at `/opt/data/` on every container.
|
||
Full inventory in `data/tessera_usb_inventory.txt`.
|
||
|
||
### Drive 1: TESSERA_APR26 (/dev/sdb1, 29GB, 21GB free)
|
||
| Dataset | Path | Size | Fields |
|
||
|---|---|---|---|
|
||
| GEBCO 2025 | `gebco/` | 6.8GB | `elev_cm` |
|
||
| HydroSHEDS v1.1 | `hydrosheds/` | 240MB | `hydro` |
|
||
| USGS MRDS | `mrds/mrds.csv` | 16MB | `geo_dep` |
|
||
|
||
### Drive 2: TESSERA_WORLD (/dev/sdd1, 29GB, 7GB free)
|
||
| Dataset | Path | Size | Fields |
|
||
|---|---|---|---|
|
||
| ESA WorldCover 2021 v200 | `worldcover/` | 22GB | `terrain` |
|
||
|
||
---
|
||
|
||
## 4. The restoration layer — critical concept
|
||
|
||
**`terrain` in the database is modern WorldCover 2021. It is wrong
|
||
for historical periods.**
|
||
|
||
WorldCover reflects 2021 land cover — cities, airports, drained
|
||
marshes, reservoirs. For all five OTIVM waypoints, the majority of
|
||
H9 cells within urban zones are classified as built-up or cropland.
|
||
In Roman times (14 BCE epoch) and Mesolithic times (8000 BCE epoch),
|
||
those same cells were overwhelmingly forested.
|
||
|
||
The Mediterranean basin was 60–70% forested in both periods. Today
|
||
it is not.
|
||
|
||
The restoration layer corrects this at query time using two datasets
|
||
not yet on the drives:
|
||
- **HYDE 3.3** — historical land use per epoch (what was actually there)
|
||
- **KK10** — potential natural vegetation (what would grow without humans)
|
||
|
||
Until these datasets are loaded and the restoration pipeline stage
|
||
is written, `terrain` is a modern snapshot, not a historical one.
|
||
The game development assistant has been informed. The game must not
|
||
present `terrain` values as historically accurate for any epoch
|
||
until the restoration layer is active.
|
||
|
||
**This is the most important pending pipeline work after the drive
|
||
additions are complete.**
|
||
|
||
---
|
||
|
||
## 5. What is missing from the drives — priority additions
|
||
|
||
These four datasets must be downloaded and added to Drive 1 before
|
||
the per-H5 pipeline can be built. Total: ~5.2GB, fits in 21GB free.
|
||
|
||
| Priority | Dataset | Size | Why needed |
|
||
|---|---|---|---|
|
||
| 1 | BGR IGME5000 shapefile | ~200MB | `geo_flag` currently depends on live API — must be local |
|
||
| 2 | HYDE 3.3 historical land use | ~4GB | Restoration layer — required |
|
||
| 3 | KK10 potential natural vegetation | ~500MB | Restoration layer — required alongside HYDE |
|
||
| 4 | HydroRivers Europe + Africa | ~500MB | Accurate river placement for `hydro` |
|
||
|
||
Download sources in `docs/TESSERA-dataset-registry.md`.
|
||
|
||
Drives are read-only when mounted. To add data:
|
||
1. Unmount from Proxmox host
|
||
2. Remount read-write on a machine with ext4 write access
|
||
3. Copy data
|
||
4. Remount read-only
|
||
5. Verify with inventory check before proceeding
|
||
|
||
**Do not begin pipeline design until all four additions are on Drive 1.**
|
||
|
||
---
|
||
|
||
## 6. The per-H5 pipeline — not yet built
|
||
|
||
The new pipeline replaces the old batch pipeline entirely. Key facts:
|
||
|
||
- Processes one H5 hex at a time
|
||
- Reads all data from USB drives (no live API calls)
|
||
- Writes to `staging_otivm.sqlite3` only
|
||
- Follows RFC-TESSERA-4.0-001 pipeline contract:
|
||
draft → validate → promote → copy to production
|
||
- Manually triggered with project owner approval
|
||
- Supersede support built in — can update existing H5 rows when
|
||
a source dataset improves
|
||
|
||
### Read strategy — mandatory
|
||
Always crop raster to H5 bounding box before sampling. Load the crop
|
||
into a numpy array in RAM. Sample all 2401 H9 centroids from the
|
||
array. Never seek 2401 individual points from USB.
|
||
|
||
Without this: GEBCO reads at ~25s per H5 (USB random seek speed).
|
||
With this: GEBCO reads at ~1-2s per H5 (one sequential crop + RAM).
|
||
|
||
### RAM allocation
|
||
- Baseline container RAM: 2GB
|
||
- Pipeline mode: 24GB (non-essential containers suspended on dev box)
|
||
- Relevant tile sizes: GEBCO tile ~891MB, WorldCover tile ~100MB
|
||
- In-memory strategy: load relevant tiles at pipeline start,
|
||
release at end
|
||
- Three Proxmox boxes: dev (pipeline work), staging (validation),
|
||
production (live game) — transfer via WireGuard mesh
|
||
|
||
### Python venv
|
||
- Path: `/home/otivm/pipeline-venv`
|
||
- Packages: h3, requests, numpy, rasterio, shapely, pyproj
|
||
- Do not use `/home/otivm/venv` — that belongs to the game assistant
|
||
|
||
### Pipeline scripts (committed, not yet functional for new pipeline)
|
||
- `pipeline/seed_extract.py` — old Dell-based extractor, do not re-run
|
||
- `pipeline/seed_promote.py` — old promotion script, do not re-run
|
||
- New per-H5 scripts to be written after drive additions complete
|
||
|
||
---
|
||
|
||
## 7. Infrastructure
|
||
|
||
### OTIVM container (CT 1105, proliant-dev, 10.0.0.23)
|
||
- App user: `otivm`
|
||
- Repo: `/home/otivm/OTIVM`
|
||
- Pipeline venv: `/home/otivm/pipeline-venv`
|
||
- Production DB: `data/otivm.sqlite3`
|
||
- Staging DB: `data/staging_otivm.sqlite3` (not in git)
|
||
- Claude Code runs here as `otivm` via `work` alias
|
||
|
||
### Three Proxmox boxes
|
||
- **proliant-dev (srv-a, 10.0.0.11)** — development and pipeline work
|
||
- **staging box** — validation before production
|
||
- **production box** — live game, never touched by pipeline directly
|
||
|
||
### Gitea
|
||
- Repo: `https://gitea.barternetwork.us/TheRON/OTIVM`
|
||
- Branch: `main`
|
||
- MCP: `mcp.civicus.us` — read any file directly from Claude chat
|
||
|
||
---
|
||
|
||
## 8. Hard rules
|
||
|
||
- Never write to `data/otivm.sqlite3` directly — always via staging
|
||
- Never commit `*.sqlite3` files — both databases are gitignored
|
||
- Never run pipeline without project owner approval and supervision
|
||
- Never modify `tessera.db` — it no longer exists (Dell decommissioned)
|
||
- Never touch game code (`src/`, `server/`, `public/`)
|
||
- Read `TESSERA-dataset-registry.md` before evaluating any new source
|
||
- One file at a time. One confirmation before proceeding.
|
||
- Do not start pipeline coding without explicit project owner instruction
|
||
|
||
---
|
||
|
||
## 9. Pending work — in order
|
||
|
||
1. **Drive additions** — project owner downloads and mounts four datasets
|
||
2. **Pipeline architecture document** — design before any code
|
||
3. **Per-H5 pipeline scripts** — one file at a time, supervised
|
||
4. **Restoration layer** — HYDE + KK10 integration into terrain field
|
||
5. **Stage 06 (occ_flag)** — archaeological sources, deferred until
|
||
simulation track begins
|
||
|
||
---
|
||
|
||
*Handover 2026-04-27 — dataset assistant track*
|
||
*Database seeded, paleo_epochs added, drives inventoried.*
|
||
*Pipeline not yet built. Drive additions required first.*
|
||
*The restoration layer is the most important pending concept.*
|