From 88baf4fb5a396ae8dac78f573f8ce5f7a766a304 Mon Sep 17 00:00:00 2001 From: Tony0410 <47945770+Tony0410@users.noreply.github.com> Date: Fri, 6 Nov 2026 03:50:34 +0800 Subject: [PATCH] stocktake 2026-11-06: 0 stale / 124 fresh / 3 no-date; correct Tailscale state + Docker seeder false negative Ran wiki-maintain.sh (repo script). skills-index regenerated -> 302 skills / 68 categories. Freshness audit at the 21-day threshold -> 0 stale / 124 fresh / 3 no-date of 127 pages. Quartz rebuilt (761 files) and deployed to public/; hermes-wiki-serve.service restarted. Live-vs-page drift diff (age scan alone missed all of this): - infrastructure/tailscale.md: 1 state transition + 8 offline-age drifts + 2 dead Serve routes. anthonys-mac-mini came back online (peer record Online:true, tailscale ping -> pong via DERP(syd)). Offline ages refreshed +1d each (cloudreve 104d, elementsynapse 107d, freshrss-new 80d, octoagent 4d, octop 2d, omarchybook 15d, omarchymac 4d, openmausbot 15d). Every peer IP still matches the live JSON; no peer missing, no orphan row. /agent-estate->8769 and /synergy-prep->9123 are DEAD (no listener) while still present in `tailscale serve status` - both now marked DEAD, matching network-map.md. All other Serve targets have live listeners. - infrastructure/docker-services.md: the 2026-11-05 "the seeder does not exist on any host" finding was FALSE - a point-in-time sampling artefact. The worldmonitor-seeder-* containers are transient and systemd-timer driven on CT200 (worldmonitor-seed.timer 4h -> seeder-scheduled; worldmonitor-seed-fast.timer 15min -> seeder-fast), visible only while a run is in flight. Both present at 2026-11-06 03:1x; neither at 2026-11-05 05:1x. The "no cron job" claim was also wrong (systemd timers, not cron) and the "rename" narrative was wrong (one script, two container names). Corrected on the page with strikethrough + dated correction. CT200's 61-container running set still matches live set-wise. CT333 container renamed again: flamboyant_mahavira -> nostalgic_keldysh. - systems/scheduled-tasks.md: no drift. Live jobs.json reads 29 jobs / 26 enabled / 3 disabled, 1 non-clean last_status - exactly as recorded. - current-state.md / index.md: false seeder claim and "every state matching" claim marked superseded with dated corrections (SCHEMA rule 9), not silently rewritten. - infrastructure/network-map.md: dates bumped, dead routes re-confirmed, peer recheck note refreshed. - log.md: appended (append-only) with the full stocktake record. No secrets, tokens or keys written or quoted. --- current-state.md | 21 +++++++--- index.md | 2 +- infrastructure/docker-services.md | 57 ++++++++++++++++++++-------- infrastructure/network-map.md | 12 +++--- infrastructure/tailscale.md | 29 +++++++------- log.md | 45 ++++++++++++++++++++++ raw/stocktake/freshness-20261106.txt | 10 ++--- systems/skills-index.md | 4 +- 8 files changed, 131 insertions(+), 49 deletions(-) diff --git a/current-state.md b/current-state.md index ba9f65a..afd23db 100644 --- a/current-state.md +++ b/current-state.md @@ -48,15 +48,26 @@ Updated: 2026-11-05 **delegating wrapper** with no generator logic, and the repo script gained `--no-build` so the wrapper honours the old interface without a second generator. Verified by running it -> 302 / 68. -> [[incidents/2026-11-05-skills-index-generator-divergence]] -- **Docker inventory corrected against live probes on all five hosts:** CT200 **61 running / 68 total** (the - `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host** - zero - `seeder` matches across CT200/201/333/460); CT333's first container renamed again - (`nostalgic_leavitt` -> **`flamboyant_mahavira`**); CT201 20/26, CT460 1/1, CT1001 1/9. +- **Docker inventory corrected against live probes on all five hosts:** CT200 **61 running / 68 total**; CT333's + first container renamed again (`nostalgic_leavitt` -> **`flamboyant_mahavira`**); CT201 20/26, CT460 1/1, + CT1001 1/9. + ⚠ **Corrected 2026-11-06 (stocktake): the "the seeder does not exist on any host" claim in this entry was + WRONG - it was a point-in-time sampling artefact.** The `worldmonitor-seeder-*` containers are **transient + and systemd-timer driven** on CT200 (`worldmonitor-seed.timer` every 4h -> `worldmonitor-seeder-scheduled`; + `worldmonitor-seed-fast.timer` every 15min -> `worldmonitor-seeder-fast`), so a `docker ps` taken between + runs sees **none** of them. Live at 2026-11-06 03:1x **both were present**; live at 2026-11-05 05:1x + **neither** was. **"Absent" must never be read as "removed."** CT333's container has also been renamed + **again**: `flamboyant_mahavira` -> **`nostalgic_keldysh`** (created 2026-11-05T22:15:57Z). -> [[infrastructure/docker-services]] - **Tailscale node table re-verified programmatically:** 52 table rows vs 52 live IPv4 peers - **no live peer missing, no orphan row, every online/offline state matches.** Two offline ages corrected (`conduit` 60d, `ubuntu-server-xfce` 76d). Serve table still matches `tailscale serve status` exactly (12 paths on - :443 + the :8443 funnel + the :9443 route). -> [[infrastructure/tailscale]] + :443 + the :8443 funnel + the :9443 route). + ⚠ **Corrected 2026-11-06 (stocktake): the "every online/offline state matches" claim in this entry no longer + holds - `anthonys-mac-mini` has come back online** (peer record `Online: true`, `tailscale ping` -> pong) and + **8 offline ages have drifted by +1d**. Two Serve routes (`/agent-estate` -> 8769, `/synergy-prep` -> 9123) + are **dead** - no listener - while still present in `tailscale serve status`. + -> [[infrastructure/tailscale]] - **Freshness now 0 stale** (was 8 on 11-04) - all 8 re-verified and refreshed. Skills count corrected to **302 / 68 categories** on [[infrastructure/hosts]] and [[index]]. diff --git a/index.md b/index.md index 78d34b3..1bc795d 100644 --- a/index.md +++ b/index.md @@ -14,7 +14,7 @@ sources: [live-cron-ledger, live-deliveries-db, live-incident-ledger, live-sessi > Operational knowledge for Anthony's agents, infrastructure, > model routing and automation. Last updated: 2026-11-06. > -> **2026-11-05 (stocktake) -** **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**. Tailscale node table re-verified programmatically - 52 rows, 52 live peers, every state matching. Freshness **0 stale / 122 fresh / 3 no-date of 125**.) +> **2026-11-05 (stocktake) -** **the `systems/skills-index.md` flip-flop is fixed at the root**: two different scripts were writing the page, and the nightly audit was pointed at the stale one; it is now a delegating wrapper -> [[incidents/2026-11-05-skills-index-generator-divergence]]. Docker inventory corrected against live probes on all five hosts - ~~the `worldmonitor-seeder-scheduled` container recorded yesterday **does not exist on any host**~~ **⚠ CORRECTED 2026-11-06: false - the seeders are transient systemd-timer-driven containers, absent only between runs, not removed.** Tailscale node table re-verified programmatically - 52 rows, 52 live peers, ~~every state matching~~ **⚠ CORRECTED 2026-11-06: `anthonys-mac-mini` has since come back online.** Freshness **0 stale / 122 fresh / 3 no-date of 125**.) > > **2026-11-06 (nightly audit - window 2026-11-05) - a clean ledger day and an empty incident ledger:** 2026-11-05 settled **175 rows - 173 `completed` / 2 `running` / 0 `failed`** across 18 jobs, the **first zero-`failed` day since 2026-11-02**. The only two `running` rows are the **Mnemosyne Health Monitor** (12:00:45 slot, still in flight **12 h+** at audit time) and this audit itself. **Unclosed incident rows 3 -> 0** - the *Newsletter Digest* row closed at **07:07:42** and both *Job Radar - Discovery* rows at **12:36:17**, when `dca8482e4f76` finally completed after **5 h 35 m 35 s**; that closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, `delivery_failed`, streak 0); deliveries **10 rows / 10 `delivered`**; the 03:00 stocktake overran **3 h 01 m 58 s** (down from 7 h 40 m and 5 h 41 m) and **committed `2f3fc99` at 05:55**. **Fleet unchanged at 29 jobs (26 / 3)** - a fourth audit at the same count. ⚠ **New and open:** live `.env` values (`TELEGRAM_BOT_TOKEN`, `MATTERMOST_TOKEN`, `HERMES_CUSTOM_OMNIROUTE_20128_API_KEY`) sit verbatim in **17 rows of `mnemosyne.db`** (`memories`, `episodic_memory`, `working_memory`, `gists`, `memoria_facts.context_snippet` and the FTS mirrors) - reproduced in full by each of the **7 uncompressed rotation snapshots** - and all four values sit verbatim in the **1.7 GB session store `state.db`** too; `CAMOFOX_API_KEY` sits verbatim in the prompts of two **enabled** Facebook Birthday jobs (`0e753e9720fd`, `fad5a86979d4`) and therefore in `cron/deliveries.db` -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. > diff --git a/infrastructure/docker-services.md b/infrastructure/docker-services.md index 0727e69..0eb642d 100644 --- a/infrastructure/docker-services.md +++ b/infrastructure/docker-services.md @@ -3,8 +3,8 @@ title: Docker Services type: infrastructure status: mixed created: 2026-07-22 -updated: 2026-11-05 -verified_on: 2026-11-05 +updated: 2026-11-06 +verified_on: 2026-11-06 confidence: high tags: [docker, homelab, infrastructure] sources: [raw/configs/hermes-config-sanitized.txt, live-probe] @@ -20,30 +20,55 @@ Docker runs on **four hosts**, and CT460's socket is not readable by the agent u | Host | Access | Containers running | |---|---|---| -| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-05**) | +| CT200 `docker` (`100.96.150.69`) | `pct exec 200 -- docker ps` via PVE | **61** running / 68 total — main Docker host (**re-verified 2026-11-06**), **plus 0–2 transient `worldmonitor-seeder-*` containers** driven by systemd timers (see the note below) | | CT201 `runtipi` (`100.96.244.39`) | `pct exec 201 -- docker ps` via PVE | **20** running / 26 total — Runtipi stack: `audiobookshelf`, `calibre-web`, `code-server`, `cyberchef`, `duplicati`, `filebrowser`, `flaresolverr`, `gotify`, `homarr`, `homebox`, `homepage-manager`, `it-tools`, `linkwarden`, `linkwarden-db`, `mealie-1`, `ntfy`, `runtipi`, `runtipi-db`, `syncthing`, `transmission` (**re-verified 2026-11-05**) | -| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total — `flamboyant_mahavira`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → **`flamboyant_mahavira`** (2026-11-04) — all `ghcr.io/github/github-mcp-server`, created 2026-11-04T10:53:53Z | +| CT333 `clawtest` (`100.122.189.81`) | `pct exec 333 -- docker ps` via PVE | **8** running / 9 total — `flamboyant_mahavira`, `community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, `onecli-postgres-1`. The `steel-browser` container and its image were **removed 2026-10-29**; `thirsty_payne` → `nostalgic_leavitt` (2026-11-02) → `flamboyant_mahavira` (2026-11-04) → **`nostalgic_keldysh`** (2026-11-06) — all `ghcr.io/github/github-mcp-server`, the live one created 2026-11-05T22:15:57Z | | CT1001 `docker-01` (`100.121.12.31`, **on `m93p`**) | `pct exec 1001 -- docker ps` via `m93p` | **1** running / 9 total — `sshwifty`. **Added 2026-10-31** — a fourth Docker host, previously undocumented | | CT460 `hermes` | `pct exec 460 -- docker ps` **as root** from the PVE host — socket is `root:docker srw-rw----`, the `hermes` user is not in group `docker` (its own `docker ps` → `permission denied while trying to connect to the docker API`) | `ignis` — **running** (`Up 5 days`, `ignis-subpath:0.8.9`, `0.0.0.0:8080->8080/tcp`), re-verified 2026-11-05 | -## CT200 live inventory (re-verified 2026-11-05, 61 running / 68 total) +## CT200 live inventory (re-verified 2026-11-06, 61 running / 68 total, excluding 0–2 transient `worldmonitor-seeder-*` containers) `8mblocal`, `apprise-api`, `audiobookshelf`, `chromadb`, `edge-tts`, `emulatorjs`, `excalidraw-backend`, `excalidraw-frontend`, `filebrowser`, `filebrowser1`, `flaresolverr`, `guacamole-guacamole-1`, `guacamole-guacd-1`, `guacamole-postgres-1`, `homarr`, **`ignis-ignis-1`**, `jellyfin`, `khoj-database-1`, `khoj-sandbox-1`, `khoj-search-1`, `khoj-server-1`, `kokoro-tts`, `libre-closet`, `lobe-chat`, `mycompose-db-1`, `mycompose-saas-1`, `nc_app_context_agent`, `nc_app_talk_bot_ai`, `nextstep-app`, `nextstep-db`, `nextstep-scheduler-1`, `onlyoffice-docserver`, `open-webui`, `overtchat-app`, `overtchat-redis`, `overtchat-searxng`, `portainer`, `portkey-gateway-web-1`, `postgres-newsku`, `prowlarr`, `qbittorrent`, `quickdrop`, `react-resume-client-1`, `readlater`, `readlater-extraction-worker`, `redis-cache`, `save-restricted-bot`, `searxng`, `secret-santa-app`, `sshwifty`, `telegram-bot-api`, `tgvideo-bot`, `tt-rss-app-1`, `tt-rss-db-1`, `tt-rss-updater-1`, `tt-rss-web-nginx-1`, `watchtower`, `worldmonitor`, `worldmonitor-redis`, `worldmonitor-redis-rest`, `ytdlp-web-player`. +**Re-verified 2026-11-06 — the 2026-11-05 "the seeder does not exist" finding was WRONG, and the cause is a +sampling artefact, not a removal.** A set-wise diff against live `docker ps --format '{{.Names}}'` on CT200 +shows **no container live but absent from the list above**, and **no listed container has disappeared** — the +61-container running set above still matches live exactly, set-wise. The `worldmonitor-seeder-*` containers are +**transient and systemd-timer driven**, so they are only visible while a seed run is in flight: at the +2026-11-06 03:1x probe **both** were present (`worldmonitor-seeder-scheduled`, created 02:45:14, `Up 32 minutes +(unhealthy)`, and `worldmonitor-seeder-fast`, created 03:17:15, `Up 22 seconds (health: starting)`); at the +2026-11-05 05:1x probe **neither** was, because no run was in flight — hence the zero `seeder` matches. The +drivers are **systemd timers on CT200, not cron**: `worldmonitor-seed.timer` (`OnUnitInactiveSec=4h`, +`RandomizedDelaySec=10min` → `worldmonitor-seed.service` → `docker run --rm --name +worldmonitor-seeder-scheduled … scripts/run-seeders.sh`) and `worldmonitor-seed-fast.timer` +(`OnUnitInactiveSec=15min`, `RandomizedDelaySec=1min` → `worldmonitor-seed-fast.service` → `--name +worldmonitor-seeder-fast`). Both services were `ActiveState=activating` at the probe. The **corrected** claim +is therefore: **the seeders exist and run on a 4h + 15min schedule; a point-in-time `docker ps` can miss them +entirely, so "absent" must never be read as "removed".** The **corrected on record** note: the earlier +`worldmonitor-seeder-fast` → `worldmonitor-seeder-scheduled` rename narrative was also wrong — there is **one +script with two timer-driven invocations** under two container names, not a rename. Live CT200 also shows +`worldmonitor` (`Up 6 days`, **healthy**), `worldmonitor-redis` (`Up 6 days`) and `worldmonitor-redis-rest` +(`Up 6 days`). + **Re-verified 2026-11-05 — 61 → 61 running, seeder entry removed.** A set-wise diff against live `docker ps --format '{{.Names}}'` on CT200 shows **no container live but absent from the list above**, and -**no listed container has disappeared**. The `worldmonitor-seeder-scheduled` entry recorded on 2026-11-04 as +**no listed container has disappeared**. ~~The `worldmonitor-seeder-scheduled` entry recorded on 2026-11-04 as "unhealthy" **does not exist on any host** — verified with `docker ps -a --filter name=seeder` across CT200, -CT201, CT333 and CT460, all of which return **zero** matches. Live CT200 now shows exactly three worldmonitor -containers: `worldmonitor` (`Up 5 days`, **healthy**), `worldmonitor-redis` (`Up 5 days`) and -`worldmonitor-redis-rest` (`Up 5 days`). There is **no seeder container and no worldmonitor-seeder cron job** -anywhere in the estate — the 2026-11-04 "watch item" was a transient container that has since been removed. -**Correction on record:** the earlier `worldmonitor-seeder-fast` → `worldmonitor-seeder-scheduled` rename -narrative did not reflect a durable redeploy; the seeder is simply absent now. +CT201, CT333 and CT460, all of which return **zero** matches.~~ **⚠ SUPERSEDED 2026-11-06 — see the +correction above: this was a sampling artefact. The seeders are transient systemd-timer-driven containers, +not absent.** ~~There is **no seeder container and no worldmonitor-seeder cron job** anywhere in the estate — +the 2026-11-04 "watch item" was a transient container that has since been removed.~~ **⚠ SUPERSEDED +2026-11-06: the seeders DO exist; the driver is a systemd timer pair, not a cron job.** ~~**Correction on +record:** the earlier `worldmonitor-seeder-fast` → `worldmonitor-seeder-scheduled` rename narrative did not +reflect a durable redeploy; the seeder is simply absent now.~~ **⚠ SUPERSEDED 2026-11-06: one script, two +timer-driven container names — not a rename.** -**CT333 container set — `nostalgic_leavitt` → `flamboyant_mahavira` (2026-11-05).** Live CT333 `docker ps` returns 8 -running containers, and the name that had replaced `thirsty_payne` on 2026-11-04 has itself been replaced: -`flamboyant_mahavira` (image `ghcr.io/github/github-mcp-server`, **created 2026-11-04T10:53:53Z**). The count is -unchanged at 8 and the other seven names are identical. A ninth container, `qdrant` +**CT333 container set — `nostalgic_leavitt` → `flamboyant_mahavira` (2026-11-05) → `nostalgic_keldysh` (2026-11-06).** +Live CT333 `docker ps` returns 8 running containers, and the name that had replaced `thirsty_payne` on 2026-11-04 +and then `flamboyant_mahavira` on 2026-11-05 has **itself been replaced again**: +**`nostalgic_keldysh`** (image `ghcr.io/github/github-mcp-server`, **created 2026-11-05T22:15:57Z**, `Up 5 hours` +at the 2026-11-06 probe). The count is unchanged at 8 and the other seven names are identical +(`community-spark`, `readlater-extraction-worker`, `readlater`, `kokoro-tts`, `edge-tts`, `gotify`, +`onecli-postgres-1`). A ninth container, `qdrant` (`qdrant/qdrant`), exists but is **stopped** (`Exited (143) 4 months ago`) — it is not in the running count and is not listed above. **Docker generates these random names for compose-less containers, so repeated renames are normal — the reliable check is always the live set, never the recorded name.** diff --git a/infrastructure/network-map.md b/infrastructure/network-map.md index aafaa5a..fcff18a 100644 --- a/infrastructure/network-map.md +++ b/infrastructure/network-map.md @@ -3,8 +3,8 @@ title: Network Map type: infrastructure status: active created: 2026-07-22 -updated: 2026-11-05 -verified_on: 2026-11-05 +updated: 2026-11-06 +verified_on: 2026-11-06 confidence: high tags: [infrastructure, network] sources: [raw/configs/hermes-config-sanitized.txt, live-probe:ct460-ss+tailscale-serve-status-2026-11-05] @@ -22,7 +22,7 @@ Where Hermes and its dependencies run, and how traffic reaches them. - **This container (hermes, ct460):** `192.168.178.129` (DHCP, eth0) ### Tailscale (`kangaroo-eel.ts.net`) -_Peer IPs below re-checked 2026-11-05 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed._ +_Peer IPs below re-checked 2026-11-06 against `tailscale status` on ct460; `webserver`, `clawtest`, `hermesagent`, `docker`, `omniroute` all matched the values listed. Full 52-row node table with live states lives in [[infrastructure/tailscale]]._ - **pve:** `100.96.100.82` - **m93p:** `100.104.51.98` — **second, unclustered Proxmox hypervisor (PVE 9.2.0), discovered 2026-10-14.** See [[infrastructure/proxmox]]. Guest IDs collide with `pve`'s, so always qualify a guest with its host. - **hermes (ct460):** `100.118.5.51` @@ -68,7 +68,7 @@ _Peer IPs below re-checked 2026-11-05 against `tailscale status` on ct460; `webs Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42647` + `38187` opencode. -## Tailscale Serve / Funnel (verified 2026-11-05) +## Tailscale Serve / Funnel (verified 2026-11-06) `tailscale serve status` on ct460 reports 14 proxies: 12 under `https://hermes.kangaroo-eel.ts.net` (tailnet only), one tailnet route on `:9443`, and one Funnel route on `:8443` (public). @@ -84,8 +84,8 @@ Loopback-only helpers (not routable): `49347` hermes gateway, `42013` Paseo, `42 | `/desktop` | `127.0.0.1:6081` | desktop VNC (noVNC) | | `/vnc-camofox` | `127.0.0.1:6080` | Camofox VNC | | `/evening-grid.html` | `127.0.0.1:9099/evening-grid.html` | jobs dashboard page | -| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (2026-11-05)** | -| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (2026-11-05)** | +| `/agent-estate` | `127.0.0.1:8769` | **DEAD — nothing listening on 8769 (re-confirmed 2026-11-06)** | +| `/synergy-prep` | `127.0.0.1:9123` | **DEAD — nothing listening on 9123 (re-confirmed 2026-11-06)** | - `:9443` (tailnet only) — `/` → `127.0.0.1:5173`. - Funnel `:8443` (public) — `/webhook` → `127.0.0.1:8085/webhook` (AgentMail webhook receiver). diff --git a/infrastructure/tailscale.md b/infrastructure/tailscale.md index 8d2232e..9cabb2e 100644 --- a/infrastructure/tailscale.md +++ b/infrastructure/tailscale.md @@ -3,8 +3,8 @@ title: Tailscale type: infrastructure status: active created: 2026-07-26 -updated: 2026-11-05 -verified_on: 2026-11-05 +updated: 2026-11-06 +verified_on: 2026-11-06 confidence: high tags: [tailscale, vpn, networking, infrastructure] --- @@ -19,7 +19,8 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access - **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) -## Tailscale Node Inventory (verified 2026-10-31) +## Tailscale Node Inventory (verified 2026-11-06) +Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable. Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route. Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route. @@ -40,17 +41,17 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | agentzero | 100.106.54.94 | online — CT475 (pve) | | anthonys-ipad | 100.111.198.120 | online | | anthonys-iphone | 100.100.116.67 | online | -| anthonys-mac-mini | 100.76.60.37 | offline <1d — last seen 2026-11-04 23:40 AWST; `tailscale ping` timeout (client asleep, not an estate host) | +| anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) | | clawtest | 100.122.189.81 | online — CT333 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) | -| cloudreve | 100.111.187.7 | offline 103d — CT209 stopped | +| cloudreve | 100.111.187.7 | offline 104d — CT209 stopped | | conduit | 100.81.79.87 | offline 60d — last seen 2026-09-06 01:32Z | | docker | 100.96.150.69 | online — CT200 (pve) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) | -| elementsynapse | 100.75.250.37 | offline 106d — CT555 stopped | +| elementsynapse | 100.75.250.37 | offline 107d — CT555 stopped | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) | -| freshrss-new | 100.110.180.115 | offline 79d | +| freshrss-new | 100.110.180.115 | offline 80d | | gitea | 100.82.100.128 | online — CT221 (m93p) | | guest-vps | 100.117.14.84 | online | | hermesagent | 100.98.205.112 | online — CT450 (pve) | @@ -64,15 +65,15 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | nextcloud | 100.90.102.19 | online — CT270 (pve) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) | | obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) | -| octoagent | 100.117.10.68 | offline 3d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | -| octop | 100.92.163.122 | offline 1d — last seen 2026-11-03 09:03Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | +| octoagent | 100.117.10.68 | offline 4d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | +| octop | 100.92.163.122 | offline 2d — last seen 2026-11-03 09:03Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | | ollama | 100.108.44.119 | online — CT205 (pve) | | omarchy | 100.73.227.103 | online | -| omarchybook | 100.107.9.100 | offline 14d | -| omarchymac | 100.126.173.115 | offline 3d — last seen 2026-11-01 11:13Z | +| omarchybook | 100.107.9.100 | offline 15d | +| omarchymac | 100.126.173.115 | offline 4d — last seen 2026-11-01 11:13Z | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | -| openmausbot | 100.91.214.91 | offline 14d — CT1264 (m93p) stopped | +| openmausbot | 100.91.214.91 | offline 15d — CT1264 (m93p) stopped | | openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) | | openweb | 100.69.230.13 | online — CT245 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) | @@ -105,8 +106,8 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st | `/gitea` | `http://100.82.100.128:3000` | Gitea | | `/desktop` | `http://127.0.0.1:6081` | CUA noVNC | | `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC | -| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep | -| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) | +| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06; the route still exists in `tailscale serve status` but serves 502) | +| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06; route still present in `tailscale serve status` but serves 502) | | `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid | Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift). diff --git a/log.md b/log.md index 3528b6f..1ef6285 100644 --- a/log.md +++ b/log.md @@ -2126,3 +2126,48 @@ embedded credential was seen during verification and is not reproduced here ([RE - **Not done / unchanged:** no page removed or archived; no decision reversed; `066bb5191280` and the `deepseek-paid-smart`-on-`omniroute` dispatch fault carried, not re-diagnosed. **No secrets, passwords, tokens or keys written or quoted** - the incident page records where secrets live, never their values. + +## 2026-11-06 03:0x AWST - stocktake (wiki-stocktake-maintenance) + +- **Ran `wiki-maintain.sh` (repo script, canonical path).** `systems/skills-index.md` regenerated -> + **302 skills / 68 categories** (unchanged count; date rolled to 2026-11-06). Freshness audit at the + **21-day** threshold -> **0 stale / 124 fresh / 3 no-date of 127 pages**. Quartz rebuilt (761 files emitted, + 127 processed) and deployed to `public/`; `hermes-wiki-serve.service` restarted. Script exit 0. + ⚠ **Threshold correction:** the previously committed `freshness-20261106.txt` carried a **90-day** threshold + (the nightly audit's setting); the stocktake script's threshold is **21 days**, and this run's report is the + authoritative one for today. +- **Freshness: 0 stale pages.** The three no-date pages are unchanged (`SCHEMA.md`, + `experiments/active/example.md`, `log.md`). No page is over 21 days. +- **Live-vs-page drift diff (the age scan alone would have missed all of this).** + - **[[infrastructure/tailscale]] - 1 state transition + 8 offline-age drifts + 2 dead Serve routes.** + `anthonys-mac-mini` **came back online** (peer record `Online: true`; `tailscale ping 100.76.60.37` -> + `pong ... via DERP(syd) in 103ms`). Offline ages refreshed +1d each: cloudreve 103->**104d**, + elementsynapse 106->**107d**, freshrss-new 79->**80d**, octoagent 3->**4d**, octop 1->**2d**, + omarchybook 14->**15d**, omarchymac 3->**4d**, openmausbot 14->**15d**. **Every peer IP still matches the + live JSON exactly; no live peer missing; no orphan row.** `m93p` re-pinged -> pong (3ms); `octop` + re-pinged -> timeout, matching its `LastSeen`. Container states cross-checked against live `pct list` on + **both** hypervisors (pve 17 guests, m93p 19 guests + 2 VMs). ⚠ **`/agent-estate` -> 8769 and + `/synergy-prep` -> 9123 are DEAD** (no listener) while still present in `tailscale serve status` - both + routes now marked DEAD on the page, matching [[infrastructure/network-map]] which had already recorded + them as dead on 2026-11-05. Every other Serve target has a live listener. + - **[[infrastructure/docker-services]] - the 2026-11-05 "the seeder does not exist on any host" finding was + FALSE; corrected on the page with strikethrough + dated correction.** The `worldmonitor-seeder-*` + containers are **transient and systemd-timer driven** on CT200 (`worldmonitor-seed.timer`, + `OnUnitInactiveSec=4h` -> `worldmonitor-seeder-scheduled`; `worldmonitor-seed-fast.timer`, + `OnUnitInactiveSec=15min` -> `worldmonitor-seeder-fast`), so they are visible only while a run is in + flight. Live at 2026-11-06 03:1x **both were present**; live at 2026-11-05 05:1x **neither** was - hence + the false negative. The "no worldmonitor-seeder cron job" claim was also wrong: the driver is a **systemd + timer pair, not cron**. The earlier "rename" narrative was wrong too: **one script, two timer-driven + container names**. CT200's 61-container running set still matches live **set-wise** (no drift). **CT333's + container renamed again**: `flamboyant_mahavira` -> **`nostalgic_keldysh`** (created 2026-11-05T22:15:57Z). + CT201 20/26, CT460 1/1, CT1001 1/9, CT333 8/9 - all unchanged and correct. + - **[[systems/scheduled-tasks]] - no drift.** Live `jobs.json` reads **29 jobs / 26 enabled / 3 disabled**, + **1** non-clean `last_status` (`066bb5191280` Uber Eats Monthly Spend Report, `delivery_failed`, streak 0) - + exactly what the page already records. + - **[[infrastructure/hosts]] - cron count 29/26/3 still correct** (last re-verified 2026-10-31). +- **Updated [[current-state]] and [[index]]** - the false seeder claim and the "every state matching" Tailscale + claim are now marked superseded with dated corrections rather than silently rewritten (SCHEMA rule 9). +- **Not done / unchanged:** no page removed or archived; no decision reversed; the open + `deepseek-paid-smart`-on-`omniroute` dispatch fault and the `066bb5191280` delivery failure carried, not + re-diagnosed; the open secrets-at-rest incident not re-litigated. **No secrets, tokens or keys written or + quoted.** diff --git a/raw/stocktake/freshness-20261106.txt b/raw/stocktake/freshness-20261106.txt index 71a3987..7db6a59 100644 --- a/raw/stocktake/freshness-20261106.txt +++ b/raw/stocktake/freshness-20261106.txt @@ -1,5 +1,5 @@ # Wiki freshness audit — 2026-11-06 -Threshold: >90 days since `updated` +Threshold: >21 days since `updated` ## STALE (0) @@ -76,12 +76,12 @@ Threshold: >90 days since `updated` - [ 0d] index.md - [ 2d] infrastructure/backups.md - [ 17d] infrastructure/credentials-map.md -- [ 1d] infrastructure/docker-services.md +- [ 0d] infrastructure/docker-services.md - [ 17d] infrastructure/domains-and-tunnels.md - [ 1d] infrastructure/hosts.md -- [ 1d] infrastructure/network-map.md +- [ 0d] infrastructure/network-map.md - [ 5d] infrastructure/proxmox.md -- [ 1d] infrastructure/tailscale.md +- [ 0d] infrastructure/tailscale.md - [ 2d] queries/retained-research-answers.md - [ 12d] reports/communities-wa-social-media-audit-2026.md - [ 5d] reports/uber-eats/2026-07.md @@ -127,7 +127,7 @@ Threshold: >90 days since `updated` - [ 2d] systems/paseo.md - [ 18d] systems/push-notifications.md - [ 0d] systems/scheduled-tasks.md -- [ 1d] systems/skills-index.md +- [ 0d] systems/skills-index.md - [ 17d] systems/terminal-backend.md - [ 17d] systems/tool-search.md diff --git a/systems/skills-index.md b/systems/skills-index.md index 5f491b2..c2c8b1e 100644 --- a/systems/skills-index.md +++ b/systems/skills-index.md @@ -3,8 +3,8 @@ title: Installed Skills type: system status: active created: 2026-07-22 -updated: 2026-11-05 -verified_on: 2026-11-05 +updated: 2026-11-06 +verified_on: 2026-11-06 confidence: high tags: [skills, hermes] sources: []