audit(2026-11-10): 11-09 first failed-free cron day since 11-05; stocktake 'completed' with zero edits; Job Radar Stage 1 stall (9h09m, no handoff) documented; tailscale 58 nodes; omniroute 55 combos

This commit is contained in:
Hermes
2026-11-10 00:58:22 +08:00
parent 3abd9901d3
commit 262bcc74d5
9 changed files with 196 additions and 27 deletions
+21 -14
View File
@@ -3,8 +3,8 @@ title: Tailscale
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-11-07
verified_on: 2026-11-07
updated: 2026-11-10
verified_on: 2026-11-10
confidence: high
tags: [tailscale, vpn, networking, infrastructure]
---
@@ -19,10 +19,11 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
## Tailscale Node Inventory (verified 2026-11-07)
## Tailscale Node Inventory (verified 2026-11-10)
Re-verified 2026-11-10 00:4x AWST (nightly audit): `tailscale status --json` re-read — **6 new peers added, 8 offline ages drifted, 1 state transition back *online* (`octop`); no IP changes and no missing table rows.** ⚠ **Six peers that had never been listed now sit in the tailnet** — `codex-environment` (`100.111.126.78`), `codex-environment-1b88dc` (`100.76.42.76`), `codex-environment-8dadbb` (`100.65.200.27`), `codex-environment-955866` (`100.81.119.118`), `codex-environment-9fbd61` (`100.106.215.121`) and `codex-environment-d3edbd` (`100.74.3.2`) — every one created **2026-11-07 05:09–05:56Z**, every one **`Online: false`** with `LastSeen` **2026-11-07 11:09–11:56Z**. **None is a guest on either hypervisor** (live `pct list` and `qm list` on both `pve` and `m93p` list no such guest), so they are **orphaned/ephemeral tailnet peers**, not estate hosts: six records registered inside 47 minutes and all went silent within ~6 h, the signature of a throwaway CI/dev environment rather than infrastructure. **State transition:** `octop` (`100.92.163.122`) went **online → offline 1d** (`LastSeen 2026-11-08T13:13:04Z`) after returning for a single day on 11-07. The **9 offline ages** were refreshed (+1d each): cloudreve 105d → **106d**; conduit 61d → **62d**; elementsynapse 108d → **109d**; freshrss-new 81d → **82d**; octoagent 5d → **6d**; omarchybook 16d → **17d**; openmausbot 16d → **17d**; ubuntu-server-xfce 77d → **78d**; plus **`omarchymac`, which was not aged but re-stated** — the 2026-11-07 row read *"offline 5d — last seen 2026-11-01 11:13Z"* while the live record reads **`LastSeen 2026-11-08T00:45:17Z` = offline 2d**, so **both the age and the last-seen stamp on that row were stale** and are corrected here. Node count **52 → 58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none** once the six additions land; the **~23 IPv6-only `funnel-ingress-node` peers** remain deliberately excluded (`tailscale status --json`: **80 peers total — 57 with a `100.` IPv4, 23 IPv6-only**). Container counts re-confirmed live: `pve` (100.96.100.82) **17 LXCs + 4 VMs**, `m93p` (100.104.51.98) **20 LXCs + 2 VMs** — unchanged. **Not re-probed this pass** (so the 2026-11-07 findings stand un-contradicted rather than re-verified): the `:443` Serve route table (12 paths), the two DEAD CT460 targets (8769 `/agent-estate`, 9123 `/synergy-prep`), and per-guest tailnet identity via `pct exec <vmid> -- tailscale ip -4`.
Re-verified 2026-11-07 03:1x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 9 offline-age drifts, no IP changes and no missing peers.** **`octop` came back online**: the peer record reads `Online: true` with `LastSeen 0001-01-01` and `tailscale ping 100.92.163.122` → **`pong from octop via DERP(syd) in 98ms`**, so the row moves offline → online. It remains **not an LXC or VM on either hypervisor** (live `pct list` / `qm list` on both `pve` and `m93p` list no such guest) — an **orphaned tailnet peer**, so its return is not an estate change. The **9 offline ages** were refreshed (+1d each): cloudreve 104d → **105d**; conduit 60d → **61d**; elementsynapse 107d → **108d**; freshrss-new 80d → **81d**; octoagent 4d → **5d**; omarchybook 15d → **16d**; omarchymac 4d → **5d**; openmausbot 15d → **16d**; ubuntu-server-xfce 76d → **77d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) **17 guests** and `m93p` (100.104.51.98) **20 guests + 2 VMs** — ⚠ **the m93p guest count on this page was one short**: live `pct list` on `m93p` returns **20 LXCs** (the extra is **CT107 `mem0`**, `100.73.94.53`, `mem0-mcp.service` active — confirmed from inside the guest), not 19; CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT998 `nightbridge` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). The route table still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; **both DEAD targets re-confirmed dead** — `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`) — while every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` answers **HTTP 200**.
Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable.
Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
Full node list from `tailscale status` — **58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required).
@@ -45,14 +46,20 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) |
| clawtest | 100.122.189.81 | online — CT333 (pve) |
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
| cloudreve | 100.111.187.7 | offline 105d — CT209 stopped |
| conduit | 100.81.79.87 | offline 61d — last seen 2026-09-06 01:32Z |
| cloudreve | 100.111.187.7 | offline 108d — CT209 stopped |
| codex-environment | 100.111.126.78 | offline 2d — last seen 2026-11-07 11:09Z; **created 2026-11-07 05:5xZ**, six-record burst; **not an LXC/VM on either hypervisor** (orphaned/ephemeral tailnet peer; added 2026-11-10, last-seen stamps corrected 2026-11-10) |
| codex-environment-1b88dc | 100.76.42.76 | offline 2d — last seen 2026-11-07 11:12Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-8dadbb | 100.65.200.27 | offline 2d — last seen 2026-11-07 11:46Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-955866 | 100.81.119.118 | offline 2d — last seen 2026-11-07 11:49Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-9fbd61 | 100.106.215.121 | offline 2d — last seen 2026-11-07 11:51Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-d3edbd | 100.74.3.2 | offline 2d — last seen 2026-11-07 11:56Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| conduit | 100.81.79.87 | offline 65d — last seen 2026-09-06 01:32Z |
| docker | 100.96.150.69 | online — CT200 (pve) |
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
| elementsynapse | 100.75.250.37 | offline 108d — CT555 stopped |
| elementsynapse | 100.75.250.37 | offline 111d — CT555 stopped |
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
| freshrss-new | 100.110.180.115 | offline 81d |
| freshrss-new | 100.110.180.115 | offline 84d — last seen 2026-08-17 17:04Z |
| gitea | 100.82.100.128 | online — CT221 (m93p) |
| guest-vps | 100.117.14.84 | online |
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
@@ -66,15 +73,15 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
| obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) |
| octoagent | 100.117.10.68 | offline 5d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
| octop | 100.92.163.122 | online — **returned 2026-11-07** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`); still **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
| octoagent | 100.117.10.68 | offline 8d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
| octop | 100.92.163.122 | **online** (re-verified 2026-11-10 00:4x; it had been dark since `LastSeen 2026-11-08T13:13:04Z` — online 11-07, offline 11-08, online again 11-10; still **not an LXC/VM on either hypervisor** — orphaned tailnet peer) |
| ollama | 100.108.44.119 | online — CT205 (pve) |
| omarchy | 100.73.227.103 | online |
| omarchybook | 100.107.9.100 | offline 16d |
| omarchymac | 100.126.173.115 | offline 5d — last seen 2026-11-01 11:13Z |
| omarchybook | 100.107.9.100 | offline 19d — last seen 2026-10-21 15:55Z |
| omarchymac | 100.126.173.115 | offline 2d — last seen 2026-11-08 00:45Z (⚠ **was stale at 5d / 2026-11-01**; corrected 2026-11-10) |
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
| openmausbot | 100.91.214.91 | offline 16d — CT1264 (m93p) stopped |
| openmausbot | 100.91.214.91 | offline 19d — CT1264 (m93p) stopped; last seen 2026-10-21 15:50Z |
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
| openweb | 100.69.230.13 | online — CT245 (m93p) |
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
@@ -87,7 +94,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
| springroll | 100.100.153.125 | online |
| springroll-browser | 100.109.221.91 | online |
| ubuntu-server-xfce | 100.101.26.104 | offline 77d — last seen 2026-08-20 19:36Z |
| ubuntu-server-xfce | 100.101.26.104 | offline 81d — last seen 2026-08-20 19:36Z |
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
| webserver | 100.82.40.23 | online — CT207 (pve) |