Files
agent-estate-wiki/infrastructure/tailscale.md
T

182 lines
34 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Tailscale
type: infrastructure
status: active
created: 2026-07-26
updated: 2026-11-10
verified_on: 2026-11-10
confidence: high
tags: [tailscale, vpn, networking, infrastructure]
---
# Tailscale
## Purpose
WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access via Tailscale Serve.
## Domain
- **Tailnet:** `kangaroo-eel.ts.net`
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
## Tailscale Node Inventory (verified 2026-11-10)
Re-verified 2026-11-10 00:4x AWST (nightly audit): `tailscale status --json` re-read — **6 new peers added, 8 offline ages drifted, 1 state transition back *online* (`octop`); no IP changes and no missing table rows.** ⚠ **Six peers that had never been listed now sit in the tailnet** — `codex-environment` (`100.111.126.78`), `codex-environment-1b88dc` (`100.76.42.76`), `codex-environment-8dadbb` (`100.65.200.27`), `codex-environment-955866` (`100.81.119.118`), `codex-environment-9fbd61` (`100.106.215.121`) and `codex-environment-d3edbd` (`100.74.3.2`) — every one created **2026-11-07 05:09–05:56Z**, every one **`Online: false`** with `LastSeen` **2026-11-07 11:09–11:56Z**. **None is a guest on either hypervisor** (live `pct list` and `qm list` on both `pve` and `m93p` list no such guest), so they are **orphaned/ephemeral tailnet peers**, not estate hosts: six records registered inside 47 minutes and all went silent within ~6 h, the signature of a throwaway CI/dev environment rather than infrastructure. **State transition:** `octop` (`100.92.163.122`) went **online → offline 1d** (`LastSeen 2026-11-08T13:13:04Z`) after returning for a single day on 11-07. The **9 offline ages** were refreshed (+1d each): cloudreve 105d → **106d**; conduit 61d → **62d**; elementsynapse 108d → **109d**; freshrss-new 81d → **82d**; octoagent 5d → **6d**; omarchybook 16d → **17d**; openmausbot 16d → **17d**; ubuntu-server-xfce 77d → **78d**; plus **`omarchymac`, which was not aged but re-stated** — the 2026-11-07 row read *"offline 5d — last seen 2026-11-01 11:13Z"* while the live record reads **`LastSeen 2026-11-08T00:45:17Z` = offline 2d**, so **both the age and the last-seen stamp on that row were stale** and are corrected here. Node count **52 → 58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none** once the six additions land; the **~23 IPv6-only `funnel-ingress-node` peers** remain deliberately excluded (`tailscale status --json`: **80 peers total — 57 with a `100.` IPv4, 23 IPv6-only**). Container counts re-confirmed live: `pve` (100.96.100.82) **17 LXCs + 4 VMs**, `m93p` (100.104.51.98) **20 LXCs + 2 VMs** — unchanged. **Not re-probed this pass** (so the 2026-11-07 findings stand un-contradicted rather than re-verified): the `:443` Serve route table (12 paths), the two DEAD CT460 targets (8769 `/agent-estate`, 9123 `/synergy-prep`), and per-guest tailnet identity via `pct exec <vmid> -- tailscale ip -4`.
Re-verified 2026-11-07 03:1x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 9 offline-age drifts, no IP changes and no missing peers.** **`octop` came back online**: the peer record reads `Online: true` with `LastSeen 0001-01-01` and `tailscale ping 100.92.163.122` → **`pong from octop via DERP(syd) in 98ms`**, so the row moves offline → online. It remains **not an LXC or VM on either hypervisor** (live `pct list` / `qm list` on both `pve` and `m93p` list no such guest) — an **orphaned tailnet peer**, so its return is not an estate change. The **9 offline ages** were refreshed (+1d each): cloudreve 104d → **105d**; conduit 60d → **61d**; elementsynapse 107d → **108d**; freshrss-new 80d → **81d**; octoagent 4d → **5d**; omarchybook 15d → **16d**; omarchymac 4d → **5d**; openmausbot 15d → **16d**; ubuntu-server-xfce 76d → **77d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) **17 guests** and `m93p` (100.104.51.98) **20 guests + 2 VMs** — ⚠ **the m93p guest count on this page was one short**: live `pct list` on `m93p` returns **20 LXCs** (the extra is **CT107 `mem0`**, `100.73.94.53`, `mem0-mcp.service` active — confirmed from inside the guest), not 19; CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT998 `nightbridge` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). The route table still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; **both DEAD targets re-confirmed dead** — `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`) — while every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` answers **HTTP 200**.
Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable.
Full node list from `tailscale status` — **58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required).
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
Re-verified 2026-10-30 03:3x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 IP changes** (anthonys-mac-mini: 100.74.17.15 -> 100.76.60.37; omarchymac: 100.84.180.69 -> 100.126.173.115) and **4 nodes dropped out of the tailnet entirely** (anthonys-imac (100.112.123.124), cachyos (100.124.208.15), omarchymac-1 (100.126.173.115), omni (100.93.204.4)). `omarchymac` is now `100.126.173.115` (the peer formerly listed as `omarchymac-1`; the old `omarchymac` at `100.84.180.69` no longer matches any peer) and `anthonys-mac-mini` moved to `100.76.60.37` — both same-named nodes, new addresses. The four dropped peers are Tailscale expired/offline records (`anthonys-imac`, `cachyos`, `omni`, and the old `omarchymac` address), not newly-installed hardware. **`m93p` reads `Online: false` in the peer record for the third consecutive audit, yet the host is plainly up — `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 0s` and SSH returned `m93p / up 12 days, 11:02` — so the peer record is not authoritative for hypervisor reachability and the row records it as online. Node count corrected to **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **72 peers total — 49 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Live IPv4 peers absent from the table: none.** Container identity re-confirmed from inside guests via `pct exec <vmid> -- tailscale ip -4` on `pve` (100.96.100.82) and `m93p` (100.104.51.98); CT600 is confirmed **running** and CT601 confirmed **absent from both hypervisors**. Serve tables re-probed per host: CT460's `:443` route set matches the table below exactly (12 paths), and CT201 (`pct exec 201 -- tailscale serve status`) serves only `/` → `127.0.0.1:8085` and `/mgr` → `127.0.0.1:8084` on **both** `:443` and `:8443`, with no `/steel` or `/steel-api` routes remaining.
Re-verified 2026-10-29 03:0x AWST (stocktake): State column rebuilt from `tailscale status --json` — **11 rows changed**. Two were **state transitions**, not age drift: **`sftpgo` (CT299, pve) came back online** (the 2026-10-28 audit had recorded it as `offline <1d`; live `tailscale ping 100.88.223.70` → `pong from sftpgo via 192.168.178.92:41641 in 0s`, `pct exec 299 -- systemctl is-active sftpgo` → `active`, `pct exec 299 -- tailscale ip -4` → `100.88.223.70`), and **`m93p` is online despite a stale peer record** — `tailscale status --json` still reports `Online: false` with `LastSeen 2026-10-28T18:40:00Z`, but `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 1ms` and SSH over the tailnet returned `m93p / up 11 days`. This is the **second** occurrence of the m93p stale-peer-record pattern (first seen 2026-10-26) — the peer record is not authoritative for hypervisor reachability. The remaining **9 rows were offline-age drift of exactly +1 day** (anthonys-imac 34d → 35d; cachyos 54d → 55d; cloudreve 95d → 96d — CT209 stopped; elementsynapse 98d → 99d — CT555 stopped; freshrss-new 71d → 72d; mattermost 38d → 39d — CT255 stopped; omarchybook 6d → 7d; omarchymac 10d → 11d; omni 71d → 72d — CT601 removed from both hypervisors; openmausbot 6d → 7d). Node count unchanged at **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **76 peers total — 53 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Nodes in the table that are no longer in the tailnet: none. Live IPv4 peers absent from the table: none. IP mismatches: none.** Container identity re-confirmed from **inside** every LXC via `pct exec <vmid> -- tailscale ip -4` across **both** hypervisors (`pve` 100.96.100.82, `m93p` 100.104.51.98) — 34 guests enumerated; every peer IP mapped to its CT as listed, and the three stopped guests (`CT209` cloudreve, `CT255` mattermost, `CT555` elementsynapse) and the three running-but-unconnected guests (`CT105` conduit, `CT1264` openmausbot, `CT100` obsidian-livesync) confirmed. CT601 remains absent from both hypervisors.
Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from `tailscale status --json` — **14 rows changed** (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Nodes in the table that are no longer in the tailnet: **none**. Live IPv4 peers absent from the table: **none**. **State transitions inside the audit window (not just age drift): `m93p` online → offline <1d, `sftpgo` (CT299) online → offline <1d.**
Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale status --json` — **`springroll-browser` (100.109.221.91) was missing entirely** and has been added. **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline** (online → idle/offline 3d) and `openmausbot` is no longer a live peer despite its m93p LXC (CT1264) still running `tailscale` — the node itself has stopped connecting. Every offline age was refreshed (e.g. `mattermost` 31d → **35d**, `omni` 64d → **68d**, `elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `omarchymac` 3d → **7d**). `m93p` (100.104.51.98) now reads **idle/offline 13m ago** — the peer record is stale, but the host is plainly reachable (both `pct list` probes and the SSH session to CT201/CT460 during this audit succeeded), so treat it as a Tailscale-side liveness artefact, not a hypervisor outage. `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it **offers** exit-node capability; it is not currently serving as one (the old "active; offers exit node" wording overstated the live state). Container states cross-checked again against live `pct list` on both hypervisors, and each peer's container identity confirmed by running `tailscale ip -4` **inside** the candidate LXC on both hosts — the authoritative mapping, since peer names and LXC names do not always match.
| Node | IP | State |
|---|---|---|
| hermes | 100.118.5.51 | **self** — CT460 (pve) |
| agentzero | 100.106.54.94 | online — CT475 (pve) |
| anthonys-ipad | 100.111.198.120 | online |
| anthonys-iphone | 100.100.116.67 | online |
| anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) |
| clawtest | 100.122.189.81 | online — CT333 (pve) |
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
| cloudreve | 100.111.187.7 | offline 108d — CT209 stopped |
| codex-environment | 100.111.126.78 | offline 2d — last seen 2026-11-07 11:09Z; **created 2026-11-07 05:5xZ**, six-record burst; **not an LXC/VM on either hypervisor** (orphaned/ephemeral tailnet peer; added 2026-11-10, last-seen stamps corrected 2026-11-10) |
| codex-environment-1b88dc | 100.76.42.76 | offline 2d — last seen 2026-11-07 11:12Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-8dadbb | 100.65.200.27 | offline 2d — last seen 2026-11-07 11:46Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-955866 | 100.81.119.118 | offline 2d — last seen 2026-11-07 11:49Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-9fbd61 | 100.106.215.121 | offline 2d — last seen 2026-11-07 11:51Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-d3edbd | 100.74.3.2 | offline 2d — last seen 2026-11-07 11:56Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| conduit | 100.81.79.87 | offline 65d — last seen 2026-09-06 01:32Z |
| docker | 100.96.150.69 | online — CT200 (pve) |
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
| elementsynapse | 100.75.250.37 | offline 111d — CT555 stopped |
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
| freshrss-new | 100.110.180.115 | offline 84d — last seen 2026-08-17 17:04Z |
| gitea | 100.82.100.128 | online — CT221 (m93p) |
| guest-vps | 100.117.14.84 | online |
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
| homeassistant | 100.68.139.5 | online — VM100 (pve) |
| litellm | 100.79.29.64 | online — CT500 (m93p) |
| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: true` again on 2026-11-04 — the stale-record pattern of 10-29 → 11-01 has cleared; `tailscale ping` → pong) |
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
| mattermost | 100.98.116.14 | online — CT255 (pve) |
| mem0 | 100.73.94.53 | online — CT107 (m93p) |
| n8n | 100.84.124.30 | online |
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
| obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) |
| octoagent | 100.117.10.68 | offline 8d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
| octop | 100.92.163.122 | **online** (re-verified 2026-11-10 00:4x; it had been dark since `LastSeen 2026-11-08T13:13:04Z` — online 11-07, offline 11-08, online again 11-10; still **not an LXC/VM on either hypervisor** — orphaned tailnet peer) |
| ollama | 100.108.44.119 | online — CT205 (pve) |
| omarchy | 100.73.227.103 | online |
| omarchybook | 100.107.9.100 | offline 19d — last seen 2026-10-21 15:55Z |
| omarchymac | 100.126.173.115 | offline 2d — last seen 2026-11-08 00:45Z (⚠ **was stale at 5d / 2026-11-01**; corrected 2026-11-10) |
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
| openmausbot | 100.91.214.91 | offline 19d — CT1264 (m93p) stopped; last seen 2026-10-21 15:50Z |
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
| openweb | 100.69.230.13 | online — CT245 (m93p) |
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
| pve | 100.96.100.82 | online |
| qwenpaw | 100.110.72.126 | online — CT465 (pve) |
| rclone | 100.107.116.17 | online — CT104 (m93p) |
| runtipi | 100.96.244.39 | online — CT201 (pve) |
| rustdesk | 100.95.227.57 | online — CT1008 (m93p) |
| sftpgo | 100.88.223.70 | online — CT299 (pve) — **recovered 2026-11-04 (freeze 1) and again 2026-11-05 (freeze 2, manual `ssd0` run)**: a wedged `vzdump` had frozen the guest for ~24 h; killing the task unfroze it (see [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]) |
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
| springroll | 100.100.153.125 | online |
| springroll-browser | 100.109.221.91 | online |
| ubuntu-server-xfce | 100.101.26.104 | offline 81d — last seen 2026-08-20 19:36Z |
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
| webserver | 100.82.40.23 | online — CT207 (pve) |
> Note: `tailscale status` also lists 23 `funnel-ingress-node` peers with IPv6-only addresses — these are Tailscale's Funnel infrastructure, not homelab hosts, and are deliberately excluded above.
>
> Note: Tailscale's offline list is not authoritative for container state — cross-check `pct list` on the PVE host.
## CT460 Serve Routes
| Path | Target | Notes |
|---|---|---|
| `/` | `http://127.0.0.1:9119` | Hermes WebUI |
| `/ui` | `http://127.0.0.1:5175` | Hermes UI (Vite) |
| `/jobs` | `http://127.0.0.1:9099` | Jobs Dashboard |
| `/wiki` | `http://100.118.5.51:9120` | Agent Estate Wiki (Quartz) |
| `/ignis` | `http://127.0.0.1:8080` | Ignis (browser-native Obsidian) |
| `/webui` | `http://127.0.0.1:8787` | Hermes WebUI (Standalone) |
| `/gitea` | `http://100.82.100.128:3000` | Gitea |
| `/desktop` | `http://127.0.0.1:6081` | CUA noVNC |
| `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC |
| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07**; the route still exists in `tailscale serve status` but serves 502) |
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06, **re-confirmed 2026-11-07**; route still present in `tailscale serve status` but serves 502) |
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift).
## CT460 Funnel and 9443 Routes
| Port | Target | Notes |
|---|---|---|
| `:8443/webhook` | `http://127.0.0.1:8085/webhook` | Public webhook Funnel (`# Funnel on:` confirmed) |
| `:9443` (tailnet only) `/` | `http://127.0.0.1:5173` | Vite dev server on port 5173 exposed via a **tailnet-only** Serve on port 9443 — was missing from this page, added 2026-10-11 |
Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from `tailscale status --json` — **`octoagent` (100.117.10.68) and `omarchymac-1` (100.126.173.115) were both missing entirely** and have been added; `m93p` (100.104.51.98) now reads **online** (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**, `ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**, `openmausbot` 3d → **4d**). The `omni` row now reads **"CT601 removed from both hypervisors"** rather than "CT601 stopped" — live `pct list` on both `pve` and `m93p` shows **no CT601 at all**, so the old guest is gone, not merely halted. Container identity re-confirmed from **inside** each LXC via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; every peer whose LXC has no Tailscale IP was checked against `pct list` (`CT105`/`CT1264` run but do not connect — `conduit` and `openmausbot` offline; `CT209`/`CT255`/`CT555` are `stopped`). Serve tables re-probed per host with `pct exec 201 -- tailscale serve status`.
## CT201 Serve Routes
Re-verified live on 2026-10-30 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
| Host / Port | Path | Target | Notes |
|---|---|---|---|
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-30; upstream 8082 has no listener — `curl http://127.0.0.1:8082/` → exit code 7 / `http=000`) |
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** (re-probed 2026-10-30) |
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-30) |
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** (re-probed 2026-10-30) |
**Steel routes removed 2026-10-29.** This table previously carried two more rows on `:443` — `/steel` →
`http://127.0.0.1:5173` and `/steel-api` → `http://127.0.0.1:3000`. Both were deleted with
`tailscale serve --set-path <path> off` when Steel was removed from CT201, so CT201's only remaining routes
are `/` → `127.0.0.1:8085` and `/mgr` → `http://127.0.0.1:8084`, on **both** `:443` and `:8443`.
→ [[comparisons/steel-vs-camoufox]]
Two corrections to the pre-2026-10-25 table:
1. **`:8443` is served by `tailscaled` on CT201, but so is `:443`.** `ss -ltnp` on CT201 shows `tailscaled` bound to `100.96.244.39:443`, and the serve table above lists routes on **both** ports. The old note claiming "Port 443 occupied by Traefik on CT201" is **false as of 2026-10-25** — no Traefik **process** runs on CT201 (`ps aux` returns nothing for `traefik`), and Tailscale owns 443 there. *(Evidence corrected 2026-10-30: the original wording said `docker ps -a` "came back empty for `traefik`" — it does not. Live `docker ps -a` lists `runtipi-reverse-proxy | traefik:v3.2 | Exited (255) 2 months ago`. The container is **stopped**, not absent; the conclusion that Traefik does not occupy 443 stands.)*
2. The old table routed `:8443` straight to `http://localhost:8082`. That port has **no listener**; the live target is `8085` (the `asset_proxy.py` front door), which itself proxies to the dead 8082 and returns 502. → [[systems/homepage-dashboard]]
## Atomic Restore Script
`/home/hermes/.hermes/scripts/tailscale-serve-apply.sh` — Rebuilds the complete CT460 route table, including tailnet Serve on `:443` and public Funnel on `:8443`.
`tailscale serve reset` clears Funnel state as well as Serve state. The atomic script therefore restores `/webhook` after rebuilding the 443 routes. Do not use the older Serve-only restore path for complete recovery.
## Key Commands
```bash
# Check routes
tailscale serve status
# Rebuild routes (CT460)
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh
# Add route
tailscale serve --set-path /jobs http://127.0.0.1:9099
# Remove route
tailscale serve --set-path /jobs off
```
## Notes
- **CT201 now serves Tailscale on BOTH `:443` and `:8443`** (verified 2026-10-30). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was **wrong**: no Traefik **process** runs on CT201, and `tailscaled` owns 443 there (`ss -ltnp` shows it bound to `100.96.244.39:443`). The CT201 Serve Routes table above now records both ports. *(Evidence corrected 2026-10-30: the original wording said "no Traefik container or process exists on CT201" — the container does exist, `runtipi-reverse-proxy | traefik:v3.2`, but it is **stopped** — `Exited (255) 2 months ago`. The conclusion that Tailscale owns 443 is unchanged.)*
- CT460 has no conflict — Tailscale uses 443 directly, with the public webhook Funnel on `:8443`.
- Routes must be re-added in order (restore script handles this).
- After every rebuild, verify that `tailscale serve status` contains both the `:443` Serve routes and the `:8443/webhook` Funnel route.
## Related
- [[infrastructure/domains-and-tunnels]] — Full DNS and tunnel inventory
- [[systems/homepage-dashboard]] — Homepage on CT201 via Tailscale
- [[runbooks/job-radar]] — Dashboard incident and route verification