Ran wiki-maintain.sh (repo script). skills-index regenerated -> 302 skills / 68 categories. Freshness audit at the 21-day threshold -> 0 stale / 124 fresh / 3 no-date of 127 pages. Quartz rebuilt (761 files) and deployed to public/; hermes-wiki-serve.service restarted. Live-vs-page drift diff (age scan alone missed all of this): - infrastructure/tailscale.md: 1 state transition + 8 offline-age drifts + 2 dead Serve routes. anthonys-mac-mini came back online (peer record Online:true, tailscale ping -> pong via DERP(syd)). Offline ages refreshed +1d each (cloudreve 104d, elementsynapse 107d, freshrss-new 80d, octoagent 4d, octop 2d, omarchybook 15d, omarchymac 4d, openmausbot 15d). Every peer IP still matches the live JSON; no peer missing, no orphan row. /agent-estate->8769 and /synergy-prep->9123 are DEAD (no listener) while still present in `tailscale serve status` - both now marked DEAD, matching network-map.md. All other Serve targets have live listeners. - infrastructure/docker-services.md: the 2026-11-05 "the seeder does not exist on any host" finding was FALSE - a point-in-time sampling artefact. The worldmonitor-seeder-* containers are transient and systemd-timer driven on CT200 (worldmonitor-seed.timer 4h -> seeder-scheduled; worldmonitor-seed-fast.timer 15min -> seeder-fast), visible only while a run is in flight. Both present at 2026-11-06 03:1x; neither at 2026-11-05 05:1x. The "no cron job" claim was also wrong (systemd timers, not cron) and the "rename" narrative was wrong (one script, two container names). Corrected on the page with strikethrough + dated correction. CT200's 61-container running set still matches live set-wise. CT333 container renamed again: flamboyant_mahavira -> nostalgic_keldysh. - systems/scheduled-tasks.md: no drift. Live jobs.json reads 29 jobs / 26 enabled / 3 disabled, 1 non-clean last_status - exactly as recorded. - current-state.md / index.md: false seeder claim and "every state matching" claim marked superseded with dated corrections (SCHEMA rule 9), not silently rewritten. - infrastructure/network-map.md: dates bumped, dead routes re-confirmed, peer recheck note refreshed. - log.md: appended (append-only) with the full stocktake record. No secrets, tokens or keys written or quoted.
174 lines
28 KiB
Markdown
174 lines
28 KiB
Markdown
---
|
||
title: Tailscale
|
||
type: infrastructure
|
||
status: active
|
||
created: 2026-07-26
|
||
updated: 2026-11-06
|
||
verified_on: 2026-11-06
|
||
confidence: high
|
||
tags: [tailscale, vpn, networking, infrastructure]
|
||
---
|
||
|
||
# Tailscale
|
||
|
||
## Purpose
|
||
WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access via Tailscale Serve.
|
||
|
||
## Domain
|
||
- **Tailnet:** `kangaroo-eel.ts.net`
|
||
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
|
||
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
||
|
||
## Tailscale Node Inventory (verified 2026-11-06)
|
||
Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable.
|
||
Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||
Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
|
||
Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
|
||
Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required).
|
||
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
|
||
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
|
||
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
|
||
Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
|
||
Re-verified 2026-10-30 03:3x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 IP changes** (anthonys-mac-mini: 100.74.17.15 -> 100.76.60.37; omarchymac: 100.84.180.69 -> 100.126.173.115) and **4 nodes dropped out of the tailnet entirely** (anthonys-imac (100.112.123.124), cachyos (100.124.208.15), omarchymac-1 (100.126.173.115), omni (100.93.204.4)). `omarchymac` is now `100.126.173.115` (the peer formerly listed as `omarchymac-1`; the old `omarchymac` at `100.84.180.69` no longer matches any peer) and `anthonys-mac-mini` moved to `100.76.60.37` — both same-named nodes, new addresses. The four dropped peers are Tailscale expired/offline records (`anthonys-imac`, `cachyos`, `omni`, and the old `omarchymac` address), not newly-installed hardware. **`m93p` reads `Online: false` in the peer record for the third consecutive audit, yet the host is plainly up — `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 0s` and SSH returned `m93p / up 12 days, 11:02` — so the peer record is not authoritative for hypervisor reachability and the row records it as online. Node count corrected to **50 non-funnel nodes** (49 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **72 peers total — 49 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Live IPv4 peers absent from the table: none.** Container identity re-confirmed from inside guests via `pct exec <vmid> -- tailscale ip -4` on `pve` (100.96.100.82) and `m93p` (100.104.51.98); CT600 is confirmed **running** and CT601 confirmed **absent from both hypervisors**. Serve tables re-probed per host: CT460's `:443` route set matches the table below exactly (12 paths), and CT201 (`pct exec 201 -- tailscale serve status`) serves only `/` → `127.0.0.1:8085` and `/mgr` → `127.0.0.1:8084` on **both** `:443` and `:8443`, with no `/steel` or `/steel-api` routes remaining.
|
||
Re-verified 2026-10-29 03:0x AWST (stocktake): State column rebuilt from `tailscale status --json` — **11 rows changed**. Two were **state transitions**, not age drift: **`sftpgo` (CT299, pve) came back online** (the 2026-10-28 audit had recorded it as `offline <1d`; live `tailscale ping 100.88.223.70` → `pong from sftpgo via 192.168.178.92:41641 in 0s`, `pct exec 299 -- systemctl is-active sftpgo` → `active`, `pct exec 299 -- tailscale ip -4` → `100.88.223.70`), and **`m93p` is online despite a stale peer record** — `tailscale status --json` still reports `Online: false` with `LastSeen 2026-10-28T18:40:00Z`, but `tailscale ping 100.104.51.98` → `pong from m93p via 192.168.178.118:41641 in 1ms` and SSH over the tailnet returned `m93p / up 11 days`. This is the **second** occurrence of the m93p stale-peer-record pattern (first seen 2026-10-26) — the peer record is not authoritative for hypervisor reachability. The remaining **9 rows were offline-age drift of exactly +1 day** (anthonys-imac 34d → 35d; cachyos 54d → 55d; cloudreve 95d → 96d — CT209 stopped; elementsynapse 98d → 99d — CT555 stopped; freshrss-new 71d → 72d; mattermost 38d → 39d — CT255 stopped; omarchybook 6d → 7d; omarchymac 10d → 11d; omni 71d → 72d — CT601 removed from both hypervisors; openmausbot 6d → 7d). Node count unchanged at **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself); `tailscale status --json` reports **76 peers total — 53 with a `100.` IPv4, 23 IPv6-only** (the `funnel-ingress-node` set). **Nodes in the table that are no longer in the tailnet: none. Live IPv4 peers absent from the table: none. IP mismatches: none.** Container identity re-confirmed from **inside** every LXC via `pct exec <vmid> -- tailscale ip -4` across **both** hypervisors (`pve` 100.96.100.82, `m93p` 100.104.51.98) — 34 guests enumerated; every peer IP mapped to its CT as listed, and the three stopped guests (`CT209` cloudreve, `CT255` mattermost, `CT555` elementsynapse) and the three running-but-unconnected guests (`CT105` conduit, `CT1264` openmausbot, `CT100` obsidian-livesync) confirmed. CT601 remains absent from both hypervisors.
|
||
|
||
Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from `tailscale status --json` — **14 rows changed** (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Nodes in the table that are no longer in the tailnet: **none**. Live IPv4 peers absent from the table: **none**. **State transitions inside the audit window (not just age drift): `m93p` online → offline <1d, `sftpgo` (CT299) online → offline <1d.**
|
||
Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale status --json` — **`springroll-browser` (100.109.221.91) was missing entirely** and has been added. **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline** (online → idle/offline 3d) and `openmausbot` is no longer a live peer despite its m93p LXC (CT1264) still running `tailscale` — the node itself has stopped connecting. Every offline age was refreshed (e.g. `mattermost` 31d → **35d**, `omni` 64d → **68d**, `elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `omarchymac` 3d → **7d**). `m93p` (100.104.51.98) now reads **idle/offline 13m ago** — the peer record is stale, but the host is plainly reachable (both `pct list` probes and the SSH session to CT201/CT460 during this audit succeeded), so treat it as a Tailscale-side liveness artefact, not a hypervisor outage. `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it **offers** exit-node capability; it is not currently serving as one (the old "active; offers exit node" wording overstated the live state). Container states cross-checked again against live `pct list` on both hypervisors, and each peer's container identity confirmed by running `tailscale ip -4` **inside** the candidate LXC on both hosts — the authoritative mapping, since peer names and LXC names do not always match.
|
||
|
||
| Node | IP | State |
|
||
|---|---|---|
|
||
| hermes | 100.118.5.51 | **self** — CT460 (pve) |
|
||
| agentzero | 100.106.54.94 | online — CT475 (pve) |
|
||
| anthonys-ipad | 100.111.198.120 | online |
|
||
| anthonys-iphone | 100.100.116.67 | online |
|
||
| anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) |
|
||
| clawtest | 100.122.189.81 | online — CT333 (pve) |
|
||
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
|
||
| cloudreve | 100.111.187.7 | offline 104d — CT209 stopped |
|
||
| conduit | 100.81.79.87 | offline 60d — last seen 2026-09-06 01:32Z |
|
||
| docker | 100.96.150.69 | online — CT200 (pve) |
|
||
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
|
||
| elementsynapse | 100.75.250.37 | offline 107d — CT555 stopped |
|
||
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
|
||
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
|
||
| freshrss-new | 100.110.180.115 | offline 80d |
|
||
| gitea | 100.82.100.128 | online — CT221 (m93p) |
|
||
| guest-vps | 100.117.14.84 | online |
|
||
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
|
||
| homeassistant | 100.68.139.5 | online — VM100 (pve) |
|
||
| litellm | 100.79.29.64 | online — CT500 (m93p) |
|
||
| m93p | 100.104.51.98 | online — hypervisor (peer record reads `Online: true` again on 2026-11-04 — the stale-record pattern of 10-29 → 11-01 has cleared; `tailscale ping` → pong) |
|
||
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
|
||
| mattermost | 100.98.116.14 | online — CT255 (pve) |
|
||
| mem0 | 100.73.94.53 | online — CT107 (m93p) |
|
||
| n8n | 100.84.124.30 | online |
|
||
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
|
||
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
|
||
| obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) |
|
||
| octoagent | 100.117.10.68 | offline 4d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
|
||
| octop | 100.92.163.122 | offline 2d — last seen 2026-11-03 09:03Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
|
||
| ollama | 100.108.44.119 | online — CT205 (pve) |
|
||
| omarchy | 100.73.227.103 | online |
|
||
| omarchybook | 100.107.9.100 | offline 15d |
|
||
| omarchymac | 100.126.173.115 | offline 4d — last seen 2026-11-01 11:13Z |
|
||
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
|
||
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
|
||
| openmausbot | 100.91.214.91 | offline 15d — CT1264 (m93p) stopped |
|
||
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
|
||
| openweb | 100.69.230.13 | online — CT245 (m93p) |
|
||
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
|
||
| pve | 100.96.100.82 | online |
|
||
| qwenpaw | 100.110.72.126 | online — CT465 (pve) |
|
||
| rclone | 100.107.116.17 | online — CT104 (m93p) |
|
||
| runtipi | 100.96.244.39 | online — CT201 (pve) |
|
||
| rustdesk | 100.95.227.57 | online — CT1008 (m93p) |
|
||
| sftpgo | 100.88.223.70 | online — CT299 (pve) — **recovered 2026-11-04 (freeze 1) and again 2026-11-05 (freeze 2, manual `ssd0` run)**: a wedged `vzdump` had frozen the guest for ~24 h; killing the task unfroze it (see [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]) |
|
||
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
|
||
| springroll | 100.100.153.125 | online |
|
||
| springroll-browser | 100.109.221.91 | online |
|
||
| ubuntu-server-xfce | 100.101.26.104 | offline 76d — last seen 2026-08-20 19:36Z |
|
||
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
|
||
| webserver | 100.82.40.23 | online — CT207 (pve) |
|
||
|
||
> Note: `tailscale status` also lists 23 `funnel-ingress-node` peers with IPv6-only addresses — these are Tailscale's Funnel infrastructure, not homelab hosts, and are deliberately excluded above.
|
||
>
|
||
> Note: Tailscale's offline list is not authoritative for container state — cross-check `pct list` on the PVE host.
|
||
|
||
## CT460 Serve Routes
|
||
| Path | Target | Notes |
|
||
|---|---|---|
|
||
| `/` | `http://127.0.0.1:9119` | Hermes WebUI |
|
||
| `/ui` | `http://127.0.0.1:5175` | Hermes UI (Vite) |
|
||
| `/jobs` | `http://127.0.0.1:9099` | Jobs Dashboard |
|
||
| `/wiki` | `http://100.118.5.51:9120` | Agent Estate Wiki (Quartz) |
|
||
| `/ignis` | `http://127.0.0.1:8080` | Ignis (browser-native Obsidian) |
|
||
| `/webui` | `http://127.0.0.1:8787` | Hermes WebUI (Standalone) |
|
||
| `/gitea` | `http://100.82.100.128:3000` | Gitea |
|
||
| `/desktop` | `http://127.0.0.1:6081` | CUA noVNC |
|
||
| `/vnc-camofox` | `http://127.0.0.1:6080` | Camofox VNC |
|
||
| `/synergy-prep` | `http://127.0.0.1:9123` | Synergy prep — ⚠ **DEAD: nothing listening on 9123** (live `ss -ltn` probe 2026-11-06; the route still exists in `tailscale serve status` but serves 502) |
|
||
| `/agent-estate` | `http://127.0.0.1:8769` | Role dashboard — target is the bare host root, **not** `/agent-estate-role-dashboard.html` (corrected 2026-10-11) — ⚠ **DEAD: nothing listening on 8769** (live `ss -ltn` probe 2026-11-06; route still present in `tailscale serve status` but serves 502) |
|
||
| `/evening-grid.html` | `http://127.0.0.1:9099/evening-grid.html` | Evening grid |
|
||
|
||
Re-verified against `tailscale serve status` on **2026-10-26** — the CT460 route table above matched live exactly (**12 paths on :443**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; no drift).
|
||
|
||
## CT460 Funnel and 9443 Routes
|
||
| Port | Target | Notes |
|
||
|---|---|---|
|
||
| `:8443/webhook` | `http://127.0.0.1:8085/webhook` | Public webhook Funnel (`# Funnel on:` confirmed) |
|
||
| `:9443` (tailnet only) `/` | `http://127.0.0.1:5173` | Vite dev server on port 5173 exposed via a **tailnet-only** Serve on port 9443 — was missing from this page, added 2026-10-11 |
|
||
|
||
Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from `tailscale status --json` — **`octoagent` (100.117.10.68) and `omarchymac-1` (100.126.173.115) were both missing entirely** and have been added; `m93p` (100.104.51.98) now reads **online** (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (`mattermost` 35d → **36d**, `omni` 68d → **69d**, `elementsynapse` 95d → **96d**, `ubuntu-server-xfce` 64d → **66d**, `omarchymac` 7d → **8d**, `omarchybook` 3d → **4d**, `openmausbot` 3d → **4d**). The `omni` row now reads **"CT601 removed from both hypervisors"** rather than "CT601 stopped" — live `pct list` on both `pve` and `m93p` shows **no CT601 at all**, so the old guest is gone, not merely halted. Container identity re-confirmed from **inside** each LXC via `pct exec <vmid> -- tailscale ip -4` on both hypervisors; every peer whose LXC has no Tailscale IP was checked against `pct list` (`CT105`/`CT1264` run but do not connect — `conduit` and `openmausbot` offline; `CT209`/`CT255`/`CT555` are `stopped`). Serve tables re-probed per host with `pct exec 201 -- tailscale serve status`.
|
||
|
||
## CT201 Serve Routes
|
||
Re-verified live on 2026-10-30 via `pct exec 201 -- tailscale serve status` on `pve` (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
|
||
|
||
| Host / Port | Path | Target | Notes |
|
||
|---|---|---|---|
|
||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Homepage asset proxy front door — **502** (re-probed 2026-10-30; upstream 8082 has no listener — `curl http://127.0.0.1:8082/` → exit code 7 / `http=000`) |
|
||
| `runtipi.kangaroo-eel.ts.net:8443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` container — **200** (re-probed 2026-10-30) |
|
||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/` | `http://127.0.0.1:8085` | Same asset proxy — **502** (re-probed 2026-10-30) |
|
||
| `runtipi.kangaroo-eel.ts.net:443` (tailnet only) | `/mgr` | `http://127.0.0.1:8084` | `homepage-manager` — **200** (re-probed 2026-10-30) |
|
||
|
||
**Steel routes removed 2026-10-29.** This table previously carried two more rows on `:443` — `/steel` →
|
||
`http://127.0.0.1:5173` and `/steel-api` → `http://127.0.0.1:3000`. Both were deleted with
|
||
`tailscale serve --set-path <path> off` when Steel was removed from CT201, so CT201's only remaining routes
|
||
are `/` → `127.0.0.1:8085` and `/mgr` → `http://127.0.0.1:8084`, on **both** `:443` and `:8443`.
|
||
→ [[comparisons/steel-vs-camoufox]]
|
||
|
||
Two corrections to the pre-2026-10-25 table:
|
||
1. **`:8443` is served by `tailscaled` on CT201, but so is `:443`.** `ss -ltnp` on CT201 shows `tailscaled` bound to `100.96.244.39:443`, and the serve table above lists routes on **both** ports. The old note claiming "Port 443 occupied by Traefik on CT201" is **false as of 2026-10-25** — no Traefik **process** runs on CT201 (`ps aux` returns nothing for `traefik`), and Tailscale owns 443 there. *(Evidence corrected 2026-10-30: the original wording said `docker ps -a` "came back empty for `traefik`" — it does not. Live `docker ps -a` lists `runtipi-reverse-proxy | traefik:v3.2 | Exited (255) 2 months ago`. The container is **stopped**, not absent; the conclusion that Traefik does not occupy 443 stands.)*
|
||
2. The old table routed `:8443` straight to `http://localhost:8082`. That port has **no listener**; the live target is `8085` (the `asset_proxy.py` front door), which itself proxies to the dead 8082 and returns 502. → [[systems/homepage-dashboard]]
|
||
|
||
## Atomic Restore Script
|
||
`/home/hermes/.hermes/scripts/tailscale-serve-apply.sh` — Rebuilds the complete CT460 route table, including tailnet Serve on `:443` and public Funnel on `:8443`.
|
||
|
||
`tailscale serve reset` clears Funnel state as well as Serve state. The atomic script therefore restores `/webhook` after rebuilding the 443 routes. Do not use the older Serve-only restore path for complete recovery.
|
||
|
||
## Key Commands
|
||
```bash
|
||
# Check routes
|
||
tailscale serve status
|
||
|
||
# Rebuild routes (CT460)
|
||
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run
|
||
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh
|
||
|
||
# Add route
|
||
tailscale serve --set-path /jobs http://127.0.0.1:9099
|
||
|
||
# Remove route
|
||
tailscale serve --set-path /jobs off
|
||
```
|
||
|
||
## Notes
|
||
- **CT201 now serves Tailscale on BOTH `:443` and `:8443`** (verified 2026-10-30). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was **wrong**: no Traefik **process** runs on CT201, and `tailscaled` owns 443 there (`ss -ltnp` shows it bound to `100.96.244.39:443`). The CT201 Serve Routes table above now records both ports. *(Evidence corrected 2026-10-30: the original wording said "no Traefik container or process exists on CT201" — the container does exist, `runtipi-reverse-proxy | traefik:v3.2`, but it is **stopped** — `Exited (255) 2 months ago`. The conclusion that Tailscale owns 443 is unchanged.)*
|
||
- CT460 has no conflict — Tailscale uses 443 directly, with the public webhook Funnel on `:8443`.
|
||
- Routes must be re-added in order (restore script handles this).
|
||
- After every rebuild, verify that `tailscale serve status` contains both the `:443` Serve routes and the `:8443/webhook` Funnel route.
|
||
|
||
## Related
|
||
- [[infrastructure/domains-and-tunnels]] — Full DNS and tunnel inventory
|
||
- [[systems/homepage-dashboard]] — Homepage on CT201 via Tailscale
|
||
- [[runbooks/job-radar]] — Dashboard incident and route verification
|