Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and cover letters never ran (Notion DB held 451 entries, 0 added). Incident row dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified). Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5, deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5, topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3). Also: the two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of 108) — committed here and index.md corrected; tailscale.md peer table rebuilt from tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained the voice pipeline section. New: incidents/2026-10-27-job-radar-budget-exhausted.md Nothing removed or archived. No secrets.
14 KiB
title, type, status, created, updated, verified_on, confidence, tags
| title | type | status | created | updated | verified_on | confidence | tags | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Tailscale | infrastructure | active | 2026-07-26 | 2026-10-28 | 2026-10-28 | high |
|
Tailscale
Purpose
WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access via Tailscale Serve.
Domain
- Tailnet:
kangaroo-eel.ts.net - Primary node: CT460 (
hermes.kangaroo-eel.ts.net,100.118.5.51) - Runtipi node: CT201 (
runtipi.kangaroo-eel.ts.net)
Tailscale Node Inventory (verified 2026-10-25)
Full node list from tailscale status — 54 non-funnel nodes (53 live IPv4 peers + hermes itself). Offline peers are listed with their last-seen age.
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
Re-verified 2026-10-16: omarchymac (100.84.180.69) added; mattermost state, offline ages and the omni/omarchymac rows refreshed from tailscale status.
Re-verified 2026-10-19 00:2x: three live peers had never been listed — guest-vps (100.117.14.84, online), openmausbot (100.91.214.91, online) and openmausbot-vm (100.77.107.30, online) — added; mattermost offline age 26d → 29d.
Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — octop (100.92.163.122), springroll (100.100.153.125), openmausbot and openmausbot-vm were missing from the table body (only mentioned in prose), omarchymac had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from tailscale status --json, keyed on DNS name rather than HostName (the JSON HostName for freshrss is ct260, and both omniroute nodes share the hostname omniroute). Container states cross-checked against live pct list on pve (100.96.100.82) and m93p (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from tailscale status --json — 14 rows changed (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to 54 non-funnel nodes (53 live IPv4 peers + hermes itself). Nodes in the table that are no longer in the tailnet: none. Live IPv4 peers absent from the table: none. State transitions inside the audit window (not just age drift): m93p online → offline <1d, sftpgo (CT299) online → offline <1d.
Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from tailscale status --json — springroll-browser (100.109.221.91) was missing entirely and has been added. omarchybook (100.107.9.100) and openmausbot (100.91.214.91) have both gone offline (online → idle/offline 3d) and openmausbot is no longer a live peer despite its m93p LXC (CT1264) still running tailscale — the node itself has stopped connecting. Every offline age was refreshed (e.g. mattermost 31d → 35d, omni 64d → 68d, elementsynapse 91d → 95d, ubuntu-server-xfce 60d → 64d, omarchymac 3d → 7d). m93p (100.104.51.98) now reads idle/offline 13m ago — the peer record is stale, but the host is plainly reachable (both pct list probes and the SSH session to CT201/CT460 during this audit succeeded), so treat it as a Tailscale-side liveness artefact, not a hypervisor outage. homeassistant still advertises ExitNodeOption: true but ExitNode: false — it offers exit-node capability; it is not currently serving as one (the old "active; offers exit node" wording overstated the live state). Container states cross-checked again against live pct list on both hypervisors, and each peer's container identity confirmed by running tailscale ip -4 inside the candidate LXC on both hosts — the authoritative mapping, since peer names and LXC names do not always match.
| Node | IP | State |
|---|---|---|
| hermes | 100.118.5.51 | self — CT460 (pve) |
| agentzero | 100.106.54.94 | online — CT475 (pve) |
| anthonys-imac | 100.112.123.124 | offline 34d |
| anthonys-ipad | 100.111.198.120 | online |
| anthonys-iphone | 100.100.116.67 | online |
| anthonys-mac-mini | 100.74.17.15 | online |
| cachyos | 100.124.208.15 | offline 54d |
| clawtest | 100.122.189.81 | online — CT333 (pve) |
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
| cloudreve | 100.111.187.7 | offline 95d — CT209 stopped |
| conduit | 100.81.79.87 | offline 52d |
| docker | 100.96.150.69 | online — CT200 (pve) |
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
| elementsynapse | 100.75.250.37 | offline 98d — CT555 stopped |
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
| freshrss-new | 100.110.180.115 | offline 71d |
| gitea | 100.82.100.128 | online — CT221 (m93p) |
| guest-vps | 100.117.14.84 | online |
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
| homeassistant | 100.68.139.5 | online |
| litellm | 100.79.29.64 | online — CT500 (m93p) |
| m93p | 100.104.51.98 | offline <1d |
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
| mattermost | 100.98.116.14 | offline 38d — CT255 stopped |
| n8n | 100.84.124.30 | online |
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
| octoagent | 100.117.10.68 | online |
| octop | 100.92.163.122 | online |
| ollama | 100.108.44.119 | online — CT205 (pve) |
| omarchy | 100.73.227.103 | online |
| omarchybook | 100.107.9.100 | offline 6d |
| omarchymac | 100.84.180.69 | offline 10d |
| omarchymac-1 | 100.126.173.115 | online |
| omni | 100.93.204.4 | offline 71d — CT601 removed from both hypervisors |
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
| openmausbot | 100.91.214.91 | offline 6d |
| openmausbot-vm | 100.77.107.30 | online |
| openweb | 100.69.230.13 | online — CT245 (m93p) |
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
| pve | 100.96.100.82 | online |
| qwenpaw | 100.110.72.126 | online — CT465 (pve) |
| rclone | 100.107.116.17 | online — CT104 (m93p) |
| runtipi | 100.96.244.39 | online — CT201 (pve) |
| rustdesk | 100.95.227.57 | online — CT1008 (m93p) |
| sftpgo | 100.88.223.70 | offline <1d — CT299 (pve) |
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
| springroll | 100.100.153.125 | online |
| springroll-browser | 100.109.221.91 | online |
| ubuntu-server-xfce | 100.101.26.104 | offline 68d |
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
| webserver | 100.82.40.23 | online — CT207 (pve) |
Note:
tailscale statusalso lists 23funnel-ingress-nodepeers with IPv6-only addresses — these are Tailscale's Funnel infrastructure, not homelab hosts, and are deliberately excluded above.Note: Tailscale's offline list is not authoritative for container state — cross-check
pct liston the PVE host.
CT460 Serve Routes
| Path | Target | Notes |
|---|---|---|
/ |
http://127.0.0.1:9119 |
Hermes WebUI |
/ui |
http://127.0.0.1:5175/ |
Hermes UI (Vite) |
/jobs |
http://127.0.0.1:9099 |
Jobs Dashboard |
/wiki |
http://100.118.5.51:9120 |
Agent Estate Wiki (Quartz) |
/ignis |
http://127.0.0.1:8080 |
Ignis (browser-native Obsidian) |
/webui |
http://127.0.0.1:8787/ |
Hermes WebUI (Standalone) |
/gitea |
http://100.82.100.128:3000 |
Gitea |
/desktop |
http://127.0.0.1:6081 |
CUA noVNC |
/vnc-camofox |
http://127.0.0.1:6080 |
Camofox VNC |
/synergy-prep |
http://127.0.0.1:9123/ |
Synergy prep |
/agent-estate |
http://127.0.0.1:8769 |
Role dashboard — target is the bare host root, not /agent-estate-role-dashboard.html (corrected 2026-10-11) |
/evening-grid.html |
http://127.0.0.1:9099/evening-grid.html |
Evening grid |
Re-verified against tailscale serve status on 2026-10-26 — the CT460 route table above matched live exactly (12 paths on :443, plus the :8443/webhook Funnel and the :9443 tailnet route; no drift).
CT460 Funnel and 9443 Routes
| Port | Target | Notes |
|---|---|---|
:8443/webhook |
http://127.0.0.1:8085/webhook |
Public webhook Funnel (# Funnel on: confirmed) |
:9443 (tailnet only) / |
http://127.0.0.1:5173 |
Vite dev server on port 5173 exposed via a tailnet-only Serve on port 9443 — was missing from this page, added 2026-10-11 |
Re-verified 2026-10-26 (stocktake): the node table was rebuilt again from tailscale status --json — octoagent (100.117.10.68) and omarchymac-1 (100.126.173.115) were both missing entirely and have been added; m93p (100.104.51.98) now reads online (it was "idle/offline 13m ago" — a stale Tailscale peer record, and the host was plainly reachable throughout); every offline age was refreshed (mattermost 35d → 36d, omni 68d → 69d, elementsynapse 95d → 96d, ubuntu-server-xfce 64d → 66d, omarchymac 7d → 8d, omarchybook 3d → 4d, openmausbot 3d → 4d). The omni row now reads "CT601 removed from both hypervisors" rather than "CT601 stopped" — live pct list on both pve and m93p shows no CT601 at all, so the old guest is gone, not merely halted. Container identity re-confirmed from inside each LXC via pct exec <vmid> -- tailscale ip -4 on both hypervisors; every peer whose LXC has no Tailscale IP was checked against pct list (CT105/CT1264 run but do not connect — conduit and openmausbot offline; CT209/CT255/CT555 are stopped). Serve tables re-probed per host with pct exec 201 -- tailscale serve status.
CT201 Serve Routes
Verified live on 2026-10-26 via pct exec 201 -- tailscale serve status on pve (100.96.100.82). The previous version of this table was wrong in two ways and has been corrected.
| Host / Port | Path | Target | Notes |
|---|---|---|---|
runtipi.kangaroo-eel.ts.net:8443 (tailnet only) |
/ |
http://127.0.0.1:8085 |
Homepage asset proxy front door — 502 (re-probed 2026-10-26; upstream 8082 has no listener) |
runtipi.kangaroo-eel.ts.net:8443 (tailnet only) |
/mgr |
http://127.0.0.1:8084 |
homepage-manager container — 200 |
runtipi.kangaroo-eel.ts.net:443 (tailnet only) |
/ |
http://127.0.0.1:8085 |
Same asset proxy — 502 (re-probed 2026-10-26) |
runtipi.kangaroo-eel.ts.net:443 (tailnet only) |
/mgr |
http://127.0.0.1:8084 |
homepage-manager — 200 |
runtipi.kangaroo-eel.ts.net:443 (tailnet only) |
/steel |
http://127.0.0.1:5173 |
Steel Browser UI — 200 on :443 (re-probed 2026-10-26). ⚠ /steel is NOT served on :8443 — it 502s there |
runtipi.kangaroo-eel.ts.net:443 (tailnet only) |
/steel-api |
http://127.0.0.1:3000 |
Steel Browser API (path prefix not stripped — use direct IP for API calls) |
Two corrections to the pre-2026-10-25 table:
:8443is served bytailscaledon CT201, but so is:443.ss -ltnpon CT201 showstailscaledbound to100.96.244.39:443, and the serve table above lists routes on both ports. The old note claiming "Port 443 occupied by Traefik on CT201" is false as of 2026-10-25 — no Traefik container or process exists on CT201 (docker ps -aandps auxboth came back empty fortraefik), and Tailscale owns 443 there. The Homebox/steel path prefix is not stripped on/steel-api.- The old table routed
:8443straight tohttp://localhost:8082. That port has no listener; the live target is8085(theasset_proxy.pyfront door), which itself proxies to the dead 8082 and returns 502. → systems/homepage-dashboard
Atomic Restore Script
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh — Rebuilds the complete CT460 route table, including tailnet Serve on :443 and public Funnel on :8443.
tailscale serve reset clears Funnel state as well as Serve state. The atomic script therefore restores /webhook after rebuilding the 443 routes. Do not use the older Serve-only restore path for complete recovery.
Key Commands
# Check routes
tailscale serve status
# Rebuild routes (CT460)
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh --dry-run
/home/hermes/.hermes/scripts/tailscale-serve-apply.sh
# Add route
tailscale serve --set-path /jobs http://127.0.0.1:9099
# Remove route
tailscale serve --set-path /jobs off
Notes
- CT201 now serves Tailscale on BOTH
:443and:8443(verified 2026-10-25). The old "Port 443 occupied by Traefik on CT201 — Tailscale uses port 8443 there" claim was wrong: no Traefik container or process exists on CT201, andtailscaledowns 443 there (ss -ltnpshows it bound to100.96.244.39:443). The CT201 Serve Routes table above now records both ports. - CT460 has no conflict — Tailscale uses 443 directly, with the public webhook Funnel on
:8443. - Routes must be re-added in order (restore script handles this).
- After every rebuild, verify that
tailscale serve statuscontains both the:443Serve routes and the:8443/webhookFunnel route.
Related
- infrastructure/domains-and-tunnels — Full DNS and tunnel inventory
- systems/homepage-dashboard — Homepage on CT201 via Tailscale
- runbooks/job-radar — Dashboard incident and route verification