audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76 (Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and cover letters never ran (Notion DB held 451 entries, 0 added). Incident row dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified). Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5, deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5, topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3). Also: the two incident rows the 2026-10-26 planned restart raised auto-closed 00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of 108) — committed here and index.md corrected; tailscale.md peer table rebuilt from tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained the voice pipeline section. New: incidents/2026-10-27-job-radar-budget-exhausted.md Nothing removed or archived. No secrets.
This commit is contained in:
+17
-16
@@ -3,8 +3,8 @@ title: Tailscale
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-26
|
||||
updated: 2026-10-26
|
||||
verified_on: 2026-10-26
|
||||
updated: 2026-10-28
|
||||
verified_on: 2026-10-28
|
||||
confidence: high
|
||||
tags: [tailscale, vpn, networking, infrastructure]
|
||||
---
|
||||
@@ -25,35 +25,36 @@ Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, seve
|
||||
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
|
||||
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**.
|
||||
Re-verified 2026-10-21 03:0x: the 2026-10-19 table had drifted again — **`octop` (100.92.163.122), `springroll` (100.100.153.125), `openmausbot` and `openmausbot-vm` were missing from the table body** (only mentioned in prose), `omarchymac` had gone offline (online → idle/offline 3d), and every offline age was stale. Table rebuilt from `tailscale status --json`, keyed on DNS name rather than `HostName` (the JSON `HostName` for `freshrss` is `ct260`, and both `omniroute` nodes share the hostname `omniroute`). Container states cross-checked against live `pct list` on `pve` (100.96.100.82) and `m93p` (100.104.51.98): CT255 (mattermost) stopped, CT601 (omni) stopped, CT299 (sftpgo) running, CT600 (omniroute) running.
|
||||
Re-verified 2026-10-28 00:2x (nightly audit): State column rebuilt from `tailscale status --json` — **14 rows changed** (anthonys-imac: idle/offline 32d -> offline 34d; cachyos: idle/offline 52d -> offline 54d; cloudreve: idle/offline 93d — CT209 stopped -> offline 95d — CT209 stopped; conduit: idle/offline 49d -> offline 52d; elementsynapse: idle/offline 96d — CT555 stopped -> offline 98d — CT555 stopped; freshrss-new: idle/offline 69d -> offline 71d; m93p: online -> offline <1d; mattermost: idle/offline 36d — CT255 stopped -> offline 38d — CT255 stopped; omarchybook: idle/offline 4d -> offline 6d; omarchymac: idle/offline 8d -> offline 10d; omni: idle/offline 69d — CT601 removed from both hypervisors -> offline 71d — CT601 removed from both hypervisors; openmausbot: idle/offline 4d -> offline 6d; sftpgo: online — CT299 (pve) -> offline <1d — CT299 (pve); ubuntu-server-xfce: idle/offline 66d -> offline 68d). Header count corrected to **54 non-funnel nodes** (53 live IPv4 peers + `hermes` itself). Nodes in the table that are no longer in the tailnet: **none**. Live IPv4 peers absent from the table: **none**. **State transitions inside the audit window (not just age drift): `m93p` online → offline <1d, `sftpgo` (CT299) online → offline <1d.**
|
||||
Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale status --json` — **`springroll-browser` (100.109.221.91) was missing entirely** and has been added. **`omarchybook` (100.107.9.100) and `openmausbot` (100.91.214.91) have both gone offline** (online → idle/offline 3d) and `openmausbot` is no longer a live peer despite its m93p LXC (CT1264) still running `tailscale` — the node itself has stopped connecting. Every offline age was refreshed (e.g. `mattermost` 31d → **35d**, `omni` 64d → **68d**, `elementsynapse` 91d → **95d**, `ubuntu-server-xfce` 60d → **64d**, `omarchymac` 3d → **7d**). `m93p` (100.104.51.98) now reads **idle/offline 13m ago** — the peer record is stale, but the host is plainly reachable (both `pct list` probes and the SSH session to CT201/CT460 during this audit succeeded), so treat it as a Tailscale-side liveness artefact, not a hypervisor outage. `homeassistant` still advertises `ExitNodeOption: true` but `ExitNode: false` — it **offers** exit-node capability; it is not currently serving as one (the old "active; offers exit node" wording overstated the live state). Container states cross-checked again against live `pct list` on both hypervisors, and each peer's container identity confirmed by running `tailscale ip -4` **inside** the candidate LXC on both hosts — the authoritative mapping, since peer names and LXC names do not always match.
|
||||
|
||||
| Node | IP | State |
|
||||
|---|---|---|
|
||||
| hermes | 100.118.5.51 | **self** — CT460 (pve) |
|
||||
| agentzero | 100.106.54.94 | online — CT475 (pve) |
|
||||
| anthonys-imac | 100.112.123.124 | idle/offline 32d |
|
||||
| anthonys-imac | 100.112.123.124 | offline 34d |
|
||||
| anthonys-ipad | 100.111.198.120 | online |
|
||||
| anthonys-iphone | 100.100.116.67 | online |
|
||||
| anthonys-mac-mini | 100.74.17.15 | online |
|
||||
| cachyos | 100.124.208.15 | idle/offline 52d |
|
||||
| cachyos | 100.124.208.15 | offline 54d |
|
||||
| clawtest | 100.122.189.81 | online — CT333 (pve) |
|
||||
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
|
||||
| cloudreve | 100.111.187.7 | idle/offline 93d — CT209 stopped |
|
||||
| conduit | 100.81.79.87 | idle/offline 49d |
|
||||
| cloudreve | 100.111.187.7 | offline 95d — CT209 stopped |
|
||||
| conduit | 100.81.79.87 | offline 52d |
|
||||
| docker | 100.96.150.69 | online — CT200 (pve) |
|
||||
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
|
||||
| elementsynapse | 100.75.250.37 | idle/offline 96d — CT555 stopped |
|
||||
| elementsynapse | 100.75.250.37 | offline 98d — CT555 stopped |
|
||||
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
|
||||
| freshrss | 100.83.60.93 | online — CT1260 (m93p) |
|
||||
| freshrss-new | 100.110.180.115 | idle/offline 69d |
|
||||
| freshrss-new | 100.110.180.115 | offline 71d |
|
||||
| gitea | 100.82.100.128 | online — CT221 (m93p) |
|
||||
| guest-vps | 100.117.14.84 | online |
|
||||
| hermesagent | 100.98.205.112 | online — CT450 (pve) |
|
||||
| homeassistant | 100.68.139.5 | online |
|
||||
| litellm | 100.79.29.64 | online — CT500 (m93p) |
|
||||
| m93p | 100.104.51.98 | online |
|
||||
| m93p | 100.104.51.98 | offline <1d |
|
||||
| mailflow | 100.115.60.92 | online — CT1262 (m93p) |
|
||||
| mattermost | 100.98.116.14 | idle/offline 36d — CT255 stopped |
|
||||
| mattermost | 100.98.116.14 | offline 38d — CT255 stopped |
|
||||
| n8n | 100.84.124.30 | online |
|
||||
| nextcloud | 100.90.102.19 | online — CT270 (pve) |
|
||||
| ntfy | 100.67.153.116 | online — CT1241 (m93p) |
|
||||
@@ -61,13 +62,13 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| octop | 100.92.163.122 | online |
|
||||
| ollama | 100.108.44.119 | online — CT205 (pve) |
|
||||
| omarchy | 100.73.227.103 | online |
|
||||
| omarchybook | 100.107.9.100 | idle/offline 4d |
|
||||
| omarchymac | 100.84.180.69 | idle/offline 8d |
|
||||
| omarchybook | 100.107.9.100 | offline 6d |
|
||||
| omarchymac | 100.84.180.69 | offline 10d |
|
||||
| omarchymac-1 | 100.126.173.115 | online |
|
||||
| omni | 100.93.204.4 | idle/offline 69d — CT601 removed from both hypervisors |
|
||||
| omni | 100.93.204.4 | offline 71d — CT601 removed from both hypervisors |
|
||||
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
|
||||
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
|
||||
| openmausbot | 100.91.214.91 | idle/offline 4d |
|
||||
| openmausbot | 100.91.214.91 | offline 6d |
|
||||
| openmausbot-vm | 100.77.107.30 | online |
|
||||
| openweb | 100.69.230.13 | online — CT245 (m93p) |
|
||||
| paseo | 100.72.188.93 | online — CT1261 (m93p) |
|
||||
@@ -76,11 +77,11 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
|
||||
| rclone | 100.107.116.17 | online — CT104 (m93p) |
|
||||
| runtipi | 100.96.244.39 | online — CT201 (pve) |
|
||||
| rustdesk | 100.95.227.57 | online — CT1008 (m93p) |
|
||||
| sftpgo | 100.88.223.70 | online — CT299 (pve) |
|
||||
| sftpgo | 100.88.223.70 | offline <1d — CT299 (pve) |
|
||||
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
|
||||
| springroll | 100.100.153.125 | online |
|
||||
| springroll-browser | 100.109.221.91 | online |
|
||||
| ubuntu-server-xfce | 100.101.26.104 | idle/offline 66d |
|
||||
| ubuntu-server-xfce | 100.101.26.104 | offline 68d |
|
||||
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
|
||||
| webserver | 100.82.40.23 | online — CT207 (pve) |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user