audit(2026-11-10): 11-09 first failed-free cron day since 11-05; stocktake 'completed' with zero edits; Job Radar Stage 1 stall (9h09m, no handoff) documented; tailscale 58 nodes; omniroute 55 combos

This commit is contained in:
Hermes
2026-11-10 00:58:22 +08:00
parent 3abd9901d3
commit 262bcc74d5
9 changed files with 196 additions and 27 deletions
+15 -4
View File
@@ -3,16 +3,27 @@ title: Current State
type: concept type: concept
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-09 updated: 2026-11-10
verified_on: 2026-11-09 verified_on: 2026-11-10
review_after: 2026-11-09 review_after: 2026-11-10
confidence: high confidence: high
tags: [system] tags: [system]
sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe, live-executions-db, live-cron-incidents, live-deliveries-db, live-sessions-db]
--- ---
# Current State # Current State
Updated: 2026-11-09 Updated: 2026-11-10
## Recent Changes (2026-11-10, nightly audit; live probes 2026-11-09 23:5x – 2026-11-10 01:0x AWST)
- ✅ **2026-11-09 was the first `failed`-free cron day since 2026-11-05: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit). Trend: 11-06 1 failed / 172, 11-07 3 / 163, 11-08 1 / 170. Deliveries 8/8 `delivered`, 0 `for_failure`.
- ⚠️ **The stocktake ran 9 h 10 m, closed `completed`, and landed nothing — third day running.** `1a8e90ea2271` (**03:00:43 → 12:10:47**) delivered `suppressed`; its own output file says `RUN INCOMPLETE`, tool-iteration budget exhausted during verification, **zero edits and no commit**. `failure_streak` 3 → 0 and `last_status ok` are the harness's verdict on the *process*, not the work. The job is still pinned to `custom:omni-101` / **`free-stack`** — a combo recorded broken on **2026-08-14**. → [[incidents/2026-11-09-stocktake-completes-with-zero-edits]], [[systems/scheduled-tasks]]
- ⚠️ **The Job Radar landed nothing on 11-09.** Stage 1 (`dca8482e4f76`) ran **07:00:48 → 16:10:12 = 9 h 09 m 24 s** and never wrote `radar-ref/handoff/2026-11-09.json` (newest handoff on disk is **2026-11-06** — nothing has landed since). Both Stage-2 slots (`5fc292ff0f3b`) fired **before** Stage 1 finished (11:30:38 and 14:30:34) and declined silently. The run's own last words: *"I stopped retrying because I kept running read_file 5 times without making progress."* Same silent family as 11-02/11-03, new trigger (read loop, not iteration cap), and again **no incident row was raised**. → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]
- ✅ **The 11-08 `pre_tool_call` hook cascade did not recur.** `errors.log` (1,035 retained lines) contains **no `pre_tool_call` timeout entry at all**; its only two hook lines are benign skips (`on_session_end` 2026-11-09 22:58:35, `pre_llm_call` 2026-11-10 00:02:54). No 11-09 run outside the two documented long-runners exceeded **74 min**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- **Unclosed incident rows 3 → 0** (`cron_incidents`: 134 rows, 0 open). The stocktake cluster closed **automatically at 2026-11-09T12:10:47.125205** — the instant the overrunning run finished; none of the three ever alerted. → [[systems/scheduled-tasks]]
- **Tailscale: 58 non-funnel nodes (57 peers + `hermes`), 42 online.** `octop` is **online** again (dark since `LastSeen 2026-11-08T13:13:04Z`); 8 offline ages re-drifted — cloudreve 106d → **108d**, elementsynapse 109d → **111d**, freshrss-new 82d → **84d**, conduit 62d → **65d**, octoagent 6d → **8d**, omarchybook 17d → **19d**, openmausbot 17d → **19d**, ubuntu-server-xfce 78d → **81d**; the 6 ephemeral `codex-environment-*` peers' last-seen stamps were corrected. → [[infrastructure/tailscale]]
- **OmniRoute `omniroute-1` (CT101, `100.78.63.20`): `/v1/combos` → HTTP 200, 55 combos**, `free-stack` present — one ahead of the page's last reading (54). `/v1/models` on **both** nodes and CT600's `/v1/combos` return **HTTP 401** (alive, auth required), so the **6,480-model catalog figure dated 2026-10-26 could not be re-verified this window and was not changed**. → [[systems/omniroute]]
- **Not verified this window (gap recorded, no page changed):** CT450/CT460 Hermes version strings — `pct exec 450 -- sh -lc 'hermes --version'` resolves the **known-stale root-level `/usr/local/bin/hermes` (v0.19.0)**, not the user-level install, and CT460's `hermes` is not on the PATH in that shell, so **[[infrastructure/hosts]]'s CT450/CT460 version lines stand as written**; and CT450/CT460 docker `Up` days (`lxc-attach … Failed to exec "docker"`).
## Recent Changes (2026-11-09, nightly audit; live probes 2026-11-08 23:5x – 2026-11-09 00:2x AWST) ## Recent Changes (2026-11-09, nightly audit; live probes 2026-11-08 23:5x – 2026-11-09 00:2x AWST)
@@ -91,6 +91,13 @@ the error string does not name the hook.
fault; if it recurs while the hook budget is 120 s, this hypothesis is dead. fault; if it recurs while the hook budget is 120 s, this hypothesis is dead.
- Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch. - Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch.
## Follow-up (2026-11-10 nightly audit)
- **The fix held; the cascade did not recur.** `errors.log` (1,035 retained lines) contains **no `pre_tool_call` timeout entry at all**. Its only two hook lines are benign skips — `on_session_end` 2026-11-09 22:58:35 and `pre_llm_call` 2026-11-10 00:02:54 (*"skipped after previous timeout or while still running"*).
- **No 11-09 run showed the cascade signature.** Outside the two long-runners documented separately (stocktake **9 h 10 m**, Job Radar Stage 1 **9 h 09 m** — both stalled in *tool iteration*, not in a hook), the longest 11-09 execution was **74 min**, then 40 min and 11 min.
- **2026-11-09 was the first `failed`-free cron day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1, 11-09: **0 of 174**), and unclosed incident rows went 3 → 0.
- **Residual:** the two long-runners above are the same 9-hour-turn environment this bug leaves behind, and one (`wiki-stocktake-maintenance`) is pinned to `free-stack` — a combo recorded broken on 2026-08-14. The job-iteration budget stays in force.
## See also ## See also
- [[systems/conduit]] - [[systems/conduit]]
@@ -0,0 +1,64 @@
---
title: "2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op"
type: incident
status: active
created: 2026-11-10
updated: 2026-11-10
verified_on: 2026-11-10
confidence: high
tags: [cron, job-radar, stall, silent-failure, handoff, read-loop]
sources: ["executions.db (dca8482e4f76: 2026-11-09T07:00:48+08:00 → 2026-11-09T16:10:12+08:00, status completed, delivery_outcome suppressed; 5fc292ff0f3b: 11:30:38 → 12:10:07 and 14:30:34 → 14:37:27, both completed/suppressed)", "~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md (ends INCOMPLETE: tool budget exhausted; last words \\u201cI stopped retrying because I kept running read_file 5 times without making progress.\\u201d)", "~/.hermes/radar-ref/handoff/ (newest handoff on disk: 2026-11-06; no 2026-11-07/08/09 files)", "cron_incidents (no row raised for this run)"]
---
# 2026-11-09 — Job Radar Stage 1 Stall, Silent No-Op
## Summary
On 2026-11-09 the Job Radar's Stage 1 (`dca8482e4f76`) ran for **9 h 09 m 24 s** (07:00:48 → 16:10:12), ended `INCOMPLETE: tool budget exhausted`, and **never wrote its handoff file**. Because both Stage-2 slots run on their own schedule, **both fired before Stage 1 finished** and each declined silently rather than erroring. Net effect: **the radar landed nothing on 11-09 and raised no alert.**
## Timeline (AWST, 2026-11-09)
| Time | Event |
|---|---|
| 07:00:48 | Stage 1 starts |
| 11:30:38 → 12:10:07 | Stage-2 slot 1 fires — Stage 1 still running → silent no-op (`suppressed`) |
| 14:30:34 → 14:37:27 | Stage-2 slot 2 fires — Stage 1 still running → silent no-op (`suppressed`) |
| ~16:10:1x | Stage 1 writes `INCOMPLETE: tool budget exhausted` |
| 16:10:12 | Stage 1 closes — `completed`, delivery `suppressed` |
| all day | `radar-ref/handoff/2026-11-09.json` is never created |
## Root Cause
Two compounding faults, both previously documented for this job:
1. **Stage 2 never checks that a fresh handoff exists**, so when Stage 1 fails it proceeds and self-suppresses instead of failing loudly. Two chances to notice, zero alerting.
2. **Stage 1 overran without producing output.** Its own final message: *"I stopped retrying because I kept running read_file 5 times without making progress."* This is a **new trigger** for the same silent family seen on 11-02/11-03 (there: iteration cap) — here: an unproductive read loop that consumed the run's budget.
## Evidence
- `executions.db` `dca8482e4f76`: `2026-11-09T07:00:48+08:00` → `2026-11-09T16:10:12+08:00`, `status completed`, `delivery_outcome suppressed`.
- `5fc292ff0f3b` (Stage 2): two rows on 11-09, both **before** Stage 1's finish stamp, both `completed` / `suppressed`.
- `~/.hermes/radar-ref/handoff/` — newest file is **2026-11-06**; nothing for 11-07, 11-08 or 11-09.
- `~/.hermes/cron/output/dca8482e4f76/2026-11-09_16-10-11.md` ends `INCOMPLETE: tool budget exhausted`.
- **No row in `cron_incidents`** for `dca8482e4f76` on 11-09 — the failure was never raised.
## Impact
- The Job Radar — the estate's job-discovery feed — produced **no output for a third day** (no handoff since 11-06).
- The failure is invisible: the run's ledger entry says `completed`, delivery is `suppressed`, and no incident row exists.
## Fix Applied
**None — documentation only.** No configuration change was made this window.
## Prevention
- **Alert on the artefact, not the exit code.** Stage 2 should hard-fail when the expected `radar-ref/handoff/<date>.json` is missing, rather than proceeding to a suppression. That single check converts both 11-09 slots into loud failures.
- Add a **read-progress guard** to Stage 1: when the same file has been read N times without a state change, stop and emit a partial handoff instead of burning the budget to `INCOMPLETE`.
- Note the pattern across incidents: **completion-time-only alerting hides every failure in this family** (stocktake 11-09, radar 11-09, radar 11-02/03).
## Related
- [[systems/scheduled-tasks]]
- [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]
- [[runbooks/provider-health]]
@@ -0,0 +1,65 @@
---
title: "2026-11-09 — Stocktake Completes With Zero Edits"
type: incident
status: active
created: 2026-11-10
updated: 2026-11-10
verified_on: 2026-11-10
confidence: high
tags: [cron, wiki, stocktake, tool-budget, silent-failure, omni-101, free-stack]
sources: ["executions.db (1a8e90ea2271: 2026-11-09T03:00:43+08:00 → 2026-11-09T12:10:47+08:00, status completed, delivery_outcome suppressed)", "~/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md (8,305 bytes; RUN INCOMPLETE, \\u201ctool-iteration budget exhausted during verification\\u201d)", "cron_incidents (1a8e90_628d914f1f97, 1a8e90_2f370ef764f5, 1a8e90_a9cad99cb150 — all closed 2026-11-09T12:10:47.125205+08:00)", "wiki-stocktake-maintenance job 1a8e90ea2271 (provider custom:omni-101, model free-stack)"]
---
# 2026-11-09 — Stocktake Completes With Zero Edits
## Summary
The nightly wiki stocktake (`wiki-stocktake-maintenance`, `1a8e90ea2271`) ran on 2026-11-09 from **03:00:43 to 12:10:47** — **9 h 10 m** — and the harness recorded it **`completed`** with delivery `suppressed`. Its own output file records the opposite: the run ran out of tool-iteration budget inside its verification phase, applied **zero edits**, made **no commit**, and its findings never landed.
**This is the third consecutive day the stocktake's work did not land** — 11-07 died at 2 h 02 m, 11-08 at 1 h 35 m (both `failed`), 11-09 at 9 h 10 m (`completed`). The visible status this time is *worse*, not better: `failure_streak` fell **3 → 0** and `last_status` shows **ok**, while the work stayed undone.
## Timeline (AWST, 2026-11-09)
| Time | Event |
|---|---|
| 03:00:43 | Run starts (`1a8e90ea2271`) |
| ~04:1x | Tool-iteration budget exhausted during verification; output file written with `RUN INCOMPLETE` |
| 12:10:47.125205 | Harness closes the run — `completed`, delivery `suppressed` |
| 12:10:47.125205 | The **three** long-open stocktake incident rows close at the same instant: `1a8e90_628d914f1f97` (11-07T04:29:54), `1a8e90_2f370ef764f5` (11-07T05:02:21), `1a8e90_a9cad99cb150` (11-08T04:35:38) |
## Root Cause
Same family as 11-07/11-08: **the run spends its entire tool budget before reaching the write phase.** The run's own output file records the stop as the tool-iteration budget being exhausted **during verification**, with zero edits applied — the edits it had identified were never written.
Secondary cause (confirmed again this window): the job is pinned to `custom:omni-101` / **`free-stack`** — a combo recorded broken on **2026-08-14** (see [[systems/scheduled-tasks]]).
## Evidence
- `executions.db`: `started_at 2026-11-09T03:00:43+08:00` → `finished_at 2026-11-09T12:10:47+08:00`, `status completed`, `delivery_outcome suppressed`. Previous two runs: 11-08 `failed` (03:01:11 → 04:36:08, 1 h 35 m), 11-07 `failed` (03:00:31 → 05:02:22, 2 h 02 m).
- Output file `/home/hermes/.hermes/cron/output/1a8e90ea2271/2026-11-09_12-10-46.md` (8,305 bytes) ends `RUN INCOMPLETE`, with the tool-iteration budget exhausted inside verification.
- The three stocktake incident rows all carry `closed_at = 2026-11-09T12:10:47.125205+08:00` — i.e. they closed on **completion**, not on a fix. All three have `alerted_at = null`: they never alerted.
- Git: **no stocktake commit on 11-09** — zero edits were applied, so there was nothing to commit and nothing to revert. (On 11-07/11-08 the page was left dirty and then reverted; on 11-09 no edit was ever made.)
## Impact
- Nine consecutive stalled stocktake runs (11-01 … 11-09) — the nightly wiki freshness pass is producing no durable output.
- The stocktake's *top-priority* brief item on 11-09 was the **monotonically worsening failure streak** itself. The job closed its own incident while failing that item, which is the failure mode to watch: **`completed` and `ok` now overstate the stocktake's health.**
- Three incident rows were silently cleared, so the fleet's incident board now reads **0 open** while the underlying problem is unfixed.
## Fix Applied
**None — this is a detection-and-verification record.** The page documents the failure and the misleading status so the next audit starts from truth.
Also recorded this window: the `free-stack` pin on `1a8e90ea2271` — the only deterministic configuration fault in the chain — is **still in place**, unchanged from the 11-08 record.
## Prevention
- **Alerting must fire on write progress, not completion.** This run produced 2 of 3 failure days silently and the third day produced a run that reported success while editing nothing. A check on `commit_count` / `edits_applied` for this job would have caught all three.
- Don't read `status`/`failure_streak` as the stocktake's health. Read whether a commit landed.
## Related
- [[systems/scheduled-tasks]]
- [[infrastructure/tailscale]]
- [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]
+6 -4
View File
File diff suppressed because one or more lines are too long
+21 -14
View File
@@ -3,8 +3,8 @@ title: Tailscale
type: infrastructure type: infrastructure
status: active status: active
created: 2026-07-26 created: 2026-07-26
updated: 2026-11-07 updated: 2026-11-10
verified_on: 2026-11-07 verified_on: 2026-11-10
confidence: high confidence: high
tags: [tailscale, vpn, networking, infrastructure] tags: [tailscale, vpn, networking, infrastructure]
--- ---
@@ -19,10 +19,11 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`) - **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`) - **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
## Tailscale Node Inventory (verified 2026-11-07) ## Tailscale Node Inventory (verified 2026-11-10)
Re-verified 2026-11-10 00:4x AWST (nightly audit): `tailscale status --json` re-read — **6 new peers added, 8 offline ages drifted, 1 state transition back *online* (`octop`); no IP changes and no missing table rows.** ⚠ **Six peers that had never been listed now sit in the tailnet** — `codex-environment` (`100.111.126.78`), `codex-environment-1b88dc` (`100.76.42.76`), `codex-environment-8dadbb` (`100.65.200.27`), `codex-environment-955866` (`100.81.119.118`), `codex-environment-9fbd61` (`100.106.215.121`) and `codex-environment-d3edbd` (`100.74.3.2`) — every one created **2026-11-07 05:09–05:56Z**, every one **`Online: false`** with `LastSeen` **2026-11-07 11:09–11:56Z**. **None is a guest on either hypervisor** (live `pct list` and `qm list` on both `pve` and `m93p` list no such guest), so they are **orphaned/ephemeral tailnet peers**, not estate hosts: six records registered inside 47 minutes and all went silent within ~6 h, the signature of a throwaway CI/dev environment rather than infrastructure. **State transition:** `octop` (`100.92.163.122`) went **online → offline 1d** (`LastSeen 2026-11-08T13:13:04Z`) after returning for a single day on 11-07. The **9 offline ages** were refreshed (+1d each): cloudreve 105d → **106d**; conduit 61d → **62d**; elementsynapse 108d → **109d**; freshrss-new 81d → **82d**; octoagent 5d → **6d**; omarchybook 16d → **17d**; openmausbot 16d → **17d**; ubuntu-server-xfce 77d → **78d**; plus **`omarchymac`, which was not aged but re-stated** — the 2026-11-07 row read *"offline 5d — last seen 2026-11-01 11:13Z"* while the live record reads **`LastSeen 2026-11-08T00:45:17Z` = offline 2d**, so **both the age and the last-seen stamp on that row were stale** and are corrected here. Node count **52 → 58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none** once the six additions land; the **~23 IPv6-only `funnel-ingress-node` peers** remain deliberately excluded (`tailscale status --json`: **80 peers total — 57 with a `100.` IPv4, 23 IPv6-only**). Container counts re-confirmed live: `pve` (100.96.100.82) **17 LXCs + 4 VMs**, `m93p` (100.104.51.98) **20 LXCs + 2 VMs** — unchanged. **Not re-probed this pass** (so the 2026-11-07 findings stand un-contradicted rather than re-verified): the `:443` Serve route table (12 paths), the two DEAD CT460 targets (8769 `/agent-estate`, 9123 `/synergy-prep`), and per-guest tailnet identity via `pct exec <vmid> -- tailscale ip -4`.
Re-verified 2026-11-07 03:1x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 9 offline-age drifts, no IP changes and no missing peers.** **`octop` came back online**: the peer record reads `Online: true` with `LastSeen 0001-01-01` and `tailscale ping 100.92.163.122` → **`pong from octop via DERP(syd) in 98ms`**, so the row moves offline → online. It remains **not an LXC or VM on either hypervisor** (live `pct list` / `qm list` on both `pve` and `m93p` list no such guest) — an **orphaned tailnet peer**, so its return is not an estate change. The **9 offline ages** were refreshed (+1d each): cloudreve 104d → **105d**; conduit 60d → **61d**; elementsynapse 107d → **108d**; freshrss-new 80d → **81d**; octoagent 4d → **5d**; omarchybook 15d → **16d**; omarchymac 4d → **5d**; openmausbot 15d → **16d**; ubuntu-server-xfce 76d → **77d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) **17 guests** and `m93p` (100.104.51.98) **20 guests + 2 VMs** — ⚠ **the m93p guest count on this page was one short**: live `pct list` on `m93p` returns **20 LXCs** (the extra is **CT107 `mem0`**, `100.73.94.53`, `mem0-mcp.service` active — confirmed from inside the guest), not 19; CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT998 `nightbridge` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). The route table still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; **both DEAD targets re-confirmed dead** — `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`) — while every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` answers **HTTP 200**. Re-verified 2026-11-07 03:1x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 9 offline-age drifts, no IP changes and no missing peers.** **`octop` came back online**: the peer record reads `Online: true` with `LastSeen 0001-01-01` and `tailscale ping 100.92.163.122` → **`pong from octop via DERP(syd) in 98ms`**, so the row moves offline → online. It remains **not an LXC or VM on either hypervisor** (live `pct list` / `qm list` on both `pve` and `m93p` list no such guest) — an **orphaned tailnet peer**, so its return is not an estate change. The **9 offline ages** were refreshed (+1d each): cloudreve 104d → **105d**; conduit 60d → **61d**; elementsynapse 107d → **108d**; freshrss-new 80d → **81d**; octoagent 4d → **5d**; omarchybook 15d → **16d**; omarchymac 4d → **5d**; openmausbot 15d → **16d**; ubuntu-server-xfce 76d → **77d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) **17 guests** and `m93p` (100.104.51.98) **20 guests + 2 VMs** — ⚠ **the m93p guest count on this page was one short**: live `pct list` on `m93p` returns **20 LXCs** (the extra is **CT107 `mem0`**, `100.73.94.53`, `mem0-mcp.service` active — confirmed from inside the guest), not 19; CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT998 `nightbridge` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). The route table still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; **both DEAD targets re-confirmed dead** — `ss -ltn` still shows **no listener on 8769** (`/agent-estate`) and **none on 9123** (`/synergy-prep`) — while every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` answers **HTTP 200**.
Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable. Re-verified 2026-11-06 03:0x–03:2x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition, 8 offline-age drifts, no IP changes and no missing peers.** **`anthonys-mac-mini` came back online**: the peer record reads `Online: true` and `tailscale ping 100.76.60.37` → `pong from anthonys-mac-mini via DERP(syd) in 103ms`, so the row moves offline → online (client device, not an estate host). The **8 offline ages** were refreshed (+1d each): cloudreve 103d → **104d**; elementsynapse 106d → **107d**; freshrss-new 79d → **80d**; octoagent 3d → **4d**; octop 1d → **2d**; omarchybook 14d → **15d**; omarchymac 3d → **4d**; openmausbot 14d → **15d**. **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none** (52 rows = 51 live IPv4 peers + `hermes` itself; the ~23 IPv6-only `funnel-ingress-node` peers remain deliberately excluded). `m93p` re-pinged → `pong from m93p via 192.168.178.250:41641 in 3ms` (online, peer record agrees), and `octop` re-pinged → **timeout** (genuinely offline, matching its `LastSeen`). Container states cross-checked against live `pct list` on **both** hypervisors: `pve` (100.96.100.82) 17 guests and `m93p` (100.104.51.98) 19 guests + 2 VMs — CT209 `cloudreve` **stopped**, CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT105 `conduit` **running but not connected to the tailnet**, CT299 `sftpgo` **running**, CT600 `omniroute` **running** (CT601 absent from both hosts). ⚠ **Two CT460 Serve routes are dead but still present in `tailscale serve status`**: `/agent-estate` → `127.0.0.1:8769` and `/synergy-prep` → `127.0.0.1:9123` have **no listener** (live `ss -ltn` probe) and therefore serve 502 — both rows are now marked DEAD here, matching [[infrastructure/network-map]] which had already recorded them as dead on 2026-11-05. The route table itself still matches `tailscale serve status` **exactly (12 paths on :443)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route; every other Serve target (9119, 5175, 9099, 5173, 8080, 8085, 8787, 6080, 6081, 9120) has a live listener, and the `/gitea` target `100.82.100.128:3000` is reachable.
Full node list from `tailscale status` — **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age. Full node list from `tailscale status` — **58 non-funnel nodes** (57 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route. Re-verified 2026-11-04 03:0x–05:5x AWST (stocktake): table rebuilt from `tailscale status --json` — **2 missing peers added, 3 state transitions, 8 offline-age drifts.** **Two live peers had never been listed** and are now in the table: **`mem0`** (`100.73.94.53`, CT107 on `m93p` — confirmed by `pct exec 107 -- tailscale ip -4`) and **`obsidian-livesync`** (`100.116.244.84`, CT100 on `m93p` — confirmed by `pct exec 100 -- tailscale ip -4`). Node count corrected to **52 non-funnel nodes** (51 live IPv4 peers + `hermes` itself). **Three genuine transitions, each confirmed by a ping rather than the peer record alone:** **`octoagent`** (100.117.10.68) went **online → offline 2d** (`LastSeen 2026-11-01T13:48:20Z`, `tailscale ping` → timeout) and **`octop`** (100.92.163.122) went **online → offline <1d** (`LastSeen 2026-11-03T09:03:02Z`, ping → timeout); **neither is an LXC or VM on either hypervisor** (`pct list` / `qm list` on both `pve` and `m93p` list no such guest) — they are **orphaned tailnet peers**, so their loss is not an estate outage. **`sftpgo`** (CT299) went **offline → online** during this audit: it had been **frozen for ~24 h by a wedged `vzdump` task** whose NFS write path to `fred-qnap` had hung; killing the task unfroze the guest and `pct exec 299 -- systemctl is-active sftpgo` → **`active`** with `LISTEN *:8080` → [[incidents/2026-11-04-vzdump-freeze-ct299-sftpgo-outage]]. **`omarchymac`** aged from `offline <1d` (last seen 2026-10-31 13:20Z) to **offline 2d** (`LastSeen` is actually `2026-11-01T11:13:06Z`). The **8 offline ages** were refreshed (cloudreve 99d → **102d**; conduit 55d → **58d**; elementsynapse 102d → **105d**; freshrss-new 75d → **78d**; omarchybook 10d → **13d**; openmausbot 10d → **13d**; ubuntu-server-xfce 72d → **75d**; omarchymac <1d → **2d**). **`m93p`'s peer record reads `Online: true` again** — the stale-record run of 10-29 → 11-01 (four consecutive false reads) has cleared, and `tailscale ping 100.104.51.98` → pong. Container identities re-confirmed from **inside** the guests (`pct exec <vmid> -- tailscale ip -4` on both hypervisors): CT107 `mem0`, CT100 `obsidian-livesync`, CT255 `mattermost` (**running**, `systemctl is-active mattermost` → `active`, `curl http://100.98.116.14:8065/api/v4/system/ping` → **HTTP 200**), CT333 `clawtest` (**running**), CT207 `webserver` (**running**, `pct exec 207 -- tailscale ip -4` → `100.82.40.23`), CT299 `sftpgo` (**running**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route. Re-verified 2026-10-31 05:4x AWST (stocktake): table rebuilt from `tailscale status --json` — **one state transition and seven offline-age drifts, no IP changes and no missing peers.** **`mattermost` came back online**: the peer record reads `Online: true` for the first time since the outage, and the guest behind it is genuinely healthy — live `pct exec 255 -- systemctl is-active mattermost` → **`active`**, `ss -ltnp` shows `mattermost` listening on `*:8065`, `pct exec 255 -- tailscale ip -4` → `100.98.116.14`, and the gateway's own `gateway.log` records **`✓ mattermost connected` at 2026-10-30 16:36:17** (after an earlier successful reconnect at 07:19:37 the same day). CT255 is therefore **running** on `pve` — the 40-day outage is over. The **7 offline ages** were refreshed (cloudreve 97d → **98d**; conduit 53d → **55d**; elementsynapse 100d → **102d**; freshrss-new 73d → **74d**; omarchybook 8d → **9d**; openmausbot 8d → **9d**; ubuntu-server-xfce 70d → **71d**). **Every peer IP in the table matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Two container annotations were added for peers that had none: `openmausbot` (**CT1264, m93p — stopped**) and `openmausbot-vm` (**VM106, m93p**), both confirmed from live `pct list` / `qm list` on `m93p`. The `m93p` row's "peer record stale" note was updated — its peer record read `Online: false` on 10-29 and 10-30 and reads **`Online: true`** today, the third consecutive day the record disagreed with the (reachable) host. CT460's `:443` Serve route table still matches `tailscale serve status` **exactly (12 paths)**, plus the `:8443/webhook` Funnel and the `:9443` tailnet route.
Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required). Re-verified 2026-11-01 03:0x–04:0x AWST (stocktake): table rebuilt from `tailscale status --json` — **1 state transition and 5 offline-age drifts, no IP changes and no missing peers.** **`omarchymac` went offline**: the peer record reads `Online: false` with `LastSeen 2026-10-31T13:20:00Z` and `tailscale ping 100.126.173.115` times out, so this is a **genuine** transition (the JSON `OS` field reads `linux`, not a Mac). **`m93p`'s peer record reads `Online: false` again** — the **4th consecutive** stale record — while `tailscale ping 100.104.51.98` → `pong from m93p via DERP(syd) in 98ms` and SSH over the tailnet returned `up 1 day, 12:02`, so the row stays **online** and the peer record remains non-authoritative for hypervisor reachability. The **5 offline ages** were refreshed (cloudreve 98d → **99d**; freshrss-new 74d → **75d**; omarchybook 9d → **10d**; openmausbot 9d → **10d**; ubuntu-server-xfce 71d → **72d**). **`conduit` (55d) and `elementsynapse` (102d) were already correct** and are unchanged. **Every peer IP in the table still matches the live JSON exactly; live IPv4 peers absent from the table: none; table rows with no live peer: none.** Container identity re-confirmed from inside the guests: `pct exec 201 -- tailscale ip -4` → `100.96.244.39` (CT201, runtipi), `pct exec 1001 -- docker ps` → `sshwifty` (CT1001 on `m93p`), CT209 `cloudreve` and CT555 `elementsynapse` **stopped**, CT1264 `openmausbot` **stopped**, CT255 `mattermost` **running** with `systemctl is-active mattermost` → `active` and `LISTEN *:8065`. Both OmniRoute nodes re-probed: `100.88.81.19` (CT600) and `100.78.63.20` (CT101) both return **401** on `/v1/models` (alive, auth required).
@@ -45,14 +46,20 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) | | anthonys-mac-mini | 100.76.60.37 | online — **returned 2026-11-06** (peer record `Online: true`, `tailscale ping` → `pong from anthonys-mac-mini (100.76.60.37) via DERP(syd) in 103ms`; client device, not an estate host) |
| clawtest | 100.122.189.81 | online — CT333 (pve) | | clawtest | 100.122.189.81 | online — CT333 (pve) |
| cliproxyapi | 100.83.112.94 | online — CT103 (pve) | | cliproxyapi | 100.83.112.94 | online — CT103 (pve) |
| cloudreve | 100.111.187.7 | offline 105d — CT209 stopped | | cloudreve | 100.111.187.7 | offline 108d — CT209 stopped |
| conduit | 100.81.79.87 | offline 61d — last seen 2026-09-06 01:32Z | | codex-environment | 100.111.126.78 | offline 2d — last seen 2026-11-07 11:09Z; **created 2026-11-07 05:5xZ**, six-record burst; **not an LXC/VM on either hypervisor** (orphaned/ephemeral tailnet peer; added 2026-11-10, last-seen stamps corrected 2026-11-10) |
| codex-environment-1b88dc | 100.76.42.76 | offline 2d — last seen 2026-11-07 11:12Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-8dadbb | 100.65.200.27 | offline 2d — last seen 2026-11-07 11:46Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-955866 | 100.81.119.118 | offline 2d — last seen 2026-11-07 11:49Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-9fbd61 | 100.106.215.121 | offline 2d — last seen 2026-11-07 11:51Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| codex-environment-d3edbd | 100.74.3.2 | offline 2d — last seen 2026-11-07 11:56Z; ephemeral peer (added 2026-11-10; stamp corrected 2026-11-10) |
| conduit | 100.81.79.87 | offline 65d — last seen 2026-09-06 01:32Z |
| docker | 100.96.150.69 | online — CT200 (pve) | | docker | 100.96.150.69 | online — CT200 (pve) |
| docker-01 | 100.121.12.31 | online — CT1001 (m93p) | | docker-01 | 100.121.12.31 | online — CT1001 (m93p) |
| elementsynapse | 100.75.250.37 | offline 108d — CT555 stopped | | elementsynapse | 100.75.250.37 | offline 111d — CT555 stopped |
| freellmapi | 100.127.208.75 | online — CT1006 (m93p) | | freellmapi | 100.127.208.75 | online — CT1006 (m93p) |
| freshrss | 100.83.60.93 | online — CT1260 (m93p) | | freshrss | 100.83.60.93 | online — CT1260 (m93p) |
| freshrss-new | 100.110.180.115 | offline 81d | | freshrss-new | 100.110.180.115 | offline 84d — last seen 2026-08-17 17:04Z |
| gitea | 100.82.100.128 | online — CT221 (m93p) | | gitea | 100.82.100.128 | online — CT221 (m93p) |
| guest-vps | 100.117.14.84 | online | | guest-vps | 100.117.14.84 | online |
| hermesagent | 100.98.205.112 | online — CT450 (pve) | | hermesagent | 100.98.205.112 | online — CT450 (pve) |
@@ -66,15 +73,15 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| nextcloud | 100.90.102.19 | online — CT270 (pve) | | nextcloud | 100.90.102.19 | online — CT270 (pve) |
| ntfy | 100.67.153.116 | online — CT1241 (m93p) | | ntfy | 100.67.153.116 | online — CT1241 (m93p) |
| obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) | | obsidian-livesync | 100.116.244.84 | online — CT100 (m93p) |
| octoagent | 100.117.10.68 | offline 5d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | | octoagent | 100.117.10.68 | offline 8d — last seen 2026-11-01 13:48Z; **not an LXC/VM on either hypervisor** (orphaned tailnet peer) |
| octop | 100.92.163.122 | online — **returned 2026-11-07** (peer record `Online: true`, `tailscale ping` → `pong from octop via DERP(syd) in 98ms`); still **not an LXC/VM on either hypervisor** (orphaned tailnet peer) | | octop | 100.92.163.122 | **online** (re-verified 2026-11-10 00:4x; it had been dark since `LastSeen 2026-11-08T13:13:04Z` — online 11-07, offline 11-08, online again 11-10; still **not an LXC/VM on either hypervisor** — orphaned tailnet peer) |
| ollama | 100.108.44.119 | online — CT205 (pve) | | ollama | 100.108.44.119 | online — CT205 (pve) |
| omarchy | 100.73.227.103 | online | | omarchy | 100.73.227.103 | online |
| omarchybook | 100.107.9.100 | offline 16d | | omarchybook | 100.107.9.100 | offline 19d — last seen 2026-10-21 15:55Z |
| omarchymac | 100.126.173.115 | offline 5d — last seen 2026-11-01 11:13Z | | omarchymac | 100.126.173.115 | offline 2d — last seen 2026-11-08 00:45Z (⚠ **was stale at 5d / 2026-11-01**; corrected 2026-11-10) |
| omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute | | omniroute | 100.88.81.19 | online — CT600 (pve), live OmniRoute |
| omniroute-1 | 100.78.63.20 | online — CT101 (m93p) | | omniroute-1 | 100.78.63.20 | online — CT101 (m93p) |
| openmausbot | 100.91.214.91 | offline 16d — CT1264 (m93p) stopped | | openmausbot | 100.91.214.91 | offline 19d — CT1264 (m93p) stopped; last seen 2026-10-21 15:50Z |
| openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) | | openmausbot-vm | 100.77.107.30 | online — VM106 (m93p) |
| openweb | 100.69.230.13 | online — CT245 (m93p) | | openweb | 100.69.230.13 | online — CT245 (m93p) |
| paseo | 100.72.188.93 | online — CT1261 (m93p) | | paseo | 100.72.188.93 | online — CT1261 (m93p) |
@@ -87,7 +94,7 @@ Re-verified 2026-10-25 03:1x (stocktake): table rebuilt again from `tailscale st
| social-dashboard | 100.70.183.37 | online — CT1263 (m93p) | | social-dashboard | 100.70.183.37 | online — CT1263 (m93p) |
| springroll | 100.100.153.125 | online | | springroll | 100.100.153.125 | online |
| springroll-browser | 100.109.221.91 | online | | springroll-browser | 100.109.221.91 | online |
| ubuntu-server-xfce | 100.101.26.104 | offline 77d — last seen 2026-08-20 19:36Z | | ubuntu-server-xfce | 100.101.26.104 | offline 81d — last seen 2026-08-20 19:36Z |
| vaultwarden | 100.123.225.90 | online — CT102 (m93p) | | vaultwarden | 100.123.225.90 | online — CT102 (m93p) |
| webserver | 100.82.40.23 | online — CT207 (pve) | | webserver | 100.82.40.23 | online — CT207 (pve) |
+11
View File
@@ -2442,3 +2442,14 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
`index.md` (new entry + incident list + `state.db` figure, confidence dropped), and this log. `index.md` (new entry + incident list + `state.db` figure, confidence dropped), and this log.
- **No wiki page removed or archived.** No system obsoleted; no decision reversed. - **No wiki page removed or archived.** No system obsoleted; no decision reversed.
- **No secrets, keys, tokens or passwords written** — no credential value appears anywhere in this audit. - **No secrets, keys, tokens or passwords written** — no credential value appears anywhere in this audit.
### 2026-11-10 — Nightly audit (run 2026-11-09 23:59 AWST; window: 2026-11-09, live probes 00:0x–01:0x)
- **2026-11-09 is the first `failed`-free cron day since 2026-11-05: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit). Deliveries **8/8 `delivered`**, 0 `for_failure`. Unclosed incident rows **3 → 0** (`cron_incidents`: 134 rows, 0 open). → `current-state.md`, [[systems/scheduled-tasks]]
- ⚠️ **The stocktake's `failure_streak` 3 → 0 / `last_status ok` are cosmetic.** `1a8e90ea2271` ran **03:00:43 → 12:10:47 = 9 h 10 m**, closed `completed` (`suppressed`), and landed **zero edits and no commit** — its own output file says `RUN INCOMPLETE`, tool-iteration budget exhausted inside verification. **Third consecutive day the stocktake's work did not land** (11-07, 11-08, 11-09); it is still pinned to `custom:omni-101` / `free-stack`. New page: [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. The three long-open incident rows (`1a8e90_628d914f1f97`, `_2f370ef764f5`, `_a9cad99cb150`) all closed at **12:10:47.125205**, the instant the run ended — hot-loop alert closing, not a fix.
- ⚠️ **New finding — Job Radar Stage 1 (11-09)** `dca8482e4f76` ran **07:00:48 → 16:10:12 = 9 h 09 m**, never wrote `radar-ref/handoff/2026-11-09.json` (newest handoff on disk: **2026-11-06**), and both Stage-2 slots (`5fc292ff0f3b`) fired **before** it finished and declined silently. Same silent family as 11-02/11-03 with a new trigger (read loop). No incident row raised. New page: [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]].
- ✅ **11-08 `pre_tool_call` hook cascade did not recur** — no `pre_tool_call` timeout line in `errors.log`; only two benign skips; no 11-09 run outside the two long-runners exceeded 74 min. Follow-up appended to [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
- **Tailscale:** 58 non-funnel nodes (57 peers + `hermes`), 42 online. `octop` **online** again; 8 offline ages refreshed (cloudreve 108d, elementsynapse 111d, freshrss-new 84d, conduit 65d, octoagent 8d, omarchybook 19d, openmausbot 19d, ubuntu-server-xfce 81d); `codex-environment-*` last-seen stamps corrected. → [[infrastructure/tailscale]]
- **OmniRoute:** `omniroute-1` CT101 `/v1/combos` → HTTP 200, **55 combos**, `free-stack` present. `/v1/models` on both nodes and CT600 `/v1/combos` → **HTTP 401**, so the **6,480-model catalog figure (2026-10-26) stays unverified this window; no page changed.** → [[systems/omniroute]]
- **Not verified (gap recorded):** CT450/CT460 Hermes version strings (the `pct exec` probe resolves the stale root-level `/usr/local/bin/hermes`; CT460's `hermes` isn't on PATH in that shell) and their docker `Up` days (`lxc-attach … Failed to exec "docker"`) — [[infrastructure/hosts]] and [[infrastructure/docker-services]] version/uptime lines stand as written. Unchanged.
- No secrets, keys, tokens or passwords written.
+3 -2
View File
@@ -3,8 +3,8 @@ title: OmniRoute
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-01 updated: 2026-11-10
verified_on: 2026-11-01 verified_on: 2026-11-10
confidence: high confidence: high
tags: [routing, provider, model, inference] tags: [routing, provider, model, inference]
sources: [raw/configs/hermes-config-sanitized.txt] sources: [raw/configs/hermes-config-sanitized.txt]
@@ -101,6 +101,7 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor, t
## AutoRoute variants (2026-08-12) ## AutoRoute variants (2026-08-12)
- `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** AutoRoute variants — they do NOT appear in the persisted combo list (`/api/combos` returns 54 combos, none named these). - `auto/best-free` (Auto Cheap) and `auto/coding-free` are **virtual** AutoRoute variants — they do NOT appear in the persisted combo list (`/api/combos` returns 54 combos, none named these).
- **Live re-read 2026-11-10 00:4x AWST (nightly audit):** `omniroute-1` (`100.78.63.20:20128`) `/v1/combos` → **HTTP 200, 55 combos**, `free-stack` present — one ahead of the last reading above. `/v1/models` on **both** nodes and CT600's `/v1/combos` return **HTTP 401** (endpoint live, auth required), so the **6,480-model catalog figure dated 2026-10-26 stands unverified this window and is unchanged.**
- The LKGP engine resolves them dynamically: it scores the operator's **live connected provider accounts** (e.g. `antigravity`, `opencode-zen`, `nvidia`, `openrouter`) rather than a hardcoded candidate list. - The LKGP engine resolves them dynamically: it scores the operator's **live connected provider accounts** (e.g. `antigravity`, `opencode-zen`, `nvidia`, `openrouter`) rather than a hardcoded candidate list.
- Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request. - Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request.
- Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map. - Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map.
+4 -3
View File
@@ -3,9 +3,9 @@ title: Scheduled Tasks
type: system type: system
status: active status: active
created: 2026-07-22 created: 2026-07-22
updated: 2026-11-09 updated: 2026-11-10
verified_on: 2026-11-09 verified_on: 2026-11-10
review_after: 2026-11-09 review_after: 2026-11-10
confidence: high confidence: high
tags: [cron, scheduled, automation] tags: [cron, scheduled, automation]
sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db] sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db, live-deliveries-db, live-sessions-db]
@@ -14,6 +14,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db, live-executions-db
# Scheduled Tasks # Scheduled Tasks
## Count ## Count
- **Current as of 2026-11-10 (window 2026-11-09; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-10 00:0x–01:0x AWST):** **29 jobs (26 enabled / 3 disabled** — `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`), fifth consecutive audit at the same count. **2026-11-09 settled: 174 rows — 173 `completed` / 0 `failed` / 1 `running`** (this audit) — the **first `failed`-free day since 2026-11-05** (11-06: 1 failed, 11-07: 3, 11-08: 1). Deliveries **8/8 `delivered`**, 0 `for_failure` (same as 11-08). **Unclosed incident rows 3 → 0** (134 rows total): the whole stocktake cluster — `1a8e90_628d914f1f97` (11-07T04:29:54, *"Restart-safe cron worker exited with status 1 after adopting…"*), `1a8e90_2f370ef764f5` (11-07T05:02:21, *"…lenovo… lost its…"*), `1a8e90_a9cad99cb150` (11-08T04:35:38, *"No reply: the turn stopped while a tool…"*) — **closed 2026-11-09T12:10:47.125205, all three at once, the instant the overrunning run finished**; none ever alerted. The most recent incident activity is now the `85edf6772bca` row `85edf6_0877ca1e0eac` (*"Cron execution unverified"*, alerted 2026-11-08T07:23:07, closed 08:06:54). **The stocktake's `failure_streak` 3 → 0 and `last_status ok` are cosmetic:** `1a8e90ea2271` ran **03:00:43 → 12:10:47 = 9 h 10 m** (its 11-07 and 11-08 runs died at 2 h 02 m and 1 h 35 m), closed `completed` with delivery `suppressed`, and landed **zero edits and no commit** — the run's own output file reads `RUN INCOMPLETE`, tool-iteration budget exhausted inside verification. **Third consecutive day the stocktake's work did not land** (11-07, 11-08, 11-09). → [[incidents/2026-11-09-stocktake-completes-with-zero-edits]]. **The Job Radar landed nothing on 11-09 either:** Stage 1 (`dca8482e4f76`) ran **07:00:48 → 16:10:12 = 9 h 09 m** and never wrote `radar-ref/handoff/2026-11-09.json` (newest handoff on disk: **2026-11-06**); both Stage-2 slots (`5fc292ff0f3b`, 11:30:38→12:10:07 and 14:30:34→14:37:27) fired **before** Stage 1 finished and declined silently (`suppressed`). → [[incidents/2026-11-09-job-radar-stage1-stall-silent-noop]]. **No other 11-09 run exceeded 74 min** (stocktake 550 min, Stage 1 549 min, then 74 / 40 / 11 min), and the 11-08 `pre_tool_call` hook cascade did not recur — see the follow-up on [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. The stocktake remains pinned to `custom:omni-101` / **`free-stack`**, a combo recorded broken on **2026-08-14**; this audit itself runs `custom:omni-101` / `deepseek-paid-smart`.
- **Current as of 2026-11-08 (window 2026-11-07; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-09 00:0x-00:1x AWST):** **29 jobs (26 enabled / 3 disabled** — `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; the count and split are unchanged across **six** consecutive audits, but fleet `updated_at` **moved** to **2026-11-09T00:06:50+08:00** — treat that as a run-state stamp, **not** a definition change, because no job was added, retired, disabled or repinned: re-verified every field on the nine jobs looked at and the enabled/disabled split is identical). **11-08 settled: 169 rows across 16 jobs — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33). The **only** failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08 (1 h 34 m 57 s)**, and it is a **new terminal signature for that job**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` — the previous two red days both read *"lost its durable fire claim ownership"*. **`failure_streak` is now 3** — **three consecutive failed days**. *Working hypothesis, not proven:* the `conduit_push` `pre_tool_call` hook that was **timing out at 30 s 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03** holds a tool result open in exactly the way this error describes, but the string does not name the hook → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. The hole in the day was **the hook, not the schedule**: *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)**, *FreshRSS Midday Briefing* `ecc1fc13655a` **12:00:50 → 14:10:43 (2 h 09 m 53 s)**, both delivered. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`** (11-07 was 7 / 7). **Unclosed incident rows 2 → 3** — a **third** silent `detected` row appeared (`1a8e90_a9cad99cb150`, `alerted_at: null`), and the two 11-07 stocktake rows are still `detected`; **the alerting path is not the fault** (the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window — its `database is locked` admission, closed when the run completed). Nothing added, retired, disabled or repinned. - **Current as of 2026-11-08 (window 2026-11-07; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db`, read 2026-11-09 00:0x-00:1x AWST):** **29 jobs (26 enabled / 3 disabled** — `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; the count and split are unchanged across **six** consecutive audits, but fleet `updated_at` **moved** to **2026-11-09T00:06:50+08:00** — treat that as a run-state stamp, **not** a definition change, because no job was added, retired, disabled or repinned: re-verified every field on the nine jobs looked at and the enabled/disabled split is identical). **11-08 settled: 169 rows across 16 jobs — 168 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:33). The **only** failure is **`1a8e90ea2271` (wiki-stocktake-maintenance), 03:01:11 → 04:36:08 (1 h 34 m 57 s)**, and it is a **new terminal signature for that job**: `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model produced no follow-up text.` — the previous two red days both read *"lost its durable fire claim ownership"*. **`failure_streak` is now 3** — **three consecutive failed days**. *Working hypothesis, not proven:* the `conduit_push` `pre_tool_call` hook that was **timing out at 30 s 58 times between 2026-11-07 14:14:03 and 2026-11-08 08:21:03** holds a tool result open in exactly the way this error describes, but the string does not name the hook → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]. The hole in the day was **the hook, not the schedule**: *Mnemosyne Health Monitor* ran **00:01:20 → 08:29:05 (8 h 27 m)** and **12:00:51 → 18:00:42 (5 h 59 m 51 s)**, *FreshRSS Midday Briefing* `ecc1fc13655a` **12:00:50 → 14:10:43 (2 h 09 m 53 s)**, both delivered. Deliveries **8 rows / 8 `delivered` / 0 `for_failure`** (11-07 was 7 / 7). **Unclosed incident rows 2 → 3** — a **third** silent `detected` row appeared (`1a8e90_a9cad99cb150`, `alerted_at: null`), and the two 11-07 stocktake rows are still `detected`; **the alerting path is not the fault** (the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted **07:23:07** and `resolved` **08:06:54** in the same window — its `database is locked` admission, closed when the run completed). Nothing added, retired, disabled or repinned.
- **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned. - **Current as of 2026-11-07 (window 2026-11-06; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-07 00:3x-00:5x AWST):** **29 jobs (26 enabled / 3 disabled** - `Important-mail monitor`, `Rhino AgentMail Inbox Watcher`, `AgentMail Rhino Inbox Poll`; count and split unchanged across five consecutive audits, fleet `updated_at` **2026-11-07T00:39:46+08:00**). **11-06 settled: 172 rows, 16 distinct jobs - 170 `completed` / 1 `failed` / 1 `running`** (this audit, 23:59:31). The single `failed` row is a **new, non-restart failure signature**: the Job Radar's **11:30 Stage-2 slot** (`95347fa5…`) was **discarded as stale** at **13:26:44** on `Fire claim ownership lost; stale result was discarded.` because a **`direct` fire of the same job, started 08:27:09, was still holding the claim** - that fire completed **13:26:22** and delivered, and the ledger records **no gateway restart** in the window -> [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]. Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**; the radar still landed the day in full at 14:30 (`MISSING_COVER_LETTER_COUNT 0`, 4 same-day strong rows, 4 cover letters, 20 non-empty artifacts, *Run status: COMPLETE*). Fleet non-clean `last_status` unchanged at **1** (`066bb5191280`, `delivery_failed`, streak 0). **Unclosed incident rows 0 -> 0** - no ledger row has `last_seen_at` on or after 2026-11-06 (newest activity remains the 2026-11-05T12:36:17 closures), so **the discarded run raised no incident row and no alert**; [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] remains the one open page. The stocktake's overrun streak **fully unwound**: `1a8e90ea2271` ran **03:00:32 -> 04:11:35 (1 h 11 m 03 s)** against **7 h 40 m -> 5 h 41 m -> 3 h 02 m** on 11-03/11-04/11-05. Nothing added, retired, disabled or repinned.
- **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]. - **Current as of 2026-11-06 (window 2026-11-05; live `jobs.json` + `executions.db` + `cron_incidents` + `deliveries.db` + `sessions`, read 2026-11-06 00:1x-00:4x AWST):** **29 jobs (26 enabled / 3 disabled)** - fourth consecutive audit at the same count, same three disabled jobs, no add/retire/repin/re-enable. **11-05 settled: 175 rows, 18 distinct jobs - 173 `completed` / 2 `running` / 0 `failed`** - the **first zero-`failed` ledger day since 2026-11-02**. Both `running` rows are explained and neither is lost work: `ea79a2107e36` (**Mnemosyne Health Monitor**, 12:00:45 slot) was still in flight **12 h+** with no delivery at audit time, and `cbfe434a3b5e` is **this audit** (23:59:31). Deliveries **10 rows / 10 `delivered` / 0 `for_failure`**. Fleet non-clean `last_status` **3 -> 1** (only `066bb5191280`, **Uber Eats Monthly Spend Report**, `delivery_failed`, streak 0). **Unclosed incident rows 3 -> 0** - all three closed in-window: `85edf6_c52a11a62044` (*Tech & AI Newsletter Digest*, `unknown`) at **07:07:42** and both Job Radar rows (`dca848_dfd4bea84f2a` `timeout` + `dca848_3b1bb1e9b003` `unknown`) at **12:36:17**. That empties the ledger for the first time since 2026-11-02 and closes out [[incidents/2026-11-04-upstream-stream-failure-0700-cohort]] and [[incidents/2026-11-03-job-radar-stream-active-timeout]] in full. New this window: a **live-secret exposure** -> [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].