audit 2026-10-18: new incident page (omniroute-1 503 resource pressure, 9 calls / 7 jobs), fleet re-verified 28 jobs / 25 enabled / 8 non-clean, Mosquito Tracker removed, radar bot token dead

This commit is contained in:
hermes-audit
2026-10-18 00:41:38 +08:00
parent 76de919e6a
commit 4137dd6945
7 changed files with 276 additions and 23 deletions
+65
View File
@@ -780,3 +780,68 @@ a second provider pin on a subset of jobs, or router-side failover. **Not applie
decision, not an audit one.
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
## [2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead
**Window:** cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran
2026-10-18 00:31 → 00:5x AWST. **Sessions reviewed:** 30, **all `cron`, zero interactive** — fifth
consecutive window with no interactive session, so every finding below is from live system state, not
conversation.
### Summary
- **`omniroute-1` (`custom:omni-101`) has a second failure mode: 503 load-shedding.** **9 model calls across
7 jobs** died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry
shortly.` between **10-16 08:00:37** and **10-17 03:00:28** (~19 h). The control plane stayed up the whole
time (`/api/health` `200 {"status":"ok"}`, `/v1/models` `401` — re-probed 10-18 00:33), so a green health
endpoint is not evidence that the model path is serving. This is **not** a repeat of the 10-14 storm:
that was reachability (`Connection error.`, Errno 111), this was the router serving and refusing under
load. → `incidents/2026-10-16-omniroute-503-resource-pressure` (new).
- **The burst ended on its own and the fleet is green again.** `10-16 = 165 completed / 8 failed`,
`10-17 = 162 / 5` (all five before 03:01), `10-18` so far `7 / 0`; **no failure row of any kind since
10-17 03:00**. Every affected job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01,
`08eb86f02d37` 10-17 08:31, `ecc1fc13655a` + `ea79a2107e36` both 10-17 12:02, each auto-closing its own
incident. **No change was made on the node**; nothing was restarted or repinned.
- **Mosquito Tracker removed — fleet 29 → 28 (`1b31187397a5`).** Absent from `jobs.json`
(`updated_at 2026-10-18T00:36:17+08:00`), last execution 10-16 15:00 (`completed`, delivered). The three
disabled jobs are unchanged, so **25 enabled / 3 disabled**. Non-clean `last_status` **12 → 8**; only
`1b23c042e17d` (`error`, streak 2, next slot Fri 10-23 08:00) is structurally stuck.
- **The watchdog double-counts.** `96eb95ce12f2` exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 +
00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one
sentinel. Read failures per job, not per row.
- **Second consecutive stocktake hole, again left alone deliberately.** The 10-17 03:00 slot failed, so
`raw/stocktake/freshness-20261017.txt` is missing and `systems/skills-index.md` is stamped 10-16;
10-15 is missing too (storm). The **10-18 03:00** slot closes it — no manual regeneration, because a
hand-run would race the scheduled one.
- **GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in its `config.json`
returns **HTTP 401** on `getMe` (probe 10-18 00:35); the job's GitHub PAT still returns **200**, so only
the delivery credential is gone. The cron row stays green (`last_status` from the 10-12 09:00 run,
`last_run_at 2026-10-12T09:00:12`). Next exercise **Mon 2026-10-19 09:00**. **Not repaired** — a new bot
token is Anthony's call, and the secret is not written here.
- **Incidents:** **9 opened, 6 closed** in-window; open **12 → 8** (5 `alerted` / 3 `detected`). The ledger
files **7 of the 9** 503 rows `failure_type: unknown` despite the explicit message (the two script-exit
rows are typed `script`), so `failure_type` still cannot be used to triage.
- **Mnemosyne:** both monitors rode the burst (`ea79a2107e36` 10-17 00:00, `b63ed5817212` 10-16 12:03),
both clean on every slot since — including 10-18 00:00 (`delivery_outcome: suppressed`). **No fresh
coverage figures exist** (they report to Telegram, not the ledger); **last explicit read stays
2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %)**, above the 95 % reindex
threshold, so no reindex is due.
**Pages touched:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (**new**);
`systems/scheduled-tasks.md` (frontmatter + Count bullet + stocktake row + Known Issues entry);
`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
new Re-verified section); `current-state.md` (frontmatter + `Updated:` line + new top section);
`index.md` (`Last updated`, new incident entry, scheduled-tasks bullet, skills-index note);
`log.md` (this entry). **Nothing removed or archived** — no page became obsolete in the window.
**Bridge housekeeping:** the shared pages are byte-identical between the authoring tree
(`/home/hermes/workspace/agent-estate-wiki`) and the vault copy (`/home/hermes/wiki/agent-estate-wiki`,
pages nested under `wiki/`), so this run mirrored the touched pages into the vault *after* writing the
authoring tree, and left the vault-only pages (`systems/ignis.md`, `systems/nextcloud.md`, three
`incidents/` pages, `raw/stocktake/`, `agents/`) untouched. No deletions, no one-way sync.
**Durable fix still open (needs Anthony's call):** the single-node topology — all 18 model-calling jobs on
`custom:omni-101` with no failover, now seen failing two different ways in five days. Options remain a
second provider pin on a subset of jobs, or router-side failover. **Not applied** — a routing decision, not
an audit one.
No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only.