audit 2026-10-18: new incident page (omniroute-1 503 resource pressure, 9 calls / 7 jobs), fleet re-verified 28 jobs / 25 enabled / 8 non-clean, Mosquito Tracker removed, radar bot token dead
This commit is contained in:
@@ -780,3 +780,68 @@ a second provider pin on a subset of jobs, or router-side failover. **Not applie
|
||||
decision, not an audit one.
|
||||
|
||||
No secrets written; host names, IPs, VMIDs and job/incident IDs only.
|
||||
|
||||
## [2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead
|
||||
|
||||
**Window:** cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran
|
||||
2026-10-18 00:31 → 00:5x AWST. **Sessions reviewed:** 30, **all `cron`, zero interactive** — fifth
|
||||
consecutive window with no interactive session, so every finding below is from live system state, not
|
||||
conversation.
|
||||
|
||||
### Summary
|
||||
- **`omniroute-1` (`custom:omni-101`) has a second failure mode: 503 load-shedding.** **9 model calls across
|
||||
7 jobs** died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry
|
||||
shortly.` between **10-16 08:00:37** and **10-17 03:00:28** (~19 h). The control plane stayed up the whole
|
||||
time (`/api/health` `200 {"status":"ok"}`, `/v1/models` `401` — re-probed 10-18 00:33), so a green health
|
||||
endpoint is not evidence that the model path is serving. This is **not** a repeat of the 10-14 storm:
|
||||
that was reachability (`Connection error.`, Errno 111), this was the router serving and refusing under
|
||||
load. → `incidents/2026-10-16-omniroute-503-resource-pressure` (new).
|
||||
- **The burst ended on its own and the fleet is green again.** `10-16 = 165 completed / 8 failed`,
|
||||
`10-17 = 162 / 5` (all five before 03:01), `10-18` so far `7 / 0`; **no failure row of any kind since
|
||||
10-17 03:00**. Every affected job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01,
|
||||
`08eb86f02d37` 10-17 08:31, `ecc1fc13655a` + `ea79a2107e36` both 10-17 12:02, each auto-closing its own
|
||||
incident. **No change was made on the node**; nothing was restarted or repinned.
|
||||
- **Mosquito Tracker removed — fleet 29 → 28 (`1b31187397a5`).** Absent from `jobs.json`
|
||||
(`updated_at 2026-10-18T00:36:17+08:00`), last execution 10-16 15:00 (`completed`, delivered). The three
|
||||
disabled jobs are unchanged, so **25 enabled / 3 disabled**. Non-clean `last_status` **12 → 8**; only
|
||||
`1b23c042e17d` (`error`, streak 2, next slot Fri 10-23 08:00) is structurally stuck.
|
||||
- **The watchdog double-counts.** `96eb95ce12f2` exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 +
|
||||
00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one
|
||||
sentinel. Read failures per job, not per row.
|
||||
- **Second consecutive stocktake hole, again left alone deliberately.** The 10-17 03:00 slot failed, so
|
||||
`raw/stocktake/freshness-20261017.txt` is missing and `systems/skills-index.md` is stamped 10-16;
|
||||
10-15 is missing too (storm). The **10-18 03:00** slot closes it — no manual regeneration, because a
|
||||
hand-run would race the scheduled one.
|
||||
- **GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in its `config.json`
|
||||
returns **HTTP 401** on `getMe` (probe 10-18 00:35); the job's GitHub PAT still returns **200**, so only
|
||||
the delivery credential is gone. The cron row stays green (`last_status` from the 10-12 09:00 run,
|
||||
`last_run_at 2026-10-12T09:00:12`). Next exercise **Mon 2026-10-19 09:00**. **Not repaired** — a new bot
|
||||
token is Anthony's call, and the secret is not written here.
|
||||
- **Incidents:** **9 opened, 6 closed** in-window; open **12 → 8** (5 `alerted` / 3 `detected`). The ledger
|
||||
files **7 of the 9** 503 rows `failure_type: unknown` despite the explicit message (the two script-exit
|
||||
rows are typed `script`), so `failure_type` still cannot be used to triage.
|
||||
- **Mnemosyne:** both monitors rode the burst (`ea79a2107e36` 10-17 00:00, `b63ed5817212` 10-16 12:03),
|
||||
both clean on every slot since — including 10-18 00:00 (`delivery_outcome: suppressed`). **No fresh
|
||||
coverage figures exist** (they report to Telegram, not the ledger); **last explicit read stays
|
||||
2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %)**, above the 95 % reindex
|
||||
threshold, so no reindex is due.
|
||||
|
||||
**Pages touched:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (**new**);
|
||||
`systems/scheduled-tasks.md` (frontmatter + Count bullet + stocktake row + Known Issues entry);
|
||||
`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter +
|
||||
new Re-verified section); `current-state.md` (frontmatter + `Updated:` line + new top section);
|
||||
`index.md` (`Last updated`, new incident entry, scheduled-tasks bullet, skills-index note);
|
||||
`log.md` (this entry). **Nothing removed or archived** — no page became obsolete in the window.
|
||||
|
||||
**Bridge housekeeping:** the shared pages are byte-identical between the authoring tree
|
||||
(`/home/hermes/workspace/agent-estate-wiki`) and the vault copy (`/home/hermes/wiki/agent-estate-wiki`,
|
||||
pages nested under `wiki/`), so this run mirrored the touched pages into the vault *after* writing the
|
||||
authoring tree, and left the vault-only pages (`systems/ignis.md`, `systems/nextcloud.md`, three
|
||||
`incidents/` pages, `raw/stocktake/`, `agents/`) untouched. No deletions, no one-way sync.
|
||||
|
||||
**Durable fix still open (needs Anthony's call):** the single-node topology — all 18 model-calling jobs on
|
||||
`custom:omni-101` with no failover, now seen failing two different ways in five days. Options remain a
|
||||
second provider pin on a subset of jobs, or router-side failover. **Not applied** — a routing decision, not
|
||||
an audit one.
|
||||
|
||||
No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only.
|
||||
|
||||
Reference in New Issue
Block a user