audit 2026-10-18: new incident page (omniroute-1 503 resource pressure, 9 calls / 7 jobs), fleet re-verified 28 jobs / 25 enabled / 8 non-clean, Mosquito Tracker removed, radar bot token dead

This commit is contained in:
hermes-audit
2026-10-18 00:41:38 +08:00
parent 76de919e6a
commit 4137dd6945
7 changed files with 276 additions and 23 deletions
@@ -0,0 +1,119 @@
---
title: OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst
type: incident
status: monitoring
created: 2026-10-18
updated: 2026-10-18
verified_on: 2026-10-18
review_after: 2026-10-25
confidence: high
tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
---
# OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst (2026-10-16 → 2026-10-17)
**Window:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
**Status:** **Monitoring.** Every slot from the 2026-10-17 08:30 run onward completed green
(10-16 = **165 completed / 8 failed**, 10-17 = **162 / 5**, and the 10-17 failures all pre-date 03:01).
The node answers `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` (auth-required) — re-probed
2026-10-18 00:33 AWST.
**Impact:** **13 failed execution rows** — **9 genuine model-call failures across 7 jobs**, plus **4 derivative
Mnemosyne Watchdog rows**. Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean day
(10-12 = 175 / 0). Unlike the 10-14 storm, **the node was up and serving throughout** — model calls failed with
`503` resource-pressure responses rather than TCP refusals.
## What happened
Every model call on `custom:omni-101` (`http://100.78.63.20:20128/v1`) between the 10-16 08:00 and
10-17 03:00 slots died with the identical error:
```
RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.
```
| Job | Slot(s) AWST | Rows | Outcome |
|---|---|---|---|
| `1b23c042e17d` Enhanced Weekend Adventure Planner | 10-16 08:00 (Fri slot) | 1 | failed; streak **2**; next slot **Fri 10-23 08:00** |
| `08eb86f02d37` Gmail Action Digest | 10-16 08:30 | 1 | failed; cleared 10-17 08:31 (own next run) |
| `b63ed5817212` Mnemosyne Embedding Coverage Monitor | 10-16 12:00, 10-17 00:00 | 2 | failed; cleared **10-17 06:01** |
| `ecc1fc13655a` FreshRSS Midday News Briefing | 10-16 12:00 | 1 | failed; cleared 10-17 12:02 |
| `ea79a2107e36` Mnemosyne Health Monitor | 10-16 12:00, 10-17 00:00 | 2 | failed; cleared 10-17 12:02 |
| `cbfe434a3b5e` Wiki Audit | 10-16 23:59 | 1 | failed; incident **open** (`detected`) — the 10-17 23:59 run is the one that produced this audit |
| `1a8e90ea2271` Wiki Stocktake | 10-17 03:00 | 1 | failed; incident **open** (`detected`) |
| `96eb95ce12f2` Mnemosyne Watchdog | 10-16 12:15 + 12:30, 10-17 00:15 + 00:30 | 4 | **derivative** — `Script exited with code 1`, its stdout quoting the watchdog's own log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…` |
**8 rows on 10-16, 5 on 10-17.** Reading per job rather than per row: **7 jobs actually failed**, and four of
the 13 rows are one sentinel watching another job's 503 — the same double-counting trap as the 10-14 storm
(see [[incidents/2026-10-14-omniroute-node-connection-refused-storm]], lesson 4).
Every 503 row sits on **`custom:omni-101`** — one alias, one node, again the single pin.
No other job failed in this window: `85edf6772bca` (Tech & AI) and `dca8482e4f76` (Job Radar) both ran
clean through it.
## Evidence
- `~/.hermes/cron/executions.db` — the 9 model-call rows each carry the literal
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`
The 4 watchdog rows carry `Script exited with code 1` plus the quoting stdout above.
- `~/.hermes/cron/executions.db:cron_incidents` — **9 incidents opened in the window, 6 closed, 3 still open**
(`1b23c0_103cbe04a79b` `alerted`; `cbfe43_9b89947a046a` and `1a8e90_3c9c1d573cde` `detected`).
**The ledger tags 7 of the 9 `failure_type: unknown`** even though the message was explicit (the two
watchdog rows are classed `script`, correctly — they *are* script exits); for the seven real 503s the
classifier had the message in hand and still filed no type. Open incidents overall: **12 → 8**
(5 `alerted` / 3 `detected`).
- **The node itself was never down.** `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` on
`100.78.63.20:20128` at 2026-10-18 00:33 AWST. A 503 is a *serving* failure, so reachability probes
(which the nightly audit and the operator both run) could not have caught this — only job-level error text did.
- **No node-side telemetry was retained.** Nothing on the Hermes side records host CPU/RAM/queue depth at
10-16 08:00, so the burst can be described but not explained. Treat the root cause as **unknown**.
## Why this is not a repeat of the 10-14 storm
| | 2026-10-14 → 10-15 | 2026-10-16 → 10-17 |
|---|---|---|
| Failure class | `ConnectError: [Errno 111] Connection refused` | HTTP `503` resource pressure |
| Node state | TCP-unreachable for ~34 h | up and answering probes throughout |
| Blast radius | 12 jobs, 24 model-call failures, every call for 34 h | 7 jobs, 9 model-call failures, bursty, self-cleared |
| Recovery | manual node recovery | none needed — cleared on its own |
Both are the **same single-node topology problem**: all enabled model-calling cron jobs ride
`custom:omni-101` with no failover, three days apart from the previous outage.
## Consequences
- **One stocktake hole:** `raw/stocktake/freshness-20261017.txt` was not written and
`systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 03:00 slot succeeded, so
`freshness-20261016.txt` exists). The **2026-10-18 03:00 slot** closes it — nothing was regenerated by hand,
because a manual run would race the scheduled one. A failure there would be the second consecutive miss.
- **The Wiki Audit lost its 10-16 slot**, so no wiki pass covered the 10-14→10-17 span until this run.
- **Stale flags clear only at the next success, and some slots are a week apart.** `1b23c042e17d` is a Friday
job: it will sit at `last_status: error`, streak 2, with an open incident until **Fri 2026-10-23 08:00**.
On a weekly cadence a one-slot failure reads as a week-long outage.
## Lessons
1. **Same topology, second failure mode in three days.** A 503 from a *live* node cannot be detected by
reachability probes; only per-job error text distinguishes "node down" from "node overloaded". Both
classes gate the whole scheduled estate.
2. **Watchdog rows inflate the count — read per job.** 4 of the 13 rows are one log sentinel reacting to
another job's failure, not independent faults.
3. **The incident classifier drops the failure type.** All **seven** genuine 503 incidents are filed
`failure_type: unknown` despite an explicit message in the execution row (only the two script-exit
watchdog incidents get a type), so `failure_type` cannot be used to triage — filter on the error text.
4. **A 503 burst leaves no evidence behind.** No host telemetry is captured at failure time, so the "resource
pressure" cause is taken on the router's word. If this recurs, capture the node's host metrics *during* the
burst, not afterwards.
## Open item (needs Anthony's call, unchanged from 10-14)
The durable fix is the same one flagged in [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]:
a second provider pin on a subset of jobs, or router-side failover between `omniroute-1` and CT600
`omniroute`. **Not applied** — that is a routing decision, not an audit one.
## Related
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — the first multi-day storm from this topology.
- [[systems/omniroute]] — node layout, providers and pins.
- [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger.