7.6 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst | incident | monitoring | 2026-10-18 | 2026-10-18 | 2026-10-18 | 2026-10-25 | high |
|
|
OmniRoute omni-101 Resource-Pressure 503s — Cron Failure Burst (2026-10-16 → 2026-10-17)
Window: first 503 failure row 2026-10-16 08:00:37, last 2026-10-17 03:00:28 AWST — 19 hours.
Status: Monitoring. Every slot from the 2026-10-17 08:30 run onward completed green
(10-16 = 165 completed / 8 failed, 10-17 = 162 / 5, and the 10-17 failures all pre-date 03:01).
The node answers /api/health → 200 {"status":"ok"} and /v1/models → 401 (auth-required) — re-probed
2026-10-18 00:33 AWST.
Impact: 13 failed execution rows — 9 genuine model-call failures across 7 jobs, plus 4 derivative
Mnemosyne Watchdog rows. Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean day
(10-12 = 175 / 0). Unlike the 10-14 storm, the node was up and serving throughout — model calls failed with
503 resource-pressure responses rather than TCP refusals.
What happened
Every model call on custom:omni-101 (http://100.78.63.20:20128/v1) between the 10-16 08:00 and
10-17 03:00 slots died with the identical error:
RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.
| Job | Slot(s) AWST | Rows | Outcome |
|---|---|---|---|
1b23c042e17d Enhanced Weekend Adventure Planner |
10-16 08:00 (Fri slot) | 1 | failed; streak 2; next slot Fri 10-23 08:00 |
08eb86f02d37 Gmail Action Digest |
10-16 08:30 | 1 | failed; cleared 10-17 08:31 (own next run) |
b63ed5817212 Mnemosyne Embedding Coverage Monitor |
10-16 12:00, 10-17 00:00 | 2 | failed; cleared 10-17 06:01 |
ecc1fc13655a FreshRSS Midday News Briefing |
10-16 12:00 | 1 | failed; cleared 10-17 12:02 |
ea79a2107e36 Mnemosyne Health Monitor |
10-16 12:00, 10-17 00:00 | 2 | failed; cleared 10-17 12:02 |
cbfe434a3b5e Wiki Audit |
10-16 23:59 | 1 | failed; incident open (detected) — the 10-17 23:59 run is the one that produced this audit |
1a8e90ea2271 Wiki Stocktake |
10-17 03:00 | 1 | failed; incident open (detected) |
96eb95ce12f2 Mnemosyne Watchdog |
10-16 12:15 + 12:30, 10-17 00:15 + 00:30 | 4 | derivative — Script exited with code 1, its stdout quoting the watchdog's own log sentinel: 🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503… |
8 rows on 10-16, 5 on 10-17. Reading per job rather than per row: 7 jobs actually failed, and four of the 13 rows are one sentinel watching another job's 503 — the same double-counting trap as the 10-14 storm (see incidents/2026-10-14-omniroute-node-connection-refused-storm, lesson 4).
Every 503 row sits on custom:omni-101 — one alias, one node, again the single pin.
No other job failed in this window: 85edf6772bca (Tech & AI) and dca8482e4f76 (Job Radar) both ran
clean through it.
Evidence
~/.hermes/cron/executions.db— the 9 model-call rows each carry the literalRuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.The 4 watchdog rows carryScript exited with code 1plus the quoting stdout above.~/.hermes/cron/executions.db:cron_incidents— 9 incidents opened in the window, 6 closed, 3 still open (1b23c0_103cbe04a79balerted;cbfe43_9b89947a046aand1a8e90_3c9c1d573cdedetected). The ledger tags 7 of the 9failure_type: unknowneven though the message was explicit (the two watchdog rows are classedscript, correctly — they are script exits); for the seven real 503s the classifier had the message in hand and still filed no type. Open incidents overall: 12 → 8 (5alerted/ 3detected).- The node itself was never down.
/api/health→200 {"status":"ok"}and/v1/models→401on100.78.63.20:20128at 2026-10-18 00:33 AWST. A 503 is a serving failure, so reachability probes (which the nightly audit and the operator both run) could not have caught this — only job-level error text did. - No node-side telemetry was retained. Nothing on the Hermes side records host CPU/RAM/queue depth at 10-16 08:00, so the burst can be described but not explained. Treat the root cause as unknown.
Why this is not a repeat of the 10-14 storm
| 2026-10-14 → 10-15 | 2026-10-16 → 10-17 | |
|---|---|---|
| Failure class | ConnectError: [Errno 111] Connection refused |
HTTP 503 resource pressure |
| Node state | TCP-unreachable for ~34 h | up and answering probes throughout |
| Blast radius | 12 jobs, 24 model-call failures, every call for 34 h | 7 jobs, 9 model-call failures, bursty, self-cleared |
| Recovery | manual node recovery | none needed — cleared on its own |
Both are the same single-node topology problem: all enabled model-calling cron jobs ride
custom:omni-101 with no failover, three days apart from the previous outage.
Consequences
- One stocktake hole:
raw/stocktake/freshness-20261017.txtwas not written andsystems/skills-index.mdstays stamped 2026-10-16 (the 10-16 03:00 slot succeeded, sofreshness-20261016.txtexists). The 2026-10-18 03:00 slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. A failure there would be the second consecutive miss. - The Wiki Audit lost its 10-16 slot, so no wiki pass covered the 10-14→10-17 span until this run.
- Stale flags clear only at the next success, and some slots are a week apart.
1b23c042e17dis a Friday job: it will sit atlast_status: error, streak 2, with an open incident until Fri 2026-10-23 08:00. On a weekly cadence a one-slot failure reads as a week-long outage.
Lessons
- Same topology, second failure mode in three days. A 503 from a live node cannot be detected by reachability probes; only per-job error text distinguishes "node down" from "node overloaded". Both classes gate the whole scheduled estate.
- Watchdog rows inflate the count — read per job. 4 of the 13 rows are one log sentinel reacting to another job's failure, not independent faults.
- The incident classifier drops the failure type. All seven genuine 503 incidents are filed
failure_type: unknowndespite an explicit message in the execution row (only the two script-exit watchdog incidents get a type), sofailure_typecannot be used to triage — filter on the error text. - A 503 burst leaves no evidence behind. No host telemetry is captured at failure time, so the "resource pressure" cause is taken on the router's word. If this recurs, capture the node's host metrics during the burst, not afterwards.
Open item (needs Anthony's call, unchanged from 10-14)
The durable fix is the same one flagged in incidents/2026-10-14-omniroute-node-connection-refused-storm:
a second provider pin on a subset of jobs, or router-side failover between omniroute-1 and CT600
omniroute. Not applied — that is a routing decision, not an audit one.
Related
- incidents/2026-10-14-omniroute-node-connection-refused-storm — the TCP-refusal storm, two days earlier, same node.
- incidents/2026-10-09-cron-provider-timeout-storm — the first multi-day storm from this topology.
- systems/omniroute — node layout, providers and pins.
- systems/scheduled-tasks — per-window fleet tally and incident ledger.