- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day - burst-2 table names/schedules/aliases corrected from live jobs.json - non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated) - 2026-10-17 tail corrected 162/5 -> 163/5
14 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts | incident | active | 2026-10-18 | 2026-10-20 | 2026-10-20 | 2026-10-27 | high |
|
|
OmniRoute omni-101 Resource-Pressure 503s — Recurring Cron Failure Bursts (burst 1: 2026-10-16 → 10-17; burst 2: 2026-10-19)
Status: ACTIVE — this is a recurrence, not a closed incident. The failure mode came back twice in four days (10-16 → 10-17 and 10-19) with an identical signature on an identical node, and the exact same 9-model-call blast radius each time. The node was healthy before, between and after both bursts.
Burst 1: first 503 failure row 2026-10-16 08:00:37, last 2026-10-17 03:00:28 AWST — 19 hours.
10-16 = 165 completed / 8 failed, 10-17 = 163 / 5 (all five pre-03:01).
Burst 2: 2026-10-19, two clusters — 06:01 (single job) and 07:01 → 12:01 (six slots across seven
jobs). 10-19 = 159 completed / 13 failed against the previous day's 168 / 1.
The node answers /api/health → 200 {"status":"ok"} and /v1/models → 401 (auth-required) — re-probed
2026-10-20 00:03 AWST, mid-recovery, still healthy.
Impact: burst 1 = 13 failed execution rows (9 genuine model-call failures across 7 jobs + 4 derivative
Mnemosyne Watchdog rows); burst 2 = 13 failed execution rows (9 genuine model-call failures across 8 jobs
- 4 derivative Mnemosyne Watchdog rows). Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean
day (10-12 = 175 / 0). Unlike the 10-14 storm, the node was up and serving throughout both bursts — model
calls failed with
503resource-pressure responses rather than TCP refusals.
What happened
Every model call on custom:omni-101 (http://100.78.63.20:20128/v1) between the 10-16 08:00 and
10-17 03:00 slots died with the identical error:
RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.
| Job | Slot(s) AWST | Rows | Outcome |
|---|---|---|---|
1b23c042e17d Enhanced Weekend Adventure Planner |
10-16 08:00 (Fri slot) | 1 | failed; streak 2; next slot Fri 10-23 08:00 |
08eb86f02d37 Gmail Action Digest |
10-16 08:30 | 1 | failed; cleared 10-17 08:31 (own next run) |
b63ed5817212 Mnemosyne Embedding Coverage Monitor |
10-16 12:00, 10-17 00:00 | 2 | failed; cleared 10-17 06:01 |
ecc1fc13655a FreshRSS Midday News Briefing |
10-16 12:00 | 1 | failed; cleared 10-17 12:02 |
ea79a2107e36 Mnemosyne Health Monitor |
10-16 12:00, 10-17 00:00 | 2 | failed; cleared 10-17 12:02 |
cbfe434a3b5e Wiki Audit |
10-16 23:59 | 1 | failed; incident open (detected) — the 10-17 23:59 run is the one that produced this audit |
1a8e90ea2271 Wiki Stocktake |
10-17 03:00 | 1 | failed; incident open (detected) |
96eb95ce12f2 Mnemosyne Watchdog |
10-16 12:15 + 12:30, 10-17 00:15 + 00:30 | 4 | derivative — Script exited with code 1, its stdout quoting the watchdog's own log sentinel: 🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503… |
8 rows on 10-16, 5 on 10-17. Reading per job rather than per row: 7 jobs actually failed, and four of the 13 rows are one sentinel watching another job's 503 — the same double-counting trap as the 10-14 storm (see incidents/2026-10-14-omniroute-node-connection-refused-storm, lesson 4).
Every 503 row sits on custom:omni-101 — one alias, one node, again the single pin.
No other job failed in this window: 85edf6772bca (Tech & AI) and dca8482e4f76 (Job Radar) both ran
clean through it.
Evidence
~/.hermes/cron/executions.db— the 9 model-call rows each carry the literalRuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.The 4 watchdog rows carryScript exited with code 1plus the quoting stdout above.~/.hermes/cron/executions.db:cron_incidents— 9 incidents opened in the window, 6 closed, 3 still open (1b23c0_103cbe04a79balerted;cbfe43_9b89947a046aand1a8e90_3c9c1d573cdedetected). The ledger tags 7 of the 9failure_type: unknowneven though the message was explicit (the two watchdog rows are classedscript, correctly — they are script exits); for the seven real 503s the classifier had the message in hand and still filed no type. Open incidents overall: 12 → 8 (5alerted/ 3detected).- The node itself was never down.
/api/health→200 {"status":"ok"}and/v1/models→401on100.78.63.20:20128at 2026-10-18 00:33 AWST. A 503 is a serving failure, so reachability probes (which the nightly audit and the operator both run) could not have caught this — only job-level error text did. - No node-side telemetry was retained. Nothing on the Hermes side records host CPU/RAM/queue depth at 10-16 08:00, so the burst can be described but not explained. Treat the root cause as unknown.
Recurrence — burst 2 (2026-10-19)
An identical failure returned 44 hours after burst 1 ended, on the same node, with the same error string and the same number of genuine model-call failures. Nothing on the Hermes side changed between the two bursts.
| AWST (2026-10-19) | Job (name and schedule per jobs.json) |
Slot | Result | Failure streak after |
|---|---|---|---|---|
| 06:01:05 | b63ed5817212 Mnemosyne Embedding Coverage Monitor |
0 */6 * * * |
RuntimeError: HTTP 503 … resource pressure |
1 |
| 07:01 | dca8482e4f76 Daily Marketing Job Radar |
0 7 * * 1-5 |
same | 1 |
| 07:01 | 85edf6772bca Tech & AI Newsletter Digest (Analytical) |
0 7 * * * |
same | 1 |
| 08:01 | 0e519874d280 Political News Digest (Mon/Thu) |
0 8 * * 1,4 |
same | 1 → 2 |
| 08:31 | 08eb86f02d37 Gmail Action Digest |
30 8 * * * |
same | 1 |
| 09:01 | c0a9fa314542 GitHub Trending Radar — Weekly Roundup |
0 9 * * 1 |
same | 1 |
| 12:01:16 | b63ed5817212 Mnemosyne Embedding Coverage Monitor |
0 */6 * * * |
same (2nd slot of the day) | 2 |
| 12:01 | ea79a2107e36 Mnemosyne Health Monitor |
0 */12 * * * |
same | 1 |
| 12:01 | ecc1fc13655a FreshRSS Midday News Briefing |
0 12 * * * |
same | 1 → 2 |
| 06:16, 06:31, 12:16, 12:31 | 96eb95ce12f2 Mnemosyne Watchdog |
*/15 * * * * |
Script exited with code 1, stdout quoting b63ed5817212's 503 — derivative |
1 |
9 genuine model-call failures across 8 jobs (burst 1 was 9 across 7), plus the usual 4 derivative watchdog rows = 13 failed execution rows, the same total as burst 1. Whole day: 159 completed / 13 failed (10-18 was 168 / 1; 10-17 was 163 / 5).
- Failed rows by hour: 06:xx → 3 (one model call + two derivative watchdog reds), 07:xx → 2, 08:xx → 2, 09:xx → 1, 12:xx → 5. Nothing after 12:31 failed — the 15:00 → 23:00 slots and the 10-20 00:00 slots all completed.
- Not one failure landed in the
no_agent/ script jobs' own logic. Every 503 was on anagent-mode job whose model call sits oncustom:omni-101, exactly as in burst 1. - The blast radius is now the stable signature: the same nine job/slot combinations that carry the heaviest prompts fail first, while short-prompt jobs on the same node come through. A node under load-shedding is dropping the large-context calls preferentially.
- Single pin, now verified: as of 2026-10-20 grep of
jobs.jsonshows every one of these eight jobs readingprovider: custom:omni-101,model: smart-route— no alias diversity at all. ⚠ The burst-1 note above records a wider alias mix (deepseek-paid-smart,free-IA,dark);jobs.jsonno longer matches that, so the pins may have been consolidated since 10-16. Confirm with Anthony before relying on either reading. - Three incident rows auto-closed —
96eb95_565ec1deb1c7at 12:46:11 (the Watchdog's 12:30 slot recovered once its subject job recovered),b63ed5_bbe44c89aac1at 18:01:56 (the 18:01 coverage slot succeeded) andea79a2_747b27556fd0at 2026-10-20 00:04:24 (the Health Monitor's 00:01 slot succeeded). Four new incident rows opened (dca848_867d9caa96a8,85edf6_256c5e150bc4,0e5198_ec2fac98a91d,c0a9fa_764558b5ae83) because those four jobs had never carried this error signature — the other five 503s folded into pre-existing rows, and08eb86_472c7b609c80+ecc1fc_c9976ea24553were refreshed from 2026-10-16, not reopened. Unclosed ledger rows: 13 (12alerted/ 1detected) as of 2026-10-20 00:09; 4 opened and 2 closed on 10-19. (Caution: the 10-16/10-18/10-19 entries reported12,8and7open — those figures used a narrower filter than a rawclosed_at IS NULLcount, which reads 12 for 10-19 00:20 and 13 now. Compare like with like from here on.) - Every failure streak is still 1 or 2. No job has consecutively failed enough to be suppressed, and the
10-20 00:00 slots all ran clean. Non-clean
last_statusacross the fleet moved 7 → 12 (10error, 2delivery_failed) — see systems/scheduled-tasks. - Re-probed 2026-10-20 00:03 AWST:
omniroute-1→/api/health200 {"status":"ok"},/v1/models401. Healthy, as it was during both bursts. A green health endpoint still proves nothing about the model path.
What this changes
- The 10-18 assessment — "it recovered on its own; nothing was changed on the node; monitoring" — is
superseded. A self-clearing fault that returns twice in four days with an identical blast radius is a
standing operational risk, not a closed incident. Status moved
monitoring→active. - Burst 1 left no captured host telemetry; burst 2 was likewise not instrumented while it was happening, for the second time. The "capture metrics during the burst" lesson from 10-18 was not acted on. The next occurrence must be caught live (see systems/scheduled-tasks for the probe to run).
github-trending-radar's 2026-10-19 09:01 slot — its scheduled Monday run (0 9 * * 1) — was lost to the 503, so this week's roundup never went out. Its revoked Telegram delivery token (recorded in systems/scheduled-tasks Known Issues and current-state) remains untested for a second week, because the job has not reached its own code on a Monday since 2026-10-12. The next Monday slot (2026-10-26) is the first real test.
Why this is not a repeat of the 10-14 storm
| 2026-10-14 → 10-15 | 2026-10-16 → 10-17 | |
|---|---|---|
| Failure class | ConnectError: [Errno 111] Connection refused |
HTTP 503 resource pressure |
| Node state | TCP-unreachable for ~34 h | up and answering probes throughout |
| Blast radius | 12 jobs, 24 model-call failures, every call for 34 h | 7 jobs, 9 model-call failures, bursty, self-cleared |
| Recovery | manual node recovery | none needed — cleared on its own |
Both are the same single-node topology problem: all enabled model-calling cron jobs ride
custom:omni-101 with no failover, three days apart from the previous outage.
Consequences
- One stocktake hole:
raw/stocktake/freshness-20261017.txtwas not written andsystems/skills-index.mdstays stamped 2026-10-16 (the 10-16 03:00 slot succeeded, sofreshness-20261016.txtexists). The 2026-10-18 03:00 slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. A failure there would be the second consecutive miss. - The Wiki Audit lost its 10-16 slot, so no wiki pass covered the 10-14→10-17 span until this run.
- Stale flags clear only at the next success, and some slots are a week apart.
1b23c042e17dis a Friday job: it will sit atlast_status: error, streak 2, with an open incident until Fri 2026-10-23 08:00. On a weekly cadence a one-slot failure reads as a week-long outage.
Lessons
- Same topology, second failure mode in three days. A 503 from a live node cannot be detected by reachability probes; only per-job error text distinguishes "node down" from "node overloaded". Both classes gate the whole scheduled estate.
- Watchdog rows inflate the count — read per job. 4 of the 13 rows are one log sentinel reacting to another job's failure, not independent faults.
- The incident classifier drops the failure type. All seven genuine 503 incidents are filed
failure_type: unknowndespite an explicit message in the execution row (only the two script-exit watchdog incidents get a type), sofailure_typecannot be used to triage — filter on the error text. - A 503 burst leaves no evidence behind. No host telemetry is captured at failure time, so the "resource pressure" cause is taken on the router's word. If this recurs, capture the node's host metrics during the burst, not afterwards.
- A self-clearing burst is not a resolved incident. Burst 1 closed as "monitoring" on 10-18 and the
identical fault returned within 44 hours. Recurrence with an identical signature and an identical blast
radius is the signal that a fault is systemic — the second occurrence, not the first, is the one that
should change a page's
status. Two occurrences, same shape, four days apart, on a topology with no failover: treat the next one as expected.
Open item (needs Anthony's call, unchanged from 10-14)
The durable fix is the same one flagged in incidents/2026-10-14-omniroute-node-connection-refused-storm:
a second provider pin on a subset of jobs, or router-side failover between omniroute-1 and CT600
omniroute. Not applied — that is a routing decision, not an audit one. The 10-19 recurrence raises the
cost of leaving it open: two of the last four days took double-digit failures from this one node, and every
model-calling cron job still has exactly one route.
Related
- incidents/2026-10-14-omniroute-node-connection-refused-storm — the TCP-refusal storm, two days earlier, same node.
- incidents/2026-10-09-cron-provider-timeout-storm — the first multi-day storm from this topology.
- systems/omniroute — node layout, providers and pins.
- systems/scheduled-tasks — per-window fleet tally and incident ledger.