Files
agent-estate-wiki/incidents/2026-10-16-omniroute-503-resource-pressure.md
T

17 KiB
Raw Blame History

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts incident monitoring 2026-10-18 2026-10-21 2026-10-21 2026-10-28 high
cron
router
provider
infrastructure
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-jobs-json
live-probe

OmniRoute omni-101 Resource-Pressure 503s — Recurring Cron Failure Bursts (burst 1: 2026-10-16 → 10-17; burst 2: 2026-10-19)

Status: MONITORING as of 2026-10-21 — not closed. The failure mode came back twice in four days (10-16 → 10-17 and 10-19) with an identical signature on an identical node, and the exact same 9-model-call blast radius each time. The node was healthy before, between and after both bursts.

2026-10-21 update — burst 2 is over; the page returns to monitoring on the 18–24 h rule it set itself. 2026-10-20 ran 168 completed / 0 failed (plus one unknown orphan — see incidents/2026-10-20-gateway-unclean-exit-orphaned-run), the first zero-failure day since 2026-10-12, and no execution row carries the resource-pressure string (the last such failure remains 2026-10-19). 4 incident rows closed on 10-20 (dca848_867d9caa96a8, 85edf6_256c5e150bc4, ecc1fc_c9976ea24553, ecc1fc_21a72aa11cc4), taking unclosed rows 13 → 9 (8 alerted / 1 detected). Nothing was changed on the node — the recovery is self-clearing, exactly as in burst 1, and the underlying exposure is untouched: the faults are still invisible to /api/health, still single-node, and a fresh burst would still take out most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing risk, not a resolved incident.

Burst 1: first 503 failure row 2026-10-16 08:00:37, last 2026-10-17 03:00:28 AWST — 19 hours. 10-16 = 165 completed / 8 failed, 10-17 = 163 / 5 (all five pre-03:01). Burst 2: 2026-10-19, two clusters — 06:01 (single job) and 07:01 → 12:01 (six slots across seven jobs). 10-19 = 159 completed / 13 failed against the previous day's 168 / 1. The node answers /api/health → 200 {"status":"ok"} and /v1/models → 401 (auth-required) — re-probed 2026-10-20 00:03 AWST, mid-recovery, still healthy. Impact: burst 1 = 13 failed execution rows (9 genuine model-call failures across 7 jobs + 4 derivative Mnemosyne Watchdog rows); burst 2 = 13 failed execution rows (9 genuine model-call failures across 8 jobs

  • 4 derivative Mnemosyne Watchdog rows). Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean day (10-12 = 175 / 0). Unlike the 10-14 storm, the node was up and serving throughout both bursts — model calls failed with 503 resource-pressure responses rather than TCP refusals.

What happened

Every model call on custom:omni-101 (http://100.78.63.20:20128/v1) between the 10-16 08:00 and 10-17 03:00 slots died with the identical error:

RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.
Job Slot(s) AWST Rows Outcome
1b23c042e17d Enhanced Weekend Adventure Planner 10-16 08:00 (Fri slot) 1 failed; streak 2; next slot Fri 10-23 08:00
08eb86f02d37 Gmail Action Digest 10-16 08:30 1 failed; cleared 10-17 08:31 (own next run)
b63ed5817212 Mnemosyne Embedding Coverage Monitor 10-16 12:00, 10-17 00:00 2 failed; cleared 10-17 06:01
ecc1fc13655a FreshRSS Midday News Briefing 10-16 12:00 1 failed; cleared 10-17 12:02
ea79a2107e36 Mnemosyne Health Monitor 10-16 12:00, 10-17 00:00 2 failed; cleared 10-17 12:02
cbfe434a3b5e Wiki Audit 10-16 23:59 1 failed; incident open (detected) — the 10-17 23:59 run is the one that produced this audit
1a8e90ea2271 Wiki Stocktake 10-17 03:00 1 failed; incident open (detected)
96eb95ce12f2 Mnemosyne Watchdog 10-16 12:15 + 12:30, 10-17 00:15 + 00:30 4 derivative — Script exited with code 1, its stdout quoting the watchdog's own log sentinel: 🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…

8 rows on 10-16, 5 on 10-17. Reading per job rather than per row: 7 jobs actually failed, and four of the 13 rows are one sentinel watching another job's 503 — the same double-counting trap as the 10-14 storm (see incidents/2026-10-14-omniroute-node-connection-refused-storm, lesson 4).

Every 503 row sits on custom:omni-101 — one alias, one node, again the single pin. No other job failed in this window: 85edf6772bca (Tech & AI) and dca8482e4f76 (Job Radar) both ran clean through it.

Evidence

  • ~/.hermes/cron/executions.db — the 9 model-call rows each carry the literal RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. The 4 watchdog rows carry Script exited with code 1 plus the quoting stdout above.
  • ~/.hermes/cron/executions.db:cron_incidents — 9 incidents opened in the window, 6 closed, 3 still open (1b23c0_103cbe04a79b alerted; cbfe43_9b89947a046a and 1a8e90_3c9c1d573cde detected). The ledger tags 7 of the 9 failure_type: unknown even though the message was explicit (the two watchdog rows are classed script, correctly — they are script exits); for the seven real 503s the classifier had the message in hand and still filed no type. Open incidents overall: 12 → 8 (5 alerted / 3 detected).
  • The node itself was never down. /api/health → 200 {"status":"ok"} and /v1/models → 401 on 100.78.63.20:20128 at 2026-10-18 00:33 AWST. A 503 is a serving failure, so reachability probes (which the nightly audit and the operator both run) could not have caught this — only job-level error text did.
  • No node-side telemetry was retained. Nothing on the Hermes side records host CPU/RAM/queue depth at 10-16 08:00, so the burst can be described but not explained. Treat the root cause as unknown.

Recurrence — burst 2 (2026-10-19)

An identical failure returned 44 hours after burst 1 ended, on the same node, with the same error string and the same number of genuine model-call failures. Nothing on the Hermes side changed between the two bursts.

AWST (2026-10-19) Job (name and schedule per jobs.json) Slot Result Failure streak after
06:01:05 b63ed5817212 Mnemosyne Embedding Coverage Monitor 0 */6 * * * RuntimeError: HTTP 503 … resource pressure 1
07:01 dca8482e4f76 Daily Marketing Job Radar 0 7 * * 1-5 same 1
07:01 85edf6772bca Tech & AI Newsletter Digest (Analytical) 0 7 * * * same 1
08:01 0e519874d280 Political News Digest (Mon/Thu) 0 8 * * 1,4 same 1 → 2
08:31 08eb86f02d37 Gmail Action Digest 30 8 * * * same 1
09:01 c0a9fa314542 GitHub Trending Radar — Weekly Roundup 0 9 * * 1 same 1
12:01:16 b63ed5817212 Mnemosyne Embedding Coverage Monitor 0 */6 * * * same (2nd slot of the day) 2
12:01 ea79a2107e36 Mnemosyne Health Monitor 0 */12 * * * same 1
12:01 ecc1fc13655a FreshRSS Midday News Briefing 0 12 * * * same 1 → 2
06:16, 06:31, 12:16, 12:31 96eb95ce12f2 Mnemosyne Watchdog */15 * * * * Script exited with code 1, stdout quoting b63ed5817212's 503 — derivative 1

9 genuine model-call failures across 8 jobs (burst 1 was 9 across 7), plus the usual 4 derivative watchdog rows = 13 failed execution rows, the same total as burst 1. Whole day: 159 completed / 13 failed (10-18 was 168 / 1; 10-17 was 163 / 5).

  • Failed rows by hour: 06:xx → 3 (one model call + two derivative watchdog reds), 07:xx → 2, 08:xx → 2, 09:xx → 1, 12:xx → 5. Nothing after 12:31 failed — the 15:00 → 23:00 slots and the 10-20 00:00 slots all completed.
  • Not one failure landed in the no_agent / script jobs' own logic. Every 503 was on an agent-mode job whose model call sits on custom:omni-101, exactly as in burst 1.
  • The blast radius is now the stable signature: the same nine job/slot combinations that carry the heaviest prompts fail first, while short-prompt jobs on the same node come through. A node under load-shedding is dropping the large-context calls preferentially.
  • Single pin, now verified: as of 2026-10-20 grep of jobs.json shows every one of these eight jobs reading provider: custom:omni-101, model: smart-route — no alias diversity at all. ⚠ The burst-1 note above records a wider alias mix (deepseek-paid-smart, free-IA, dark); jobs.json no longer matches that, so the pins may have been consolidated since 10-16. Confirm with Anthony before relying on either reading.
  • Three incident rows auto-closed — 96eb95_565ec1deb1c7 at 12:46:11 (the Watchdog's 12:30 slot recovered once its subject job recovered), b63ed5_bbe44c89aac1 at 18:01:56 (the 18:01 coverage slot succeeded) and ea79a2_747b27556fd0 at 2026-10-20 00:04:24 (the Health Monitor's 00:01 slot succeeded). Four new incident rows opened (dca848_867d9caa96a8, 85edf6_256c5e150bc4, 0e5198_ec2fac98a91d, c0a9fa_764558b5ae83) because those four jobs had never carried this error signature — the other five 503s folded into pre-existing rows, and 08eb86_472c7b609c80 + ecc1fc_c9976ea24553 were refreshed from 2026-10-16, not reopened. Unclosed ledger rows: 13 (12 alerted / 1 detected) as of 2026-10-20 00:09; 4 opened and 2 closed on 10-19. (Caution: the 10-16/10-18/10-19 entries reported 12, 8 and 7 open — those figures used a narrower filter than a raw closed_at IS NULL count, which reads 12 for 10-19 00:20 and 13 now. Compare like with like from here on.)
  • Every failure streak is still 1 or 2. No job has consecutively failed enough to be suppressed, and the 10-20 00:00 slots all ran clean. Non-clean last_status across the fleet moved 7 → 12 (10 error, 2 delivery_failed) — see systems/scheduled-tasks.
  • Re-probed 2026-10-20 00:03 AWST: omniroute-1 → /api/health 200 {"status":"ok"}, /v1/models 401. Healthy, as it was during both bursts. A green health endpoint still proves nothing about the model path.

What this changes

  • The 10-18 assessment — "it recovered on its own; nothing was changed on the node; monitoring" — is superseded. A self-clearing fault that returns twice in four days with an identical blast radius is a standing operational risk, not a closed incident. Status moved monitoring → active.
  • Burst 1 left no captured host telemetry; burst 2 was likewise not instrumented while it was happening, for the second time. The "capture metrics during the burst" lesson from 10-18 was not acted on. The next occurrence must be caught live (see systems/scheduled-tasks for the probe to run).
  • github-trending-radar's 2026-10-19 09:01 slot — its scheduled Monday run (0 9 * * 1) — was lost to the 503, so this week's roundup never went out. Its revoked Telegram delivery token (recorded in systems/scheduled-tasks Known Issues and current-state) remains untested for a second week, because the job has not reached its own code on a Monday since 2026-10-12. The next Monday slot (2026-10-26) is the first real test.

Why this is not a repeat of the 10-14 storm

2026-10-14 → 10-15 2026-10-16 → 10-17
Failure class ConnectError: [Errno 111] Connection refused HTTP 503 resource pressure
Node state TCP-unreachable for ~34 h up and answering probes throughout
Blast radius 12 jobs, 24 model-call failures, every call for 34 h 7 jobs, 9 model-call failures, bursty, self-cleared
Recovery manual node recovery none needed — cleared on its own

Both are the same single-node topology problem: all enabled model-calling cron jobs ride custom:omni-101 with no failover, three days apart from the previous outage.

Consequences

  • One stocktake hole: raw/stocktake/freshness-20261017.txt was not written and systems/skills-index.md stays stamped 2026-10-16 (the 10-16 03:00 slot succeeded, so freshness-20261016.txt exists). The 2026-10-18 03:00 slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. A failure there would be the second consecutive miss.
  • The Wiki Audit lost its 10-16 slot, so no wiki pass covered the 10-14→10-17 span until this run.
  • Stale flags clear only at the next success, and some slots are a week apart. 1b23c042e17d is a Friday job: it will sit at last_status: error, streak 2, with an open incident until Fri 2026-10-23 08:00. On a weekly cadence a one-slot failure reads as a week-long outage.

Lessons

  1. Same topology, second failure mode in three days. A 503 from a live node cannot be detected by reachability probes; only per-job error text distinguishes "node down" from "node overloaded". Both classes gate the whole scheduled estate.
  2. Watchdog rows inflate the count — read per job. 4 of the 13 rows are one log sentinel reacting to another job's failure, not independent faults.
  3. The incident classifier drops the failure type. All seven genuine 503 incidents are filed failure_type: unknown despite an explicit message in the execution row (only the two script-exit watchdog incidents get a type), so failure_type cannot be used to triage — filter on the error text.
  4. A 503 burst leaves no evidence behind. No host telemetry is captured at failure time, so the "resource pressure" cause is taken on the router's word. If this recurs, capture the node's host metrics during the burst, not afterwards.
  5. A self-clearing burst is not a resolved incident. Burst 1 closed as "monitoring" on 10-18 and the identical fault returned within 44 hours. Recurrence with an identical signature and an identical blast radius is the signal that a fault is systemic — the second occurrence, not the first, is the one that should change a page's status. Two occurrences, same shape, four days apart, on a topology with no failover: treat the next one as expected.

Open item (needs Anthony's call, unchanged from 10-14)

The durable fix is the same one flagged in incidents/2026-10-14-omniroute-node-connection-refused-storm: a second provider pin on a subset of jobs, or router-side failover between omniroute-1 and CT600 omniroute. Not applied — that is a routing decision, not an audit one. The 10-19 recurrence raises the cost of leaving it open: two of the last four days took double-digit failures from this one node, and every model-calling cron job still has exactly one route.

Update 2026-10-21 (audit of the 2026-10-20 window): the burst did not resume

The 2026-10-19 burst was followed by 2026-10-20's 168 completed / 0 failed — a clean day and the first zero-failure day since 2026-10-12, with zero rows carrying the resource-pressure string and zero new incident rows opened for the whole window. The 503 fault's last recorded failure is therefore 2026-10-19, and per systems/scheduled-tasks the page status returns to monitoring (18–24 h of clean running, not a fix).

What did not change, and matters more than the clean day:

  • No failover was added. 20 of 28 jobs still pin custom:omni-101 (smart-route ×11, free-stack ×6, deepseek-paid-smart ×1) or custom:omni ×2. One quiet day on a single-route topology is luck, not margin, and this page has already been wrong once about the burst being over (it was downgraded 44 h before burst 2).
  • The 8 jobs with no model pin no longer ride the router at all. The global provider default moved to nous direct (see below), so the fleet is now split across two upstream paths for the first time — worth watching as a separate risk, since a Nous-side fault now has its own blast radius.
  • Global default changed between 2026-10-09 and 2026-10-20: provider: custom → nous (https://inference-api.nousresearch.com/v1), model.default: smart-route → meituan/longcat-2.5-preview:free. Pinned jobs are unaffected; unpinned jobs are. → systems/current-profile