255 lines
19 KiB
Markdown
255 lines
19 KiB
Markdown
---
|
||
title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts
|
||
type: incident
|
||
status: monitoring
|
||
created: 2026-10-18
|
||
updated: 2026-10-27
|
||
verified_on: 2026-10-27
|
||
review_after: 2026-11-03
|
||
confidence: high
|
||
tags: [cron, router, provider, infrastructure]
|
||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
|
||
---
|
||
|
||
# OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts (burst 1: 2026-10-16 → 10-17; burst 2: 2026-10-19)
|
||
|
||
**Status: `MONITORING` as of 2026-10-21 — not closed.** The failure mode came back **twice in four
|
||
days** (10-16 → 10-17 and 10-19) with an identical signature on an identical node, and the **exact same
|
||
9-model-call blast radius** each time. The node was healthy before, between and after both bursts.
|
||
|
||
> **2026-10-21 update — burst 2 is over; the page returns to `monitoring` on the 18–24 h rule it set itself.**
|
||
> **2026-10-20 ran 168 `completed` / 0 `failed`** (plus one `unknown` orphan — see
|
||
> [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), the **first zero-failure day since 2026-10-12**,
|
||
> and **no execution row carries the resource-pressure string** (the last such failure remains 2026-10-19).
|
||
> **4 incident rows closed on 10-20** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`,
|
||
> `ecc1fc_21a72aa11cc4`), taking unclosed rows **13 → 9** (8 `alerted` / 1 `detected`). **Nothing was changed on
|
||
> the node** — the recovery is self-clearing, exactly as in burst 1, and the underlying exposure is untouched:
|
||
> the faults are still invisible to `/api/health`, still single-node, and a fresh burst would still take out
|
||
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
|
||
> risk, not a resolved incident.
|
||
|
||
> **2026-10-26 update — every incident row from this failure mode is now closed.** The last open row,
|
||
> `c0a9fa_764558b5ae83` (`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.
|
||
> Retry shortly.`, job `c0a9fa314542` GitHub Trending Radar, `alerted` since 2026-10-19 09:02), **auto-closed
|
||
> 2026-10-26 09:13:00** when that job's 09:00 slot ran clean. **No execution row carries the resource-pressure
|
||
> string on 2026-10-25 or 2026-10-26.** Status stays **`monitoring`** for the same reason as 10-21: the rows are
|
||
> shut, the single-node exposure is not.
|
||
|
||
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
|
||
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
|
||
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
|
||
jobs). 10-19 = **159 completed / 13 failed** against the previous day's 168 / 1.
|
||
The node answers `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` (auth-required) — re-probed
|
||
**2026-10-20 00:03 AWST**, mid-recovery, still healthy.
|
||
**Impact:** burst 1 = **13 failed execution rows** (**9 genuine model-call failures across 7 jobs** + 4 derivative
|
||
Mnemosyne Watchdog rows); burst 2 = **13 failed execution rows** (**9 genuine model-call failures across 8 jobs**
|
||
+ 4 derivative Mnemosyne Watchdog rows). Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean
|
||
day (10-12 = 175 / 0). Unlike the 10-14 storm, **the node was up and serving throughout both bursts** — model
|
||
calls failed with `503` resource-pressure responses rather than TCP refusals.
|
||
|
||
## What happened
|
||
|
||
Every model call on `custom:omni-101` (`http://100.78.63.20:20128/v1`) between the 10-16 08:00 and
|
||
10-17 03:00 slots died with the identical error:
|
||
|
||
```
|
||
RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.
|
||
```
|
||
|
||
| Job | Slot(s) AWST | Rows | Outcome |
|
||
|---|---|---|---|
|
||
| `1b23c042e17d` Enhanced Weekend Adventure Planner | 10-16 08:00 (Fri slot) | 1 | failed; streak **2**; next slot **Fri 10-23 08:00** |
|
||
| `08eb86f02d37` Gmail Action Digest | 10-16 08:30 | 1 | failed; cleared 10-17 08:31 (own next run) |
|
||
| `b63ed5817212` Mnemosyne Embedding Coverage Monitor | 10-16 12:00, 10-17 00:00 | 2 | failed; cleared **10-17 06:01** |
|
||
| `ecc1fc13655a` FreshRSS Midday News Briefing | 10-16 12:00 | 1 | failed; cleared 10-17 12:02 |
|
||
| `ea79a2107e36` Mnemosyne Health Monitor | 10-16 12:00, 10-17 00:00 | 2 | failed; cleared 10-17 12:02 |
|
||
| `cbfe434a3b5e` Wiki Audit | 10-16 23:59 | 1 | failed; incident **open** (`detected`) — the 10-17 23:59 run is the one that produced this audit |
|
||
| `1a8e90ea2271` Wiki Stocktake | 10-17 03:00 | 1 | failed; incident **open** (`detected`) |
|
||
| `96eb95ce12f2` Mnemosyne Watchdog | 10-16 12:15 + 12:30, 10-17 00:15 + 00:30 | 4 | **derivative** — `Script exited with code 1`, its stdout quoting the watchdog's own log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…` |
|
||
|
||
**8 rows on 10-16, 5 on 10-17.** Reading per job rather than per row: **7 jobs actually failed**, and four of
|
||
the 13 rows are one sentinel watching another job's 503 — the same double-counting trap as the 10-14 storm
|
||
(see [[incidents/2026-10-14-omniroute-node-connection-refused-storm]], lesson 4).
|
||
|
||
Every 503 row sits on **`custom:omni-101`** — one alias, one node, again the single pin.
|
||
No other job failed in this window: `85edf6772bca` (Tech & AI) and `dca8482e4f76` (Job Radar) both ran
|
||
clean through it.
|
||
|
||
## Evidence
|
||
|
||
- `~/.hermes/cron/executions.db` — the 9 model-call rows each carry the literal
|
||
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`
|
||
The 4 watchdog rows carry `Script exited with code 1` plus the quoting stdout above.
|
||
- `~/.hermes/cron/executions.db:cron_incidents` — **9 incidents opened in the window, 6 closed, 3 still open**
|
||
(`1b23c0_103cbe04a79b` `alerted`; `cbfe43_9b89947a046a` and `1a8e90_3c9c1d573cde` `detected`).
|
||
**The ledger tags 7 of the 9 `failure_type: unknown`** even though the message was explicit (the two
|
||
watchdog rows are classed `script`, correctly — they *are* script exits); for the seven real 503s the
|
||
classifier had the message in hand and still filed no type. Open incidents overall: **12 → 8**
|
||
(5 `alerted` / 3 `detected`).
|
||
- **The node itself was never down.** `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` on
|
||
`100.78.63.20:20128` at 2026-10-18 00:33 AWST. A 503 is a *serving* failure, so reachability probes
|
||
(which the nightly audit and the operator both run) could not have caught this — only job-level error text did.
|
||
- **No node-side telemetry was retained.** Nothing on the Hermes side records host CPU/RAM/queue depth at
|
||
10-16 08:00, so the burst can be described but not explained. Treat the root cause as **unknown**.
|
||
|
||
## Recurrence — burst 2 (2026-10-19)
|
||
|
||
An **identical** failure returned 44 hours after burst 1 ended, on the same node, with the same error string and
|
||
the same number of genuine model-call failures. Nothing on the Hermes side changed between the two bursts.
|
||
|
||
| AWST (2026-10-19) | Job (name and schedule per `jobs.json`) | Slot | Result | Failure streak after |
|
||
|---|---|---|---|---|
|
||
| 06:01:05 | `b63ed5817212` Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | `RuntimeError: HTTP 503 … resource pressure` | 1 |
|
||
| 07:01 | `dca8482e4f76` Daily Marketing Job Radar | `0 7 * * 1-5` | same | 1 |
|
||
| 07:01 | `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | `0 7 * * *` | same | 1 |
|
||
| 08:01 | `0e519874d280` Political News Digest (Mon/Thu) | `0 8 * * 1,4` | same | 1 → **2** |
|
||
| 08:31 | `08eb86f02d37` Gmail Action Digest | `30 8 * * *` | same | 1 |
|
||
| 09:01 | `c0a9fa314542` GitHub Trending Radar — Weekly Roundup | `0 9 * * 1` | same | 1 |
|
||
| 12:01:16 | `b63ed5817212` Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | same (2nd slot of the day) | **2** |
|
||
| 12:01 | `ea79a2107e36` Mnemosyne Health Monitor | `0 */12 * * *` | same | 1 |
|
||
| 12:01 | `ecc1fc13655a` FreshRSS Midday News Briefing | `0 12 * * *` | same | 1 → **2** |
|
||
| 06:16, 06:31, 12:16, 12:31 | `96eb95ce12f2` Mnemosyne Watchdog | `*/15 * * * *` | `Script exited with code 1`, stdout quoting `b63ed5817212`'s 503 — **derivative** | 1 |
|
||
|
||
**9 genuine model-call failures across 8 jobs** (burst 1 was 9 across 7), plus the usual 4 derivative watchdog
|
||
rows = **13 failed execution rows**, the same total as burst 1. Whole day: **159 completed / 13 failed** (10-18 was
|
||
168 / 1; 10-17 was 163 / 5).
|
||
|
||
- **Failed rows by hour:** 06:xx → 3 (one model call + two derivative watchdog reds), 07:xx → 2, 08:xx → 2,
|
||
09:xx → 1, 12:xx → 5. Nothing after 12:31 failed — the 15:00 → 23:00 slots and the 10-20 00:00 slots all
|
||
completed.
|
||
- **Not one failure landed in the `no_agent` / script jobs' own logic.** Every 503 was on an `agent`-mode job
|
||
whose model call sits on `custom:omni-101`, exactly as in burst 1.
|
||
- **The blast radius is now the stable signature**: the same nine job/slot combinations that carry the
|
||
heaviest prompts fail first, while short-prompt jobs on the same node come through. A node under load-shedding
|
||
is dropping the *large-context* calls preferentially.
|
||
- **Single pin, now verified:** as of 2026-10-20 grep of `jobs.json` shows **every one of these eight jobs reading
|
||
`provider: custom:omni-101`, `model: smart-route`** — no alias diversity at all. ⚠ The burst-1 note above
|
||
records a wider alias mix (`deepseek-paid-smart`, `free-IA`, `dark`); `jobs.json` no longer matches that, so
|
||
the pins may have been consolidated since 10-16. **Confirm with Anthony before relying on either reading.**
|
||
- **Three incident rows auto-closed** — `96eb95_565ec1deb1c7` at **12:46:11** (the Watchdog's 12:30 slot
|
||
recovered once its subject job recovered), `b63ed5_bbe44c89aac1` at **18:01:56** (the 18:01 coverage slot
|
||
succeeded) and `ea79a2_747b27556fd0` at **2026-10-20 00:04:24** (the Health Monitor's 00:01 slot succeeded).
|
||
**Four new incident rows opened** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `0e5198_ec2fac98a91d`,
|
||
`c0a9fa_764558b5ae83`) because those four jobs had never carried this error signature — the other five 503s
|
||
folded into pre-existing rows, and `08eb86_472c7b609c80` + `ecc1fc_c9976ea24553` were **refreshed from
|
||
2026-10-16**, not reopened. **Unclosed ledger rows: 13** (12 `alerted` / 1 `detected`) as of 2026-10-20 00:09;
|
||
4 opened and 2 closed on 10-19. *(Caution: the 10-16/10-18/10-19 entries reported `12`, `8` and `7` open —
|
||
those figures used a narrower filter than a raw `closed_at IS NULL` count, which reads 12 for 10-19 00:20 and
|
||
13 now. Compare like with like from here on.)*
|
||
- **Every failure streak is still 1 or 2.** No job has consecutively failed enough to be suppressed, and the
|
||
10-20 00:00 slots all ran clean. Non-clean `last_status` across the fleet moved **7 → 12** (10 `error`,
|
||
2 `delivery_failed`) — see [[systems/scheduled-tasks]].
|
||
- **Re-probed 2026-10-20 00:03 AWST:** `omniroute-1` → `/api/health` `200 {"status":"ok"}`, `/v1/models` `401`.
|
||
Healthy, as it was during both bursts. **A green health endpoint still proves nothing about the model path.**
|
||
|
||
### What this changes
|
||
|
||
- The 10-18 assessment — *"it recovered on its own; nothing was changed on the node; monitoring"* — is
|
||
**superseded**. A self-clearing fault that returns twice in four days with an identical blast radius is a
|
||
**standing operational risk**, not a closed incident. Status moved `monitoring` → **`active`**.
|
||
- Burst 1 left **no captured host telemetry**; burst 2 was likewise *not* instrumented while it was happening,
|
||
for the second time. The "capture metrics during the burst" lesson from 10-18 was not acted on. **The next
|
||
occurrence must be caught live** (see [[systems/scheduled-tasks]] for the probe to run).
|
||
- **`github-trending-radar`'s 2026-10-19 09:01 slot — its scheduled Monday run (`0 9 * * 1`) — was lost to
|
||
the 503**, so this week's roundup never went out. Its revoked Telegram delivery token (recorded in
|
||
[[systems/scheduled-tasks]] Known Issues and [[current-state]]) **remains untested for a second week**,
|
||
because the job has not reached its own code on a Monday since 2026-10-12. The next Monday slot
|
||
(2026-10-26) is the first real test.
|
||
|
||
## Why this is not a repeat of the 10-14 storm
|
||
|
||
| | 2026-10-14 → 10-15 | 2026-10-16 → 10-17 |
|
||
|---|---|---|
|
||
| Failure class | `ConnectError: [Errno 111] Connection refused` | HTTP `503` resource pressure |
|
||
| Node state | TCP-unreachable for ~34 h | up and answering probes throughout |
|
||
| Blast radius | 12 jobs, 24 model-call failures, every call for 34 h | 7 jobs, 9 model-call failures, bursty, self-cleared |
|
||
| Recovery | manual node recovery | none needed — cleared on its own |
|
||
|
||
Both are the **same single-node topology problem**: all enabled model-calling cron jobs ride
|
||
`custom:omni-101` with no failover, three days apart from the previous outage.
|
||
|
||
## Consequences
|
||
|
||
- **One stocktake hole:** `raw/stocktake/freshness-20261017.txt` was not written and
|
||
`systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 03:00 slot succeeded, so
|
||
`freshness-20261016.txt` exists). The **2026-10-18 03:00 slot** closes it — nothing was regenerated by hand,
|
||
because a manual run would race the scheduled one. A failure there would be the second consecutive miss.
|
||
- **The Wiki Audit lost its 10-16 slot**, so no wiki pass covered the 10-14→10-17 span until this run.
|
||
- **Stale flags clear only at the next success, and some slots are a week apart.** `1b23c042e17d` is a Friday
|
||
job: it will sit at `last_status: error`, streak 2, with an open incident until **Fri 2026-10-23 08:00**.
|
||
On a weekly cadence a one-slot failure reads as a week-long outage.
|
||
|
||
## Lessons
|
||
|
||
1. **Same topology, second failure mode in three days.** A 503 from a *live* node cannot be detected by
|
||
reachability probes; only per-job error text distinguishes "node down" from "node overloaded". Both
|
||
classes gate the whole scheduled estate.
|
||
2. **Watchdog rows inflate the count — read per job.** 4 of the 13 rows are one log sentinel reacting to
|
||
another job's failure, not independent faults.
|
||
3. **The incident classifier drops the failure type.** All **seven** genuine 503 incidents are filed
|
||
`failure_type: unknown` despite an explicit message in the execution row (only the two script-exit
|
||
watchdog incidents get a type), so `failure_type` cannot be used to triage — filter on the error text.
|
||
4. **A 503 burst leaves no evidence behind.** No host telemetry is captured at failure time, so the "resource
|
||
pressure" cause is taken on the router's word. If this recurs, capture the node's host metrics *during* the
|
||
burst, not afterwards.
|
||
5. **A self-clearing burst is not a resolved incident.** Burst 1 closed as "monitoring" on 10-18 and the
|
||
identical fault returned within 44 hours. **Recurrence with an identical signature and an identical blast
|
||
radius is the signal that a fault is systemic** — the second occurrence, not the first, is the one that
|
||
should change a page's `status`. Two occurrences, same shape, four days apart, on a topology with no
|
||
failover: treat the next one as expected.
|
||
|
||
## Open item (needs Anthony's call, unchanged from 10-14)
|
||
|
||
The durable fix is the same one flagged in [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]:
|
||
a second provider pin on a subset of jobs, or router-side failover between `omniroute-1` and CT600
|
||
`omniroute`. **Not applied** — that is a routing decision, not an audit one. The 10-19 recurrence raises the
|
||
cost of leaving it open: **two of the last four days took double-digit failures from this one node**, and every
|
||
model-calling cron job still has exactly one route.
|
||
|
||
## Update 2026-10-21 (audit of the 2026-10-20 window): the burst did **not** resume
|
||
|
||
The 2026-10-19 burst was followed by **2026-10-20's 168 `completed` / 0 `failed`** — a clean day and the
|
||
first zero-failure day since 2026-10-12, with **zero** rows carrying the resource-pressure string and **zero
|
||
new incident rows** opened for the whole window. The 503 fault's last recorded failure is therefore
|
||
**2026-10-19**, and per [[systems/scheduled-tasks]] the page status returns to **`monitoring`** (18–24 h of
|
||
clean running, not a fix).
|
||
|
||
**What did *not* change, and matters more than the clean day:**
|
||
|
||
- **No failover was added.** 20 of 28 jobs still pin `custom:omni-101` (`smart-route` ×11, `free-stack` ×6,
|
||
`deepseek-paid-smart` ×1) or `custom:omni` ×2. One quiet day on a single-route topology is luck, not
|
||
margin, and this page has already been wrong once about the burst being over (it was downgraded 44 h
|
||
before burst 2).
|
||
- **The 8 jobs with no model pin no longer ride the router at all.** The global provider default moved to
|
||
`nous` direct (see below), so the fleet is now split across **two** upstream paths for the first time —
|
||
worth watching as a separate risk, since a Nous-side fault now has its own blast radius.
|
||
- **Global default changed between 2026-10-09 and 2026-10-20:** `provider: custom` →
|
||
`nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` →
|
||
`meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]]
|
||
|
||
## Update 2026-10-21 (window 2026-10-20 → 10-21)
|
||
|
||
- **A third failure mode from the same pin — and it is not 503.** The only failed cron row on **2026-10-21**
|
||
was `dca8482e4f76` (Daily Marketing Job Radar, `custom:omni-101` / `smart-route`), which died after
|
||
**40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
|
||
filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].
|
||
- **The resource-pressure string did not appear at all on 2026-10-21** (zero ledger rows; last occurrence
|
||
remains 2026-10-19 09:02), so the second burst stays closed and this page stays `status: monitoring`.
|
||
- **One row closed, one opened.** `08eb86_472c7b609c80` (the Gmail Action Digest 503s, first seen 10-16 08:33)
|
||
**resolved 2026-10-21 08:37:21** — the job's own 08:30 slot ran clean that morning (its 10-20 run was the
|
||
orphan; no incident row was ever raised for it), which also cleared `08eb86f02d37` from the non-clean table.
|
||
`dca848_014affeb6954` opened 07:40:42 and is **still `alerted`**. Unclosed rows **stay at 9**
|
||
(8 `alerted` / 1 `detected`).
|
||
- **Windows:** 2026-10-20 settled at **169 `completed` / 1 `unknown`** (the gateway-orphaned run; the 10-21
|
||
audit's 168 / 0 was a mid-flight snapshot taken before the 23:59 slot landed). **2026-10-21: 171 `completed`
|
||
/ 1 `failed`** (+ this audit).
|
||
- **Still no failover.** Every job that has failed on this node since 2026-10-14 still reads
|
||
`provider: custom:omni-101` with `model: smart-route` or `free-stack`, and none has a second pin.
|
||
|
||
## Related
|
||
|
||
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.
|
||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — the first multi-day storm from this topology.
|
||
- [[systems/omniroute]] — node layout, providers and pins.
|
||
- [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger.
|