Files
agent-estate-wiki/incidents/2026-10-16-omniroute-503-resource-pressure.md
T
Tony0410 3919ec8557 🦉 wiki audit 2026-10-22: omni-101 pre-dispatch filter skip (3rd failure mode, new page), Hermes 0.22.3 update with snapshot skipped, Gitea backup backlog 76
- new: incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md
- systems/hermes-agent.md: version -> 0.22.3 (5b147ab036), 22:47-22:57 update, snapshot stage skipped, advisories unactioned
- systems/scheduled-tasks.md: 2026-10-21 = 171 completed / 1 failed; non-clean stays 9 with 08eb86f02d37 out, dca8482e4f76 in
- systems/omniroute.md: one pin, three signatures in eight days
- incidents/2026-10-16-omniroute-503-resource-pressure.md: window update, stays monitoring
- incidents/2026-08-08-gitea-config-backup-push-dead.md: 2026-10-21 run, backlog 75 -> 76, message_id 8449
- infrastructure/backups.md: 76 commits; pre_update_backup:false now exercised
- current-state.md, index.md, log.md updated
2026-10-22 00:16:39 +08:00

248 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts
type: incident
status: monitoring
created: 2026-10-18
updated: 2026-10-22
verified_on: 2026-10-22
review_after: 2026-10-29
confidence: high
tags: [cron, router, provider, infrastructure]
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
---
# OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts (burst 1: 2026-10-16 → 10-17; burst 2: 2026-10-19)
**Status: `MONITORING` as of 2026-10-21 — not closed.** The failure mode came back **twice in four
days** (10-16 → 10-17 and 10-19) with an identical signature on an identical node, and the **exact same
9-model-call blast radius** each time. The node was healthy before, between and after both bursts.
> **2026-10-21 update — burst 2 is over; the page returns to `monitoring` on the 18–24 h rule it set itself.**
> **2026-10-20 ran 168 `completed` / 0 `failed`** (plus one `unknown` orphan — see
> [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), the **first zero-failure day since 2026-10-12**,
> and **no execution row carries the resource-pressure string** (the last such failure remains 2026-10-19).
> **4 incident rows closed on 10-20** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `ecc1fc_c9976ea24553`,
> `ecc1fc_21a72aa11cc4`), taking unclosed rows **13 → 9** (8 `alerted` / 1 `detected`). **Nothing was changed on
> the node** — the recovery is self-clearing, exactly as in burst 1, and the underlying exposure is untouched:
> the faults are still invisible to `/api/health`, still single-node, and a fresh burst would still take out
> most of the fleet. Two bursts four days apart, both self-clearing, is the pattern to watch; this is a standing
> risk, not a resolved incident.
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
jobs). 10-19 = **159 completed / 13 failed** against the previous day's 168 / 1.
The node answers `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` (auth-required) — re-probed
**2026-10-20 00:03 AWST**, mid-recovery, still healthy.
**Impact:** burst 1 = **13 failed execution rows** (**9 genuine model-call failures across 7 jobs** + 4 derivative
Mnemosyne Watchdog rows); burst 2 = **13 failed execution rows** (**9 genuine model-call failures across 8 jobs**
+ 4 derivative Mnemosyne Watchdog rows). Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean
day (10-12 = 175 / 0). Unlike the 10-14 storm, **the node was up and serving throughout both bursts** — model
calls failed with `503` resource-pressure responses rather than TCP refusals.
## What happened
Every model call on `custom:omni-101` (`http://100.78.63.20:20128/v1`) between the 10-16 08:00 and
10-17 03:00 slots died with the identical error:
```
RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.
```
| Job | Slot(s) AWST | Rows | Outcome |
|---|---|---|---|
| `1b23c042e17d` Enhanced Weekend Adventure Planner | 10-16 08:00 (Fri slot) | 1 | failed; streak **2**; next slot **Fri 10-23 08:00** |
| `08eb86f02d37` Gmail Action Digest | 10-16 08:30 | 1 | failed; cleared 10-17 08:31 (own next run) |
| `b63ed5817212` Mnemosyne Embedding Coverage Monitor | 10-16 12:00, 10-17 00:00 | 2 | failed; cleared **10-17 06:01** |
| `ecc1fc13655a` FreshRSS Midday News Briefing | 10-16 12:00 | 1 | failed; cleared 10-17 12:02 |
| `ea79a2107e36` Mnemosyne Health Monitor | 10-16 12:00, 10-17 00:00 | 2 | failed; cleared 10-17 12:02 |
| `cbfe434a3b5e` Wiki Audit | 10-16 23:59 | 1 | failed; incident **open** (`detected`) — the 10-17 23:59 run is the one that produced this audit |
| `1a8e90ea2271` Wiki Stocktake | 10-17 03:00 | 1 | failed; incident **open** (`detected`) |
| `96eb95ce12f2` Mnemosyne Watchdog | 10-16 12:15 + 12:30, 10-17 00:15 + 00:30 | 4 | **derivative** — `Script exited with code 1`, its stdout quoting the watchdog's own log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…` |
**8 rows on 10-16, 5 on 10-17.** Reading per job rather than per row: **7 jobs actually failed**, and four of
the 13 rows are one sentinel watching another job's 503 — the same double-counting trap as the 10-14 storm
(see [[incidents/2026-10-14-omniroute-node-connection-refused-storm]], lesson 4).
Every 503 row sits on **`custom:omni-101`** — one alias, one node, again the single pin.
No other job failed in this window: `85edf6772bca` (Tech & AI) and `dca8482e4f76` (Job Radar) both ran
clean through it.
## Evidence
- `~/.hermes/cron/executions.db` — the 9 model-call rows each carry the literal
`RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`
The 4 watchdog rows carry `Script exited with code 1` plus the quoting stdout above.
- `~/.hermes/cron/executions.db:cron_incidents` — **9 incidents opened in the window, 6 closed, 3 still open**
(`1b23c0_103cbe04a79b` `alerted`; `cbfe43_9b89947a046a` and `1a8e90_3c9c1d573cde` `detected`).
**The ledger tags 7 of the 9 `failure_type: unknown`** even though the message was explicit (the two
watchdog rows are classed `script`, correctly — they *are* script exits); for the seven real 503s the
classifier had the message in hand and still filed no type. Open incidents overall: **12 → 8**
(5 `alerted` / 3 `detected`).
- **The node itself was never down.** `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` on
`100.78.63.20:20128` at 2026-10-18 00:33 AWST. A 503 is a *serving* failure, so reachability probes
(which the nightly audit and the operator both run) could not have caught this — only job-level error text did.
- **No node-side telemetry was retained.** Nothing on the Hermes side records host CPU/RAM/queue depth at
10-16 08:00, so the burst can be described but not explained. Treat the root cause as **unknown**.
## Recurrence — burst 2 (2026-10-19)
An **identical** failure returned 44 hours after burst 1 ended, on the same node, with the same error string and
the same number of genuine model-call failures. Nothing on the Hermes side changed between the two bursts.
| AWST (2026-10-19) | Job (name and schedule per `jobs.json`) | Slot | Result | Failure streak after |
|---|---|---|---|---|
| 06:01:05 | `b63ed5817212` Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | `RuntimeError: HTTP 503 … resource pressure` | 1 |
| 07:01 | `dca8482e4f76` Daily Marketing Job Radar | `0 7 * * 1-5` | same | 1 |
| 07:01 | `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | `0 7 * * *` | same | 1 |
| 08:01 | `0e519874d280` Political News Digest (Mon/Thu) | `0 8 * * 1,4` | same | 1 → **2** |
| 08:31 | `08eb86f02d37` Gmail Action Digest | `30 8 * * *` | same | 1 |
| 09:01 | `c0a9fa314542` GitHub Trending Radar — Weekly Roundup | `0 9 * * 1` | same | 1 |
| 12:01:16 | `b63ed5817212` Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | same (2nd slot of the day) | **2** |
| 12:01 | `ea79a2107e36` Mnemosyne Health Monitor | `0 */12 * * *` | same | 1 |
| 12:01 | `ecc1fc13655a` FreshRSS Midday News Briefing | `0 12 * * *` | same | 1 → **2** |
| 06:16, 06:31, 12:16, 12:31 | `96eb95ce12f2` Mnemosyne Watchdog | `*/15 * * * *` | `Script exited with code 1`, stdout quoting `b63ed5817212`'s 503 — **derivative** | 1 |
**9 genuine model-call failures across 8 jobs** (burst 1 was 9 across 7), plus the usual 4 derivative watchdog
rows = **13 failed execution rows**, the same total as burst 1. Whole day: **159 completed / 13 failed** (10-18 was
168 / 1; 10-17 was 163 / 5).
- **Failed rows by hour:** 06:xx → 3 (one model call + two derivative watchdog reds), 07:xx → 2, 08:xx → 2,
09:xx → 1, 12:xx → 5. Nothing after 12:31 failed — the 15:00 → 23:00 slots and the 10-20 00:00 slots all
completed.
- **Not one failure landed in the `no_agent` / script jobs' own logic.** Every 503 was on an `agent`-mode job
whose model call sits on `custom:omni-101`, exactly as in burst 1.
- **The blast radius is now the stable signature**: the same nine job/slot combinations that carry the
heaviest prompts fail first, while short-prompt jobs on the same node come through. A node under load-shedding
is dropping the *large-context* calls preferentially.
- **Single pin, now verified:** as of 2026-10-20 grep of `jobs.json` shows **every one of these eight jobs reading
`provider: custom:omni-101`, `model: smart-route`** — no alias diversity at all. ⚠ The burst-1 note above
records a wider alias mix (`deepseek-paid-smart`, `free-IA`, `dark`); `jobs.json` no longer matches that, so
the pins may have been consolidated since 10-16. **Confirm with Anthony before relying on either reading.**
- **Three incident rows auto-closed** — `96eb95_565ec1deb1c7` at **12:46:11** (the Watchdog's 12:30 slot
recovered once its subject job recovered), `b63ed5_bbe44c89aac1` at **18:01:56** (the 18:01 coverage slot
succeeded) and `ea79a2_747b27556fd0` at **2026-10-20 00:04:24** (the Health Monitor's 00:01 slot succeeded).
**Four new incident rows opened** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `0e5198_ec2fac98a91d`,
`c0a9fa_764558b5ae83`) because those four jobs had never carried this error signature — the other five 503s
folded into pre-existing rows, and `08eb86_472c7b609c80` + `ecc1fc_c9976ea24553` were **refreshed from
2026-10-16**, not reopened. **Unclosed ledger rows: 13** (12 `alerted` / 1 `detected`) as of 2026-10-20 00:09;
4 opened and 2 closed on 10-19. *(Caution: the 10-16/10-18/10-19 entries reported `12`, `8` and `7` open —
those figures used a narrower filter than a raw `closed_at IS NULL` count, which reads 12 for 10-19 00:20 and
13 now. Compare like with like from here on.)*
- **Every failure streak is still 1 or 2.** No job has consecutively failed enough to be suppressed, and the
10-20 00:00 slots all ran clean. Non-clean `last_status` across the fleet moved **7 → 12** (10 `error`,
2 `delivery_failed`) — see [[systems/scheduled-tasks]].
- **Re-probed 2026-10-20 00:03 AWST:** `omniroute-1` → `/api/health` `200 {"status":"ok"}`, `/v1/models` `401`.
Healthy, as it was during both bursts. **A green health endpoint still proves nothing about the model path.**
### What this changes
- The 10-18 assessment — *"it recovered on its own; nothing was changed on the node; monitoring"* — is
**superseded**. A self-clearing fault that returns twice in four days with an identical blast radius is a
**standing operational risk**, not a closed incident. Status moved `monitoring` → **`active`**.
- Burst 1 left **no captured host telemetry**; burst 2 was likewise *not* instrumented while it was happening,
for the second time. The "capture metrics during the burst" lesson from 10-18 was not acted on. **The next
occurrence must be caught live** (see [[systems/scheduled-tasks]] for the probe to run).
- **`github-trending-radar`'s 2026-10-19 09:01 slot — its scheduled Monday run (`0 9 * * 1`) — was lost to
the 503**, so this week's roundup never went out. Its revoked Telegram delivery token (recorded in
[[systems/scheduled-tasks]] Known Issues and [[current-state]]) **remains untested for a second week**,
because the job has not reached its own code on a Monday since 2026-10-12. The next Monday slot
(2026-10-26) is the first real test.
## Why this is not a repeat of the 10-14 storm
| | 2026-10-14 → 10-15 | 2026-10-16 → 10-17 |
|---|---|---|
| Failure class | `ConnectError: [Errno 111] Connection refused` | HTTP `503` resource pressure |
| Node state | TCP-unreachable for ~34 h | up and answering probes throughout |
| Blast radius | 12 jobs, 24 model-call failures, every call for 34 h | 7 jobs, 9 model-call failures, bursty, self-cleared |
| Recovery | manual node recovery | none needed — cleared on its own |
Both are the **same single-node topology problem**: all enabled model-calling cron jobs ride
`custom:omni-101` with no failover, three days apart from the previous outage.
## Consequences
- **One stocktake hole:** `raw/stocktake/freshness-20261017.txt` was not written and
`systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 03:00 slot succeeded, so
`freshness-20261016.txt` exists). The **2026-10-18 03:00 slot** closes it — nothing was regenerated by hand,
because a manual run would race the scheduled one. A failure there would be the second consecutive miss.
- **The Wiki Audit lost its 10-16 slot**, so no wiki pass covered the 10-14→10-17 span until this run.
- **Stale flags clear only at the next success, and some slots are a week apart.** `1b23c042e17d` is a Friday
job: it will sit at `last_status: error`, streak 2, with an open incident until **Fri 2026-10-23 08:00**.
On a weekly cadence a one-slot failure reads as a week-long outage.
## Lessons
1. **Same topology, second failure mode in three days.** A 503 from a *live* node cannot be detected by
reachability probes; only per-job error text distinguishes "node down" from "node overloaded". Both
classes gate the whole scheduled estate.
2. **Watchdog rows inflate the count — read per job.** 4 of the 13 rows are one log sentinel reacting to
another job's failure, not independent faults.
3. **The incident classifier drops the failure type.** All **seven** genuine 503 incidents are filed
`failure_type: unknown` despite an explicit message in the execution row (only the two script-exit
watchdog incidents get a type), so `failure_type` cannot be used to triage — filter on the error text.
4. **A 503 burst leaves no evidence behind.** No host telemetry is captured at failure time, so the "resource
pressure" cause is taken on the router's word. If this recurs, capture the node's host metrics *during* the
burst, not afterwards.
5. **A self-clearing burst is not a resolved incident.** Burst 1 closed as "monitoring" on 10-18 and the
identical fault returned within 44 hours. **Recurrence with an identical signature and an identical blast
radius is the signal that a fault is systemic** — the second occurrence, not the first, is the one that
should change a page's `status`. Two occurrences, same shape, four days apart, on a topology with no
failover: treat the next one as expected.
## Open item (needs Anthony's call, unchanged from 10-14)
The durable fix is the same one flagged in [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]:
a second provider pin on a subset of jobs, or router-side failover between `omniroute-1` and CT600
`omniroute`. **Not applied** — that is a routing decision, not an audit one. The 10-19 recurrence raises the
cost of leaving it open: **two of the last four days took double-digit failures from this one node**, and every
model-calling cron job still has exactly one route.
## Update 2026-10-21 (audit of the 2026-10-20 window): the burst did **not** resume
The 2026-10-19 burst was followed by **2026-10-20's 168 `completed` / 0 `failed`** — a clean day and the
first zero-failure day since 2026-10-12, with **zero** rows carrying the resource-pressure string and **zero
new incident rows** opened for the whole window. The 503 fault's last recorded failure is therefore
**2026-10-19**, and per [[systems/scheduled-tasks]] the page status returns to **`monitoring`** (18–24 h of
clean running, not a fix).
**What did *not* change, and matters more than the clean day:**
- **No failover was added.** 20 of 28 jobs still pin `custom:omni-101` (`smart-route` ×11, `free-stack` ×6,
`deepseek-paid-smart` ×1) or `custom:omni` ×2. One quiet day on a single-route topology is luck, not
margin, and this page has already been wrong once about the burst being over (it was downgraded 44 h
before burst 2).
- **The 8 jobs with no model pin no longer ride the router at all.** The global provider default moved to
`nous` direct (see below), so the fleet is now split across **two** upstream paths for the first time —
worth watching as a separate risk, since a Nous-side fault now has its own blast radius.
- **Global default changed between 2026-10-09 and 2026-10-20:** `provider: custom` →
`nous` (`https://inference-api.nousresearch.com/v1`), `model.default: smart-route` →
`meituan/longcat-2.5-preview:free`. Pinned jobs are unaffected; unpinned jobs are. → [[systems/current-profile]]
## Update 2026-10-21 (window 2026-10-20 → 10-21)
- **A third failure mode from the same pin — and it is not 503.** The only failed cron row on **2026-10-21**
was `dca8482e4f76` (Daily Marketing Job Radar, `custom:omni-101` / `smart-route`), which died after
**40 m 36 s** on `RuntimeError: Service temporarily unavailable: all targets were skipped by pre-dispatch
filters reset after 19m 20s` → [[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]].
- **The resource-pressure string did not appear at all on 2026-10-21** (zero ledger rows; last occurrence
remains 2026-10-19 09:02), so the second burst stays closed and this page stays `status: monitoring`.
- **One row closed, one opened.** `08eb86_472c7b609c80` (the Gmail Action Digest 503s, first seen 10-16 08:33)
**resolved 2026-10-21 08:37:21** — the job's own 08:30 slot ran clean that morning (its 10-20 run was the
orphan; no incident row was ever raised for it), which also cleared `08eb86f02d37` from the non-clean table.
`dca848_014affeb6954` opened 07:40:42 and is **still `alerted`**. Unclosed rows **stay at 9**
(8 `alerted` / 1 `detected`).
- **Windows:** 2026-10-20 settled at **169 `completed` / 1 `unknown`** (the gateway-orphaned run; the 10-21
audit's 168 / 0 was a mid-flight snapshot taken before the 23:59 slot landed). **2026-10-21: 171 `completed`
/ 1 `failed`** (+ this audit).
- **Still no failover.** Every job that has failed on this node since 2026-10-14 still reads
`provider: custom:omni-101` with `model: smart-route` or `free-stack`, and none has a second pin.
## Related
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node.
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — the first multi-day storm from this topology.
- [[systems/omniroute]] — node layout, providers and pins.
- [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger.