audit(2026-10-20): omni-101 503 burst 2 — incident back to active; scheduled-tasks rebuilt from jobs.json
- burst 2 (2026-10-19): 9 model calls across 8 jobs, 159/13 day - burst-2 table names/schedules/aliases corrected from live jobs.json - non-clean last_status 7 -> 12; unclosed incident rows 13 (query stated) - 2026-10-17 tail corrected 162/5 -> 163/5
This commit is contained in:
@@ -1,27 +1,33 @@
|
||||
---
|
||||
title: OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst
|
||||
title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts
|
||||
type: incident
|
||||
status: monitoring
|
||||
status: active
|
||||
created: 2026-10-18
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
review_after: 2026-10-25
|
||||
updated: 2026-10-20
|
||||
verified_on: 2026-10-20
|
||||
review_after: 2026-10-27
|
||||
confidence: high
|
||||
tags: [cron, router, provider, infrastructure]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe]
|
||||
---
|
||||
|
||||
# OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst (2026-10-16 → 2026-10-17)
|
||||
# OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts (burst 1: 2026-10-16 → 10-17; burst 2: 2026-10-19)
|
||||
|
||||
**Window:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
|
||||
**Status:** **Monitoring.** Every slot from the 2026-10-17 08:30 run onward completed green
|
||||
(10-16 = **165 completed / 8 failed**, 10-17 = **162 / 5**, and the 10-17 failures all pre-date 03:01).
|
||||
**Status: ACTIVE — this is a recurrence, not a closed incident.** The failure mode came back **twice in four
|
||||
days** (10-16 → 10-17 and 10-19) with an identical signature on an identical node, and the **exact same
|
||||
9-model-call blast radius** each time. The node was healthy before, between and after both bursts.
|
||||
|
||||
**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**.
|
||||
10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01).
|
||||
**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven
|
||||
jobs). 10-19 = **159 completed / 13 failed** against the previous day's 168 / 1.
|
||||
The node answers `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` (auth-required) — re-probed
|
||||
2026-10-18 00:33 AWST.
|
||||
**Impact:** **13 failed execution rows** — **9 genuine model-call failures across 7 jobs**, plus **4 derivative
|
||||
Mnemosyne Watchdog rows**. Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean day
|
||||
(10-12 = 175 / 0). Unlike the 10-14 storm, **the node was up and serving throughout** — model calls failed with
|
||||
`503` resource-pressure responses rather than TCP refusals.
|
||||
**2026-10-20 00:03 AWST**, mid-recovery, still healthy.
|
||||
**Impact:** burst 1 = **13 failed execution rows** (**9 genuine model-call failures across 7 jobs** + 4 derivative
|
||||
Mnemosyne Watchdog rows); burst 2 = **13 failed execution rows** (**9 genuine model-call failures across 8 jobs**
|
||||
+ 4 derivative Mnemosyne Watchdog rows). Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean
|
||||
day (10-12 = 175 / 0). Unlike the 10-14 storm, **the node was up and serving throughout both bursts** — model
|
||||
calls failed with `503` resource-pressure responses rather than TCP refusals.
|
||||
|
||||
## What happened
|
||||
|
||||
@@ -68,6 +74,70 @@ clean through it.
|
||||
- **No node-side telemetry was retained.** Nothing on the Hermes side records host CPU/RAM/queue depth at
|
||||
10-16 08:00, so the burst can be described but not explained. Treat the root cause as **unknown**.
|
||||
|
||||
## Recurrence — burst 2 (2026-10-19)
|
||||
|
||||
An **identical** failure returned 44 hours after burst 1 ended, on the same node, with the same error string and
|
||||
the same number of genuine model-call failures. Nothing on the Hermes side changed between the two bursts.
|
||||
|
||||
| AWST (2026-10-19) | Job (name and schedule per `jobs.json`) | Slot | Result | Failure streak after |
|
||||
|---|---|---|---|---|
|
||||
| 06:01:05 | `b63ed5817212` Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | `RuntimeError: HTTP 503 … resource pressure` | 1 |
|
||||
| 07:01 | `dca8482e4f76` Daily Marketing Job Radar | `0 7 * * 1-5` | same | 1 |
|
||||
| 07:01 | `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | `0 7 * * *` | same | 1 |
|
||||
| 08:01 | `0e519874d280` Political News Digest (Mon/Thu) | `0 8 * * 1,4` | same | 1 → **2** |
|
||||
| 08:31 | `08eb86f02d37` Gmail Action Digest | `30 8 * * *` | same | 1 |
|
||||
| 09:01 | `c0a9fa314542` GitHub Trending Radar — Weekly Roundup | `0 9 * * 1` | same | 1 |
|
||||
| 12:01:16 | `b63ed5817212` Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | same (2nd slot of the day) | **2** |
|
||||
| 12:01 | `ea79a2107e36` Mnemosyne Health Monitor | `0 */12 * * *` | same | 1 |
|
||||
| 12:01 | `ecc1fc13655a` FreshRSS Midday News Briefing | `0 12 * * *` | same | 1 → **2** |
|
||||
| 06:16, 06:31, 12:16, 12:31 | `96eb95ce12f2` Mnemosyne Watchdog | `*/15 * * * *` | `Script exited with code 1`, stdout quoting `b63ed5817212`'s 503 — **derivative** | 1 |
|
||||
|
||||
**9 genuine model-call failures across 8 jobs** (burst 1 was 9 across 7), plus the usual 4 derivative watchdog
|
||||
rows = **13 failed execution rows**, the same total as burst 1. Whole day: **159 completed / 13 failed** (10-18 was
|
||||
168 / 1; 10-17 was 163 / 5).
|
||||
|
||||
- **Failed rows by hour:** 06:xx → 3 (one model call + two derivative watchdog reds), 07:xx → 2, 08:xx → 2,
|
||||
09:xx → 1, 12:xx → 5. Nothing after 12:31 failed — the 15:00 → 23:00 slots and the 10-20 00:00 slots all
|
||||
completed.
|
||||
- **Not one failure landed in the `no_agent` / script jobs' own logic.** Every 503 was on an `agent`-mode job
|
||||
whose model call sits on `custom:omni-101`, exactly as in burst 1.
|
||||
- **The blast radius is now the stable signature**: the same nine job/slot combinations that carry the
|
||||
heaviest prompts fail first, while short-prompt jobs on the same node come through. A node under load-shedding
|
||||
is dropping the *large-context* calls preferentially.
|
||||
- **Single pin, now verified:** as of 2026-10-20 grep of `jobs.json` shows **every one of these eight jobs reading
|
||||
`provider: custom:omni-101`, `model: smart-route`** — no alias diversity at all. ⚠ The burst-1 note above
|
||||
records a wider alias mix (`deepseek-paid-smart`, `free-IA`, `dark`); `jobs.json` no longer matches that, so
|
||||
the pins may have been consolidated since 10-16. **Confirm with Anthony before relying on either reading.**
|
||||
- **Three incident rows auto-closed** — `96eb95_565ec1deb1c7` at **12:46:11** (the Watchdog's 12:30 slot
|
||||
recovered once its subject job recovered), `b63ed5_bbe44c89aac1` at **18:01:56** (the 18:01 coverage slot
|
||||
succeeded) and `ea79a2_747b27556fd0` at **2026-10-20 00:04:24** (the Health Monitor's 00:01 slot succeeded).
|
||||
**Four new incident rows opened** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `0e5198_ec2fac98a91d`,
|
||||
`c0a9fa_764558b5ae83`) because those four jobs had never carried this error signature — the other five 503s
|
||||
folded into pre-existing rows, and `08eb86_472c7b609c80` + `ecc1fc_c9976ea24553` were **refreshed from
|
||||
2026-10-16**, not reopened. **Unclosed ledger rows: 13** (12 `alerted` / 1 `detected`) as of 2026-10-20 00:09;
|
||||
4 opened and 2 closed on 10-19. *(Caution: the 10-16/10-18/10-19 entries reported `12`, `8` and `7` open —
|
||||
those figures used a narrower filter than a raw `closed_at IS NULL` count, which reads 12 for 10-19 00:20 and
|
||||
13 now. Compare like with like from here on.)*
|
||||
- **Every failure streak is still 1 or 2.** No job has consecutively failed enough to be suppressed, and the
|
||||
10-20 00:00 slots all ran clean. Non-clean `last_status` across the fleet moved **7 → 12** (10 `error`,
|
||||
2 `delivery_failed`) — see [[systems/scheduled-tasks]].
|
||||
- **Re-probed 2026-10-20 00:03 AWST:** `omniroute-1` → `/api/health` `200 {"status":"ok"}`, `/v1/models` `401`.
|
||||
Healthy, as it was during both bursts. **A green health endpoint still proves nothing about the model path.**
|
||||
|
||||
### What this changes
|
||||
|
||||
- The 10-18 assessment — *"it recovered on its own; nothing was changed on the node; monitoring"* — is
|
||||
**superseded**. A self-clearing fault that returns twice in four days with an identical blast radius is a
|
||||
**standing operational risk**, not a closed incident. Status moved `monitoring` → **`active`**.
|
||||
- Burst 1 left **no captured host telemetry**; burst 2 was likewise *not* instrumented while it was happening,
|
||||
for the second time. The "capture metrics during the burst" lesson from 10-18 was not acted on. **The next
|
||||
occurrence must be caught live** (see [[systems/scheduled-tasks]] for the probe to run).
|
||||
- **`github-trending-radar`'s 2026-10-19 09:01 slot — its scheduled Monday run (`0 9 * * 1`) — was lost to
|
||||
the 503**, so this week's roundup never went out. Its revoked Telegram delivery token (recorded in
|
||||
[[systems/scheduled-tasks]] Known Issues and [[current-state]]) **remains untested for a second week**,
|
||||
because the job has not reached its own code on a Monday since 2026-10-12. The next Monday slot
|
||||
(2026-10-26) is the first real test.
|
||||
|
||||
## Why this is not a repeat of the 10-14 storm
|
||||
|
||||
| | 2026-10-14 → 10-15 | 2026-10-16 → 10-17 |
|
||||
@@ -104,12 +174,19 @@ Both are the **same single-node topology problem**: all enabled model-calling cr
|
||||
4. **A 503 burst leaves no evidence behind.** No host telemetry is captured at failure time, so the "resource
|
||||
pressure" cause is taken on the router's word. If this recurs, capture the node's host metrics *during* the
|
||||
burst, not afterwards.
|
||||
5. **A self-clearing burst is not a resolved incident.** Burst 1 closed as "monitoring" on 10-18 and the
|
||||
identical fault returned within 44 hours. **Recurrence with an identical signature and an identical blast
|
||||
radius is the signal that a fault is systemic** — the second occurrence, not the first, is the one that
|
||||
should change a page's `status`. Two occurrences, same shape, four days apart, on a topology with no
|
||||
failover: treat the next one as expected.
|
||||
|
||||
## Open item (needs Anthony's call, unchanged from 10-14)
|
||||
|
||||
The durable fix is the same one flagged in [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]:
|
||||
a second provider pin on a subset of jobs, or router-side failover between `omniroute-1` and CT600
|
||||
`omniroute`. **Not applied** — that is a routing decision, not an audit one.
|
||||
`omniroute`. **Not applied** — that is a routing decision, not an audit one. The 10-19 recurrence raises the
|
||||
cost of leaving it open: **two of the last four days took double-digit failures from this one node**, and every
|
||||
model-calling cron job still has exactly one route.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
Reference in New Issue
Block a user