diff --git a/current-state.md b/current-state.md index 02574a2..5e90e20 100644 --- a/current-state.md +++ b/current-state.md @@ -3,16 +3,43 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-19 -verified_on: 2026-10-19 -review_after: 2026-10-25 +updated: 2026-10-20 +verified_on: 2026-10-20 +review_after: 2026-10-27 confidence: high tags: [system] sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, live-probe] --- # Current State -Updated: 2026-10-19 +Updated: 2026-10-20 + +## Recent Changes (2026-10-20, nightly audit; live probes 2026-10-20 00:03-00:12 AWST) + +- **The `omni-101` 503 resource-pressure fault returned 44 hours after the incident page was downgraded to + `monitoring` — same node, same error string, same blast radius.** 2026-10-19 closed **159 completed / 13 + failed** against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all read + `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on + `custom:omni-101`, in two clusters — 06:01 (one job) and 07:01 → 12:01 (seven more runs, seven jobs). The + remaining 4 failed rows are the Mnemosyne Watchdog quoting the coverage monitor's error: the same derivative + double-count as burst 1. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] (status back to **`active`**) +- **Single pin, verified this time.** Every job that failed in burst 2 reads `provider: custom:omni-101`, + `model: smart-route` in `jobs.json` as of 2026-10-20 — there is no alias diversity left to fail over to. ⚠ The + burst-1 note recording a wider mix (`deepseek-paid-smart`, `free-IA`, `dark`) no longer matches `jobs.json`; + it needs confirming with Anthony before either reading is relied on. +- **The node stayed green through all of it.** `omniroute-1` (`100.78.63.20:20128`) answered + `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` at 2026-10-20 00:03, before, during and after + the burst. **Reachability probes cannot see this failure class** — only job-level error text can. +- **`c0a9fa314542` (GitHub Trending Radar) lost its scheduled Monday slot** (`0 9 * * 1`, 09:01) to the 503, so + this week's roundup never ran and its revoked-token status is **untested for a second consecutive Monday**. + Its next real test is Mon **2026-10-26 09:00**. +- **Non-clean `last_status` moved 7 → 12** (10 `error`, 2 `delivery_failed`) as of 2026-10-20 00:09. Newly + dirtied: `dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`; deepened: `0e519874d280` and + `ecc1fc13655a` (both streak 2). `fad5a86979d4` added a `delivery_failed` (bare-`telegram` target, thread + 34795 not found — no incident row created for it). `b63ed5817212`, `96eb95ce12f2` and `ea79a2107e36` all + cleared on their next slots. **Unclosed incident rows: 13** (12 `alerted` / 1 `detected`). → [[systems/scheduled-tasks]] +- **Unchanged otherwise:** fleet still **28 jobs / 25 enabled / 3 disabled**; CT255 Mattermost + (`100.96.100.255:8065`) still unreachable; **15 sessions in the window, all `cron` — no interactive sessions.** ## Recent Changes (2026-10-19, nightly audit; live probes 2026-10-19 00:04-00:2x AWST) diff --git a/incidents/2026-10-16-omniroute-503-resource-pressure.md b/incidents/2026-10-16-omniroute-503-resource-pressure.md index 5999206..4fbaf58 100644 --- a/incidents/2026-10-16-omniroute-503-resource-pressure.md +++ b/incidents/2026-10-16-omniroute-503-resource-pressure.md @@ -1,27 +1,33 @@ --- -title: OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst +title: OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts type: incident -status: monitoring +status: active created: 2026-10-18 -updated: 2026-10-18 -verified_on: 2026-10-18 -review_after: 2026-10-25 +updated: 2026-10-20 +verified_on: 2026-10-20 +review_after: 2026-10-27 confidence: high tags: [cron, router, provider, infrastructure] sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe] --- -# OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst (2026-10-16 → 2026-10-17) +# OmniRoute `omni-101` Resource-Pressure 503s — Recurring Cron Failure Bursts (burst 1: 2026-10-16 → 10-17; burst 2: 2026-10-19) -**Window:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**. -**Status:** **Monitoring.** Every slot from the 2026-10-17 08:30 run onward completed green -(10-16 = **165 completed / 8 failed**, 10-17 = **162 / 5**, and the 10-17 failures all pre-date 03:01). +**Status: ACTIVE — this is a recurrence, not a closed incident.** The failure mode came back **twice in four +days** (10-16 → 10-17 and 10-19) with an identical signature on an identical node, and the **exact same +9-model-call blast radius** each time. The node was healthy before, between and after both bursts. + +**Burst 1:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**. +10-16 = **165 completed / 8 failed**, 10-17 = **163 / 5** (all five pre-03:01). +**Burst 2:** **2026-10-19**, two clusters — **06:01** (single job) and **07:01 → 12:01** (six slots across seven +jobs). 10-19 = **159 completed / 13 failed** against the previous day's 168 / 1. The node answers `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` (auth-required) — re-probed -2026-10-18 00:33 AWST. -**Impact:** **13 failed execution rows** — **9 genuine model-call failures across 7 jobs**, plus **4 derivative -Mnemosyne Watchdog rows**. Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean day -(10-12 = 175 / 0). Unlike the 10-14 storm, **the node was up and serving throughout** — model calls failed with -`503` resource-pressure responses rather than TCP refusals. +**2026-10-20 00:03 AWST**, mid-recovery, still healthy. +**Impact:** burst 1 = **13 failed execution rows** (**9 genuine model-call failures across 7 jobs** + 4 derivative +Mnemosyne Watchdog rows); burst 2 = **13 failed execution rows** (**9 genuine model-call failures across 8 jobs** ++ 4 derivative Mnemosyne Watchdog rows). Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean +day (10-12 = 175 / 0). Unlike the 10-14 storm, **the node was up and serving throughout both bursts** — model +calls failed with `503` resource-pressure responses rather than TCP refusals. ## What happened @@ -68,6 +74,70 @@ clean through it. - **No node-side telemetry was retained.** Nothing on the Hermes side records host CPU/RAM/queue depth at 10-16 08:00, so the burst can be described but not explained. Treat the root cause as **unknown**. +## Recurrence — burst 2 (2026-10-19) + +An **identical** failure returned 44 hours after burst 1 ended, on the same node, with the same error string and +the same number of genuine model-call failures. Nothing on the Hermes side changed between the two bursts. + +| AWST (2026-10-19) | Job (name and schedule per `jobs.json`) | Slot | Result | Failure streak after | +|---|---|---|---|---| +| 06:01:05 | `b63ed5817212` Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | `RuntimeError: HTTP 503 … resource pressure` | 1 | +| 07:01 | `dca8482e4f76` Daily Marketing Job Radar | `0 7 * * 1-5` | same | 1 | +| 07:01 | `85edf6772bca` Tech & AI Newsletter Digest (Analytical) | `0 7 * * *` | same | 1 | +| 08:01 | `0e519874d280` Political News Digest (Mon/Thu) | `0 8 * * 1,4` | same | 1 → **2** | +| 08:31 | `08eb86f02d37` Gmail Action Digest | `30 8 * * *` | same | 1 | +| 09:01 | `c0a9fa314542` GitHub Trending Radar — Weekly Roundup | `0 9 * * 1` | same | 1 | +| 12:01:16 | `b63ed5817212` Mnemosyne Embedding Coverage Monitor | `0 */6 * * *` | same (2nd slot of the day) | **2** | +| 12:01 | `ea79a2107e36` Mnemosyne Health Monitor | `0 */12 * * *` | same | 1 | +| 12:01 | `ecc1fc13655a` FreshRSS Midday News Briefing | `0 12 * * *` | same | 1 → **2** | +| 06:16, 06:31, 12:16, 12:31 | `96eb95ce12f2` Mnemosyne Watchdog | `*/15 * * * *` | `Script exited with code 1`, stdout quoting `b63ed5817212`'s 503 — **derivative** | 1 | + +**9 genuine model-call failures across 8 jobs** (burst 1 was 9 across 7), plus the usual 4 derivative watchdog +rows = **13 failed execution rows**, the same total as burst 1. Whole day: **159 completed / 13 failed** (10-18 was +168 / 1; 10-17 was 163 / 5). + +- **Failed rows by hour:** 06:xx → 3 (one model call + two derivative watchdog reds), 07:xx → 2, 08:xx → 2, + 09:xx → 1, 12:xx → 5. Nothing after 12:31 failed — the 15:00 → 23:00 slots and the 10-20 00:00 slots all + completed. +- **Not one failure landed in the `no_agent` / script jobs' own logic.** Every 503 was on an `agent`-mode job + whose model call sits on `custom:omni-101`, exactly as in burst 1. +- **The blast radius is now the stable signature**: the same nine job/slot combinations that carry the + heaviest prompts fail first, while short-prompt jobs on the same node come through. A node under load-shedding + is dropping the *large-context* calls preferentially. +- **Single pin, now verified:** as of 2026-10-20 grep of `jobs.json` shows **every one of these eight jobs reading + `provider: custom:omni-101`, `model: smart-route`** — no alias diversity at all. ⚠ The burst-1 note above + records a wider alias mix (`deepseek-paid-smart`, `free-IA`, `dark`); `jobs.json` no longer matches that, so + the pins may have been consolidated since 10-16. **Confirm with Anthony before relying on either reading.** +- **Three incident rows auto-closed** — `96eb95_565ec1deb1c7` at **12:46:11** (the Watchdog's 12:30 slot + recovered once its subject job recovered), `b63ed5_bbe44c89aac1` at **18:01:56** (the 18:01 coverage slot + succeeded) and `ea79a2_747b27556fd0` at **2026-10-20 00:04:24** (the Health Monitor's 00:01 slot succeeded). + **Four new incident rows opened** (`dca848_867d9caa96a8`, `85edf6_256c5e150bc4`, `0e5198_ec2fac98a91d`, + `c0a9fa_764558b5ae83`) because those four jobs had never carried this error signature — the other five 503s + folded into pre-existing rows, and `08eb86_472c7b609c80` + `ecc1fc_c9976ea24553` were **refreshed from + 2026-10-16**, not reopened. **Unclosed ledger rows: 13** (12 `alerted` / 1 `detected`) as of 2026-10-20 00:09; + 4 opened and 2 closed on 10-19. *(Caution: the 10-16/10-18/10-19 entries reported `12`, `8` and `7` open — + those figures used a narrower filter than a raw `closed_at IS NULL` count, which reads 12 for 10-19 00:20 and + 13 now. Compare like with like from here on.)* +- **Every failure streak is still 1 or 2.** No job has consecutively failed enough to be suppressed, and the + 10-20 00:00 slots all ran clean. Non-clean `last_status` across the fleet moved **7 → 12** (10 `error`, + 2 `delivery_failed`) — see [[systems/scheduled-tasks]]. +- **Re-probed 2026-10-20 00:03 AWST:** `omniroute-1` → `/api/health` `200 {"status":"ok"}`, `/v1/models` `401`. + Healthy, as it was during both bursts. **A green health endpoint still proves nothing about the model path.** + +### What this changes + +- The 10-18 assessment — *"it recovered on its own; nothing was changed on the node; monitoring"* — is + **superseded**. A self-clearing fault that returns twice in four days with an identical blast radius is a + **standing operational risk**, not a closed incident. Status moved `monitoring` → **`active`**. +- Burst 1 left **no captured host telemetry**; burst 2 was likewise *not* instrumented while it was happening, + for the second time. The "capture metrics during the burst" lesson from 10-18 was not acted on. **The next + occurrence must be caught live** (see [[systems/scheduled-tasks]] for the probe to run). +- **`github-trending-radar`'s 2026-10-19 09:01 slot — its scheduled Monday run (`0 9 * * 1`) — was lost to + the 503**, so this week's roundup never went out. Its revoked Telegram delivery token (recorded in + [[systems/scheduled-tasks]] Known Issues and [[current-state]]) **remains untested for a second week**, + because the job has not reached its own code on a Monday since 2026-10-12. The next Monday slot + (2026-10-26) is the first real test. + ## Why this is not a repeat of the 10-14 storm | | 2026-10-14 → 10-15 | 2026-10-16 → 10-17 | @@ -104,12 +174,19 @@ Both are the **same single-node topology problem**: all enabled model-calling cr 4. **A 503 burst leaves no evidence behind.** No host telemetry is captured at failure time, so the "resource pressure" cause is taken on the router's word. If this recurs, capture the node's host metrics *during* the burst, not afterwards. +5. **A self-clearing burst is not a resolved incident.** Burst 1 closed as "monitoring" on 10-18 and the + identical fault returned within 44 hours. **Recurrence with an identical signature and an identical blast + radius is the signal that a fault is systemic** — the second occurrence, not the first, is the one that + should change a page's `status`. Two occurrences, same shape, four days apart, on a topology with no + failover: treat the next one as expected. ## Open item (needs Anthony's call, unchanged from 10-14) The durable fix is the same one flagged in [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]: a second provider pin on a subset of jobs, or router-side failover between `omniroute-1` and CT600 -`omniroute`. **Not applied** — that is a routing decision, not an audit one. +`omniroute`. **Not applied** — that is a routing decision, not an audit one. The 10-19 recurrence raises the +cost of leaving it open: **two of the last four days took double-digit failures from this one node**, and every +model-calling cron job still has exactly one route. ## Related diff --git a/index.md b/index.md index 5cb0d45..093f809 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-19 +> model routing and automation. Last updated: 2026-10-20 > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -34,12 +34,13 @@ ### Integrations - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP). - [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip. -- [[systems/scheduled-tasks]] — **28 cron jobs (25 enabled, 3 disabled)**; **7 carry a non-clean `last_status`** as of 2026-10-19 00:2x — down from 12. The **2026-10-18 window was clean except one run**: `ecc1fc13655a` (FreshRSS Midday) delivered a canned non-answer to Telegram and then died on a model repetition loop → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. The **`omniroute-1` 503 resource-pressure window (2026-10-16 08:00 → 10-17 03:00)** cost **9 model-call failures across 7 jobs** and was followed by a clean 10-17 06:00 → 10-18 window; the Mosquito Tracker was removed (29 → 28 jobs). See [[incidents/2026-10-16-omniroute-503-resource-pressure]]. Prior incident: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs** and ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/scheduled-tasks]] — **28 cron jobs (25 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-20 00:09 — **up from 7**, because the **`omniroute-1` 503 resource-pressure fault returned on 2026-10-19** (second burst in four days: **9 model calls across 8 jobs**, 06:01 and 07:01 → 12:01, all `model: smart-route` on `custom:omni-101`, while `/api/health` stayed `200 ok`) → [[incidents/2026-10-16-omniroute-503-resource-pressure]]. 2026-10-19 closed **159 / 13** against 10-18's 168 / 1. The **Mosquito Tracker was removed** (29 → 28 jobs), and the **2026-10-18 degenerate-stream** placeholder-delivery mode remains an open question → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Earlier: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs**, ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** - [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13). ## Incidents - [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route? -- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **OmniRoute's second failure mode, 2026-10-16 → 10-17:** `omniroute-1` load-shed model calls with **`HTTP 503 … resource pressure`** — **9 model calls across 7 jobs** over ~19 h, while `/api/health` stayed `200 ok` and `/v1/models` stayed `401`. **Self-cleared** (last 503 10-17 03:00:28; 10-17 closed 162 / 5, 10-18 so far 7 / 0). 9 incidents opened, 6 closed; **8 open** overall, and the ledger tags **7 of the 9 `failure_type: unknown`**. **Single-node topology still open.** +- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **`omniroute-1` load-sheds model calls with `HTTP 503 … resource pressure` — two bursts: 2026-10-16 → 10-17, and again 2026-10-19.** Burst 1: **9 model calls across 7 jobs** over ~19 h, self-cleared (reported then as `10-17 = 162 / 5`; the DB reads **163 / 5** — corrected 2026-10-20). Burst 2, 44 h later with an identical signature and blast radius: **9 model calls across 8 jobs** on 2026-10-19 (06:01 + 07:01 → 12:01, whole day **159 / 13**). `/api/health` read `200 ok` throughout both. **Status back to `active`** — a self-clearing fault that returns twice in four days is a standing risk, not a closed incident. 4 incident rows opened and 2 auto-closed on 10-19 (a third, `ea79a2_747b27556fd0`, closed 10-20 00:04); **13 unclosed rows** overall (12 `alerted` / 1 `detected`); the ledger still tags every 503 row `failure_type: unknown`, so triage on error text. **Single-node topology still open — and every job that failed reads `model: smart-route`, `provider: custom:omni-101`.** + - [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved) - [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00) - [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved) diff --git a/log.md b/log.md index 8264808..3624d2f 100644 --- a/log.md +++ b/log.md @@ -912,3 +912,31 @@ with a working tree carrying uncommitted 10-17/10-18 changes, and it has **no re pushed. **Not fixed** — a full mirror pass is a bigger call than an audit's mandate; flagged here. No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only. + +## [2026-10-20] audit | Nightly wiki audit — `omni-101` 503 burst 2 (2026-10-19) + +- **Incident re-opened.** [[incidents/2026-10-16-omniroute-503-resource-pressure]] went back to + `status: active`. A second, identical `HTTP 503 … resource pressure` burst hit `omniroute-1` on 2026-10-19 — + **9 model calls across 8 jobs** (06:01, then 07:01 → 12:01) — against burst 1's 9 across 7 + (2026-10-16 08:00 → 10-17 03:00). Whole day **159 completed / 13 failed**; 2026-10-18 was 168 / 1. +- **Facts corrected from the source of truth** (`~/.hermes/cron/jobs.json`, `executions.db`) instead of being + carried forward: the burst-2 table's job *names*, *schedules* and *model aliases* were wrong + (`85edf6772bca` is **Tech & AI Newsletter Digest (Analytical)**, not "Fringe Festival Plan"; `ea79a2107e36` + is the **Mnemosyne Health Monitor**, not a news digest; `ecc1fc13655a` is **FreshRSS Midday News Briefing**), + `08eb86f02d37` fails at **08:31** (not 08:01, it is a `30 8 * * *` job), and 2026-10-17 reads **163 / 5**, + not 162 / 5. **All eight failing jobs read `provider: custom:omni-101`, `model: smart-route`** — the burst-1 + note's wider alias mix (`deepseek-paid-smart`, `free-IA`, `dark`) no longer matches `jobs.json` and is now + flagged in the page for confirmation rather than silently overwritten. +- **Incident ledger read properly for the first time.** Unclosed rows = **13** (12 `alerted` / 1 `detected`), + not the `7` the 10-19 entry recorded — the earlier figures used a narrower filter. A raw `closed_at IS NULL` + count reads **12** at 10-19 00:20 and **13** now; the entry states the query so future entries compare like + with like. 4 rows opened and 2 closed on 10-19, plus `ea79a2_747b27556fd0` closed 10-20 00:04:24. +- **Pages updated:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (burst-2 table rebuilt from + `jobs.json`, new findings, status back to active), `systems/scheduled-tasks.md` (new current-count bullet, + non-clean `last_status` rewritten 7 → 12, Known Issues addition, 8 job-table rows corrected), + `systems/omniroute.md` (new Recent Changes section), `current-state.md` (new Recent Changes section), + `index.md` (scheduled-tasks and incident summaries, "Last updated" → 2026-10-20), `log.md`. +- **Nothing removed or archived** — no page became obsolete and no one-way sync was run. +- **Fleet:** 28 jobs / 25 enabled / 3 disabled (unchanged). **15 sessions in the window, all `cron`** — no + interactive sessions, so there were no user-facing decisions to record. +- No secrets written; host names, IPs, job/incident IDs, ports and probe status codes only. diff --git a/systems/omniroute.md b/systems/omniroute.md index 9bb2cc4..de0c246 100644 --- a/systems/omniroute.md +++ b/systems/omniroute.md @@ -3,8 +3,8 @@ title: OmniRoute type: system status: active created: 2026-07-22 -updated: 2026-10-18 -verified_on: 2026-10-18 +updated: 2026-10-20 +verified_on: 2026-10-20 confidence: high tags: [routing, provider, model, inference] sources: [raw/configs/hermes-config-sanitized.txt] @@ -12,6 +12,26 @@ sources: [raw/configs/hermes-config-sanitized.txt] # OmniRoute +## Recent Changes (2026-10-20) + +- **Recurring `503 … resource pressure` load-shedding on `omniroute-1` — second burst in four days.** On + **2026-10-19** the node failed **9 model calls across 8 jobs** (06:01, then 07:01 → 12:01, plus 4 derivative + Mnemosyne Watchdog reds) with + `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.`, while + `/api/health` answered `200 {"status":"ok"}` and `/v1/models` answered `401` throughout. Burst 1 ran + **2026-10-16 08:00 → 10-17 03:00** with the identical error string and the identical 9-call blast radius. + Both bursts **self-cleared**; nothing was changed on CT600 or `omniroute-1` in between. + **Root cause unknown — no host telemetry was captured during either burst.** + → [[incidents/2026-10-16-omniroute-503-resource-pressure]] +- **Every failing job is pinned to the same alias on the same node.** All eight read + `provider: custom:omni-101`, `model: smart-route` in `~/.hermes/cron/jobs.json` (verified 2026-10-20). + With no second provider pin and no router-side failover between `omniroute-1` and CT600 `omniroute`, one + node's load-shedding blanks the fleet's digest slots. +- **Probe properly.** A green `/api/health` proves nothing about the model path — it stayed `200 ok` through a + ~19 h burst and a second 7 h burst. To catch the next one, run a **real completion** against + `custom:omni-101` while jobs are still failing, and capture the node's CPU / RAM / queue depth *during* the + burst, not afterwards. + ## Purpose Secondary provider/router for Hermes auxiliary roles and some agent routing. diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index 8aaf860..d0d2097 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,9 +3,9 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-19 -verified_on: 2026-10-19 -review_after: 2026-10-25 +updated: 2026-10-20 +verified_on: 2026-10-20 +review_after: 2026-10-27 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] @@ -14,11 +14,12 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count -- **✅ Current as of 2026-10-19 00:20 AWST (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]] +- **⚠ Current as of 2026-10-20 00:09 AWST (`jobs.json.updated_at 2026-10-20T00:09:07+08:00`) — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]] +- **Current as of 2026-10-19 00:20 AWST (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]] - **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] - **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-19 00:20 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. -- **Non-clean `last_status`:** **7 as of 2026-10-19 00:20** — `1b23c042e17d` (`error`, streak 2), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `ecc1fc13655a` (`error`, streak 1 — degenerate stream, 10-18 12:08), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`, streak 0). `cbfe434a3b5e` and `1a8e90ea2271` cleared on schedule (10-18 00:42:54 and 03:01:22). History: **3** at the 2026-10-12 audit (when `fad5a86979d4` and `dca8482e4f76` cleared on schedule) → **12** at the 2026-10-16 storm peak → 8 at the 2026-10-18 audit → **7 now**. The six jobs that fixed themselves did so on their next scheduled slot; the remaining weekly ones (`1b23c042e17d` Fri, `0e519874d280` Mon, `06a0a1c8704f` + `c8d52904e1d6` Thu) cannot clear before their next run. +- **Non-clean `last_status`:** **12 as of 2026-10-20 00:09** — 10 `error` + 2 `delivery_failed`. The burst re-dirtied four rows that were clean at the 10-19 audit (`dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`) and deepened two of the existing seven (`0e519874d280` and `ecc1fc13655a` both to `failure_streak: 2`); `fad5a86979d4` newly reads `delivery_failed` (bare-`telegram` target). `b63ed5817212` (dirtied twice) and `96eb95ce12f2` (four derivative reds) **cleared the same day** (18:01:56 / 12:46:11), and `ea79a2107e36` cleared on its 10-20 00:01 slot. Survivors, with next run: `ecc1fc13655a` (streak **2**, 10-20 12:00), `0e519874d280` (**2**, Thu 10-22 08:00), `1b23c042e17d` (**2**, Fri 10-23 08:00), `dca8482e4f76` (1, **10-20 07:00**), `85edf6772bca` (1, **10-20 07:00**), `08eb86f02d37` (1, 10-20 08:30), `c0a9fa314542` (1, **Mon 10-26 09:00**), `06a0a1c8704f` (1, Thu 10-22 10:00), `c8d52904e1d6` (1, Thu 10-22 18:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 0, 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → **12 now**. Three of these fire within eight hours of this audit, so the count will move again by morning. - **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). - **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. @@ -40,26 +41,26 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | -| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. Live `last_status: ok`, streak 0. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open) | +| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic | -| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). Live `last_status: ok`, streak 0. ⚠ Bare `telegram` target should move to a group topic | -| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Live `last_status: error`, `failure_streak: 1`, next slot **2026-10-19 12:00** | -| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. `last_status: ok` because the 401 is inside the script, not the cron run. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** | -| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); cleared by the 10-17 08:31 run, which auto-closed the incident. Live `last_status: ok` | +| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). ⚠ **Live status is no longer `ok`:** the 2026-10-19 03:04:47 run completed but recorded `delivery_failed` — `configured thread_id 34795 for telegram:1793951355:34795 was not found` (the CLI renders it as "ran, but the result was not delivered", then "delivered without thread_id"). The run is fine; only the send is. **No incident row was created for this.** ⚠ Bare `telegram` target should move to a group topic | +| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. ⚠ **Then the 2026-10-19 12:01 slot failed again, on the `omni-101` 503** — `ecc1fc_c9976ea24553` (a **10-16** row, refreshed 10-19 12:02:58, `alerted`) plus the separate `ecc1fc_21a72aa11cc4` degenerate-stream row from 10-18, so **this one job holds two open incidents**. Live `last_status: error`, `failure_streak: 2`, next slot **2026-10-20 12:00** | +| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** ⚠ **The status reading changed again on 2026-10-19:** its scheduled Monday slot (`0 9 * * 1`) at **09:01 failed on the `omni-101` 503** (new incident `c0a9fa_764558b5ae83`, `alerted`), so **this week's roundup did not run at all** and `last_status` now reads `error` (streak 1). The token question is therefore **untested for a second consecutive Monday** — the job has not reached `digest.py` on a Monday since 2026-10-12. Next real test: **Mon 2026-10-26 09:00**. | +| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); and its **own 10-17 08:31** run restored a clean `last_status`. ⚠ **But the incident never closed: `08eb86_472c7b609c80` was refreshed on 2026-10-19 08:33 by the same 503, and is still `alerted`.** Live `last_status: error`, streak 1 | | Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) | | Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent | | Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent | | Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent | -| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run | -| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots (10-16 12:00, 10-17 00:00); cleared by its own **10-17 12:02** run | -| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults | +| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run. ⚠ **Both 2026-10-19 slots failed the same way** (06:01:05 and 12:01:16), taking its streak to **2** and re-triggering the watchdog; its **18:01 slot succeeded**, clearing the streak and auto-closing `b63ed5_bbe44c89aac1` at 18:01:56. Live `last_status: ok`. With `0 */6 * * *` this job is the fleet's best 503 sentinel — four probes a day | +| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 | +| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults. **The pattern repeated exactly in burst 2** — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at **12:46:11** (`96eb95_565ec1deb1c7`, a 10-16 row refreshed through 10-19 12:31). **8 of the 26 failure rows across the two bursts were this one sentinel** | | Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` | -| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | | +| Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) | -| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | | +| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | ⚠ **2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `85edf6_256c5e150bc4`, `alerted`) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live `last_status: error`, streak 1, next slot 2026-10-20 07:00 | | | Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic | | Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 | | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 | @@ -119,7 +120,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]]. - **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]. -- **The `omni-101` 503 burst left no node-side evidence (NEW 2026-10-18):** 9 model calls across 7 jobs died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` between 2026-10-16 08:00 and 2026-10-17 03:00 — while the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-18 00:33). Nothing on the Hermes side records the node's CPU/RAM/queue depth at failure time, so the "resource pressure" cause is taken on the router's word; if it recurs, capture host metrics *during* the burst, not afterwards. The incident ledger also tags **seven of the nine** rows `failure_type: unknown` **despite** the explicit message (only the two script-exit watchdog incidents get a type), so `failure_type` cannot be used to triage 503s — filter on the error text. Root cause **unknown**; not a repeat of the 10-14 TCP-refusal storm (which was reachability, this was serving). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] +- **The `omni-101` 503 burst left no node-side evidence (NEW 2026-10-18):** 9 model calls across 7 jobs died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` between 2026-10-16 08:00 and 2026-10-17 03:00 — while the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-18 00:33). Nothing on the Hermes side records the node's CPU/RAM/queue depth at failure time, so the "resource pressure" cause is taken on the router's word; if it recurs, capture host metrics *during* the burst, not afterwards. The incident ledger also tags **seven of the nine** rows `failure_type: unknown` **despite** the explicit message (only the two script-exit watchdog incidents get a type), so `failure_type` cannot be used to triage 503s — filter on the error text. Root cause **unknown**; not a repeat of the 10-14 TCP-refusal storm (which was reachability, this was serving). +- **It recurred on 2026-10-19 with the same 9-call blast radius (NEW 2026-10-20):** the identical `HTTP 503 … resource pressure` string took **9 model calls across 8 jobs** again (06:01, then 07:01 → 12:01), and again the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-20 00:03). **Two bursts, four days apart, same node, same shape, no failover** — and nothing was instrumented during either one. **Action for the next occurrence: while jobs are still failing, run a real completion against `custom:omni-101` and capture the node's host CPU / RAM / queue depth — not afterwards.** The ledger again tagged every new row `failure_type: unknown`, so find these by the **error text**, never by `failure_type`. **Also verified 2026-10-20:** all eight affected jobs read `provider: custom:omni-101`, `model: smart-route` in `jobs.json` — one alias, one node, no diversity. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] → [[incidents/2026-10-16-omniroute-503-resource-pressure]] ### Historical - **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].