diff --git a/current-state.md b/current-state.md index d8c0633..6723045 100644 --- a/current-state.md +++ b/current-state.md @@ -3,8 +3,8 @@ title: Current State type: concept status: active created: 2026-07-22 -updated: 2026-10-16 -verified_on: 2026-10-16 +updated: 2026-10-18 +verified_on: 2026-10-18 review_after: 2026-10-25 confidence: high tags: [system] @@ -12,7 +12,42 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv --- # Current State -Updated: 2026-10-16 +Updated: 2026-10-18 + +## Recent Changes (2026-10-18, nightly audit; live probes 2026-10-18 00:31-00:36 AWST) + +- **The `omniroute-1` node is up — and this time it shed load instead of refusing connections.** Three days + after the 34-hour TCP blackout, the same node produced a second, different failure mode: **9 model calls + across 7 jobs died with `RuntimeError: HTTP 503: Service temporarily unavailable due to resource + pressure.`** between **10-16 08:00:37 and 10-17 03:00:28** (~19 h), while `/api/health` answered + `200 {"status":"ok"}` and `/v1/models` answered `401` before, during and after (re-probed 10-18 00:33). + → [[incidents/2026-10-16-omniroute-503-resource-pressure]] +- **It cleared itself and the fleet is green again.** `10-16 = 165 completed / 8 failed`, `10-17 = 162 / 5` + (**all five before 03:01**), `10-18` so far `7 / 0` — **no failure row of any kind since 10-17 03:00**. + Every job that failed re-ran clean on its next slot: `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 + 08:31, `ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, each of those auto-closing its own incident. +- **Fleet count fell 29 → 28: the Mosquito Tracker was removed.** `1b31187397a5` is absent from + `jobs.json` (file `updated_at 2026-10-18T00:36:17+08:00`); its last execution was **10-16 15:00** + (`completed`, delivered). The three disabled jobs are unchanged — **25 enabled / 3 disabled**. +- **Non-clean `last_status`: 12 → 8.** Two of the eight cannot self-heal before their next slot — + `1b23c042e17d` sits at `error` (streak 2) until **Fri 10-23 08:00**; `0e519874d280`, `06a0a1c8704f`, + `c8d52904e1d6`, `0e753e9720fd` and `066bb5191280` are unchanged. **Open incidents 12 → 8** + (5 `alerted` / 3 `detected`); **9 opened and 6 closed** inside this window. → [[systems/scheduled-tasks]] +- **The watchdog double-counts — 4 of the 13 failure rows are one sentinel.** `96eb95ce12f2` exited 1 on + 10-16 12:15 + 12:30 and 10-17 00:15 + 00:30, each row quoting the coverage monitor's own 503 error. + Read failures per job, not per row. → [[systems/mnemosyne]] +- **Second consecutive stocktake hole.** The 10-17 03:00 slot failed (503), so + `raw/stocktake/freshness-20261017.txt` does not exist and `systems/skills-index.md` is stamped **10-16** + (10-15 is missing too, from the storm — `RuntimeError: Connection error.`). The **10-18 03:00** slot + closes it; nothing was regenerated by hand, because a manual run would race the scheduled one. +- **Fifth consecutive window with no interactive session** — **30 sessions** across the two days, all + `cron`, and **zero** non-cron sessions since 2026-10-15. Every finding here comes from live system + state, not conversation. +- **The GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in + `~/workspace/github-trending-radar/config.json` returns **HTTP 401** on `getMe` (probed 10-18 00:35); + the job's GitHub PAT still answers **200**, so only the delivery credential is gone. The cron row stays + green because `last_status` was set by the **10-12 09:00** run (`last_run_at 2026-10-12T09:00:12`). + Next exercise: **Mon 2026-10-19 09:00**. **Not repaired** — a replacement bot token is Anthony's call. ## Recent Changes (2026-10-16, nightly audit; live probes 2026-10-16 00:0x–00:2x AWST) diff --git a/incidents/2026-10-16-omniroute-503-resource-pressure.md b/incidents/2026-10-16-omniroute-503-resource-pressure.md new file mode 100644 index 0000000..5999206 --- /dev/null +++ b/incidents/2026-10-16-omniroute-503-resource-pressure.md @@ -0,0 +1,119 @@ +--- +title: OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst +type: incident +status: monitoring +created: 2026-10-18 +updated: 2026-10-18 +verified_on: 2026-10-18 +review_after: 2026-10-25 +confidence: high +tags: [cron, router, provider, infrastructure] +sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-jobs-json, live-probe] +--- + +# OmniRoute `omni-101` Resource-Pressure 503s — Cron Failure Burst (2026-10-16 → 2026-10-17) + +**Window:** first 503 failure row **2026-10-16 08:00:37**, last **2026-10-17 03:00:28** AWST — **19 hours**. +**Status:** **Monitoring.** Every slot from the 2026-10-17 08:30 run onward completed green +(10-16 = **165 completed / 8 failed**, 10-17 = **162 / 5**, and the 10-17 failures all pre-date 03:01). +The node answers `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` (auth-required) — re-probed +2026-10-18 00:33 AWST. +**Impact:** **13 failed execution rows** — **9 genuine model-call failures across 7 jobs**, plus **4 derivative +Mnemosyne Watchdog rows**. Against the 10-14/10-15 storm tail (10-15 = 152 / 24) and the last clean day +(10-12 = 175 / 0). Unlike the 10-14 storm, **the node was up and serving throughout** — model calls failed with +`503` resource-pressure responses rather than TCP refusals. + +## What happened + +Every model call on `custom:omni-101` (`http://100.78.63.20:20128/v1`) between the 10-16 08:00 and +10-17 03:00 slots died with the identical error: + +``` +RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly. +``` + +| Job | Slot(s) AWST | Rows | Outcome | +|---|---|---|---| +| `1b23c042e17d` Enhanced Weekend Adventure Planner | 10-16 08:00 (Fri slot) | 1 | failed; streak **2**; next slot **Fri 10-23 08:00** | +| `08eb86f02d37` Gmail Action Digest | 10-16 08:30 | 1 | failed; cleared 10-17 08:31 (own next run) | +| `b63ed5817212` Mnemosyne Embedding Coverage Monitor | 10-16 12:00, 10-17 00:00 | 2 | failed; cleared **10-17 06:01** | +| `ecc1fc13655a` FreshRSS Midday News Briefing | 10-16 12:00 | 1 | failed; cleared 10-17 12:02 | +| `ea79a2107e36` Mnemosyne Health Monitor | 10-16 12:00, 10-17 00:00 | 2 | failed; cleared 10-17 12:02 | +| `cbfe434a3b5e` Wiki Audit | 10-16 23:59 | 1 | failed; incident **open** (`detected`) — the 10-17 23:59 run is the one that produced this audit | +| `1a8e90ea2271` Wiki Stocktake | 10-17 03:00 | 1 | failed; incident **open** (`detected`) | +| `96eb95ce12f2` Mnemosyne Watchdog | 10-16 12:15 + 12:30, 10-17 00:15 + 00:30 | 4 | **derivative** — `Script exited with code 1`, its stdout quoting the watchdog's own log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…` | + +**8 rows on 10-16, 5 on 10-17.** Reading per job rather than per row: **7 jobs actually failed**, and four of +the 13 rows are one sentinel watching another job's 503 — the same double-counting trap as the 10-14 storm +(see [[incidents/2026-10-14-omniroute-node-connection-refused-storm]], lesson 4). + +Every 503 row sits on **`custom:omni-101`** — one alias, one node, again the single pin. +No other job failed in this window: `85edf6772bca` (Tech & AI) and `dca8482e4f76` (Job Radar) both ran +clean through it. + +## Evidence + +- `~/.hermes/cron/executions.db` — the 9 model-call rows each carry the literal + `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` + The 4 watchdog rows carry `Script exited with code 1` plus the quoting stdout above. +- `~/.hermes/cron/executions.db:cron_incidents` — **9 incidents opened in the window, 6 closed, 3 still open** + (`1b23c0_103cbe04a79b` `alerted`; `cbfe43_9b89947a046a` and `1a8e90_3c9c1d573cde` `detected`). + **The ledger tags 7 of the 9 `failure_type: unknown`** even though the message was explicit (the two + watchdog rows are classed `script`, correctly — they *are* script exits); for the seven real 503s the + classifier had the message in hand and still filed no type. Open incidents overall: **12 → 8** + (5 `alerted` / 3 `detected`). +- **The node itself was never down.** `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` on + `100.78.63.20:20128` at 2026-10-18 00:33 AWST. A 503 is a *serving* failure, so reachability probes + (which the nightly audit and the operator both run) could not have caught this — only job-level error text did. +- **No node-side telemetry was retained.** Nothing on the Hermes side records host CPU/RAM/queue depth at + 10-16 08:00, so the burst can be described but not explained. Treat the root cause as **unknown**. + +## Why this is not a repeat of the 10-14 storm + +| | 2026-10-14 → 10-15 | 2026-10-16 → 10-17 | +|---|---|---| +| Failure class | `ConnectError: [Errno 111] Connection refused` | HTTP `503` resource pressure | +| Node state | TCP-unreachable for ~34 h | up and answering probes throughout | +| Blast radius | 12 jobs, 24 model-call failures, every call for 34 h | 7 jobs, 9 model-call failures, bursty, self-cleared | +| Recovery | manual node recovery | none needed — cleared on its own | + +Both are the **same single-node topology problem**: all enabled model-calling cron jobs ride +`custom:omni-101` with no failover, three days apart from the previous outage. + +## Consequences + +- **One stocktake hole:** `raw/stocktake/freshness-20261017.txt` was not written and + `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 03:00 slot succeeded, so + `freshness-20261016.txt` exists). The **2026-10-18 03:00 slot** closes it — nothing was regenerated by hand, + because a manual run would race the scheduled one. A failure there would be the second consecutive miss. +- **The Wiki Audit lost its 10-16 slot**, so no wiki pass covered the 10-14→10-17 span until this run. +- **Stale flags clear only at the next success, and some slots are a week apart.** `1b23c042e17d` is a Friday + job: it will sit at `last_status: error`, streak 2, with an open incident until **Fri 2026-10-23 08:00**. + On a weekly cadence a one-slot failure reads as a week-long outage. + +## Lessons + +1. **Same topology, second failure mode in three days.** A 503 from a *live* node cannot be detected by + reachability probes; only per-job error text distinguishes "node down" from "node overloaded". Both + classes gate the whole scheduled estate. +2. **Watchdog rows inflate the count — read per job.** 4 of the 13 rows are one log sentinel reacting to + another job's failure, not independent faults. +3. **The incident classifier drops the failure type.** All **seven** genuine 503 incidents are filed + `failure_type: unknown` despite an explicit message in the execution row (only the two script-exit + watchdog incidents get a type), so `failure_type` cannot be used to triage — filter on the error text. +4. **A 503 burst leaves no evidence behind.** No host telemetry is captured at failure time, so the "resource + pressure" cause is taken on the router's word. If this recurs, capture the node's host metrics *during* the + burst, not afterwards. + +## Open item (needs Anthony's call, unchanged from 10-14) + +The durable fix is the same one flagged in [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]: +a second provider pin on a subset of jobs, or router-side failover between `omniroute-1` and CT600 +`omniroute`. **Not applied** — that is a routing decision, not an audit one. + +## Related + +- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — the TCP-refusal storm, two days earlier, same node. +- [[incidents/2026-10-09-cron-provider-timeout-storm]] — the first multi-day storm from this topology. +- [[systems/omniroute]] — node layout, providers and pins. +- [[systems/scheduled-tasks]] — per-window fleet tally and incident ledger. diff --git a/index.md b/index.md index 157e2d6..c1caa5b 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ # Anthony's Systems Wiki > Operational knowledge for Anthony's agents, infrastructure, -> model routing and automation. Last updated: 2026-10-16 +> model routing and automation. Last updated: 2026-10-18 > > ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered > hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake. @@ -33,10 +33,11 @@ ### Integrations - [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. -- [[systems/scheduled-tasks]] — **29 cron jobs (26 enabled, 3 disabled)**; **12 carry a non-clean `last_status`** as of 2026-10-16 00:1x — the **`omniroute-1` connection-refused storm (2026-10-14 → 10-15)** cost **24 model-call failures across 12 jobs** and blanked ~34 h of digests; **all 18 model-calling jobs sit on `custom:omni-101` with no failover**. See [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]. Prior audit: **4 carried a non-clean `last_status`** as of 2026-10-14 00:15. **The clean streak broke on 2026-10-13: `167 completed` / 1 `failed`**, after 10-12 finalised `175 / 0` (10-11 `172 / 0`, 10-10 `175 / 0`) — `1a8e90ea2271` (wiki-stocktake-maintenance, 03:00) hit `HTTP 404: No active credentials for provider: opencode`, its **third failure in six days**, so **open incidents rose 2 → 3**. Two rows cleared: `fad5a86979d4` (10-12 03:05), `dca8482e4f76` (10-13 07:02). The DHL monitor `62619aa6bcb3` was **retired** on 2026-10-10. ⚠ A green cron row is not proof of delivery: GitHub Trending Radar's Telegram token is revoked (`getMe` → 401) while the row stays green — it next runs Mon 09:00. See [[incidents/2026-10-09-cron-provider-timeout-storm]] -- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13). +- [[systems/scheduled-tasks]] — **28 cron jobs (25 enabled, 3 disabled)**; **8 carry a non-clean `last_status`** as of 2026-10-18 00:3x — down from 12. The **`omniroute-1` 503 resource-pressure window (2026-10-16 08:00 → 10-17 03:00)** cost **9 model-call failures across 7 jobs** and was followed by a clean 10-17 06:00 → 10-18 window; the Mosquito Tracker was removed (29 → 28 jobs). See [[incidents/2026-10-16-omniroute-503-resource-pressure]]. Prior incident: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs** and ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.** +- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ **2026-10-17** stocktake slot failed (`HTTP 503`), so `freshness-20261017.txt` is missing and the index is stamped **10-16**; the **10-18 03:00** slot closes it. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13). ## Incidents +- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **OmniRoute's second failure mode, 2026-10-16 → 10-17:** `omniroute-1` load-shed model calls with **`HTTP 503 … resource pressure`** — **9 model calls across 7 jobs** over ~19 h, while `/api/health` stayed `200 ok` and `/v1/models` stayed `401`. **Self-cleared** (last 503 10-17 03:00:28; 10-17 closed 162 / 5, 10-18 so far 7 / 0). 9 incidents opened, 6 closed; **8 open** overall, and the ledger tags **7 of the 9 `failure_type: unknown`**. **Single-node topology still open.** - [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved) - [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00) - [[incidents/2026-10-09-memory-recall-relevance]] — Recall relevance: a stale conversational episode outranked current-state facts on an open question; capture and vectorisation verified healthy, the ranking fix was never persisted in config (unresolved) diff --git a/log.md b/log.md index 74aa2c9..0787da4 100644 --- a/log.md +++ b/log.md @@ -780,3 +780,68 @@ a second provider pin on a subset of jobs, or router-side failover. **Not applie decision, not an audit one. No secrets written; host names, IPs, VMIDs and job/incident IDs only. + +## [2026-10-18] audit | nightly review of 2026-10-16 -> 2026-10-17 — OmniRoute's second failure mode (503 resource pressure), Mosquito Tracker removed, radar bot token dead + +**Window:** cron executions 2026-10-16 00:00 → 2026-10-17 24:00 (+ 10-18 00:00 → 00:36 live probes), ran +2026-10-18 00:31 → 00:5x AWST. **Sessions reviewed:** 30, **all `cron`, zero interactive** — fifth +consecutive window with no interactive session, so every finding below is from live system state, not +conversation. + +### Summary +- **`omniroute-1` (`custom:omni-101`) has a second failure mode: 503 load-shedding.** **9 model calls across + 7 jobs** died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry + shortly.` between **10-16 08:00:37** and **10-17 03:00:28** (~19 h). The control plane stayed up the whole + time (`/api/health` `200 {"status":"ok"}`, `/v1/models` `401` — re-probed 10-18 00:33), so a green health + endpoint is not evidence that the model path is serving. This is **not** a repeat of the 10-14 storm: + that was reachability (`Connection error.`, Errno 111), this was the router serving and refusing under + load. → `incidents/2026-10-16-omniroute-503-resource-pressure` (new). +- **The burst ended on its own and the fleet is green again.** `10-16 = 165 completed / 8 failed`, + `10-17 = 162 / 5` (all five before 03:01), `10-18` so far `7 / 0`; **no failure row of any kind since + 10-17 03:00**. Every affected job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, + `08eb86f02d37` 10-17 08:31, `ecc1fc13655a` + `ea79a2107e36` both 10-17 12:02, each auto-closing its own + incident. **No change was made on the node**; nothing was restarted or repinned. +- **Mosquito Tracker removed — fleet 29 → 28 (`1b31187397a5`).** Absent from `jobs.json` + (`updated_at 2026-10-18T00:36:17+08:00`), last execution 10-16 15:00 (`completed`, delivered). The three + disabled jobs are unchanged, so **25 enabled / 3 disabled**. Non-clean `last_status` **12 → 8**; only + `1b23c042e17d` (`error`, streak 2, next slot Fri 10-23 08:00) is structurally stuck. +- **The watchdog double-counts.** `96eb95ce12f2` exited 1 four times (10-16 12:15 + 12:30, 10-17 00:15 + + 00:30), each row quoting the coverage monitor's own 503 — 4 of the 13 failure rows in the window are one + sentinel. Read failures per job, not per row. +- **Second consecutive stocktake hole, again left alone deliberately.** The 10-17 03:00 slot failed, so + `raw/stocktake/freshness-20261017.txt` is missing and `systems/skills-index.md` is stamped 10-16; + 10-15 is missing too (storm). The **10-18 03:00** slot closes it — no manual regeneration, because a + hand-run would race the scheduled one. +- **GitHub Trending Radar is dead at the delivery end.** The Telegram bot token in its `config.json` + returns **HTTP 401** on `getMe` (probe 10-18 00:35); the job's GitHub PAT still returns **200**, so only + the delivery credential is gone. The cron row stays green (`last_status` from the 10-12 09:00 run, + `last_run_at 2026-10-12T09:00:12`). Next exercise **Mon 2026-10-19 09:00**. **Not repaired** — a new bot + token is Anthony's call, and the secret is not written here. +- **Incidents:** **9 opened, 6 closed** in-window; open **12 → 8** (5 `alerted` / 3 `detected`). The ledger + files **7 of the 9** 503 rows `failure_type: unknown` despite the explicit message (the two script-exit + rows are typed `script`), so `failure_type` still cannot be used to triage. +- **Mnemosyne:** both monitors rode the burst (`ea79a2107e36` 10-17 00:00, `b63ed5817212` 10-16 12:03), + both clean on every slot since — including 10-18 00:00 (`delivery_outcome: suppressed`). **No fresh + coverage figures exist** (they report to Telegram, not the ledger); **last explicit read stays + 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 (99.4 %)**, above the 95 % reindex + threshold, so no reindex is due. + +**Pages touched:** `incidents/2026-10-16-omniroute-503-resource-pressure.md` (**new**); +`systems/scheduled-tasks.md` (frontmatter + Count bullet + stocktake row + Known Issues entry); +`systems/omniroute.md` (frontmatter + new Recent Changes section); `systems/mnemosyne.md` (frontmatter + +new Re-verified section); `current-state.md` (frontmatter + `Updated:` line + new top section); +`index.md` (`Last updated`, new incident entry, scheduled-tasks bullet, skills-index note); +`log.md` (this entry). **Nothing removed or archived** — no page became obsolete in the window. + +**Bridge housekeeping:** the shared pages are byte-identical between the authoring tree +(`/home/hermes/workspace/agent-estate-wiki`) and the vault copy (`/home/hermes/wiki/agent-estate-wiki`, +pages nested under `wiki/`), so this run mirrored the touched pages into the vault *after* writing the +authoring tree, and left the vault-only pages (`systems/ignis.md`, `systems/nextcloud.md`, three +`incidents/` pages, `raw/stocktake/`, `agents/`) untouched. No deletions, no one-way sync. + +**Durable fix still open (needs Anthony's call):** the single-node topology — all 18 model-calling jobs on +`custom:omni-101` with no failover, now seen failing two different ways in five days. Options remain a +second provider pin on a subset of jobs, or router-side failover. **Not applied** — a routing decision, not +an audit one. + +No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only. diff --git a/systems/mnemosyne.md b/systems/mnemosyne.md index ee7d547..2bed290 100644 --- a/systems/mnemosyne.md +++ b/systems/mnemosyne.md @@ -3,8 +3,8 @@ title: Mnemosyne type: system status: active created: 2026-07-22 -updated: 2026-10-16 -verified_on: 2026-10-16 +updated: 2026-10-18 +verified_on: 2026-10-18 confidence: high tags: [memory, mnemosyne, backend] sources: [live-probe, mnemosyne-db, config, ops-profile-config] @@ -141,3 +141,17 @@ Primary agent memory store for Hermes. mode `0755`, so the monitor's non-executable finding is resolved. - **Open, unverified:** 13 orphan rows in `memory_embeddings` (oldest 2026-08-22) reported by a 2026-10-10 monitor run - neither confirmed nor cleaned by this audit. + +## Re-verified 2026-10-18 (audit window; live probes 2026-10-18 00:30-00:36 AWST) + +- **Both monitors rode the `omniroute-1` 503 window; both are green again.** `ea79a2107e36` (health, every + 12 h) failed **10-17 00:00** with `RuntimeError: HTTP 503 ... resource pressure` and delivered its alert; + `b63ed5817212` (coverage, every 6 h) failed **10-16 12:03** the same way. Since **10-17 12:00** every slot + of both has completed with `delivery_outcome: suppressed` (nothing to report) — including **10-18 00:00**. +- **No fresh coverage figures exist in the ledger.** These jobs report to Telegram, not to the execution + row, so the **last explicit read remains 2026-10-12 12:02 — working 760/786 (96.7 %), episodic 1050/1056 + (99.4 %)**, both above the 95 % reindex threshold. Nothing here indicates a reindex is due. +- **The watchdog triple-checks the same thing.** `96eb95ce12f2` (every 15 min) exited 1 four times — + 10-16 12:15 + 12:30, 10-17 00:15 + 00:30 — each row quoting the **coverage monitor's own 503 error**, i.e. + one sentinel producing four failure rows. Read failures **per job**, not per row. It has been clean on + every slot since (latest 10-18 00:30). diff --git a/systems/omniroute.md b/systems/omniroute.md index cec0b5a..9bb2cc4 100644 --- a/systems/omniroute.md +++ b/systems/omniroute.md @@ -3,8 +3,8 @@ title: OmniRoute type: system status: active created: 2026-07-22 -updated: 2026-10-16 -verified_on: 2026-10-16 +updated: 2026-10-18 +verified_on: 2026-10-18 confidence: high tags: [routing, provider, model, inference] sources: [raw/configs/hermes-config-sanitized.txt] @@ -66,6 +66,21 @@ The wiki previously recorded CT600 as retired and CT601 as the live successor. * - Scoring signals: health, quota/token-reset windows, latency, consecutive errors, and cost-inverse across all connected accounts; failover to next-best on throttle/error without dropping the request. - Example observed: an `auto/best-free` request routed to Antigravity (Google AI Plus) because that account was active, healthy, zero-cost, and highest priority — proving live dynamic scoring, not a static map. +## Recent Changes (2026-10-18) — `omniroute-1` 503 resource pressure (2026-10-16 -> 2026-10-17) + +- **A second, different failure mode from the same node.** Three days after the 34-hour TCP blackout, + `omniroute-1` answered model calls with **`HTTP 503: Service temporarily unavailable due to resource + pressure. Retry shortly.`** — **9 model calls across 7 jobs**, between **10-16 08:00:37** and + **10-17 03:00:28** (~19 h). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] +- **The control plane looked healthy throughout.** `/api/health` returned `200 {"status":"ok"}` and + `/v1/models` returned `401` (auth-required) before, during and after the window — re-probed 2026-10-18 + 00:33. A green health endpoint does **not** mean the model path is serving: 503 here is load-shedding + under resource pressure, not a dead process, so process checks miss it. +- **It recovered on its own; nothing was changed on the node.** Last 503 **10-17 03:00:28**; `10-17` + closed **162 completed / 5 failed** (all five before 03:01) and `10-18` is green so far. Every affected + job re-ran clean on its next slot — `b63ed5817212` 10-17 06:01, `08eb86f02d37` 10-17 08:31, + `ecc1fc13655a` and `ea79a2107e36` both 10-17 12:02, auto-closing their own incidents. + ## Recent Changes (2026-10-16) — `omniroute-1` connection-refused storm - **`omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) was TCP-unreachable from 2026-10-14 diff --git a/systems/scheduled-tasks.md b/systems/scheduled-tasks.md index fda80f4..07a73e4 100644 --- a/systems/scheduled-tasks.md +++ b/systems/scheduled-tasks.md @@ -3,8 +3,8 @@ title: Scheduled Tasks type: system status: active created: 2026-07-22 -updated: 2026-10-16 -verified_on: 2026-10-16 +updated: 2026-10-18 +verified_on: 2026-10-18 review_after: 2026-10-25 confidence: high tags: [cron, scheduled, automation] @@ -14,9 +14,10 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] # Scheduled Tasks ## Count +- **✅ Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] -- **Total jobs:** **29 — 26 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json`, re-verified 2026-10-13 — unchanged; the count fell from 30 when the redundant DHL monitor `62619aa6bcb3` was retired on 2026-10-10) -- **Non-clean `last_status`:** **3** (was 5 at the 2026-10-12 audit) — `0e753e9720fd` (`error`, streak 0), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). `fad5a86979d4` and `dca8482e4f76` cleared on **2026-10-12** when both ran clean on schedule. +- **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-18 00:31 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. +- **Non-clean `last_status`:** **8 as of 2026-10-18 00:31** — `1b23c042e17d` (`error`, streak 2), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`, streak 0). History: **3** at the 2026-10-12 audit (when `fad5a86979d4` and `dca8482e4f76` cleared on schedule) → **12** at the 2026-10-16 storm peak → 8 now. The six jobs that fixed themselves did so on their next scheduled slot; the remaining weekly ones (`1b23c042e17d` Fri, `0e519874d280` Mon, `06a0a1c8704f` + `c8d52904e1d6` Thu) cannot clear before their next run. - **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). - **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. @@ -40,21 +41,20 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | | Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. Live `last_status: ok`, streak 0. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open) | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | -| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1. **Next scheduled slot: Fri 2026-10-16 08:00**, which should clear it | +| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic | | Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). Live `last_status: ok`, streak 0. ⚠ Bare `telegram` target should move to a group topic | -| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | | +| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed the incident. Live `last_status: ok` | | GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. `last_status: ok` because the 401 is inside the script, not the cron run. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** | -| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | | +| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); cleared by the 10-17 08:31 run, which auto-closed the incident. Live `last_status: ok` | | Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) | | Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent | | Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent | | Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent | -| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95% | -| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception | -| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30) | -| Mosquito Tracker | `1b31187397a5` | `0 7,15,22 * * *` | telegram:-1004321904721:273 | — | `mosquito_tracker.py` | Excludes 23:00–07:00 | +| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run | +| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots (10-16 12:00, 10-17 00:00); cleared by its own **10-17 12:02** run | +| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults | | Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` | | Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | — | | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) | @@ -62,8 +62,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] | Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic | | Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 | | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 | -| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes | -| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken | +| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text | +| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message | | 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent | ## Removed Jobs @@ -71,6 +71,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] |---|---|---| | Workday start reminder | 2026-08-26 | No longer present in live cron list | | CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected | +| DHL Tracking - Parcel 8052838382 | 2026-10-10 | `62619aa6bcb3` — parcel delivered, monitor redundant | +| Mosquito Tracker | 2026-10-16 (after the 15:00 run) | `1b31187397a5` — absent from `jobs.json` at the 2026-10-18 audit; last execution 2026-10-16 15:00 was `completed` / `delivered` | | Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up | ## Disabled Jobs (not removed) @@ -116,6 +118,8 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db] - **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]]. - **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]. +- **The `omni-101` 503 burst left no node-side evidence (NEW 2026-10-18):** 9 model calls across 7 jobs died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` between 2026-10-16 08:00 and 2026-10-17 03:00 — while the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-18 00:33). Nothing on the Hermes side records the node's CPU/RAM/queue depth at failure time, so the "resource pressure" cause is taken on the router's word; if it recurs, capture host metrics *during* the burst, not afterwards. The incident ledger also tags **seven of the nine** rows `failure_type: unknown` **despite** the explicit message (only the two script-exit watchdog incidents get a type), so `failure_type` cannot be used to triage 503s — filter on the error text. Root cause **unknown**; not a repeat of the 10-14 TCP-refusal storm (which was reachability, this was serving). → [[incidents/2026-10-16-omniroute-503-resource-pressure]] + ### Historical - **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]]. - **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]].