--- title: Scheduled Tasks type: system status: active created: 2026-07-22 updated: 2026-10-20 verified_on: 2026-10-20 review_after: 2026-10-27 confidence: high tags: [cron, scheduled, automation] sources: [live-cron-list, live-jobs-json, cron-executions-db] --- # Scheduled Tasks ## Count - **⚠ Current as of 2026-10-20 00:09 AWST (`jobs.json.updated_at 2026-10-20T00:09:07+08:00`) — the `omni-101` 503 fault is BACK, and it is now the fleet's defining risk.** Window 2026-10-19: **159 `completed` / 13 `failed`**, against 10-18's 168 / 1 and 10-17's 163 / 5. **9 genuine model-call failures across 8 jobs** all carry the identical `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` on `custom:omni-101`, in two clusters — **06:01** (one job) and **07:01 → 12:01** (seven more runs across seven jobs). The other **4 rows are derivative Mnemosyne Watchdog exits** (`96eb95ce12f2` at 06:16, 06:31, 12:16, 12:31) that merely quote the coverage monitor's error. This is the **second burst in four days** on the same node with the same blast radius, while `/api/health` read `200 {"status":"ok"}` throughout — so [[incidents/2026-10-16-omniroute-503-resource-pressure]] is back to **`status: active`** (it had been downgraded to `monitoring` 44 h earlier). **Fleet unchanged at 28 jobs — 25 enabled / 3 disabled.** **Non-clean `last_status` 7 → 12** (10 `error` + 2 `delivery_failed`); **unclosed incident rows 13** (12 `alerted` / 1 `detected`) — 4 opened and 2 closed on 10-19. Every `failure_streak` still reads ≤ 2 and the 10-20 00:00 slots ran clean, so nothing is being suppressed yet. **15 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **Current as of 2026-10-19 00:20 AWST (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]] - **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] - **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] - **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-19 00:20 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. - **Non-clean `last_status`:** **12 as of 2026-10-20 00:09** — 10 `error` + 2 `delivery_failed`. The burst re-dirtied four rows that were clean at the 10-19 audit (`dca8482e4f76`, `85edf6772bca`, `08eb86f02d37`, `c0a9fa314542`) and deepened two of the existing seven (`0e519874d280` and `ecc1fc13655a` both to `failure_streak: 2`); `fad5a86979d4` newly reads `delivery_failed` (bare-`telegram` target). `b63ed5817212` (dirtied twice) and `96eb95ce12f2` (four derivative reds) **cleared the same day** (18:01:56 / 12:46:11), and `ea79a2107e36` cleared on its 10-20 00:01 slot. Survivors, with next run: `ecc1fc13655a` (streak **2**, 10-20 12:00), `0e519874d280` (**2**, Thu 10-22 08:00), `1b23c042e17d` (**2**, Fri 10-23 08:00), `dca8482e4f76` (1, **10-20 07:00**), `85edf6772bca` (1, **10-20 07:00**), `08eb86f02d37` (1, 10-20 08:30), `c0a9fa314542` (1, **Mon 10-26 09:00**), `06a0a1c8704f` (1, Thu 10-22 10:00), `c8d52904e1d6` (1, Thu 10-22 18:00), `fad5a86979d4` (`delivery_failed`, Mon 10-26 03:00), `0e753e9720fd` (`error`, 0, 11-01 09:00), `066bb5191280` (`delivery_failed`, 0, 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → **12 now**. Three of these fire within eight hours of this audit, so the count will move again by morning. - **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]]. - Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04). - **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window. - **Repin applied 2026-10-05:** `free-stack` ×6 (`06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`, `1a8e90ea2271`, `fad5a86979d4`, `0e753e9720fd`) and `smart-route` ×1 (`dca8482e4f76`), all on `custom:omni-101`. Routing aliases were used deliberately: the fault was raw-pin brittleness, so re-pinning to another raw model name would have re-created it. `free-stack` is in service again after appearing on the broken-combo list as recently as 2026-08-14 — it was re-verified before being handed six jobs. - **Verified live, not assumed:** `79f9bb41ecf5` succeeded on a manual run 2026-10-05 23:39 and `06a0a1c8704f` at 23:45 (both incidents auto-resolved); `dca8482e4f76` completed 23:56 and delivered to `telegram:-1004321904721:1030` (incident resolved 2026-10-06 00:05:07); and `1a8e90ea2271` completed automatically on **2026-10-06 03:00**. Four of the seven are therefore verified by live run. The remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are repinned but not yet re-run — they are due on their normal schedules. `1a8e90ea2271` has since run **twice more on schedule** (2026-10-06 03:00 → `33f4e95`; 2026-10-07 03:00 → `8c63ef2`), so the stocktake repin is verified across consecutive automatic runs. - **Streak counting note:** the `N failures in a row` figure in `hermes cron list` comes from the `failure_streak` counter in `jobs.json`. `executions.db` retains only rows from **2026-09-30 onward**, so DB-based recounts undercount older streaks. Live counters as of 2026-10-06: **all 0**. - **Re-verified 2026-10-06 AWST:** job count unchanged at 30 (27 enabled / 3 disabled). `066bb5191280` and `0e753e9720fd` still carry the bare-`telegram` target warning (`configured thread_id 34795 for telegram:1793951355 was not found`). - **Re-verified 2026-10-07 AWST:** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — all 30 `failure_streak` counters read 0. Audit window 2026-10-06 00:00 → 2026-10-07 00:00: 176 `completed`, 0 `failed`, 1 `running` (the audit itself); 14 sessions, all `cron` — **no interactive (non-cron) sessions**. The stocktake job `1a8e90ea2271` ran on schedule at 03:00 and regenerated [[systems/skills-index]] (344 → 346 skills) plus `raw/stocktake/freshness-20261006.txt`, committing to Gitea as `33f4e95` — the first fully automatic stocktake since its repin. Only three jobs now carry a stale pre-repin `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`). - **A stale notice, not a live outage, drove this review.** The interactive session on 2026-10-05 23:16 began because Anthony replied to a `85edf6772bca` failure notice that had been sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice looked current only because its wording matches the live 2026-09-28 managed-runtime incident. Check `executions.started_at` (and the `cron_incidents` state) before acting on a notice. - **Re-verified 2026-10-08 AWST (window 2026-10-07 00:00 → 2026-10-08 00:00):** job count unchanged at 30 (27 enabled / 3 disabled); **0 failing** — every `failure_streak` reads 0. Window tally (`executions.db`): **176 `completed`, 1 `failed`, 1 `running`** (the audit itself). The single failure is `62619aa6bcb3` (DHL Tracking, 08:00) — a Camofox HTTP read timeout (`ensure_browser()` → `TimeoutError: timed out`), notified 08:00:58 and **auto-resolved 11:01:13** when the next 3-hourly run succeeded (incident `62619a_f485ccfd4a0d`). 13 sessions started in the window, all `source = cron` — **no interactive sessions**; 12 deliveries, **0 delivery failures**. The same three stale pre-repin `last_status: error` jobs remain (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`); the stocktake job `1a8e90ea2271` ran on schedule at 03:00 for the second consecutive day. - Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03) → 30 (2026-10-05) → 30, **0 failing** (2026-10-06) → 30, **0 failing** (2026-10-07) → 30, **0 failing** (2026-10-08) → **29, 0 failing** (2026-10-10) → **29, 0 failing** (2026-10-11) → **29, 0 failing** (2026-10-12) - **Re-verified 2026-10-12 AWST (window 2026-10-11 00:00 → 2026-10-12 00:00):** 29 jobs (26/3), **0 failing**, **5** non-clean `last_status`; window tally **172 `completed`, 0 `failed`** (the 2026-10-11 day finalised one run higher than the previous audit's snapshot, which caught it mid-flight at 171 + 1 `running`). 18 sessions — 17 `cron` + 1 interactive. - **Re-verified 2026-10-13 AWST (window 2026-10-12 00:00 → 2026-10-13 00:00):** 29 jobs (26/3), **0 failing**, non-clean `last_status` down to **3**. Window tally **174 `completed`, 0 `failed`** — **third consecutive clean day** (10-10 `175/0`, 10-11 `172/0`, 10-12 `174/0`). **5 open incidents → 2:** three closed on 2026-10-12 (see [[#Resolved 2026-10-12]]). 18 sessions in the window, **all `cron`, no interactive sessions**; 0 new incident pages required. Two findings led the window: the **GitHub Trending Radar delivery token is dead** (`getMe` → 401, live-verified) and the **Daily Marketing Job Radar run hit Camofox memory pressure** (2.5 GB peak, heavy swap, self-recovered by restart). ## Job Table | Name | ID | Schedule | Delivery | Skills | Script | Notes | |---|---|---|---|---|---|---| | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) | | AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was `custom:Omniroute:20128` / `cliprox/terra`, HTTP 401). Verified by a live manual run at 23:45 — the incident auto-resolved. Live `last_status: ok`, streak 0 | | Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | | | Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `smart-route` (was `command-code` / `gpt-5.6-terra`, 403 `MODEL_NOT_IN_PLAN` ×7). ✅ **First scheduled success on the repin 2026-10-12 07:00** (finished 07:33:50, delivered) — its 2026-10-09 timeout incident `dca848_23fcd2b18aaa` auto-closed 07:33:41. ⚠ The 2026-10-12 run hit **Camofox memory pressure** — the service reached its containment ceiling (2.5 GB peak, `MemoryHigh=2G`, swap 1 GB fully consumed, `available: 0B`), so `/tabs` calls hung; the agent restarted Camofox and finished one tab at a time. See Known Issues (Open). ⚠ **The 2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `dca848_867d9caa96a8`, `alerted`) — the second time this job has been caught by that node's faults. Live `last_status: error`, `failure_streak: 1`, next slot 2026-10-20 07:00 | | DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24. ⚠ The 2026-10-07 08:00 run failed on a Camofox read timeout (`ensure_browser()` → `TimeoutError: timed out`) — incident `62619a_f485ccfd4a0d`, auto-resolved 11:01:13 when the 11:00 run succeeded. The 14:00 run reported waybill 8052838382 **Delivered** — the parcel has arrived, so the monitor is now redundant | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the failed `cliprox/terra` pin, HTTP 401 ×1). ⚠ Last run **2026-10-09 08:00 failed** — `RuntimeError: Request timed out.` (incident `1b23c0_c35703f637b8`, still `alerted`, opened 2026-10-09 08:17) — so it still carries `last_status: error`, streak 1 → now **streak 2**. ⚠ **The Fri 2026-10-16 08:00 slot failed too** — `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` (incident `1b23c0_103cbe04a79b`, `alerted`). It cannot clear before **Fri 2026-10-23 08:00**, so a single missed slot now reads as a two-week-old failure. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] | | Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent | | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic | | Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). ⚠ **Live status is no longer `ok`:** the 2026-10-19 03:04:47 run completed but recorded `delivery_failed` — `configured thread_id 34795 for telegram:1793951355:34795 was not found` (the CLI renders it as "ran, but the result was not delivered", then "delivered without thread_id"). The run is fine; only the send is. **No incident row was created for this.** ⚠ Bare `telegram` target should move to a group topic | | FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. ⚠ **Then the 2026-10-19 12:01 slot failed again, on the `omni-101` 503** — `ecc1fc_c9976ea24553` (a **10-16** row, refreshed 10-19 12:02:58, `alerted`) plus the separate `ecc1fc_21a72aa11cc4` degenerate-stream row from 10-18, so **this one job holds two open incidents**. Live `last_status: error`, `failure_streak: 2`, next slot **2026-10-20 12:00** | | GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** ⚠ **The status reading changed again on 2026-10-19:** its scheduled Monday slot (`0 9 * * 1`) at **09:01 failed on the `omni-101` 503** (new incident `c0a9fa_764558b5ae83`, `alerted`), so **this week's roundup did not run at all** and `last_status` now reads `error` (streak 1). The token question is therefore **untested for a second consecutive Monday** — the job has not reached `digest.py` on a Monday since 2026-10-12. Next real test: **Mon 2026-10-26 09:00**. | | Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); and its **own 10-17 08:31** run restored a clean `last_status`. ⚠ **But the incident never closed: `08eb86_472c7b609c80` was refreshed on 2026-10-19 08:33 by the same 503, and is still `alerted`.** Live `last_status: error`, streak 1 | | Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) | | Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent | | Mnemosyne Auto-Consolidate | `0c771463c3cd` | `every 30m` | local | — | `consolidate_mnemosyne.sh` | no-agent | | Mnemosyne DB Backup | `e5ed215fc91c` | `0 3 * * 0` | telegram:1793951355 | — | `mnemosyne-backup.sh` | no-agent | | Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run. ⚠ **Both 2026-10-19 slots failed the same way** (06:01:05 and 12:01:16), taking its streak to **2** and re-triggering the watchdog; its **18:01 slot succeeded**, clearing the streak and auto-closing `b63ed5_bbe44c89aac1` at 18:01:56. Live `last_status: ok`. With `0 */6 * * *` this job is the fleet's best 503 sentinel — four probes a day | | Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 | | Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults. **The pattern repeated exactly in burst 2** — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at **12:46:11** (`96eb95_565ec1deb1c7`, a 10-16 row refreshed through 10-19 12:31). **8 of the 26 failure rows across the two bursts were this one sentinel** | | Perth New Restaurant & Cafe Openings (Weekly) | `c8d52904e1d6` | `0 18 * * 4` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30; verified `delivered`, status `ok` | | Political News Digest (Mon/Thu) | `0e519874d280` | `0 8 * * 1,4` | telegram:-1004321904721:346 | — | ⚠ Two node faults, no code fault: 2026-10-15 08:06 `RuntimeError: Connection error.` (`0e5198_be2291809e56`) and **2026-10-19 08:01 `HTTP 503 … resource pressure`** (new incident `0e5198_ec2fac98a91d`, `alerted`). Both are open; `failure_streak: 2`. Next slot Thu 2026-10-22 08:00 | | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) | | Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | ⚠ **2026-10-19 07:01 slot failed on the `omni-101` 503** (new incident `85edf6_256c5e150bc4`, `alerted`) — its first failure since its 10-15 connection-refused red cleared on 10-16. Live `last_status: error`, streak 1, next slot 2026-10-20 07:00 | | | Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic | | Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | ⚠ Failed 2026-10-04 18:00 on the `cliprox/terra` pin (HTTP 401), which confirmed that pin 3/3 broken. ✅ **Repinned 2026-10-05 23:39** → `custom:omni-101` / `free-stack` and **verified by a live manual run**; the incident auto-resolved. Now `last_status: ok`, streak 0 | | weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 | | Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes. ⚠ **The 2026-10-16 23:59 slot failed** on the `omni-101` 503, so the 2026-10-15 → 2026-10-17 span went unaudited; incident `cbfe43_9b89947a046a` is **`detected`** and clears only on the next successful 23:59 run (the 2026-10-17 run, which produced this entry). **Not** a stale session lock — the execution row carries the explicit 503 error text | | wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the retired `qwen3.5:397b`, HTTP 410 ×11). ✅ **Verified by live run 2026-10-06 03:00** — first success since 2026-09-25; regenerated skills-index (344 → 346 skills) and the freshness audit, committed to Gitea as `33f4e95`. Live `last_status: ok`, streak 0. Regenerates skills-index, freshness audit and commits to Gitea. ⚠ **The Quartz rebuild step is a no-op in practice** — no `quartz.config.ts` exists and `npx` is unavailable, so the static site in `public/` is never refreshed (see Known Issues). Ran automatically again 2026-10-07 03:00 (346 skills, 62 stale / 21 fresh of 90 pages, commit `8c63ef2`) — a **second consecutive automatic success**, verified. The script itself was never broken. ⚠ **The 2026-10-17 03:00 slot failed** on the same `omni-101` 503 — the first miss since the **2026-10-15** storm slot (which died with `RuntimeError: Connection error.`; the 10-16 slot succeeded). `raw/stocktake/freshness-20261017.txt` was not written and `systems/skills-index.md` stays stamped **2026-10-16** (the 10-16 slot succeeded, so `freshness-20261016.txt` exists); the **2026-10-18 03:00** slot closes it — nothing was regenerated by hand, because a manual run would race the scheduled one. Incident `1a8e90_3c9c1d573cde` `detected`. Note the ledger tags this row `failure_type: unknown` even though the execution carries an explicit 503 message | | 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent | ## Removed Jobs | Name | Removed | Reason | |---|---|---| | Workday start reminder | 2026-08-26 | No longer present in live cron list | | CT 221 Miner Watchdog | 2026-08-26 | Job `d5516d0a5147` removed; CT221 status unknown/inspected | | DHL Tracking - Parcel 8052838382 | 2026-10-10 | `62619aa6bcb3` — parcel delivered, monitor redundant | | Mosquito Tracker | 2026-10-16 (after the 15:00 run) | `1b31187397a5` — absent from `jobs.json` at the 2026-10-18 audit; last execution 2026-10-16 15:00 was `completed` / `delivered` | | Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up | ## Disabled Jobs (not removed) | Name | ID | Disabled since | Reason | |---|---|---|---| | Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. | | Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | | AgentMail Rhino Inbox Poll | `0a7f933d06e2` | 2026-07-27 | Legacy AgentMail polling; superseded by newer patterns. | ## Known Issues ### Resolved 2026-10-12 - **Both `deepseek-v4-flash` incidents closed by a clean scheduled run (`fad5a86979d4`, 03:00):** the Facebook Birthday Weekly Registry Sync completed and delivered on 2026-10-12 (03:00:45 → 03:05:47), and incidents `fad5a8_a13bbc837e1c` (open since 2026-09-28, every fallback provider exhausted) and `fad5a8_c80b59a907f0` (since 2026-09-30, `deepseek-v4-flash` HTTP 400) **both auto-closed at 03:05:39**. Its stale pre-repin `last_status: error` cleared with them, after carrying it for two weeks — the repin is now verified by a real scheduled run, not only by the manual one. - **`dca8482e4f76` timeout incident closed (`dca848_23fcd2b18aaa`):** the Daily Marketing Job Radar ran on schedule 2026-10-12 07:00:47 → 07:33:50 and delivered to topic 1030; the 2026-10-09 timeout incident that belonged to the provider storm **auto-closed at 07:33:41**. Of the ten storm incidents, nine have now closed and **one remains**: `1b23c0_c35703f637b8` (`1b23c042e17d`, still `alerted`, next slot Fri 2026-10-16 08:00). See [[incidents/2026-10-09-cron-provider-timeout-storm]]. - **Net effect: open cron incidents 5 → 2.** The two survivors are `0e753e_e7f8194a7d06` (`detected` since 2026-10-01, untouched because its job does not run until 2026-11-01) and `1b23c0_c35703f637b8` (`alerted`, cleared on the next Friday slot). ### Resolved 2026-10-07 - **DHL Tracking `62619aa6bcb3` — single Camofox timeout, self-recovered (2026-10-07 08:00 → 11:01):** the 08:00 run died in `ensure_browser()` with `TimeoutError: timed out` on a browser HTTP call (`dhl_tracking_monitor.py:233`), i.e. Camofox was momentarily unresponsive. The failure notice was sent at 08:00:58 and the incident (`62619a_f485ccfd4a0d`, `failure_type: timeout`) **auto-resolved at 11:01:13** when the 11:00 run completed; the 14:00, 17:00 and 20:00 runs also succeeded. **This is the first detect → notify → auto-resolve incident cycle since the 2026-09-30 managed-runtime event** — the incident machinery itself worked as designed. No action needed; a repeat would justify a Camofox health probe inside the script. - **The tracked parcel arrived.** The 14:00 run on 2026-10-07 reported waybill **8052838382 `Delivered`** (Kensington → Perth). The monitor has therefore achieved its purpose and runs 5×/day for no remaining benefit — a candidate for removal, flagged rather than removed (the job is Anthony's to retire). ### Resolved 2026-10-05 - **Dead model pins across 7 jobs — repinned to routing aliases (resolved 2026-10-05 23:16 → 2026-10-06 00:05):** Seven jobs were each pinned to a raw model name that had died upstream — `cliprox/terra` (HTTP 401, connection auth expired, 3 jobs), `ollama-cloud/qwen3.5:397b` (HTTP 410, retired 2026-09-25), `nvidia/deepseek-ai/deepseek-v4-flash` (HTTP 400, dropped from the live catalog, 2 jobs) and `command-code/gpt-5.6-terra` (HTTP 403 `MODEL_NOT_IN_PLAN`) — across 2026-10-01 → 2026-10-05. All seven were repinned to provider `custom:omni-101` using **routing aliases** — `free-stack` ×6, `smart-route` ×1 — deliberately, because the fault was raw-pin brittleness and re-pinning to another raw name would have re-created it. `failure_streak` is now 0 on all 30 jobs, down from 7 failing. Three were verified by live run: `79f9bb41ecf5` (23:39), `06a0a1c8704f` (23:45) and `dca8482e4f76` (23:56, delivered to topic 1030 — its first success since 2026-09-25). The remaining four are repinned but not yet re-run; the stale `last_status: error` field clears on each one's next scheduled run. See [[incidents/2026-10-05-dead-model-pin-repin]]. - The `cliprox/terra` incident is **closed for cron purposes**: no enabled job references that pin any more. The underlying proxy connection was never re-authenticated — the jobs were moved off it instead. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]]. - `wiki-stocktake-maintenance` resumes its 03:00 schedule, so `wiki-maintain.sh` no longer needs a daily manual run. - **GitHub Trending Radar `c0a9fa314542` — token-revocation claim refuted (2026-10-05): SUPERSEDED 2026-10-13.** The Mon 09:00 run *completed* on 2026-10-05 and 2026-10-12, which was read as proof the `getMe` → 401 diagnosis was wrong. It was not: a cron run can be green while the script's own delivery hop is dead, and the 2026-10-13 live `getMe` against the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **401**. Corrected above and in Open items. **Lesson: a completed run does not clear a credential fault unless the run exercised the same call that failed.** - **A stale failure notice, not a live outage, opened the 2026-10-05 review:** the interactive session began because Anthony acted on a `85edf6772bca` failure notice sitting in topic `346` since the **2026-09-29 07:00** run. That job completed on every run from 2026-09-30 17:53 onward and its three incidents were already `resolved` on 2026-09-30 17:55:13. The notice read as current only because its wording matches the live 2026-09-28 managed-runtime incident. **Check `executions.started_at` and the `cron_incidents` state before acting on a notice.** ### Resolved 2026-09-30 - **Telegram thread delivery failures — root cause found and fixed:** jobs `fad5a86979d4`, `1b23c042e17d`, `c8d52904e1d6` reported `configured thread_id 34795 for telegram:1793951355 was not found` / `Message thread not found`. The target was wrong, not the topic: a **bare `deliver: "telegram"` resolves to the home DM (`1793951355`)**, where a group forum topic ID cannot exist — so the send hard-failed and the message was lost (an earlier "benign, delivered without thread" reading was wrong). `1b23c042e17d` and `c8d52904e1d6` were retargeted to `telegram:-1004321904721:102` and verified end-to-end. See [[systems/messaging-integrations]]. - **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. All managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]]. - **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". The script existed at `/home/hermes/scripts/` but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now `ok`, streak 0. - **Residual pre-fix dispatch errors cleared:** `cbfe434a3b5e` returned to `ok` on its next run as predicted. The `1a8e90ea2271` error of the same era also cleared — its current failure is a *different*, upstream problem (below). - **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`. ### Open - **GitHub Trending Radar's delivery token is revoked — needs a new @BotFather token (NEW 2026-10-13):** live `getMe` on the token in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401**, so `digest.py`'s Telegram send fails every Monday while the cron run itself stays green (`last_status: ok`). A *completed* run therefore does not mean the digest was sent by the script — the 2026-10-12 run only reached the DM because the agent recovered the content with `--dry-run` and its own response was delivered. Fix: issue a fresh token from @BotFather, replace it in `config.json`, and re-run `getMe` to confirm; do not re-read a green cron status as proof of delivery. **This corrects the 2026-10-05 "claim refuted" entry, which was wrong.** See [[systems/scheduled-tasks]] job `c0a9fa314542`. - **Camofox memory pressure during long browser runs (NEW 2026-10-13):** the 2026-10-12 07:01 Daily Marketing Job Radar run drove Camofox into the ceiling of its own systemd containment drop-in (`~/.config/systemd/user/camofox-browser.service.d/resource-containment.conf`: `MemoryHigh=2G`, `MemoryMax=3G`, swap max 1G). Live probe mid-run read `Memory: 2.5G (high: 2G, max: 3G, swap: 1023.9M, available: 0B, peak: 2.5G)` with 383 tasks — swap fully consumed and no reclaimable headroom, which is why the `/tabs` endpoint and browser evals started hanging. The run still delivered: the agent stopped piling on memory, restarted the service, and finished one tab at a time with immediate close (post-restart: 2.1 GB used of 14 GB on CT460). Any job that opens many tabs in one pass is exposed to the same wall. Keep multi-tab extraction bounded, close each tab before opening the next, and consider raising the containment limits only if the host headroom justifies it. See [[systems/browser-backend]]. - **Stocktake's Quartz rebuild step is dead code (NEW 2026-10-07):** `wiki-maintain.sh` documents a third step — rebuild the Quartz site — but the 2026-10-07 03:00 run logged it as skipped: no `quartz.config.ts` is present and `npx` is unavailable, so `public/` has not been refreshed. The skills-index and freshness artifacts are regenerated and committed correctly; only the published-HTML step is inert. Either restore the Quartz toolchain or drop the step from the script and the job description. - **`skills-index` under-counts by one skill (NEW 2026-10-07):** live `find ~/.hermes/skills -name SKILL.md -type f` returns **347**, but the generated page says **346** (70 categories). The generator de-dups on skill basename within a category, so the nested duplicate `productivity/obsidian-git-sync/obsidian-git-sync/SKILL.md` collapses into its parent and is silently dropped. The other duplicated basename (`hermes-api-vault`, at the root and under `devops/`) survives because the two land in different categories. Remove the nested duplicate directory, or key the generator on path rather than basename. - **A delivered notice is not a successful job:** `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output. - **Stale provider key `custom:omni` (1 enabled + 1 disabled):** the provider was removed from `custom_providers` (live set on 2026-10-01: `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`). Re-verified 2026-10-06: `e530c89f2be0` (enabled) and `d40085da631c` (disabled) still name it; both run `smart-route`, and the default `custom` provider resolves to the same base URL (`http://100.78.63.20:20128`), so no failure has been observed — but the references should be normalised to `custom:omni-101`. The 2026-10-05 repin moved seven jobs to an explicit `custom:omni-101`, leaving these two as the only remaining `custom:omni` references. - **Bare `telegram` delivery targets (3 jobs) — demonstrably breaking sends:** `0e753e9720fd`, `fad5a86979d4`, `066bb5191280` deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with `configured thread_id 34795 for telegram:1793951355 was not found`. Work generated correctly; the delivery did not. Retarget all three to `telegram:-1004321904721:`. See [[systems/messaging-integrations]]. - **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]]. - **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]. - **The `omni-101` 503 burst left no node-side evidence (NEW 2026-10-18):** 9 model calls across 7 jobs died on `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure. Retry shortly.` between 2026-10-16 08:00 and 2026-10-17 03:00 — while the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-18 00:33). Nothing on the Hermes side records the node's CPU/RAM/queue depth at failure time, so the "resource pressure" cause is taken on the router's word; if it recurs, capture host metrics *during* the burst, not afterwards. The incident ledger also tags **seven of the nine** rows `failure_type: unknown` **despite** the explicit message (only the two script-exit watchdog incidents get a type), so `failure_type` cannot be used to triage 503s — filter on the error text. Root cause **unknown**; not a repeat of the 10-14 TCP-refusal storm (which was reachability, this was serving). - **It recurred on 2026-10-19 with the same 9-call blast radius (NEW 2026-10-20):** the identical `HTTP 503 … resource pressure` string took **9 model calls across 8 jobs** again (06:01, then 07:01 → 12:01), and again the node answered `/api/health` → `200 {"status":"ok"}` and `/v1/models` → `401` before, during and after (re-probed 2026-10-20 00:03). **Two bursts, four days apart, same node, same shape, no failover** — and nothing was instrumented during either one. **Action for the next occurrence: while jobs are still failing, run a real completion against `custom:omni-101` and capture the node's host CPU / RAM / queue depth — not afterwards.** The ledger again tagged every new row `failure_type: unknown`, so find these by the **error text**, never by `failure_type`. **Also verified 2026-10-20:** all eight affected jobs read `provider: custom:omni-101`, `model: smart-route` in `jobs.json` — one alias, one node, no diversity. → [[incidents/2026-10-16-omniroute-503-resource-pressure]] → [[incidents/2026-10-16-omniroute-503-resource-pressure]] ### Historical - **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]]. - **Session storage failures (2026-08-10):** Four jobs failed with TimeoutError / "session storage could not be written" at 83% disk. See [[current-state]]. - **Model repinning (2026-08-08):** 13 jobs repinned from dead/stale provider combos to `smart-route` or `gpt-5.6-terra`. - **Wiki Audit model drift (2026-08-07):** Global inference config change caused `RuntimeError: Skipped to prevent unintended spend` on an unpinned job. ## Related - [[systems/mnemosyne]] - [[runbooks/job-radar]] - [[systems/hermes-agent]] - [[systems/messaging-integrations]] - [[incidents/2026-09-28-hermes-managed-runtime-deps]] - [[incidents/2026-10-01-cliproxy-connection-auth-expired]] - [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] --- ## Re-verified 2026-10-09 (audit window 2026-10-08 → 2026-10-09) **Fleet:** 30 jobs · 27 enabled · 3 disabled · **10 carrying a non-clean `last_status`** at 2026-10-10 00:22 (`jobs.json.updated_at 2026-10-10T00:22:17+08:00`). Executions by AWST day: **10-07** {'completed': 177, 'failed': 1} · **10-08** {'completed': 178, 'failed': 2} · **10-09** {'completed': 162, 'failed': 15, 'running': 1}. **The 2026-10-09 timeout storm.** 16 failed executions across 11 distinct jobs, every one `RuntimeError: Request timed out.` on the routing aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; every run from 18:00 onward completed. Full timeline, job list and incident table: [[incidents/2026-10-09-cron-provider-timeout-storm]]. - Four incidents auto-closed on the next clean run (`96eb95` 07:01, `b63ed5` 18:01, `1b3118` 22:05, `ea79a2` **2026-10-10 00:03**); **eight remain open** — `1a8e90_789224ad589c`, `cbfe43_fa81052baaf7`, `1a8e90_dc6b2d40ec44`, `dca848_23fcd2b18aaa`, `85edf6_18bedbeec5cd`, `1b23c0_c35703f637b8`, `08eb86_634f1a296f78`, `ecc1fc_58b6c38296f3`. - `1a8e90ea2271` is the only job still at **`failure_streak: 2`** (10-08 HTTP 403 + 10-09 timeout); its 03:00 run today decides whether the streak clears. - `96eb95ce12f2` (watchdog) failed twice as a **cascade** — it exits 1 when it finds ERROR lines in `agent.log`, so it reported the storm rather than suffering it. **The 2026-10-08 window has no audit entry.** The nightly audit's own 23:59 run on 2026-10-08 failed (`RuntimeError: Request timed out.`, incident `cbfe43_fa81052baaf7`, first seen 2026-10-09T00:32:09) and committed nothing; the last committed audit covers the 2026-10-07 window (commit `659356a`, 2026-10-08 00:12). The 2026-10-09 run covers both days. **Stocktake: two-day gap, artifacts recovered.** The 2026-10-08 03:01 run left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261008.txt` untracked; both were stranded (the job's own commit step never ran) and are committed by this audit. The 2026-10-09 03:00 run timed out before the generator ran, so **no `freshness-20261009.txt` exists** and the skills index is still the 2026-10-08 output. **DHL monitor — retirement instructed, still enabled.** Anthony replied "Delete this job" to the 20:26 DHL delivery notice on 2026-10-09; the session replied that `62619aa6bcb3` was *"stopped and deleted"*. Live `jobs.json` at 2026-10-10T00:22 shows `62619aa6bcb3` **still enabled** — `paused_at: null`, `last_status: ok`, schedule `0 8,11,14,17,20 * * *` — so it runs again at 08:00. The parcel (`8052838382`) was reported `Delivered` on 2026-10-07, making the monitor redundant. **This audit did not mutate cron state**; retiring the job is open item #1 below. **Memory recall relevance.** "Are you up to date" returned a stale Uber Eats episode on 2026-10-09; capture and vector health were verified good (19 new working rows since 10-08; 760/760 working, 1,043/1,047 episodic embedded) and the session's "config saved" ranking fix is **not in live config**. See [[incidents/2026-10-09-memory-recall-relevance]]. ### Open items (2026-10-09) 1. ~~**Retire `62619aa6bcb3`** (DHL Tracking — Parcel `8052838382`) — instructed 2026-10-09 21:24, not executed.~~ **DONE 2026-10-10** — verified retired by ID lookup; fleet dropped to 29 jobs (26 enabled). See [[log.md]] and the 2026-10-10 audit block below. 2. **Decide and persist the recall-ranking change** in `memory.mnemosyne`; re-run the trigger question as a regression test. 3. ~~**Close the eight open cron incidents** once their next clean runs land; watch `1a8e90ea2271` (streak 2).~~ **DONE** — all eight cleared: five on 2026-10-10 and `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41. (A different pair, `1b23c0_c35703f637b8` + `0e753e_e7f8194a7d06`, remains open for unrelated reasons — see `### Open` above.) 4. **Stale delivery targets** — `0e753e9720fd`, `fad5a86979d4` (`last_status: error`) and `066bb5191280` (`delivery_failed`) unchanged since the 2026-10-07 audit. 5. **Quartz site rebuild is skipped by `wiki-maintain.sh` step 3** (no `quartz.config.ts`, no `npx`) — carried from 2026-10-07, still open. 6. **Skills index off by one** (347 `SKILL.md` on disk vs 346 reported; the generator de-dups a nested duplicate) — carried from 2026-10-07, still open. --- ## Re-verified 2026-10-10 (audit window 2026-10-09 -> 2026-10-10; live probes 2026-10-11 00:14-00:20 AWST) **Fleet: 29 jobs - 26 enabled - 3 disabled** (`jobs.json.updated_at 2026-10-11T00:19:44+08:00`). The count fell from 30 to 29 because **`62619aa6bcb3` (DHL Tracking - Parcel `8052838382`) is gone**: open item #1 of the 2026-10-09 audit is **CLOSED** (below). Disabled: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll). **Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` - 10-08 `178 / 2` - 10-09 `163 / 15` - **10-10 `174 completed / 0 failed`** (+1 left `running`). The 2026-10-10 fleet was wholly clean: the first zero-failure day in the four on record here, and the end of the 10-09 storm. **Non-clean `last_status`: 5 jobs, down from 10.** | Job | ID | `last_status` | streak | |---|---|---|---| | Facebook Birthday Monthly Scrape | `0e753e9720fd` | `error` | 0 | | Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `error` | 0 | | Daily Marketing Job Radar | `dca8482e4f76` | `error` | 1 | | Enhanced Weekend Adventure Planner | `1b23c042e17d` | `error` | 1 | | Uber Eats Monthly Spend Report | `066bb5191280` | `delivery_failed` | 0 | The three `streak 0` rows are stale `last_status` fields, not live failure streaks. `1a8e90ea2271` (stocktake) is back to `last_status: ok`. **Incidents: 7 closed during 2026-10-10, 5 open.** The 10-09 storm unwound as each job next ran clean (`ea79a2` 00:03, `cbfe43` 00:27, `1a8e90ea2271` x2 03:06, `85edf6` 07:04, `08eb86` 08:32, `ecc1fc` 12:05). Still open: `1b23c042e17d` + `dca8482e4f76` (`timeout`, `alerted`, both first seen 2026-10-09) and `0e753e9720fd` + `fad5a86979d4` (`agent`; open since 2026-10-01 / 2026-09-30). **Open item #1 CLOSED - the DHL monitor is retired.** Anthony instructed removal again on the evening of 2026-10-10 and the job is **absent from live `jobs.json`**, verified 2026-10-11 00:19 by direct ID lookup (no job id contains `62619`; no job name matches dhl/parcel). This is the third "deleted" claim for this job - the 2026-10-07 and 2026-10-09 reports were both **false** because the check was the absence of job output rather than a positive lookup. Removals must be verified by ID lookup. **Still open:** the Quartz rebuild step in `wiki-maintain.sh` (no `quartz.config.ts`) - skills-index off-by-one (347 `SKILL.md` vs 346 reported) - the 5 non-clean `last_status` rows above - Mattermost platform unreachable ([[systems/messaging-integrations]]) - the gateway runtime env is outside the 15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]). --- ## Re-verified 2026-10-12 (audit window 2026-10-11 00:00 → 2026-10-12 00:2x AWST; live probes 00:0x-00:2x) **Fleet:** **29 jobs — 26 enabled / 3 disabled** (`jobs.json` `updated_at` 2026-10-12T00:10:42+08:00). Disabled set unchanged: `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher), `0a7f933d06e2` (AgentMail Rhino Inbox Poll — legacy). **Executions by AWST day** (`executions.db`): 10-07 `177 completed / 1 failed` — 10-08 `178 / 2` — 10-09 `163 / 15` — 10-10 **`175 completed / 0 failed`** (finalised; the 10-11 audit saw 174 + 1 still `running`) — 10-11 **`171 completed / 0 failed`** + 1 `running` (this audit). **Two consecutive zero-failure days** after the 2026-10-09 provider-timeout storm. Heaviest jobs on 10-11: runtime-deps guard `96eb95ce12f2` 96 runs (15-min cadence), `0c771463c3cd` 46, `2fd50c87b0d0` 12, coverage monitor `b63ed5817212` 3, health monitor `ea79a2107e36` 2, stocktake `1a8e90ea2271` 1. **Non-clean `last_status`: 5 jobs — unchanged from 2026-10-10.** `0e753e9720fd` (`error`, streak 0), `fad5a86979d4` (`error`, streak 0), `dca8482e4f76` (`error`, streak 1), `1b23c042e17d` (`error`, streak 1), `066bb5191280` (`delivery_failed`, streak 0). The two streak-1 jobs are the Mon-Fri and Friday-only schedulers: their `last_error` is `RuntimeError: Request timed out.` from **Friday 2026-10-09** (the storm) and neither has had a scheduled slot since — `dca8482e4f76` next runs 07:00 Mon, `1b23c042e17d` Fri 08:00, so both should clear themselves. Everything else reads `last_status: ok` (this build reports `ok`, not `success`). **Incidents: 0 opened / 0 closed in the window — 5 unresolved, unchanged.** `alerted` 3: `fad5a8_c80b59a907f0` (weekly registry sync, scope `agent`, first seen 2026-09-30, `deepseek-v4-flash` HTTP 400), `dca848_23fcd2b18aaa` + `1b23c0_c35703f637b8` (`timeout`, both first seen 2026-10-09 — the last two left from the storm). `detected` 2: `fad5a8_a13bbc837e1c` (2026-09-28, every provider in the fallback chain exhausted), `0e753e_e7f8194a7d06` (2026-10-01, same `deepseek-v4-flash` 400). Table totals: alerted 3 / detected 2 / closed 8 / resolved 63. Last closures remain the seven on 2026-10-10. **Sessions in the window:** 18 — 17 `cron` + 1 interactive (`talaria-ios`, 2026-10-11 19:13 → 19:51, 41 messages, "best VNC clients" research). The 03:00 stocktake `1a8e90ea2271` ran clean on schedule (`completed` 03:00:40 → 03:11:29) and committed `d981e30`. **False alarm — the 2026-10-11 00:08 Mnemosyne Health Monitor EXCEPTION was a timezone mislabel, not a capture stall.** The run compared a naive-UTC `working_memory.created_at` (`2026-10-10 13:24:53`) against AWST `now` and reported "10.7 h of no captures"; the true gap was 2.7 h (the row landed 21:24:53 AWST, inside the session that produced it). No pipeline fault. The same job's 12:05 and 2026-10-12 00:02 runs returned `[SILENT]` (green). → [[systems/mnemosyne]] **Still open (unchanged):** Quartz rebuild step inert (`wiki-maintain.sh` step 3 — no `quartz.config.ts`); skills-index off-by-one (347 `SKILL.md` on disk vs 346 reported); the 5 non-clean `last_status` rows; Mattermost platform unreachable ([[systems/messaging-integrations]]); gateway runtime env outside the 15-minute dependency guard ([[incidents/2026-09-28-hermes-managed-runtime-deps]]). ## Re-verified 2026-10-13 (audit window 2026-10-12 00:00 → 2026-10-12 23:59 AWST; live probes 2026-10-13 00:05–00:45) **Headline: a clean run *hid* a dead credential.** The cron table was green — 174 completed / 0 failed — and two of the three non-clean `last_status` rows cleared overnight, yet one job that reported success on paper had its delivery hop dead the whole time. Detail below. **Cron tally:** 2026-10-11 **172 completed / 0 failed** (the one row still `running` at the last audit finished clean); 2026-10-12 **174 completed / 0 failed / 1 running** — and the running row is *this audit itself* (`cbfe434a3b5e`, started 23:59:41). **Third consecutive clean day.** The execution-log retention floor has moved up to **2026-10-07 08:15**, so 10-07 rows are now pruned (their earlier tally is no longer reproducible from this DB). **Incidents: open 5 → 2.** Three closed on 2026-10-12 — `fad5a8_a13bbc837e1c` and `fad5a8_c80b59a907f0` at **03:05:39** (both `deepseek-v4-flash`, closed by the first clean scheduled run of `fad5a86979d4`); `dca848_23fcd2b18aaa` at **07:33:41** (closed by `dca8482e4f76`'s own successful 07:00 run). Table totals now **alerted 1 / detected 1 / closed 8 / resolved 66** (76 rows). Survivors: `0e753e_e7f8194a7d06` (`detected` since 2026-10-01 — cannot clear until its job next runs on **2026-11-01**) and `1b23c0_c35703f637b8` (`alerted` since 2026-10-09, next slot **Fri 2026-10-16 08:00**). **Job-by-job (verified against the executions log, `delivery_outcome` included):** - `fad5a86979d4` — Facebook Birthday Weekly Registry Sync **03:00:45 → 03:05:47, completed, delivered**. Output: Birthday Dashboard, **448 unique birthdays**, 1 contact match. This is the repin's live proof. - `dca8482e4f76` — Daily Marketing Job Radar **07:00:47 → 07:33:50, completed, delivered** to topic 1030 (32-minute run; the 2026-10-09 timeout incident closed 9 seconds before it finished). Run was messy — see the Camofox finding — but it produced and delivered. - `c0a9fa314542` — GitHub Trending Radar **09:00:41 → 09:08:33, completed, `delivered`** — and yet **its own send failed** (finding 1). A `completed` row plus `delivery_outcome: delivered` is *not* proof the job's payload went out; here the delivered artifact was the agent's recovered digest, not the script's. - `1b23c042e17d` — Enhanced Weekend Adventure Planner **2026-10-09 08:00:38 → 08:17:46, failed** (`RuntimeError: Request timed out.`, notice delivered). No run since; next slot Fri 2026-10-16 08:00. - `0e753e9720fd` — Facebook Birthday Monthly Scrape: **no runs at all inside the retained window** (monthly, next 2026-11-01), so its `error`/streak-0 row is still untested rather than fixed. **Sessions in the window:** **18, all `cron`, no interactive session** — a quiet day, in contrast to the previous window's `talaria-ios` research session. Busiest by message count: `dca8482e4f76` (140), `1a8e90ea2271` stocktake (97), Wiki Audit (28), `c0a9fa314542` (25). The 03:00 stocktake ran clean and pushed **`0eee5f1`** to origin/master: skills-index regenerated (346 skills / 70 categories, 347 files on disk — the known off-by-one), freshness audit **54 stale / 32 fresh / 7 no-date of 93 pages**, Quartz step skipped on the documented path. **Mnemosyne (12:02 health run, green):** working **760/786 embedded = 96.7%**, episodic **1050/1056 = 99.4%** — both above the 95% floor, so no `reindex` needed. Orphan embeddings 0. FTS5 aligned exactly (`fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876). 0 unconsolidated working rows; 1 unresolved conflict (threshold 10); last consolidation 0.3 h earlier. 2252 memories total (786 working / 1056 episodic / 1 triple), 395 consolidated facts. Disk 15 GB free (81% used). No exceptions → nothing to act on, and nothing to change on [[systems/mnemosyne]] beyond the coverage figures. **Finding 1 — GitHub Trending Radar's Telegram token is revoked (live-verified).** `getMe` against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The token is a plausible length (46 chars), so it is revoked/invalid rather than malformed, and there is **no valid replacement anywhere on the box**. The fetch path is healthy — GitHub trending pulled 100 repos and wrote 26 cache files in 24 h — so only `digest.py`'s `send()` fails. Consequence: the Monday digest reached the DM **only because the agent noticed the failure and re-ran with `--dry-run`** and its own response was delivered. **The 2026-10-05 "token-revocation claim refuted" entry was wrong** and has been corrected: two green runs were read as proof the token worked, when a green cron row only means the *agent* finished. Fix: new token from @BotFather → `config.json` → re-run `getMe`. Credential present, value not recorded here. **Finding 2 — Camofox hit its memory ceiling during the Job Radar run.** Mid-run probe of `camofox-browser.service`: `Memory: 2.5G (high: 2G, max: 3G, swap max: 1G, available: 0B, peak: 2.5G, swap: 1023.9M)` with 383 tasks and the service up 9 days. Swap fully consumed, zero reclaimable headroom → `/tabs` and page evals hung. The agent restarted the service (back to ~2.1 GB used of 14 GB on CT460) and completed the extraction **one tab at a time with immediate close**. Same containment drop-in as documented (`resource-containment.conf`); the limits held, the *workload* is what needs bounding. → [[systems/browser-backend]] **Nothing added/removed:** no new system, no retired job, so `index.md` is unchanged this cycle. No page archived: the DHL monitor stays where the 2026-10-10 audit put it (redundant, in Removed Jobs). **Still open after this cycle:** GitHub Trending Radar token (new 2026-10-13); Camofox containment vs. multi-tab workloads (new 2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **3** non-clean `last_status` rows (`0e753e9720fd`, `1b23c042e17d`, `066bb5191280`); Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard. ## Re-verified 2026-10-14 Window: **2026-10-13 00:00 → 23:59 AWST**; live probes 2026-10-14 00:00–00:30 AWST. Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged this cycle. **Open incidents 2 → 3**. **Cron tally — the clean streak broke.** 2026-10-13 = **`167 completed` / 1 `failed`** (plus this audit's own row in flight at probe time, which finalises the day at 168). Tail: 10-10 `175 / 0`, 10-11 `172 / 0`, 10-12 `175 / 0` → **the first failed cron run since 2026-10-09** (a day that recorded 15). **Finding 1 — `wiki-stocktake-maintenance` (`1a8e90ea2271`) died on a dead `opencode` credential.** 03:00:43 → 03:01:57, `RuntimeError: HTTP 404: No active credentials for provider: opencode.` Pinned `custom:omni-101:free-stack`. New incident **`1a8e90_d11698bd7275`** (`state: detected`, `failure_type: agent`, `alerted_at: NULL` — failure streak 1, so under the 3-strike alert rule). This is the job's **third failure in six days, all three on the opencode path**: 10-08 `HTTP 403: OpenCode's free tier can only be used from within OpenCode` (`1a8e90_789224ad589c`), 10-09 timeout (`1a8e90_dc6b2d40ec44`), 10-13 404 no credentials (`1a8e90_d11698bd7275`). The first two resolved on 2026-10-10; this one clears only when the 2026-10-14 03:00 run succeeds — i.e. **each incident clears only on the next successful run of its own job**. → [[incidents/2026-10-05-dead-model-pin-repin]] **Finding 2 — the failure stranded its own generated artifacts; this audit committed them.** The script's step 1 (skills-index regeneration) and step 2 (freshness report) both completed and wrote to disk, then the agent/LLM turn that follows died → nothing committed, nothing pushed. Left in the working tree: `systems/skills-index.md` (modified — **date-only change: `updated`/`verified_on` 10-12 → 10-13; 346 skills / 70 categories unchanged**) and `raw/stocktake/freshness-20261013.txt` (untracked — **48 stale / 38 fresh / 7 no-date of 93 pages**, down from 54 stale / 32 fresh on 10-12). Both are in this audit's commit. Structural point: the *agent* row is what commits, so a perfectly healthy script run still publishes nothing when the LLM turn dies. There is no green-row alternative — the row went red. **Retention floor moved.** `executions.db` now holds **1,001 rows** (1,000 terminal + 1 in flight); `MAX_TERMINAL_EXECUTIONS = 1000` in `cron/executions.py`. Oldest retained terminal row: **2026-10-08 07:11**. Days above the floor are complete; older ones undercount — 10-08 now reads `126 completed / 1 failed` where its finalised tally was higher. Run-level evidence older than the floor is pruned, not recoverable. **Delivery.** 10-13: `159 suppressed` / **9 delivered** / 1 no-record. Delivered = Daily Marketing Job Radar (`dca8482e4f76`), Mosquito Tracker (`1b31187397a5` ×3), Tech & AI Newsletter Digest (`85edf6772bca`), Daily Birthday Checker (`a432898b55a4`), Gmail Action Digest (`08eb86f02d37`), FreshRSS Midday News Briefing (`ecc1fc13655a`), Evening Stand-Down Briefing (`e530c89f2be0`). No unexpected delivery; the two dead-thread delivery faults (`0e753e9720fd`, `066bb5191280`, both Telegram `thread_id 34795`) are unchanged since 10-01. **Mnemosyne.** All six monitor runs returned the silent token — Health Monitor 00:02 and 12:02, Embedding Coverage Monitor 00:02 / 06:02 / 12:01 / 18:01 → no exception, i.e. both coverages at/above the 95 % floor and no `reindex`. Last explicit figures remain the 2026-10-12 12:02 run (working 760/786 = **96.7 %**, episodic 1050/1056 = **99.4 %**). Wrinkle: the 12:02 health run emitted the token **markdown-bolded** rather than bare; the delivery layer still suppressed it (`delivery_outcome: suppressed`), so the guard held — but the matcher is tolerant of wrapping rather than literal. → [[systems/mnemosyne]] **Non-clean `last_status`: 3 → 4.** Cleared: `fad5a86979d4` (success 10-12 03:05), `dca8482e4f76` (success 10-13 07:02). Added: `1a8e90ea2271`. Remaining: `0e753e9720fd` (10-01), `1b23c042e17d` (10-09), `066bb5191280` (10-01), `1a8e90ea2271` (10-13). **Sessions.** **15 in the window, all `cron`, zero interactive** — third consecutive window with no interactive session. **Pages touched:** `systems/scheduled-tasks.md` (this block, frontmatter, non-clean list, retention note), `systems/skills-index.md` (committed stocktake artifact), `current-state.md`, `index.md`, `log.md`. **Nothing added/removed:** no new system, no retired job, no page archived → `index.md` structure unchanged. **Still open after this cycle:** `wiki-stocktake-maintenance` opencode credential (new 2026-10-13 — watch the 10-14 03:00 run: streak 3 triggers an alert, and the durable fix is to repin the job off `free-stack`-opencode or repair OmniRoute's `opencode` provider credentials); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab workloads (2026-10-13); Quartz rebuild step inert; skills-index off-by-one; **4** non-clean `last_status` rows (above); Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard. ## Re-verified 2026-10-15 / 2026-10-16 — the `omni-101` connection-refused storm Window: **2026-10-14 00:00 → 2026-10-16 00:00 AWST** (two days — the 2026-10-14 23:59 audit died inside the same outage, so the 10-14 window was never covered). Live probes 2026-10-16 00:0x–00:2x. Fleet **29 jobs (26 enabled / 3 disabled)** — unchanged. **Open incidents 3 → 12** (9 `alerted` / 3 `detected`). **The tally broke hard.** 2026-10-14 = **160 completed / 12 failed**; 2026-10-15 = **151 completed / 24 failed** / 1 running. Against a four-day clean run (10-10 `175/0`, 10-11 `172/0`, 10-12 `175/0`, 10-13 `167/1`). **36 failure rows across 13 jobs** — but only **24 are genuine model-call failures** on **12 jobs**; the other **12 rows are the Mnemosyne Watchdog reddening on a log sentinel** (symptom, not fault). Day-level, 10-15 is the worst cron day since 2026-10-09 (15). **Root cause.** `httpcore.ConnectError: [Errno 111] Connection refused` against `http://100.78.63.20:20128` — the **`omniroute-1`** node, provider **`custom:omni-101`** (`config.yaml`: custom provider `name: omni-101`). Surfaced as `openai.APIConnectionError: Connection error.` → `RuntimeError: Connection error.` 328 `Errno 111` lines in the live `errors.log`, 504 across rotations. Window 10-14 08:30:27 → 10-15 18:07, recovered by 10-15 22:01. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] **Structural finding — no model-call failover anywhere in the fleet.** Verified from `jobs.json`: **all 18 enabled agent-mode jobs are pinned to `custom:omni-101`** (`smart-route` ×10, `free-stack` ×6, `deepseek-paid-smart` ×2). The only other model-capable cron pin, `custom:omni` (CT600, `omniroute:20128`), sits on `e530c89f2be0` — which is **`no_agent: true`**, a script-only job, so its provider field is vestigial. Completing runs inside the window by provider: **210 `no_agent` / no provider + 1 `no_agent` on `custom:omni`** — that is the entire green of the fleet, and **zero model calls completed in 34 hours**. ⚠ A first pass at this window read Evening Stand-Down's clean 18:45 runs as proof CT600 stayed up. It is not: Stand-Down never calls a model. **Never read a `no_agent` job's green row as provider-health evidence.** **Worst-affected jobs:** `b63ed5817212` Mnemosyne Embedding Coverage ×6 (every slot for 30 h), `1b31187397a5` Mosquito Tracker ×4, `ea79a2107e36` Mnemosyne Health Monitor ×3, `08eb86f02d37` Gmail Action Digest ×2, `ecc1fc13655a` FreshRSS Midday ×2, then one apiece for `85edf6772bca`, `dca8482e4f76`, `0e519874d280`, `06a0a1c8704f`, `c8d52904e1d6`, `1a8e90ea2271`, `cbfe434a3b5e`. Full table in the incident page. **User-visible cost:** Digests that never went out — Tech & AI, Daily Marketing Job Radar, Political News, AI Newsletter Curation for Patrick, Gmail Action Digest ×2, FreshRSS Midday ×2, and the weekly Perth New Restaurant (missed outright; next chance 10-22). The missed audit also meant the 10-14 window went uncovered. **Recovery verified by schedule, not by probe alone.** `1b31187397a5` completed 10-15 **22:01:56** (streak 4 → 0); `b63ed5817212` 10-16 **00:02:29**; `96eb95ce12f2` 10-16 **00:01:36**; `ea79a2107e36` 10-16 **00:09:48**. Live 10-16 00:1x: `GET /v1/models` on **100.78.63.20:20128 → HTTP 401 in 10 ms** (up, auth required) and the same on `omniroute:20128` / 100.88.81.19. **Non-clean `last_status`: 3 → 12.** Cleared during the window: `b63ed5817212`, `1b31187397a5`, `ea79a2107e36` (all on successful post-storm runs). Added: the nine rows listed under **Count** above. Unchanged long-standing: `1b23c042e17d` (10-09 error), `0e753e9720fd` (10-01 error), `066bb5191280` (10-01 `delivery_failed`, dead Telegram `thread_id 34795`). **Mnemosyne.** Both monitors sat on the failing node, so **no coverage figures were refreshed in the window** — last explicit numbers remain the 2026-10-12 12:02 run (working 760/786 = 96.7 %, episodic 1050/1056 = 99.4 %); the 10-16 00:09 Health Monitor run is the first clean read since 10-12. → [[systems/mnemosyne]] **Stocktake gap (self-healing).** `1a8e90ea2271` failed 10-15 03:00 → `raw/stocktake/freshness-20261015.txt` was never written and `systems/skills-index.md` still carries the 10-14 stamp. Its next slot is 10-16 03:00, so the gap closes on its own — **not** regenerated by hand here, to avoid racing the scheduled run. **Sessions.** **36 sessions in the two-day window, all `cron`, zero interactive** — fourth consecutive window with no interactive session. **Pages touched:** `incidents/2026-10-14-omniroute-node-connection-refused-storm.md` (new), `systems/omniroute.md`, `systems/mnemosyne.md`, `systems/scheduled-tasks.md` (this block, frontmatter, Count), `current-state.md`, `index.md`, `log.md`. **Nothing removed or archived:** no retired job, no obsolete system → `index.md` structure gains one page. **Still open after this cycle:** the **no-failover topology** itself (all 18 model-calling jobs on one OmniRoute node — the durable fix is a second provider pin or a router-side failover, and it needs Anthony's call); GitHub Trending Radar token (2026-10-13, next run Mon 09:00); Camofox containment vs. multi-tab workloads; Quartz rebuild step inert; skills-index off-by-one; **12** non-clean `last_status` rows; Mattermost platform unreachable; gateway runtime env outside the 15-minute dependency guard.