with 8 rows, all `delivered`** (latest 18:46:04) — the 03:3x "zero rows / notice never delivered" reading is
superseded.
- **⚠ Superseded by the 2026-10-31 entry above — read that first.** Cron fleet changed shape overnight on 2026-10-30 — the Mnemosyne Health Monitor is paused and its failure
notice was never delivered.** `ea79a2107e36`'s 00:01:08 slot closed **`unknown`** at 01:04:22 on a
# CT460 Hermes host outage — 5 h 30 m dark, scheduler idle 5 h 39 m (2026-10-30)
**Status: RESOLVED** — the container and scheduler came back on 2026-10-30 ~16:28–16:37 and the day
closed clean (138 executions, 136 `completed`, **0 `failed`**). **Root cause of the host reboot is not
established.** No monitor alerted; the gap was found by this audit.
## Summary
The Hermes runtime went dark at **~10:58 AWST on 2026-10-30** and did not return until
**16:28** (container) / **16:37:21** (first execution). The PVE host `pve` itself rebooted — its boot
time is **2026-10-30 16:24:00** — so this was a **host-level outage, not a cron or gateway fault**.
The gateway's previous life died with no exit path, which the lifecycle ledger logs as
`SIGKILL / OOM / VM death`. Nothing noticed for **5 h 37 m**.
## Timeline (all times AWST, 2026-10-30)
| Time | Event | Evidence |
|---|---|---|
| 10:49:54 | Last execution before the gap starts | `executions.db` last row `< 16:37` |
| ~10:58:20 | Gateway life `pid=3913841` dies; no exit path ran | `gateway-exit-diag.log``gateway.previous_unclean_exit` (prior life started 2026-10-28T17:21:58Z) |
| 10:58:20 → 16:37:21 | **Zero execution rows** (5 h 39 m) | `executions.db` count in window = **0** |
| `/home/hermes/.hermes/skills/` | Skills dir | 302 skills across 69 categories (`.archive/` excluded) |
| `/home/hermes/.hermes/cron/` | Cron jobs | 29 jobs — 25 enabled / 4 disabled (the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure) |
| `/home/hermes/.hermes/cron/` | Cron jobs | **29 jobs — 26 enabled / 3 disabled** (re-verified 2026-10-31 00:3x). ⚠ Superseded: the 2026-10-30 03:3x reading was 25 / 4 — the 4th disabled job, `ea79a2107e36` Mnemosyne Health Monitor, was auto-paused 2026-10-30 01:04 after a gateway-connection failure, but **that pause lasted only until 10:02:48**, when its long-running slot closed `completed`, incident `ea79a2_d93c3acdc4f2` auto-closed `resolved`, and the job returned to `enabled: true` / `last_status: ok` / streak 0. **CT460 itself was then dark for 5 h 30 m on 2026-10-30** (PVE `pve` rebooted; host boot 16:24:00, CT460 16:28:00, scheduler resumed 16:37:21 — ≈39 cron slots lost, 0 failures) → [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]] |
- **Current as of 2026-10-30 03:3x AWST (live `jobs.json` + `executions.db`) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered.** Fleet now **29 jobs (25 enabled / 4 disabled)** — `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused at 2026-10-30 01:04:22** after its 00:01:08 slot closed **`unknown`** on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, which opened incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`). Non-clean `last_status` is now **4** — the three long-standing survivors (`0e753e9720fd``error` since 2026-10-01, `fad5a86979d4` and `066bb5191280``delivery_failed`) **plus `ea79a2107e36` (`error`, `failure_streak: 1`)** — so the standing "every `failure_streak` reads 0" claim is now false. Window **2026-10-29 settled 177 `completed` / 0 `failed`** (the page previously said 176 — the 23:59 Wiki Audit row `cbfe434a3b5e` had still been `running` at audit time and has since settled `completed`); **2026-10-28 stands at 169 / 2** (also +1, same cause). **Unclosed incident rows 1 → 2:**`dca848_e93a538ade6b` and `dca848_0a2f03b5952b`**both closed `resolved` at 10:16:12**, leaving the Facebook monthly's `0e753e_e7f8194a7d06` (`detected`, cannot clear before its 2026-11-01 slot) **plus the new `ea79a2_d93c3acdc4f2`**. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report); **zero delivery rows were created on 2026-10-30**, because the Mnemosyne Health Monitor's own failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: `dca8482e4f76` (stage 1) and `5fc292ff0f3b` (stage 2) both read `ok` with streak 0, and every previously-red row (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) has since **cleared to `ok`** on its own scheduled slot — the page's older per-row notes still claimed `error` for several of them. **Pins: 18 of the 25 enabled jobs pin `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); the other 7 enabled rows are `no_agent` jobs with no pin.
- **Current as of 2026-10-31 00:3x AWST (live `jobs.json` + `executions.db`) — the fleet is whole, but it is whole
because a 5 h 30 m HOST OUTAGE ended, not because anything was repaired.** Fleet **29 jobs — 26 enabled /
**`deliveries.db` finished 2026-10-30 with 8 rows, all `delivered`** (latest 18:46:04). 2026-10-31 opened clean
(5 runs, all `completed`).
- **⚠ Superseded by the 2026-10-31 entry above — Cron as of 2026-10-30 03:3x AWST (live `jobs.json` + `executions.db`) — the 10-29 clean day held, but the fleet changed shape overnight: the Mnemosyne Health Monitor was auto-paused and its failure notice could not be delivered.** Fleet now **29 jobs (25 enabled / 4 disabled)** — `ea79a2107e36` (Mnemosyne Health Monitor) was **auto-paused at 2026-10-30 01:04:22** after its 00:01:08 slot closed **`unknown`** on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, which opened incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`). Non-clean `last_status` is now **4** — the three long-standing survivors (`0e753e9720fd``error` since 2026-10-01, `fad5a86979d4` and `066bb5191280``delivery_failed`) **plus `ea79a2107e36` (`error`, `failure_streak: 1`)** — so the standing "every `failure_streak` reads 0" claim is now false. Window **2026-10-29 settled 177 `completed` / 0 `failed`** (the page previously said 176 — the 23:59 Wiki Audit row `cbfe434a3b5e` had still been `running` at audit time and has since settled `completed`); **2026-10-28 stands at 169 / 2** (also +1, same cause). **Unclosed incident rows 1 → 2:**`dca848_e93a538ade6b` and `dca848_0a2f03b5952b`**both closed `resolved` at 10:16:12**, leaving the Facebook monthly's `0e753e_e7f8194a7d06` (`detected`, cannot clear before its 2026-11-01 slot) **plus the new `ea79a2_d93c3acdc4f2`**. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report); **zero delivery rows were created on 2026-10-30**, because the Mnemosyne Health Monitor's own failure notice **failed to send** (`send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed`) — a failure notice that was itself never delivered. The radar's two-stage split held its ground: `dca8482e4f76` (stage 1) and `5fc292ff0f3b` (stage 2) both read `ok` with streak 0, and every previously-red row (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) has since **cleared to `ok`** on its own scheduled slot — the page's older per-row notes still claimed `error` for several of them. **Pins: 18 of the 25 enabled jobs pin `custom:omni-101`** (11 `smart-route`, 6 `free-stack`, 1 `deepseek-paid-smart`); the other 7 enabled rows are `no_agent` jobs with no pin.
- **Prior window — as of 2026-10-30 00:0x AWST *(superseded by the 2026-10-30 03:3x entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split worked end to end for the first time, and the fleet had its first clean day in a week.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status`**5 → 3**, and every `failure_streak` read **0** — the three survivors are all stale re-runs of long-standing rows (`0e753e9720fd``error` since 2026-10-01, `fad5a86979d4` and `066bb5191280``delivery_failed`). Window **2026-10-29 settled 176 `completed` / 0 `failed`** (+ this audit `running`) — the first zero-failure day since 2026-10-24; 2026-10-28 stands at **168 / 2**. **Unclosed incident rows 4 → 1:**`dca848_e93a538ade6b` and `dca848_0a2f03b5952b`**both closed `resolved` at 10:16:12**, leaving only the Facebook monthly's `0e753e_e7f8194a7d06`, which cannot clear before its 2026-11-01 slot. Delivery health: **10 deliveries on 10-29, 10/10 `delivered`, 0 failures** (one to topic 1030 — the radar report). `dca8482e4f76` (stage 1) ran **07:00:59 → 10:16:12** (**3 h 15 m 13 s**), closed **`completed`**, and wrote `~/.hermes/radar-ref/handoff/2026-10-29.json` — **170 discovered / 33 duplicates avoided / 37 verified**, `run_date 2026-10-29T08:23:51+08:00`, sources `seek ok`, `linkedin ok`, `indeed blocked`, `wa_gov ok` — opening **no** new incident row (contrast the two consecutive stage-1 failures on 10-27 and 10-28). `5fc292ff0f3b` (stage 2) fired **10:13:46**, consumed the handoff **10:40:59**, closed **10:44:48 `completed`** and delivered the same second: **20 pages / 20 `page_ids` / 2 cover letters / 2 same-day strong matches / `verified_cover_letter_count 2` / `missing_cover_letter_count 0`**; the Notion board went **455 → 475**. Both later slots no-oped as designed (**11:30 → 10 m 46 s**, **14:30 → 1 m 51 s**). Stage 1 correctly no longer delivers. See [[incidents/2026-10-27-job-radar-budget-exhausted]] and [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], now both **resolved**. **Pins unchanged** — 19 of the 26 enabled jobs pin `custom:omni-101` (12 `smart-route`, 6 `free-stack`, 1 explicit).
- **Prior window — as of 2026-10-29 00:3x AWST *(superseded by the 2026-10-30 entry above)* (live `jobs.json` + `executions.db`) — the radar's two-stage split ran for real, but stage 1 hit the agent iteration cap and wrote no handoff, so both stage-2 slots correctly no-oped.** Fleet unchanged at **29 jobs (26 enabled / 3 disabled)**; non-clean `last_status`**5 → 5**; unclosed incident rows **2 → 4**. Window **2026-10-28 settled 168 / 2**, and both failures are in-window. `dca8482e4f76` (Job Radar stage 1) ran **07:00:32 → 10:33:18** (**3 h 32 m 46 s**), closed **`failed`**, **`failure_streak: 2`**, on *"hit the agent iteration cap before the handoff could be written"*; `~/.hermes/radar-ref/handoff/` holds only `2026-10-27.json`, and stage 2 (`5fc292ff0f3b`) fired **11:30:27 → 11:31:31** and **14:30:27 → 14:31:43**, both `completed` / **`suppressed`** — the no-op path, confirmed in production → [[incidents/2026-10-28-job-radar-stage1-iteration-cap]]. The other failure is the **03:00 stocktake** (`1a8e90ea2271`): **03:00:28 → 03:37:58** (37 m 30 s), `failed`, **streak 1**, delivery `suppressed`, on `RuntimeError: [400] … duplicate field 'reasoning_content' at line 1 column 512203` — the provider could not deserialize the model's own 512 KB final response. It filed incident `1a8e90_f57ab4cafab8` in state **`detected`** and **never alerted**, and left `systems/skills-index.md` modified and `raw/stocktake/freshness-20261028.txt` untracked; the 2026-10-29 audit committed both. **Steel left the stack in-window — and was then fully removed:**`browser.cloud_provider`/`camera_provider` are back to **`camofox`** (config mtime 2026-10-29 00:15:13), the `browser-steel` plugin was **deleted** (backup `config.yaml.bak-pre-steel-plugin-removal-20261028T161324Z` written 00:13:24), and by the end of 2026-10-29 the Steel deployments on **CT201** (containers, images, network, project dir, and the Serve routes `/steel` and `/steel-api`) and **CT333** (container + image) were **removed entirely** — nothing listens on 3000, 5173 or 9223, and there is no Steel fallback path left in the tree. `STEEL_BASE_URL` is no longer in `~/.hermes/.env`; it survived only in the inherited environment of processes already running at the cleanup, and the **gateway was restarted on 2026-10-29 01:21 AWST** to clear it → [[systems/browser-backend]], [[decisions/rollback-steel-to-camofox]], [[comparisons/steel-vs-camoufox]]. **2026-10-27 settled 173 / 1** (was 172 / 1 at the 2026-10-28 audit — one late run landed after that count). **Pins: 12 `smart-route` / 6 `free-stack` / 7 unpinned / 1 `deepseek-paid-smart`.**
- **Prior window — as of 2026-10-28 00:2x AWST** (live `jobs.json` + `executions.db`) — the radar's budget overrun stopped being silent, and the radar is now two jobs.** Window 2026-10-27: **172 `completed` / 1 `failed`** (+ this audit `running`). The single failure is `dca8482e4f76` (**Daily Marketing Job Radar - Discovery**), which ran **07:01:08 → 11:03:54** (**4 h 2 m 46 s**), closed `failed`, and delivered a report that opens *"INCOMPLETE (tool-call budget exhausted before Notion write stage)"* — discovery/verify/dedup/classify finished, **Notion ingestion, cover-letter generation and the final audit did not run**. Incident row `dca848_e93a538ade6b` opened **11:03:53**, `alerted` 11:03:59, **still open** (`failure_type: auth`, which does not describe a budget fault) → [[incidents/2026-10-27-job-radar-budget-exhausted]]. **Response, same day:** the radar was split into a **two-stage pipeline** — stage 1 `dca8482e4f76` (Discovery, `0 7 * * 1-5`, `deliver: local`, failures → topic 1030) and new stage 2 **`5fc292ff0f3b`** (Daily Marketing Job Radar - Write and Letters, `30 11,14 * * 1-5`, Telegram topic 1030, created **2026-10-27T17:34:15**; two test fires the same evening, 17:35:00 → 17:36:52 and 17:38:38 → 17:43:42, both `completed`, delivery `suppressed`, `last_status: ok`) → [[runbooks/job-radar]]. **Fleet 28 → 29 jobs (26 enabled / 3 disabled).****Unclosed incident rows 3 → 2:** the two rows the 2026-10-26 planned restart raised (`ea79a2_a4661554a3c2`, `ea79a2_c3ab2d784ba3`) **auto-closed 00:38:32**, and the radar row opened. **Non-clean `last_status` 4 → 5** (churn): `ea79a2107e36` cleared to `ok`; `dca8482e4f76` joined as `error` (streak 1) and `08eb86f02d37` re-joined as `delivery_queued` — still non-clean: `0e753e9720fd` (error), `fad5a86979d4` (delivery_failed), `066bb5191280` (delivery_failed). **Pins unchanged.** The 2026-10-26 orphan class **did not recur** — every 10-27 run except the radar's completed. *(superseded by the 2026-10-29 entry above)*
- **Current as of 2026-10-19 00:20 AWST *(superseded by the 2026-10-20 entry above)* (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01; DB re-read 2026-10-20: **163 / 5** — one row was double-counted at the time) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:**`b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
- **Total jobs:****29 — 25 enabled, 4 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-30 03:3x AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
- **Non-clean `last_status`:****4 as of 2026-10-30 03:3x AWST** — 2 `error` + 2 `delivery_failed`. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault.
- **Failing (last run failed outright):****1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1`. Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots.
- **Total jobs:****29 — 26 enabled, 3 disabled** (re-verified from live `~/.hermes/cron/jobs.json` at 2026-10-31
00:3x AWST). ⚠ Superseded reading: the 2026-10-30 03:3x audit counted **25 enabled / 4 disabled**; the 4th
disabled job was `ea79a2107e36` (Mnemosyne Health Monitor), and it is **enabled again** as of 2026-10-30
10:02:48 (`last_status: ok`, streak 0) — so **every "4th disabled job" statement below is superseded**. The job
count itself is unchanged at 29. The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs. The **4th disabled job is `ea79a2107e36` (Mnemosyne Health Monitor)**, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; the other three are the long-standing `d40085da631c` (Important-mail monitor), `f77b7e46bd09` (Rhino AgentMail Inbox Watcher) and `0a7f933d06e2` (AgentMail Rhino Inbox Poll).
- **Non-clean `last_status`:****3 as of 2026-10-31 00:3x AWST** — **1 `error` + 2 `delivery_failed`**. ⚠ Superseded:
the 2026-10-30 03:3x reading was **4** (2 `error` + 2 `delivery_failed`); its first entry, `ea79a2107e36`
(Mnemosyne Health Monitor, `error`, `failure_streak: 1`), has **cleared to `ok` / 0** — its 00:01:08 slot was
still running, not failed, and closed `completed` at 10:02:48. The job list that follows is the 03:3x snapshot. Live `jobs.json` (authoritative for the per-job counters): `ea79a2107e36` (Mnemosyne Health Monitor — `error`, `failure_streak: 1`, auto-paused 2026-10-30 01:04:22 after a gateway-connection failure; incident `ea79a2_d93c3acdc4f2` open `detected`), `0e753e9720fd` (Facebook Birthday Monthly Scrape — `error`, streak 0, next slot 11-01 09:00), `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — `delivery_failed`, streak 0, most recent failure **2026-10-26 03:08:15**) and `066bb5191280` (Uber Eats Monthly Spend Report — `delivery_failed`, streak 0, next slot 11-01 02:00). History: **3** (10-12 audit) → **12** (10-16 peak) → 8 (10-18) → 7 (10-19 00:20) → 12 (10-20 00:09) → 3 (10-30 00:0x) → **4 now**, the increase being the Mnemosyne Health Monitor's overnight gateway fault.
- **Failing (last run failed outright):****0 as of 2026-10-31 00:3x AWST**. ⚠ Superseded: the 2026-10-30 03:3x
reading was **1** — `ea79a2107e36` (Mnemosyne Health Monitor), `failure_streak: 1` — and that verdict was the
reconciliation path, **not the run**: the slot closed `completed` at 10:02:48 after 10 h 01 m 40 s and the
counters reset. **Only delivery-only failures remain (2).** Its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, and the job was **auto-paused** at that moment — it is now the 4th disabled job. ⚠ **Its failure notice was never delivered**: `hermes cron list` reports *"The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"*, and `deliveries.db` holds **zero rows for 2026-10-30**. A failure notice that fails to send is the same green-row/dead-job trap as [[incidents/2026-08-08-gitea-config-backup-push-dead]] — the incident row `ea79a2_d93c3acdc4f2` is the only trace. The other previously-red jobs (`1b23c042e17d`, `ecc1fc13655a`, `08eb86f02d37`, `85edf6772bca`, `0e519874d280`, `c0a9fa314542`) have all **cleared to `ok` with streak 0** on their own scheduled slots.
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN`×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
- **Date-bucketing note (added 2026-10-25):**`executions.db` stores `started_at` with a `+08:00` offset, and SQLite's `date()` normalises to UTC — so `date(started_at)` moves every row that ran 00:00–07:59 AWST onto the **previous** day. Daily tallies on this page are bucketed with `substr(started_at,1,10)`, which preserves AWST calendar days. A recount via `date()` reads low by roughly a day's early-morning runs; this is the same class of drift that put 10-21 at 168 instead of 171.
- **Failing (delivery only):****2** — `066bb5191280` (Uber Eats Monthly Spend Report: the report itself generated; the send failed against the DM thread — bare `telegram` target, 2026-10-01) and `fad5a86979d4` (Facebook Birthday Weekly Registry Sync — the same dead-thread cause, most recently **2026-10-26 03:08:15**). Both carry `failure_streak: 0` because the *run* succeeded; only the delivery failed.
| Mnemosyne Embedding Coverage Monitor | `b63ed5817212` | `0 */6 * * *` | telegram:-1004321904721:102 | — | `check_embedding_coverage.py` | Alerts if coverage < 95%. ⚠ **Both 503-era slots failed** (10-16 12:00, 10-17 00:00 — two of the four watchdog reds quote this job's error); cleared by its own **10-17 06:01** run. ⚠ **Both 2026-10-19 slots failed the same way** (06:01:05 and 12:01:16), taking its streak to **2** and re-triggering the watchdog; its **18:01 slot succeeded**, clearing the streak and auto-closing `b63ed5_bbe44c89aac1` at 18:01:56. Live `last_status: ok`. With `0 */6 * * *` this job is the fleet's best 503 sentinel — four probes a day |
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception. ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 ⚠ **Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing:** its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`), and the scheduler **auto-paused the job at that moment**. Live `last_status: error`, `failure_streak: 1`, `enabled: false`. The "Live `last_status: ok`, streak 0" note above is **superseded**. ⚠ **The failure notice was never delivered** — `hermes cron list` reports *"live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"* and `deliveries.db` has **zero rows for 2026-10-30**. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
| Mnemosyne Health Monitor | `ea79a2107e36` | `0 */12 * * *` | telegram:-1004321904721:102 | mnemosyne-operations | — | Reports only on exception.**Live status re-verified 2026-10-31 00:3x: ENABLED, `last_status: ok`, `failure_streak: 0`; incident `ea79a2_d93c3acdc4f2` closed `resolved` 2026-10-30T10:02:48** (⚠ the 2026-10-30 03:3x DISABLED/`unknown` note later in this row is superseded by that; its 12:01 slot was lost to the CT460 host outage and fired late at 16:37:21). ⚠ Same two 503 slots in burst 1 (10-16 12:00, 10-17 00:00), then **again at 2026-10-19 12:01** — incident `ea79a2_747b27556fd0` was first opened 10-16 12:03, refreshed 10-19 12:02:55, and **auto-closed 2026-10-20 00:04:24** when the 00:01 slot succeeded. Live `last_status: ok`, streak 0 ⚠ **Live status re-verified 2026-10-30 03:3x — the job is now DISABLED and failing:** its 2026-10-30 00:01:08 slot closed **`unknown`** at 01:04:22 on *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*, opening incident `ea79a2_d93c3acdc4f2` (`failure_type: unknown`, state `detected`), and the scheduler **auto-paused the job at that moment**. Live `last_status: error`, `failure_streak: 1`, `enabled: false`. The "Live `last_status: ok`, streak 0" note above is **superseded**. ⚠ **The failure notice was never delivered** — `hermes cron list` reports *"live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed"* and `deliveries.db` has **zero rows for 2026-10-30**. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
| Mnemosyne Watchdog | `96eb95ce12f2` | `*/15 * * * *` | telegram:-1004321904721:102 | — | `mnemosyne-watchdog.py` | no-agent, silent ≤10 conflicts; **also runs `hermes-runtime-deps-guard.sh`** every 15 min (added 2026-09-30). ⚠ **Its reds are derivative — it double-counts other jobs' failures.** During the 503 burst it exited 1 on **4 slots** (10-16 12:15 + 12:30, 10-17 00:15 + 00:30), each row quoting the coverage monitor's own error from the log sentinel: `🚨 Log ERROR in agent.log/errors.log: Job 'Mnemosyne Embedding Coverage Monitor' failed: RuntimeError: HTTP 503…`. One sentinel watching another job, not four independent faults. **The pattern repeated exactly in burst 2** — four more reds on 2026-10-19 (06:16, 06:31, 12:16, 12:31), all quoting the same coverage-monitor error, all clearing at **12:46:11** (`96eb95_565ec1deb1c7`, a 10-16 row refreshed through 10-19 12:31). **8 of the 26 failure rows across the two bursts were this one sentinel** |
| Mosquito Tracker | 2026-10-16 (after the 15:00 run) | `1b31187397a5` — absent from `jobs.json` at the 2026-10-18 audit; last execution 2026-10-16 15:00 was `completed` / `delivered` |
| Review Agent Estate Phase One | 2026-08-26 | One-time job; completed 2026-08-08, now cleaned up |
## Disabled Jobs (not removed)
## Disabled Jobs (not removed) — **3 as of 2026-10-31**
| Name | ID | Disabled since | Reason |
|---|---|---|---|
| Important-mail monitor | `d40085da631c` | 2026-08-29 | Paused during Gmail read-state incident. See [[incidents/2026-08-29-gmail-read-state]]. |
| Mnemosyne Health Monitor | `ea79a2107e36` | **2026-10-30 01:04:22** |**Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
| ~~Mnemosyne Health Monitor~~ (**re-enabled 2026-10-30**) | `ea79a2107e36` | **2026-10-30 01:04:22 → re-enabled same day** | ⚠ **SUPERSEDED as of 2026-10-31 — the job is ENABLED** (`enabled: true`, `last_status: ok`, `failure_streak: 0`) and incident `ea79a2_d93c3acdc4f2` closed **`resolved` 2026-10-30T10:02:48.662+08:00**. The following describes the 03:3x snapshot only.**Auto-paused by the scheduler** after its 00:01:08 slot closed `unknown` on a gateway-connection failure (*"Cron execution unverified; reconcile `~/.hermes/cron/admissions/ad897cbc9f…json`: Gateway connection failed; no local fallback"*). Live `last_status: error`, `failure_streak: 1`, incident `ea79a2_d93c3acdc4f2` open `detected`. **Its failure notice was never delivered** — `deliveries.db` has zero rows for 2026-10-30. See [[incidents/2026-10-30-mnemosyne-health-monitor-paused]]. |
## Known Issues
### Resolved 2026-10-30
- **The Mnemosyne Health Monitor was never really down — and it is enabled again (`ea79a2107e36`).** Its 00:01:08
slot was still **running** at the 03:3x reading; the `unknown`/`error` verdict came from the admission
reconciliation, not the run. It **closed `completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**,
auto-closing incident `ea79a2_d93c3acdc4f2`**`resolved` at 2026-10-30T10:02:48.662+08:00**. Live:
alerted** — the two jobs that could have (`cbfe434a3b5e` NFS mount monitor, `1a8e90ea2271` m93p LXC boot monitor)
are disabled, and this remains the open item.
### Resolved 2026-10-29
- **The Job Radar's two-stage split finished end to end for the first time (`dca8482e4f76` + `5fc292ff0f3b`):** stage 1 ran **07:00:59 → 10:16:12** (3 h 15 m 13 s), closed **`completed`** and wrote `handoff/2026-10-29.json` (**170 discovered / 33 duplicates avoided / 37 verified**) — the first post-split run to write a handoff, after 10-27 died on the **tool-call budget** and 10-28 on the **iteration cap**. Stage 2 consumed it at **10:40:59**, closed `completed` at **10:44:48** and **delivered** to topic 1030 in the same second: **20 pages, 2 cover letters, `missing_cover_letter_count 0`**, Notion **455 → 475**. Both stage-1 ledger rows (`dca848_e93a538ade6b`, `dca848_0a2f03b5952b`) closed **`resolved` at 10:16:12**, and the run opened no new row → [[incidents/2026-10-27-job-radar-budget-exhausted]], [[incidents/2026-10-28-job-radar-stage1-iteration-cap]], [[runbooks/job-radar]]. ⚠ The discovery stage is still **slow** (3 h 15 m) and both ceilings remain reachable — a `completed` stage-1 row with **no handoff file** is the same fault in new clothing.
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.