--- title: Mnemosyne Health Monitor auto-paused after gateway-connection failure type: incident status: resolved created: 2026-10-30 updated: 2026-10-31 verified_on: 2026-10-31 confidence: high tags: [cron, mnemosyne, gateway, incident, delivery] sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents] --- # Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30) **Status: RESOLVED — 2026-10-30T10:02:48.662+08:00** (updated by the 2026-10-31 audit; everything below is the 2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day). **What actually happened:** the 00:01:08 slot was **still running**, not dead — it closed **`completed` at 10:02:48** after **10 h 01 m 40 s** and **delivered**; incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved` at 2026-10-30T10:02:48.662+08:00**, and the job is live again (`enabled: true`, `last_status: ok`, `failure_streak: 0`). `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04) — the zero-row reading was an artefact of the day having barely begun. ⚠ Its **12:01 slot was then lost to the CT460 host outage** and fired late in the 16:37:21 catch-up burst, still `running` at the 2026-10-31 audit — see [[incidents/2026-10-30-ct460-host-outage-scheduler-gap]]. **Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram. ## Summary The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its 2026-10-30 00:01 slot with a **gateway-connection** fault. The run closed with status **`unknown`**, the scheduler **auto-paused the job**, and the job's own failure notice **failed to send** — so the only trace of the fault is the incident row in `cron_incidents`. ## What happened | Field | Value | |---|---| | Job | `ea79a2107e36` — Mnemosyne Health Monitor | | Schedule | `0 */12 * * *` (every 12 h) | | Started | 2026-10-30 00:01:08.343024+08:00 | | Finished | 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s) | | Execution status | **`unknown`** | | Job `last_status` | **`error`**, `failure_streak: 1` | | Job `enabled` | **`false`** — auto-paused 2026-10-30 01:04:22 | | Incident | `ea79a2_d93c3acdc4f2`, `failure_type: unknown`, state **`detected`** | | Delivery | **failed — never sent** | Error text on the execution row: > Cron execution unverified; reconcile > `/home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`: > Gateway connection failed; no local fallback ## The two traps in this one fault ### 1. A failure notice that was itself never delivered `hermes cron list` reports for this job: > The result was not delivered: live adapter send failed: send_path_degraded; > live adapter delivery to `telegram:-1004321904721:102` failed Confirmed at the artefact level: `deliveries.db` contains **zero rows created on 2026-10-30** (2026-10-29 has 10 rows, all `delivered`). So the failure produced **no Telegram output at all** — not even a notice about the failure. This is the same class as [[incidents/2026-08-08-gitea-config-backup-push-dead]]: a job whose green row hides a broken destination. Here the destination was broken *and* the row went red, but the **alert path** was the part that was dead. If Anthony had only watched `hermes cron list` for a non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have seen nothing at all. ### 2. `unknown` is not `failed` The execution closed `unknown`, not `failed`. Per the wiki's own convention ([[systems/scheduled-tasks]]), a `status: unknown` row with **no matching `deliveries.db` row** is a probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then auto-paused the job — so the fleet lost a **monitoring** job, not a reporting job. ## Root cause **Unverified.** The error names a gateway-connection failure with "no local fallback" during the reconciliation of an admission record. The admission file referenced in the error is `~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`. This is **not** the same failure as the 2026-10-26 planned-restart orphan class ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]), though the error string is similar ("Cron execution unverified; reconcile …"). The 10-26 rows were `alerted` and auto-closed; this row is still `detected` and the job stayed paused. **No gateway restart was performed on 2026-10-30** around that window, so the trigger is not established. ## Current state (verified 2026-10-30 03:3x AWST) - Live `jobs.json`: `ea79a2107e36` → `enabled: false`, `last_status: error`, `failure_streak: 1`, `paused_reason` = the gateway-connection text above. - Live `cron_incidents` (table lives inside `executions.db`): `ea79a2_d93c3acdc4f2` is the **only** new row since 2026-10-29; `closed_at` is **NULL**, `acked_at` is **NULL**, `alerted_at` is **NULL**. - The other unclosed row is `0e753e_e7f8194a7d06` (Facebook monthly, `failure_type: agent`, `detected` since 2026-10-01, clears on its 2026-11-01 slot) — **2 unclosed rows in total**. - Fleet is now **29 jobs — 25 enabled / 4 disabled**; this job is the 4th disabled one. - The rest of the fleet was healthy in the same window: 2026-10-29 settled **177 `completed` / 0 `failed`**, and 2026-10-30 (to 05:4x) shows 37 `completed`, 1 `running` (this stocktake), 1 `unknown` (this job), 0 `failed`. ## Actions taken 1. Recorded the incident and its two distinct failure modes here. 2. Updated [[systems/scheduled-tasks]] — the job's row, the Disabled Jobs table, the fleet counts (29/25/4), the non-clean `last_status` count (3 → 4), the "Failing (last run failed outright)" bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2). 3. Updated [[infrastructure/hosts]] for the cron count. ## Actions NOT taken — and why - **The job was not re-enabled.** Re-enabling a job whose gateway connection failed an hour ago risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to whoever owns the gateway fix. - **No gateway restart was performed.** This was a stocktake run, and restarting the gateway is a disruptive action with its own orphan-risk class ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]). The fault is recorded, not papered over. ## Next step Re-enable `ea79a2107e36` **after** the gateway connection is confirmed healthy, then confirm the run completes and — critically — that a delivery row appears in `deliveries.db`. The point of this monitor is to report on exception to topic `102`; if the topic path is still degraded, re-enabling it recreates a silent monitor. ## Related - [[systems/scheduled-tasks]] — fleet ledger - [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — the earlier "execution unverified" class - [[incidents/2026-08-08-gitea-config-backup-push-dead]] — green row, dead destination - [[systems/mnemosyne]] — the subsystem being monitored