Files
agent-estate-wiki/incidents/2026-10-30-mnemosyne-health-monitor-paused.md
T

135 lines
7.1 KiB
Markdown

---
title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
type: incident
status: resolved
created: 2026-10-30
updated: 2026-10-31
verified_on: 2026-10-31
confidence: high
tags: [cron, mnemosyne, gateway, incident, delivery]
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
---
# Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
**Status: RESOLVED — 2026-10-30T10:02:48.662+08:00** (updated by the 2026-10-31 audit; everything below is the
2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day).
**What actually happened:** the 00:01:08 slot was **still running**, not dead — it closed **`completed` at
10:02:48** after **10 h 01 m 40 s** and **delivered**; incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved`
at 2026-10-30T10:02:48.662+08:00**, and the job is live again (`enabled: true`, `last_status: ok`,
`failure_streak: 0`). `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04) — the
zero-row reading was an artefact of the day having barely begun. ⚠ Its **12:01 slot was then lost to the CT460
host outage** and fired late in the 16:37:21 catch-up burst, still `running` at the 2026-10-31 audit — see
[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
## Summary
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its
2026-10-30 00:01 slot with a **gateway-connection** fault. The run closed with status **`unknown`**,
the scheduler **auto-paused the job**, and the job's own failure notice **failed to send** — so the
only trace of the fault is the incident row in `cron_incidents`.
## What happened
| Field | Value |
|---|---|
| Job | `ea79a2107e36` — Mnemosyne Health Monitor |
| Schedule | `0 */12 * * *` (every 12 h) |
| Started | 2026-10-30 00:01:08.343024+08:00 |
| Finished | 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s) |
| Execution status | **`unknown`** |
| Job `last_status` | **`error`**, `failure_streak: 1` |
| Job `enabled` | **`false`** — auto-paused 2026-10-30 01:04:22 |
| Incident | `ea79a2_d93c3acdc4f2`, `failure_type: unknown`, state **`detected`** |
| Delivery | **failed — never sent** |
Error text on the execution row:
> Cron execution unverified; reconcile
> `/home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`:
> Gateway connection failed; no local fallback
## The two traps in this one fault
### 1. A failure notice that was itself never delivered
`hermes cron list` reports for this job:
> The result was not delivered: live adapter send failed: send_path_degraded;
> live adapter delivery to `telegram:-1004321904721:102` failed
Confirmed at the artefact level: `deliveries.db` contains **zero rows created on 2026-10-30**
(2026-10-29 has 10 rows, all `delivered`). So the failure produced **no Telegram output at all** —
not even a notice about the failure.
This is the same class as [[incidents/2026-08-08-gitea-config-backup-push-dead]]: a job whose green
row hides a broken destination. Here the destination was broken *and* the row went red, but the
**alert path** was the part that was dead. If Anthony had only watched `hermes cron list` for a
non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have
seen nothing at all.
### 2. `unknown` is not `failed`
The execution closed `unknown`, not `failed`. Per the wiki's own convention
([[systems/scheduled-tasks]]), a `status: unknown` row with **no matching `deliveries.db` row** is a
probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then
auto-paused the job — so the fleet lost a **monitoring** job, not a reporting job.
## Root cause
**Unverified.** The error names a gateway-connection failure with "no local fallback" during the
reconciliation of an admission record. The admission file referenced in the error is
`~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`.
This is **not** the same failure as the 2026-10-26 planned-restart orphan class
([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]), though the error string is similar
("Cron execution unverified; reconcile …"). The 10-26 rows were `alerted` and auto-closed; this row
is still `detected` and the job stayed paused. **No gateway restart was performed on 2026-10-30**
around that window, so the trigger is not established.
## Current state (verified 2026-10-30 03:3x AWST)
- Live `jobs.json`: `ea79a2107e36` → `enabled: false`, `last_status: error`, `failure_streak: 1`,
`paused_reason` = the gateway-connection text above.
- Live `cron_incidents` (table lives inside `executions.db`): `ea79a2_d93c3acdc4f2` is the **only**
new row since 2026-10-29; `closed_at` is **NULL**, `acked_at` is **NULL**, `alerted_at` is **NULL**.
- The other unclosed row is `0e753e_e7f8194a7d06` (Facebook monthly, `failure_type: agent`, `detected`
since 2026-10-01, clears on its 2026-11-01 slot) — **2 unclosed rows in total**.
- Fleet is now **29 jobs — 25 enabled / 4 disabled**; this job is the 4th disabled one.
- The rest of the fleet was healthy in the same window: 2026-10-29 settled **177 `completed` / 0
`failed`**, and 2026-10-30 (to 05:4x) shows 37 `completed`, 1 `running` (this stocktake), 1
`unknown` (this job), 0 `failed`.
## Actions taken
1. Recorded the incident and its two distinct failure modes here.
2. Updated [[systems/scheduled-tasks]] — the job's row, the Disabled Jobs table, the fleet counts
(29/25/4), the non-clean `last_status` count (3 → 4), the "Failing (last run failed outright)"
bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2).
3. Updated [[infrastructure/hosts]] for the cron count.
## Actions NOT taken — and why
- **The job was not re-enabled.** Re-enabling a job whose gateway connection failed an hour ago
risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to
whoever owns the gateway fix.
- **No gateway restart was performed.** This was a stocktake run, and restarting the gateway is a
disruptive action with its own orphan-risk class ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]).
The fault is recorded, not papered over.
## Next step
Re-enable `ea79a2107e36` **after** the gateway connection is confirmed healthy, then confirm the
run completes and — critically — that a delivery row appears in `deliveries.db`. The point of this
monitor is to report on exception to topic `102`; if the topic path is still degraded, re-enabling it
recreates a silent monitor.
## Related
- [[systems/scheduled-tasks]] — fleet ledger
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — the earlier "execution unverified" class
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — green row, dead destination
- [[systems/mnemosyne]] — the subsystem being monitored