Files
agent-estate-wiki/incidents/2026-10-30-mnemosyne-health-monitor-paused.md
T

7.1 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Mnemosyne Health Monitor auto-paused after gateway-connection failure incident resolved 2026-10-30 2026-10-31 2026-10-31 high
cron
mnemosyne
gateway
incident
delivery
live-jobs-json
live-executions-db
live-deliveries-db
live-cron-incidents

Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)

Status: RESOLVED — 2026-10-30T10:02:48.662+08:00 (updated by the 2026-10-31 audit; everything below is the 2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day). What actually happened: the 00:01:08 slot was still running, not dead — it closed completed at 10:02:48 after 10 h 01 m 40 s and delivered; incident ea79a2_d93c3acdc4f2 auto-closed resolved at 2026-10-30T10:02:48.662+08:00, and the job is live again (enabled: true, last_status: ok, failure_streak: 0). deliveries.db finished 2026-10-30 with 8 rows, all delivered (latest 18:46:04) — the zero-row reading was an artefact of the day having barely begun. ⚠ Its 12:01 slot was then lost to the CT460 host outage and fired late in the 16:37:21 catch-up burst, still running at the 2026-10-31 audit — see incidents/2026-10-30-ct460-host-outage-scheduler-gap.

Historical snapshot (2026-10-30 03:3x): ACTIVE — the job is disabled, the incident row is open, and the failure notice never reached Telegram.

Summary

The Mnemosyne Health Monitor cron job ea79a2107e36 (0 */12 * * *, delivery topic 102) failed its 2026-10-30 00:01 slot with a gateway-connection fault. The run closed with status unknown, the scheduler auto-paused the job, and the job's own failure notice failed to send — so the only trace of the fault is the incident row in cron_incidents.

What happened

Field Value
Job ea79a2107e36 — Mnemosyne Health Monitor
Schedule 0 */12 * * * (every 12 h)
Started 2026-10-30 00:01:08.343024+08:00
Finished 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s)
Execution status unknown
Job last_status error, failure_streak: 1
Job enabled false — auto-paused 2026-10-30 01:04:22
Incident ea79a2_d93c3acdc4f2, failure_type: unknown, state detected
Delivery failed — never sent

Error text on the execution row:

Cron execution unverified; reconcile /home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json: Gateway connection failed; no local fallback

The two traps in this one fault

1. A failure notice that was itself never delivered

hermes cron list reports for this job:

The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed

Confirmed at the artefact level: deliveries.db contains zero rows created on 2026-10-30 (2026-10-29 has 10 rows, all delivered). So the failure produced no Telegram output at all — not even a notice about the failure.

This is the same class as incidents/2026-08-08-gitea-config-backup-push-dead: a job whose green row hides a broken destination. Here the destination was broken and the row went red, but the alert path was the part that was dead. If Anthony had only watched hermes cron list for a non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have seen nothing at all.

2. unknown is not failed

The execution closed unknown, not failed. Per the wiki's own convention (systems/scheduled-tasks), a status: unknown row with no matching deliveries.db row is a probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then auto-paused the job — so the fleet lost a monitoring job, not a reporting job.

Root cause

Unverified. The error names a gateway-connection failure with "no local fallback" during the reconciliation of an admission record. The admission file referenced in the error is ~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json.

This is not the same failure as the 2026-10-26 planned-restart orphan class (incidents/2026-10-26-planned-restart-orphaned-cron-run), though the error string is similar ("Cron execution unverified; reconcile …"). The 10-26 rows were alerted and auto-closed; this row is still detected and the job stayed paused. No gateway restart was performed on 2026-10-30 around that window, so the trigger is not established.

Current state (verified 2026-10-30 03:3x AWST)

  • Live jobs.json: ea79a2107e36 → enabled: false, last_status: error, failure_streak: 1, paused_reason = the gateway-connection text above.
  • Live cron_incidents (table lives inside executions.db): ea79a2_d93c3acdc4f2 is the only new row since 2026-10-29; closed_at is NULL, acked_at is NULL, alerted_at is NULL.
  • The other unclosed row is 0e753e_e7f8194a7d06 (Facebook monthly, failure_type: agent, detected since 2026-10-01, clears on its 2026-11-01 slot) — 2 unclosed rows in total.
  • Fleet is now 29 jobs — 25 enabled / 4 disabled; this job is the 4th disabled one.
  • The rest of the fleet was healthy in the same window: 2026-10-29 settled 177 completed / 0 failed, and 2026-10-30 (to 05:4x) shows 37 completed, 1 running (this stocktake), 1 unknown (this job), 0 failed.

Actions taken

  1. Recorded the incident and its two distinct failure modes here.
  2. Updated systems/scheduled-tasks — the job's row, the Disabled Jobs table, the fleet counts (29/25/4), the non-clean last_status count (3 → 4), the "Failing (last run failed outright)" bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2).
  3. Updated infrastructure/hosts for the cron count.

Actions NOT taken — and why

  • The job was not re-enabled. Re-enabling a job whose gateway connection failed an hour ago risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to whoever owns the gateway fix.
  • No gateway restart was performed. This was a stocktake run, and restarting the gateway is a disruptive action with its own orphan-risk class (incidents/2026-10-26-planned-restart-orphaned-cron-run). The fault is recorded, not papered over.

Next step

Re-enable ea79a2107e36 after the gateway connection is confirmed healthy, then confirm the run completes and — critically — that a delivery row appears in deliveries.db. The point of this monitor is to report on exception to topic 102; if the topic path is still degraded, re-enabling it recreates a silent monitor.