7.1 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mnemosyne Health Monitor auto-paused after gateway-connection failure | incident | resolved | 2026-10-30 | 2026-10-31 | 2026-10-31 | high |
|
|
Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
Status: RESOLVED — 2026-10-30T10:02:48.662+08:00 (updated by the 2026-10-31 audit; everything below is the
2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day).
What actually happened: the 00:01:08 slot was still running, not dead — it closed completed at
10:02:48 after 10 h 01 m 40 s and delivered; incident ea79a2_d93c3acdc4f2 auto-closed resolved
at 2026-10-30T10:02:48.662+08:00, and the job is live again (enabled: true, last_status: ok,
failure_streak: 0). deliveries.db finished 2026-10-30 with 8 rows, all delivered (latest 18:46:04) — the
zero-row reading was an artefact of the day having barely begun. ⚠ Its 12:01 slot was then lost to the CT460
host outage and fired late in the 16:37:21 catch-up burst, still running at the 2026-10-31 audit — see
incidents/2026-10-30-ct460-host-outage-scheduler-gap.
Historical snapshot (2026-10-30 03:3x): ACTIVE — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
Summary
The Mnemosyne Health Monitor cron job ea79a2107e36 (0 */12 * * *, delivery topic 102) failed its
2026-10-30 00:01 slot with a gateway-connection fault. The run closed with status unknown,
the scheduler auto-paused the job, and the job's own failure notice failed to send — so the
only trace of the fault is the incident row in cron_incidents.
What happened
| Field | Value |
|---|---|
| Job | ea79a2107e36 — Mnemosyne Health Monitor |
| Schedule | 0 */12 * * * (every 12 h) |
| Started | 2026-10-30 00:01:08.343024+08:00 |
| Finished | 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s) |
| Execution status | unknown |
Job last_status |
error, failure_streak: 1 |
Job enabled |
false — auto-paused 2026-10-30 01:04:22 |
| Incident | ea79a2_d93c3acdc4f2, failure_type: unknown, state detected |
| Delivery | failed — never sent |
Error text on the execution row:
Cron execution unverified; reconcile
/home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json: Gateway connection failed; no local fallback
The two traps in this one fault
1. A failure notice that was itself never delivered
hermes cron list reports for this job:
The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to
telegram:-1004321904721:102failed
Confirmed at the artefact level: deliveries.db contains zero rows created on 2026-10-30
(2026-10-29 has 10 rows, all delivered). So the failure produced no Telegram output at all —
not even a notice about the failure.
This is the same class as incidents/2026-08-08-gitea-config-backup-push-dead: a job whose green
row hides a broken destination. Here the destination was broken and the row went red, but the
alert path was the part that was dead. If Anthony had only watched hermes cron list for a
non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have
seen nothing at all.
2. unknown is not failed
The execution closed unknown, not failed. Per the wiki's own convention
(systems/scheduled-tasks), a status: unknown row with no matching deliveries.db row is a
probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then
auto-paused the job — so the fleet lost a monitoring job, not a reporting job.
Root cause
Unverified. The error names a gateway-connection failure with "no local fallback" during the
reconciliation of an admission record. The admission file referenced in the error is
~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json.
This is not the same failure as the 2026-10-26 planned-restart orphan class
(incidents/2026-10-26-planned-restart-orphaned-cron-run), though the error string is similar
("Cron execution unverified; reconcile …"). The 10-26 rows were alerted and auto-closed; this row
is still detected and the job stayed paused. No gateway restart was performed on 2026-10-30
around that window, so the trigger is not established.
Current state (verified 2026-10-30 03:3x AWST)
- Live
jobs.json:ea79a2107e36→enabled: false,last_status: error,failure_streak: 1,paused_reason= the gateway-connection text above. - Live
cron_incidents(table lives insideexecutions.db):ea79a2_d93c3acdc4f2is the only new row since 2026-10-29;closed_atis NULL,acked_atis NULL,alerted_atis NULL. - The other unclosed row is
0e753e_e7f8194a7d06(Facebook monthly,failure_type: agent,detectedsince 2026-10-01, clears on its 2026-11-01 slot) — 2 unclosed rows in total. - Fleet is now 29 jobs — 25 enabled / 4 disabled; this job is the 4th disabled one.
- The rest of the fleet was healthy in the same window: 2026-10-29 settled 177
completed/ 0failed, and 2026-10-30 (to 05:4x) shows 37completed, 1running(this stocktake), 1unknown(this job), 0failed.
Actions taken
- Recorded the incident and its two distinct failure modes here.
- Updated systems/scheduled-tasks — the job's row, the Disabled Jobs table, the fleet counts
(29/25/4), the non-clean
last_statuscount (3 → 4), the "Failing (last run failed outright)" bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2). - Updated infrastructure/hosts for the cron count.
Actions NOT taken — and why
- The job was not re-enabled. Re-enabling a job whose gateway connection failed an hour ago risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to whoever owns the gateway fix.
- No gateway restart was performed. This was a stocktake run, and restarting the gateway is a disruptive action with its own orphan-risk class (incidents/2026-10-26-planned-restart-orphaned-cron-run). The fault is recorded, not papered over.
Next step
Re-enable ea79a2107e36 after the gateway connection is confirmed healthy, then confirm the
run completes and — critically — that a delivery row appears in deliveries.db. The point of this
monitor is to report on exception to topic 102; if the topic path is still degraded, re-enabling it
recreates a silent monitor.
Related
- systems/scheduled-tasks — fleet ledger
- incidents/2026-10-26-planned-restart-orphaned-cron-run — the earlier "execution unverified" class
- incidents/2026-08-08-gitea-config-backup-push-dead — green row, dead destination
- systems/mnemosyne — the subsystem being monitored