135 lines
7.1 KiB
Markdown
135 lines
7.1 KiB
Markdown
---
|
|
title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
|
|
type: incident
|
|
status: resolved
|
|
created: 2026-10-30
|
|
updated: 2026-10-31
|
|
verified_on: 2026-10-31
|
|
confidence: high
|
|
tags: [cron, mnemosyne, gateway, incident, delivery]
|
|
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
|
|
---
|
|
|
|
# Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
|
|
|
|
**Status: RESOLVED — 2026-10-30T10:02:48.662+08:00** (updated by the 2026-10-31 audit; everything below is the
|
|
2026-10-30 03:3x snapshot and its "job is disabled / notice never delivered" reading did not survive the day).
|
|
**What actually happened:** the 00:01:08 slot was **still running**, not dead — it closed **`completed` at
|
|
10:02:48** after **10 h 01 m 40 s** and **delivered**; incident `ea79a2_d93c3acdc4f2` **auto-closed `resolved`
|
|
at 2026-10-30T10:02:48.662+08:00**, and the job is live again (`enabled: true`, `last_status: ok`,
|
|
`failure_streak: 0`). `deliveries.db` finished 2026-10-30 with **8 rows, all `delivered`** (latest 18:46:04) — the
|
|
zero-row reading was an artefact of the day having barely begun. ⚠ Its **12:01 slot was then lost to the CT460
|
|
host outage** and fired late in the 16:37:21 catch-up burst, still `running` at the 2026-10-31 audit — see
|
|
[[incidents/2026-10-30-ct460-host-outage-scheduler-gap]].
|
|
|
|
**Historical snapshot (2026-10-30 03:3x): ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
|
|
|
|
## Summary
|
|
|
|
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its
|
|
2026-10-30 00:01 slot with a **gateway-connection** fault. The run closed with status **`unknown`**,
|
|
the scheduler **auto-paused the job**, and the job's own failure notice **failed to send** — so the
|
|
only trace of the fault is the incident row in `cron_incidents`.
|
|
|
|
## What happened
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Job | `ea79a2107e36` — Mnemosyne Health Monitor |
|
|
| Schedule | `0 */12 * * *` (every 12 h) |
|
|
| Started | 2026-10-30 00:01:08.343024+08:00 |
|
|
| Finished | 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s) |
|
|
| Execution status | **`unknown`** |
|
|
| Job `last_status` | **`error`**, `failure_streak: 1` |
|
|
| Job `enabled` | **`false`** — auto-paused 2026-10-30 01:04:22 |
|
|
| Incident | `ea79a2_d93c3acdc4f2`, `failure_type: unknown`, state **`detected`** |
|
|
| Delivery | **failed — never sent** |
|
|
|
|
Error text on the execution row:
|
|
|
|
> Cron execution unverified; reconcile
|
|
> `/home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`:
|
|
> Gateway connection failed; no local fallback
|
|
|
|
## The two traps in this one fault
|
|
|
|
### 1. A failure notice that was itself never delivered
|
|
|
|
`hermes cron list` reports for this job:
|
|
|
|
> The result was not delivered: live adapter send failed: send_path_degraded;
|
|
> live adapter delivery to `telegram:-1004321904721:102` failed
|
|
|
|
Confirmed at the artefact level: `deliveries.db` contains **zero rows created on 2026-10-30**
|
|
(2026-10-29 has 10 rows, all `delivered`). So the failure produced **no Telegram output at all** —
|
|
not even a notice about the failure.
|
|
|
|
This is the same class as [[incidents/2026-08-08-gitea-config-backup-push-dead]]: a job whose green
|
|
row hides a broken destination. Here the destination was broken *and* the row went red, but the
|
|
**alert path** was the part that was dead. If Anthony had only watched `hermes cron list` for a
|
|
non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have
|
|
seen nothing at all.
|
|
|
|
### 2. `unknown` is not `failed`
|
|
|
|
The execution closed `unknown`, not `failed`. Per the wiki's own convention
|
|
([[systems/scheduled-tasks]]), a `status: unknown` row with **no matching `deliveries.db` row** is a
|
|
probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then
|
|
auto-paused the job — so the fleet lost a **monitoring** job, not a reporting job.
|
|
|
|
## Root cause
|
|
|
|
**Unverified.** The error names a gateway-connection failure with "no local fallback" during the
|
|
reconciliation of an admission record. The admission file referenced in the error is
|
|
`~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`.
|
|
|
|
This is **not** the same failure as the 2026-10-26 planned-restart orphan class
|
|
([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]), though the error string is similar
|
|
("Cron execution unverified; reconcile …"). The 10-26 rows were `alerted` and auto-closed; this row
|
|
is still `detected` and the job stayed paused. **No gateway restart was performed on 2026-10-30**
|
|
around that window, so the trigger is not established.
|
|
|
|
## Current state (verified 2026-10-30 03:3x AWST)
|
|
|
|
- Live `jobs.json`: `ea79a2107e36` → `enabled: false`, `last_status: error`, `failure_streak: 1`,
|
|
`paused_reason` = the gateway-connection text above.
|
|
- Live `cron_incidents` (table lives inside `executions.db`): `ea79a2_d93c3acdc4f2` is the **only**
|
|
new row since 2026-10-29; `closed_at` is **NULL**, `acked_at` is **NULL**, `alerted_at` is **NULL**.
|
|
- The other unclosed row is `0e753e_e7f8194a7d06` (Facebook monthly, `failure_type: agent`, `detected`
|
|
since 2026-10-01, clears on its 2026-11-01 slot) — **2 unclosed rows in total**.
|
|
- Fleet is now **29 jobs — 25 enabled / 4 disabled**; this job is the 4th disabled one.
|
|
- The rest of the fleet was healthy in the same window: 2026-10-29 settled **177 `completed` / 0
|
|
`failed`**, and 2026-10-30 (to 05:4x) shows 37 `completed`, 1 `running` (this stocktake), 1
|
|
`unknown` (this job), 0 `failed`.
|
|
|
|
## Actions taken
|
|
|
|
1. Recorded the incident and its two distinct failure modes here.
|
|
2. Updated [[systems/scheduled-tasks]] — the job's row, the Disabled Jobs table, the fleet counts
|
|
(29/25/4), the non-clean `last_status` count (3 → 4), the "Failing (last run failed outright)"
|
|
bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2).
|
|
3. Updated [[infrastructure/hosts]] for the cron count.
|
|
|
|
## Actions NOT taken — and why
|
|
|
|
- **The job was not re-enabled.** Re-enabling a job whose gateway connection failed an hour ago
|
|
risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to
|
|
whoever owns the gateway fix.
|
|
- **No gateway restart was performed.** This was a stocktake run, and restarting the gateway is a
|
|
disruptive action with its own orphan-risk class ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]).
|
|
The fault is recorded, not papered over.
|
|
|
|
## Next step
|
|
|
|
Re-enable `ea79a2107e36` **after** the gateway connection is confirmed healthy, then confirm the
|
|
run completes and — critically — that a delivery row appears in `deliveries.db`. The point of this
|
|
monitor is to report on exception to topic `102`; if the topic path is still degraded, re-enabling it
|
|
recreates a silent monitor.
|
|
|
|
## Related
|
|
|
|
- [[systems/scheduled-tasks]] — fleet ledger
|
|
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — the earlier "execution unverified" class
|
|
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — green row, dead destination
|
|
- [[systems/mnemosyne]] — the subsystem being monitored
|