Files
agent-estate-wiki/incidents/2026-10-30-mnemosyne-health-monitor-paused.md
T
Tony0410 987eba72e6 Stocktake 2026-10-30: 0 stale by age, but 5 pages drifted from live state
Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories,
unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages;
Quartz rebuilt and deployed).

Freshness is not accuracy. A live-vs-page diff found real drift on five pages.
All corrections verified against live sources first.

- infrastructure/tailscale: node table rebuilt from tailscale status --json.
  50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers
  expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on
  re-verification. m93p records online (peer record chronically stale, third
  occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and
  CT201 :443/:8443 tables re-probed and match.
- systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not
  28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30
  01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection
  failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2.
  Seven job rows carried stale error claims for jobs since cleared to ok --
  each marked superseded with a dated correction, not rewritten (SCHEMA rule 9).
- NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own
  failure notice was never delivered (deliveries.db has zero rows for
  2026-10-30), so the incident row is the only trace. Job not re-enabled and
  gateway not restarted by this run; root cause recorded as unverified.
- infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes
  is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 /
  5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of
  the tailnet entirely. Skills count of 302 left alone -- it is correct.
- systems/homepage-dashboard: live app count is 123 (was 114); grid is a
  responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid;
  tab bar is built from the service groups (renderTabs iterates state.groups),
  so the settings.yaml layout block and its tab:/columns: keys are inert --
  there is no "Overview" tab; saved theme is Boarding Pass / light, not dark
  amber; manager.py runs inside the homepage-manager container; and the "no
  Traefik container" evidence was wrong -- the container exists but is
  stopped (Exited (255) 2 months ago). The conclusion that Traefik does not
  occupy 443 stands. Same evidence error corrected in tailscale.md.

Also corrected two of my own first-pass errors before committing: the
homepage tab list (groups vs tabs) and the CT450 version (a stale root-level
/usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user
install at v0.22.5).

No secrets written -- provider names, key names and HTTP status codes only.
2026-10-30 05:41:11 +08:00

6.1 KiB

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
Mnemosyne Health Monitor auto-paused after gateway-connection failure incident active 2026-10-30 2026-10-30 2026-10-30 high
cron
mnemosyne
gateway
incident
delivery
live-jobs-json
live-executions-db
live-deliveries-db
live-cron-incidents

Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)

Status: ACTIVE — the job is disabled, the incident row is open, and the failure notice never reached Telegram.

Summary

The Mnemosyne Health Monitor cron job ea79a2107e36 (0 */12 * * *, delivery topic 102) failed its 2026-10-30 00:01 slot with a gateway-connection fault. The run closed with status unknown, the scheduler auto-paused the job, and the job's own failure notice failed to send — so the only trace of the fault is the incident row in cron_incidents.

What happened

Field Value
Job ea79a2107e36 — Mnemosyne Health Monitor
Schedule 0 */12 * * * (every 12 h)
Started 2026-10-30 00:01:08.343024+08:00
Finished 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s)
Execution status unknown
Job last_status error, failure_streak: 1
Job enabled false — auto-paused 2026-10-30 01:04:22
Incident ea79a2_d93c3acdc4f2, failure_type: unknown, state detected
Delivery failed — never sent

Error text on the execution row:

Cron execution unverified; reconcile /home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json: Gateway connection failed; no local fallback

The two traps in this one fault

1. A failure notice that was itself never delivered

hermes cron list reports for this job:

The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to telegram:-1004321904721:102 failed

Confirmed at the artefact level: deliveries.db contains zero rows created on 2026-10-30 (2026-10-29 has 10 rows, all delivered). So the failure produced no Telegram output at all — not even a notice about the failure.

This is the same class as incidents/2026-08-08-gitea-config-backup-push-dead: a job whose green row hides a broken destination. Here the destination was broken and the row went red, but the alert path was the part that was dead. If Anthony had only watched hermes cron list for a non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have seen nothing at all.

2. unknown is not failed

The execution closed unknown, not failed. Per the wiki's own convention (systems/scheduled-tasks), a status: unknown row with no matching deliveries.db row is a probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then auto-paused the job — so the fleet lost a monitoring job, not a reporting job.

Root cause

Unverified. The error names a gateway-connection failure with "no local fallback" during the reconciliation of an admission record. The admission file referenced in the error is ~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json.

This is not the same failure as the 2026-10-26 planned-restart orphan class (incidents/2026-10-26-planned-restart-orphaned-cron-run), though the error string is similar ("Cron execution unverified; reconcile …"). The 10-26 rows were alerted and auto-closed; this row is still detected and the job stayed paused. No gateway restart was performed on 2026-10-30 around that window, so the trigger is not established.

Current state (verified 2026-10-30 03:3x AWST)

  • Live jobs.json: ea79a2107e36 → enabled: false, last_status: error, failure_streak: 1, paused_reason = the gateway-connection text above.
  • Live cron_incidents (table lives inside executions.db): ea79a2_d93c3acdc4f2 is the only new row since 2026-10-29; closed_at is NULL, acked_at is NULL, alerted_at is NULL.
  • The other unclosed row is 0e753e_e7f8194a7d06 (Facebook monthly, failure_type: agent, detected since 2026-10-01, clears on its 2026-11-01 slot) — 2 unclosed rows in total.
  • Fleet is now 29 jobs — 25 enabled / 4 disabled; this job is the 4th disabled one.
  • The rest of the fleet was healthy in the same window: 2026-10-29 settled 177 completed / 0 failed, and 2026-10-30 (to 05:4x) shows 37 completed, 1 running (this stocktake), 1 unknown (this job), 0 failed.

Actions taken

  1. Recorded the incident and its two distinct failure modes here.
  2. Updated systems/scheduled-tasks — the job's row, the Disabled Jobs table, the fleet counts (29/25/4), the non-clean last_status count (3 → 4), the "Failing (last run failed outright)" bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2).
  3. Updated infrastructure/hosts for the cron count.

Actions NOT taken — and why

  • The job was not re-enabled. Re-enabling a job whose gateway connection failed an hour ago risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to whoever owns the gateway fix.
  • No gateway restart was performed. This was a stocktake run, and restarting the gateway is a disruptive action with its own orphan-risk class (incidents/2026-10-26-planned-restart-orphaned-cron-run). The fault is recorded, not papered over.

Next step

Re-enable ea79a2107e36 after the gateway connection is confirmed healthy, then confirm the run completes and — critically — that a delivery row appears in deliveries.db. The point of this monitor is to report on exception to topic 102; if the topic path is still degraded, re-enabling it recreates a silent monitor.