Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories, unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages; Quartz rebuilt and deployed). Freshness is not accuracy. A live-vs-page diff found real drift on five pages. All corrections verified against live sources first. - infrastructure/tailscale: node table rebuilt from tailscale status --json. 50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on re-verification. m93p records online (peer record chronically stale, third occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and CT201 :443/:8443 tables re-probed and match. - systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not 28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30 01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2. Seven job rows carried stale error claims for jobs since cleared to ok -- each marked superseded with a dated correction, not rewritten (SCHEMA rule 9). - NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own failure notice was never delivered (deliveries.db has zero rows for 2026-10-30), so the incident row is the only trace. Job not re-enabled and gateway not restarted by this run; root cause recorded as unverified. - infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 / 5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of the tailnet entirely. Skills count of 302 left alone -- it is correct. - systems/homepage-dashboard: live app count is 123 (was 114); grid is a responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid; tab bar is built from the service groups (renderTabs iterates state.groups), so the settings.yaml layout block and its tab:/columns: keys are inert -- there is no "Overview" tab; saved theme is Boarding Pass / light, not dark amber; manager.py runs inside the homepage-manager container; and the "no Traefik container" evidence was wrong -- the container exists but is stopped (Exited (255) 2 months ago). The conclusion that Traefik does not occupy 443 stands. Same evidence error corrected in tailscale.md. Also corrected two of my own first-pass errors before committing: the homepage tab list (groups vs tabs) and the CT450 version (a stale root-level /usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user install at v0.22.5). No secrets written -- provider names, key names and HTTP status codes only.
6.1 KiB
title, type, status, created, updated, verified_on, confidence, tags, sources
| title | type | status | created | updated | verified_on | confidence | tags | sources | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mnemosyne Health Monitor auto-paused after gateway-connection failure | incident | active | 2026-10-30 | 2026-10-30 | 2026-10-30 | high |
|
|
Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
Status: ACTIVE — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
Summary
The Mnemosyne Health Monitor cron job ea79a2107e36 (0 */12 * * *, delivery topic 102) failed its
2026-10-30 00:01 slot with a gateway-connection fault. The run closed with status unknown,
the scheduler auto-paused the job, and the job's own failure notice failed to send — so the
only trace of the fault is the incident row in cron_incidents.
What happened
| Field | Value |
|---|---|
| Job | ea79a2107e36 — Mnemosyne Health Monitor |
| Schedule | 0 */12 * * * (every 12 h) |
| Started | 2026-10-30 00:01:08.343024+08:00 |
| Finished | 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s) |
| Execution status | unknown |
Job last_status |
error, failure_streak: 1 |
Job enabled |
false — auto-paused 2026-10-30 01:04:22 |
| Incident | ea79a2_d93c3acdc4f2, failure_type: unknown, state detected |
| Delivery | failed — never sent |
Error text on the execution row:
Cron execution unverified; reconcile
/home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json: Gateway connection failed; no local fallback
The two traps in this one fault
1. A failure notice that was itself never delivered
hermes cron list reports for this job:
The result was not delivered: live adapter send failed: send_path_degraded; live adapter delivery to
telegram:-1004321904721:102failed
Confirmed at the artefact level: deliveries.db contains zero rows created on 2026-10-30
(2026-10-29 has 10 rows, all delivered). So the failure produced no Telegram output at all —
not even a notice about the failure.
This is the same class as incidents/2026-08-08-gitea-config-backup-push-dead: a job whose green
row hides a broken destination. Here the destination was broken and the row went red, but the
alert path was the part that was dead. If Anthony had only watched hermes cron list for a
non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have
seen nothing at all.
2. unknown is not failed
The execution closed unknown, not failed. Per the wiki's own convention
(systems/scheduled-tasks), a status: unknown row with no matching deliveries.db row is a
probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then
auto-paused the job — so the fleet lost a monitoring job, not a reporting job.
Root cause
Unverified. The error names a gateway-connection failure with "no local fallback" during the
reconciliation of an admission record. The admission file referenced in the error is
~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json.
This is not the same failure as the 2026-10-26 planned-restart orphan class
(incidents/2026-10-26-planned-restart-orphaned-cron-run), though the error string is similar
("Cron execution unverified; reconcile …"). The 10-26 rows were alerted and auto-closed; this row
is still detected and the job stayed paused. No gateway restart was performed on 2026-10-30
around that window, so the trigger is not established.
Current state (verified 2026-10-30 03:3x AWST)
- Live
jobs.json:ea79a2107e36→enabled: false,last_status: error,failure_streak: 1,paused_reason= the gateway-connection text above. - Live
cron_incidents(table lives insideexecutions.db):ea79a2_d93c3acdc4f2is the only new row since 2026-10-29;closed_atis NULL,acked_atis NULL,alerted_atis NULL. - The other unclosed row is
0e753e_e7f8194a7d06(Facebook monthly,failure_type: agent,detectedsince 2026-10-01, clears on its 2026-11-01 slot) — 2 unclosed rows in total. - Fleet is now 29 jobs — 25 enabled / 4 disabled; this job is the 4th disabled one.
- The rest of the fleet was healthy in the same window: 2026-10-29 settled 177
completed/ 0failed, and 2026-10-30 (to 05:4x) shows 37completed, 1running(this stocktake), 1unknown(this job), 0failed.
Actions taken
- Recorded the incident and its two distinct failure modes here.
- Updated systems/scheduled-tasks — the job's row, the Disabled Jobs table, the fleet counts
(29/25/4), the non-clean
last_statuscount (3 → 4), the "Failing (last run failed outright)" bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2). - Updated infrastructure/hosts for the cron count.
Actions NOT taken — and why
- The job was not re-enabled. Re-enabling a job whose gateway connection failed an hour ago risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to whoever owns the gateway fix.
- No gateway restart was performed. This was a stocktake run, and restarting the gateway is a disruptive action with its own orphan-risk class (incidents/2026-10-26-planned-restart-orphaned-cron-run). The fault is recorded, not papered over.
Next step
Re-enable ea79a2107e36 after the gateway connection is confirmed healthy, then confirm the
run completes and — critically — that a delivery row appears in deliveries.db. The point of this
monitor is to report on exception to topic 102; if the topic path is still degraded, re-enabling it
recreates a silent monitor.
Related
- systems/scheduled-tasks — fleet ledger
- incidents/2026-10-26-planned-restart-orphaned-cron-run — the earlier "execution unverified" class
- incidents/2026-08-08-gitea-config-backup-push-dead — green row, dead destination
- systems/mnemosyne — the subsystem being monitored