Stocktake 2026-10-30: 0 stale by age, but 5 pages drifted from live state
Ran wiki-maintain.sh (skills-index regenerated: 302 skills / 69 categories, unchanged; freshness report 0 stale / 109 fresh / 3 no-date of 112 pages; Quartz rebuilt and deployed). Freshness is not accuracy. A live-vs-page diff found real drift on five pages. All corrections verified against live sources first. - infrastructure/tailscale: node table rebuilt from tailscale status --json. 50 non-funnel nodes (49 IPv4 peers + hermes), not 54. 2 IP changes, 4 peers expired out of the tailnet. Zero IP mismatches, zero missing/extra rows on re-verification. m93p records online (peer record chronically stale, third occurrence -- ping + SSH confirm). CT460 :443 route set (12 paths) and CT201 :443/:8443 tables re-probed and match. - systems/scheduled-tasks: fleet is 29 jobs (25 enabled / 4 disabled), not 28/25/3. Mnemosyne Health Monitor ea79a2107e36 was auto-paused 2026-10-30 01:04:22 after its 00:01:08 slot closed "unknown" on a gateway-connection failure. Non-clean last_status is 4, not 3. "Failing (delivery only)" is 2. Seven job rows carried stale error claims for jobs since cleared to ok -- each marked superseded with a dated correction, not rewritten (SCHEMA rule 9). - NEW incidents/2026-10-30-mnemosyne-health-monitor-paused: the job's own failure notice was never delivered (deliveries.db has zero rows for 2026-10-30), so the incident row is the only trace. Job not re-enabled and gateway not restarted by this run; root cause recorded as unverified. - infrastructure/hosts: CT201 LAN is 192.168.178.77 (was .166); CT450 Hermes is v0.22.5 / bf0203a794a0, same build as CT460 (page said v0.22.3 / 5b147ab036d6); cron count 29/25/4; orphaned "omni" peer has expired out of the tailnet entirely. Skills count of 302 left alone -- it is correct. - systems/homepage-dashboard: live app count is 123 (was 114); grid is a responsive repeat(auto-fill, minmax(230px, 1fr)), not a fixed 5-column grid; tab bar is built from the service groups (renderTabs iterates state.groups), so the settings.yaml layout block and its tab:/columns: keys are inert -- there is no "Overview" tab; saved theme is Boarding Pass / light, not dark amber; manager.py runs inside the homepage-manager container; and the "no Traefik container" evidence was wrong -- the container exists but is stopped (Exited (255) 2 months ago). The conclusion that Traefik does not occupy 443 stands. Same evidence error corrected in tailscale.md. Also corrected two of my own first-pass errors before committing: the homepage tab list (groups vs tabs) and the CT450 version (a stale root-level /usr/local/bin/hermes reports v0.19.0; the live agent is the hermes-user install at v0.22.5). No secrets written -- provider names, key names and HTTP status codes only.
This commit is contained in:
@@ -0,0 +1,124 @@
|
||||
---
|
||||
title: Mnemosyne Health Monitor auto-paused after gateway-connection failure
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-30
|
||||
updated: 2026-10-30
|
||||
verified_on: 2026-10-30
|
||||
confidence: high
|
||||
tags: [cron, mnemosyne, gateway, incident, delivery]
|
||||
sources: [live-jobs-json, live-executions-db, live-deliveries-db, live-cron-incidents]
|
||||
---
|
||||
|
||||
# Mnemosyne Health Monitor auto-paused after gateway-connection failure (2026-10-30)
|
||||
|
||||
**Status: ACTIVE** — the job is disabled, the incident row is open, and the failure notice never reached Telegram.
|
||||
|
||||
## Summary
|
||||
|
||||
The Mnemosyne Health Monitor cron job `ea79a2107e36` (`0 */12 * * *`, delivery topic `102`) failed its
|
||||
2026-10-30 00:01 slot with a **gateway-connection** fault. The run closed with status **`unknown`**,
|
||||
the scheduler **auto-paused the job**, and the job's own failure notice **failed to send** — so the
|
||||
only trace of the fault is the incident row in `cron_incidents`.
|
||||
|
||||
## What happened
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Job | `ea79a2107e36` — Mnemosyne Health Monitor |
|
||||
| Schedule | `0 */12 * * *` (every 12 h) |
|
||||
| Started | 2026-10-30 00:01:08.343024+08:00 |
|
||||
| Finished | 2026-10-30 01:04:22.510091+08:00 (1 h 03 m 14 s) |
|
||||
| Execution status | **`unknown`** |
|
||||
| Job `last_status` | **`error`**, `failure_streak: 1` |
|
||||
| Job `enabled` | **`false`** — auto-paused 2026-10-30 01:04:22 |
|
||||
| Incident | `ea79a2_d93c3acdc4f2`, `failure_type: unknown`, state **`detected`** |
|
||||
| Delivery | **failed — never sent** |
|
||||
|
||||
Error text on the execution row:
|
||||
|
||||
> Cron execution unverified; reconcile
|
||||
> `/home/hermes/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`:
|
||||
> Gateway connection failed; no local fallback
|
||||
|
||||
## The two traps in this one fault
|
||||
|
||||
### 1. A failure notice that was itself never delivered
|
||||
|
||||
`hermes cron list` reports for this job:
|
||||
|
||||
> The result was not delivered: live adapter send failed: send_path_degraded;
|
||||
> live adapter delivery to `telegram:-1004321904721:102` failed
|
||||
|
||||
Confirmed at the artefact level: `deliveries.db` contains **zero rows created on 2026-10-30**
|
||||
(2026-10-29 has 10 rows, all `delivered`). So the failure produced **no Telegram output at all** —
|
||||
not even a notice about the failure.
|
||||
|
||||
This is the same class as [[incidents/2026-08-08-gitea-config-backup-push-dead]]: a job whose green
|
||||
row hides a broken destination. Here the destination was broken *and* the row went red, but the
|
||||
**alert path** was the part that was dead. If Anthony had only watched `hermes cron list` for a
|
||||
non-clean row, he would have seen the job had paused; if he had only watched Telegram, he would have
|
||||
seen nothing at all.
|
||||
|
||||
### 2. `unknown` is not `failed`
|
||||
|
||||
The execution closed `unknown`, not `failed`. Per the wiki's own convention
|
||||
([[systems/scheduled-tasks]]), a `status: unknown` row with **no matching `deliveries.db` row** is a
|
||||
probable non-delivery that the dashboard hides behind the previous slot's error. The scheduler then
|
||||
auto-paused the job — so the fleet lost a **monitoring** job, not a reporting job.
|
||||
|
||||
## Root cause
|
||||
|
||||
**Unverified.** The error names a gateway-connection failure with "no local fallback" during the
|
||||
reconciliation of an admission record. The admission file referenced in the error is
|
||||
`~/.hermes/cron/admissions/ad897cbc9f522c3446990c20fba2c1cf773f336805bbb09f79db7564d74e64fd.json`.
|
||||
|
||||
This is **not** the same failure as the 2026-10-26 planned-restart orphan class
|
||||
([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]), though the error string is similar
|
||||
("Cron execution unverified; reconcile …"). The 10-26 rows were `alerted` and auto-closed; this row
|
||||
is still `detected` and the job stayed paused. **No gateway restart was performed on 2026-10-30**
|
||||
around that window, so the trigger is not established.
|
||||
|
||||
## Current state (verified 2026-10-30 03:3x AWST)
|
||||
|
||||
- Live `jobs.json`: `ea79a2107e36` → `enabled: false`, `last_status: error`, `failure_streak: 1`,
|
||||
`paused_reason` = the gateway-connection text above.
|
||||
- Live `cron_incidents` (table lives inside `executions.db`): `ea79a2_d93c3acdc4f2` is the **only**
|
||||
new row since 2026-10-29; `closed_at` is **NULL**, `acked_at` is **NULL**, `alerted_at` is **NULL**.
|
||||
- The other unclosed row is `0e753e_e7f8194a7d06` (Facebook monthly, `failure_type: agent`, `detected`
|
||||
since 2026-10-01, clears on its 2026-11-01 slot) — **2 unclosed rows in total**.
|
||||
- Fleet is now **29 jobs — 25 enabled / 4 disabled**; this job is the 4th disabled one.
|
||||
- The rest of the fleet was healthy in the same window: 2026-10-29 settled **177 `completed` / 0
|
||||
`failed`**, and 2026-10-30 (to 05:4x) shows 37 `completed`, 1 `running` (this stocktake), 1
|
||||
`unknown` (this job), 0 `failed`.
|
||||
|
||||
## Actions taken
|
||||
|
||||
1. Recorded the incident and its two distinct failure modes here.
|
||||
2. Updated [[systems/scheduled-tasks]] — the job's row, the Disabled Jobs table, the fleet counts
|
||||
(29/25/4), the non-clean `last_status` count (3 → 4), the "Failing (last run failed outright)"
|
||||
bullet (0 → 1), and the "Failing (delivery only)" bullet (1 → 2).
|
||||
3. Updated [[infrastructure/hosts]] for the cron count.
|
||||
|
||||
## Actions NOT taken — and why
|
||||
|
||||
- **The job was not re-enabled.** Re-enabling a job whose gateway connection failed an hour ago
|
||||
risks an immediate re-pause, and it would close the incident's purpose. The recovery belongs to
|
||||
whoever owns the gateway fix.
|
||||
- **No gateway restart was performed.** This was a stocktake run, and restarting the gateway is a
|
||||
disruptive action with its own orphan-risk class ([[incidents/2026-10-26-planned-restart-orphaned-cron-run]]).
|
||||
The fault is recorded, not papered over.
|
||||
|
||||
## Next step
|
||||
|
||||
Re-enable `ea79a2107e36` **after** the gateway connection is confirmed healthy, then confirm the
|
||||
run completes and — critically — that a delivery row appears in `deliveries.db`. The point of this
|
||||
monitor is to report on exception to topic `102`; if the topic path is still degraded, re-enabling it
|
||||
recreates a silent monitor.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]] — fleet ledger
|
||||
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — the earlier "execution unverified" class
|
||||
- [[incidents/2026-08-08-gitea-config-backup-push-dead]] — green row, dead destination
|
||||
- [[systems/mnemosyne]] — the subsystem being monitored
|
||||
Reference in New Issue
Block a user