--- title: Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in) type: incident status: resolved created: 2026-10-27 updated: 2026-11-19 verified_on: 2026-11-19 review_after: 2026-12-19 confidence: high tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure] sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-sessions, live-probe] --- # Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in) ## Headline On **2026-10-26 at 14:33:35 AWST** the gateway received a **planned `SIGTERM`** (`under_systemd=yes`, `parent_name=systemd`) — and the shutdown drain **still interrupted a running cron execution**, because the drain reported `timed_out=True` after only **1.35 s**. The orphaned run was `ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`), which had been claimed at **12:01:13** and was **2 h 34 m** into its work. This is the **second instance** of the orphaned-cron-run class documented for 2026-10-20 ([[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]]), with three material differences: | | 2026-10-20 | 2026-10-26 | |---|---|---| | Stop type | **unclean exit** (no exit path, `suspected_oom=False`) | **planned `SIGTERM`** under systemd | | Execution closed as | `unknown` ("owner exited before a durable terminal state") | **`failed`** with a **new** error string | | Incident row raised | **no** | **yes — two** | | Failure delivered | no | **yes**, 14:35:59 | So the loud-orphan recommendation from the 10-20 page is **now partly implemented** — the run no longer disappears silently. What it did not do is *finish the work* or *avoid the interruption*. ## Timeline (all times AWST, 2026-10-26) | Time | Event | Source | |---|---|---| | 12:01:13 | `ea79a2107e36` Mnemosyne Health Monitor execution claimed, `running` | `executions.db` | | 00:01:46 → 00:10:45 | the same job's previous (00:01) slot — **completed in 9 m** | `executions.db` | | 14:33:35 | `Received SIGTERM as a planned gateway stop`, `under_systemd=yes`, `parent_name=systemd`, `loadavg_1m=2.15` | `gateway.log` | | 14:33:37 | `Thread 34795 not found` while sending the shutdown notice (same dead thread as the birthday/Uber jobs) | `gateway.log` | | 14:33:37 | incident row `ea79a2_c3ab2d784ba3` opens — *"Cron execution unverified; reconcile `~/.hermes/cron/admissions/724a69108b…json`"* (`alerted`) | `cron_incidents` | | 14:33:38 | `Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True` → `interrupting remaining work` | `gateway.log` | | 14:34:25 | `1 authority task(s) did not settle within 40s; left for recovery`; `Profile /home/hermes/.hermes still has live writers at shutdown; its admissions stay for recovery` | `gateway.log` | | 14:34:27 | `ERROR gateway.session_authority: Admitted turn 28ad127ffb2a44c0bcd17283a93aa496 failed`; cron session `local-253a96…` **paused: runtime_draining** | `gateway.log` | | 14:34:27 | that session finalises with `end_reason: cron_complete` (66 msgs, 39 tool calls, 127 API calls) | `state.db` | | 14:34:29 | `Gateway stopped (total teardown 54.06s)`; **`Skipping .clean_shutdown marker — drain timed out with interrupted agents`** | `gateway.log` | | 14:35:18 | next gateway life starts (`Starting Hermes Gateway...`) | `gateway.log` | | 14:35:59 | execution closes **`failed`**: `RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership` | `executions.db` | | 14:35:59 | incident row `ea79a2_a4661554a3c2` opens on that RuntimeError (`alerted`); execution `c5ac27e3…` **delivered** the failure | `cron_incidents`, `deliveries.db` | | 14:46:12 / 15:01:15 | Mnemosyne Watchdog `96eb95ce12f2` fails twice (`Script exited with code 1`) reacting to the log `ERROR` — rows `96eb95_0c89bd373a0a`, `96eb95_e32174134546` | `cron_incidents` | | 15:16:12 | both watchdog rows **`resolved`** (the error aged out of its 15-min log window); watchdog back to `ok` | `cron_incidents` | ## What is new or unexplained 1. **A planned stop is enough to orphan a run.** No crash was required. The drain had a **30 s** budget and reported `timed_out=True` after **1.35 s** with `cron_now=0` — i.e. the drain did **not** wait out its budget before taking the interrupt path. Logged verbatim; **not diagnosed**. 2. **New terminal error string:** `lost its durable fire claim ownership`. The 10-20 orphan closed as `unknown` with a scheduler-side note; this one closed as `failed` from the worker's own perspective. Any alerting or triage that pattern-matches on the 10-20 wording will miss this one. 3. **Two rows for one fault.** `ea79a2_a4661554a3c2` (fire claim) and `ea79a2_c3ab2d784ba3` (unverified / reconcile admission) both remain **`alerted`, unclosed** as of 2026-10-27 00:1x, and the job carries **`failure_streak: 2`** from a **single** interrupted fire. 4. **The admission was reconciled.** `~/.hermes/cron/admissions/` no longer holds `724a69108b…json` (only the 00:00 / 00:01 files from the following night), so the "reconcile" step in the row's text has been satisfied — the rows were simply never closed. 5. **The watchdog works.** This fault **did** alert, twice, and self-resolved — which is a genuine improvement over 10-20 (one wakes up via the incident ledger, not via a missing digest). ## Impact - **One 12-hourly memory consolidation cycle did not run.** `ea79a2107e36` is the Mnemosyne health/consolidation job → [[systems/mnemosyne]]. The 00:01 slot completed normally; the 12:01 slot produced no completed work. - **Day tally:** 2026-10-26 settled **169 `completed` / 3 `failed`** — and **all three failures are this one fault** (1 Health Monitor + 2 derivative Watchdog reds). 2026-10-25 was **172 / 2**. - **Session cost sunk:** the interrupted run burned 66 messages / 39 tool calls / 127 API calls (~597 k input, 254 k output tokens) before the interrupt. - **Unclosed incident rows 2 → 3** (`0e753e_e7f8194a7d06` carried over from 2026-10-01, plus the two new `ea79a2_*` rows). ## Recommended follow-ups (not yet done) 1. Decide the drain policy for **long-running cron jobs**: this job legitimately occupies 9 m → 2 h+. Either exempt a cron execution from the drain deadline, or interrupt it with a durable *"interrupted → re-fire this slot"* marker instead of a terminal `RuntimeError`. 2. **Close or ack the two `ea79a2_*` rows** — the admitted turn failed and the admission is gone, so they are historical. 3. **Verify the drain-timeout arithmetic** (`timed_out=True` at 1.35 s against a 30 s budget) — if the deadline is being bypassed whenever a writer is live, every future restart can interrupt work. 4. Confirm `failure_streak` should count the *unverified* row at all — a single interrupted fire reading **streak 2** inflates the job's apparent health. ## Related - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — first instance of this class - [[systems/scheduled-tasks]] — fleet ledger and per-window tallies - [[systems/mnemosyne]] — the interrupted job's purpose and settings - [[systems/hermes-agent]] — gateway lifecycle and restart history ## Update — 2026-10-28 (nightly audit) - **No recurrence.** Every run in the 2026-10-27 window except the radar's budget overrun completed, and the watchdog logged no alert for the day. - **Both incident rows this fault raised closed automatically** — `ea79a2_a4661554a3c2` and `ea79a2_c3ab2d784ba3` **auto-closed 2026-10-27T00:38:32+08:00** (unclosed rows fleet-wide fell 3 → 2). - The `lost its durable fire claim ownership` string did not appear in any 10-27 execution row. - Open question from the original write-up still stands: why a *planned* stop bypasses the drain deadline (`timed_out=True` at 1.35 s of a 30 s budget). ## Update — 2026-11-19 (stocktake re-probe) - **Both `ea79a2_*` rows are now closed** (follow-up 2 satisfied): `ea79a2_c3ab2d784ba3` `resolved` 2026-10-27T00:38:32, and `ea79a2_a4661554a3c2` **`closed` 2026-11-07T15:43:13** (live probe of `cron_incidents` in `~/.hermes/cron/executions.db`). - The `lost its durable fire claim ownership` error string has **no rows in the retained `executions` window** — no recurrence since 2026-10-27 (subject to the DB's ~18-day retention floor; the 10-28 update already confirmed the 10-27 window clean). - **Status → `resolved`.** The fault did not recur and its incident rows are closed. Follow-ups 1, 3 and 4 (drain-policy decision, drain-timeout arithmetic, `failure_streak` semantics) remain open design questions and were **not** actioned.