--- title: Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07) type: incident status: open created: 2026-11-08 updated: 2026-11-09 verified_on: 2026-11-09 review_after: 2026-11-15 confidence: high tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake] sources: [live-executions-db, live-cron-incidents, live-agent-log, live-systemd-journal, conduit-self-update-json, live-jobs-json] --- # Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07) **Summary.** A gateway restart was **requested at ~04:29:52 AWST on 2026-11-07**. Hermes held it in *restart-after-turn* mode for the full **1800 s** window, then timed out and drained (`#77184`). The drain killed the two long-running cron executions that were still in flight, and both were closed **`failed`** with `RuntimeError: Cron job '…' lost its durable fire claim ownership`: | job | id | run window (ledger) | closed | |---|---|---|---| | *Mnemosyne Health Monitor* | `ea79a2107e36` | 00:00:42.918 → 05:02:22.870 | `failed` | | *wiki-stocktake-maintenance* | `1a8e90ea2271` | 03:00:31.515 → 05:02:22.174 | `failed` | systemd restarted the unit at **05:01:00** (`hermes-gateway.service: Scheduled restart job, restart counter is at 3.`); the replacement process was the one that wrote both terminal `failed` rows at 05:02:22. **The two runs were collateral, not the cause** — nothing in the ledger shows them failing on their own. ## Timeline (AWST; `journalctl --user -u hermes-gateway` unless stated) | time | event | |---|---| | 04:27:55 | `conduit_push` self-update starts (`0.122.2 → 0.134.4`, [[systems/conduit]]) — `~/.hermes/conduit-self-update.json` | | ~04:29:52 | a gateway restart is requested — **inferred**, not logged: the after-turn wait that expires at 04:59:52 is 1800 s | | 04:29:54.774 / .796 | `ERROR cron.scheduler: Job 'ea79a2107e36'/'1a8e90ea2271': Restart-safe cron worker exited with status 1 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown.` → **incident rows raised** | | 04:29:5x → 04:59 | the gateway **itself keeps running** — `pid 175269` keeps logging at 04:34, 04:39 … 04:59 | | 04:32:32 | conduit's ledger marks its update `step: failed`, `failed_step: restarting_gateway`, detail *"The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host."* | | 04:59:52.433 | `WARNING gateway.run: Restart after-turn wait timed out after 1800s with 3 still active; proceeding to stop()/drain which may interrupt remaining work (#77184)` | | 05:00:43.469 | `ERROR cron.scheduler: Job 'Mnemosyne Health Monitor' failed: RuntimeError: … lost its durable fire claim ownership`; admitted turn `cc64fdf07d8c4dbd898d83dc2fa74894` fails | | 05:00:47.344 | the same for `wiki-stocktake-maintenance`; admitted turn `56c8fe136a1241e7827316299b490ecb` fails | | 05:00:50.863 | `Force-killed MCP process 175401 (cua-driver) after SIGTERM timeout` | | 05:01:00 | systemd: *Scheduled restart job, restart counter is at 3* | | 05:01:41 | gateway back up as `pid 2981474` | | 05:02:22 | both ledger rows finalised `failed` | | 05:02:41 → 05:02:59 | job `96eb95ce12f2` (agent-log ERROR watchdog) exits 1 having caught the health monitor's traceback — **it reported the event correctly, ~19 s later** | | 05:43:03/04 | `hermes-dashboard` stopped and started (the 0.22.5 → 0.23.0 update, [[systems/hermes-agent]]) | | 11:11:44 → 11:11:55 | the gateway is restarted again to load the reinstalled `hermes-hq-push` 0.9.42 routes ([[systems/hermes-hq-push]]) | | 15:43:17 / 17:08:52 | the health monitor's two incident rows are closed / resolved | | 2026-11-08 00:00 | **this audit:** both stocktake rows are still `detected`, `alerted_at: null` | ## Incident-ledger state (live read 2026-11-08 00:0x AWST) | id | job | state | first seen | alerted_at | closed | |---|---|---|---|---|---| | `1a8e90_2f370ef764f5` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T05:02:21.573 | **null** | — | | `1a8e90_628d914f1f97` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T04:29:54.815 | **null** | — | | `ea79a2_111e67a3d83c` | Mnemosyne Health Monitor | `closed` | 2026-11-07T04:29:54.795 | 04:29:57.581 | 15:43:17 | | `ea79a2_f317fcd843ae` | Mnemosyne Health Monitor | `resolved` | 2026-11-07T13:36:56.713 | 13:36:59 | 17:08:52 — *"Cron execution unverified; reconcile `…/admissions/df79590101fb3b3164e2261a1018f78473c5`"* | The day ends with **two open rows on one job, neither alerted** (`alerted_at` null) — the same *red row, no alert* shape the 2026-11-06 audit recorded for Job Radar, here on a different failure path. ## Root cause and working hypotheses **Established.** This is a **restart-drain** event, not a scheduler or model fault: the ledger error string, the 1800 s after-turn timeout, and the systemd restart on the same minute line up. The runtime behaviour is `#77184` — a requested restart is deferred until active turns finish, and when the 1800 s window expires the gateway proceeds to `stop()`/drain *"which may interrupt remaining work"*. The two cron runs were that remaining work. **Not established — the restart's origin.** The request at ~04:29:52 is: - **2 s before** both restart-safe cron workers exited status 1 (04:29:54), and - inside the `conduit_push` self-update window (`0.122.2 → 0.134.4`, 04:27:55 → 04:32:32), whose own final step is recorded as **`restarting_gateway` = failed**. *Working hypothesis:* **the conduit self-update requested the restart, could not complete it (the gateway runs under systemd), recorded failure — and the gateway went on to drain and restart 31 minutes later.** Every timestamp fits; no log line names the requester, so this is not proven. The alternative (another component requested it) cannot be excluded. The 0.22.5 → 0.23.0 updater is **excluded**: its own receipt does not begin until **05:05:13**, after the restart was already pending. **Second, independent defect — adoption lost the claim instead of resuming it.** Both workers *adopted* their executions at 04:29:54 and then exited status 1 without a durable terminal state; the adopted runs were never re-driven, and the claims were only closed by the next process at 05:00:43/47. The ledger cannot say whether their side effects ran (*"whether side effects ran is unknown"*), so the stocktake's 2026-11-07 edits are known to exist only because an in-day session committed them ([[systems/scheduled-tasks]]). ## Impact - **Mnemosyne Health Monitor** delivered nothing from its 00:00 slot. Its 11-06 slot had taken **7 h 47 m 56 s** and its 11-05 slot **12 h 13 m 54 s**, so this slot was already deep in its multi-hour tail ([[systems/mnemosyne]]). - **wiki-stocktake-maintenance** lost its 03:00 slot for a **second consecutive day** (`failure_streak: 2`, `last_status: error`) — no stocktake ran on 2026-11-07. - **Job Radar was not affected on 11-07:** no slot failed and the day's report landed normally, unlike 2026-11-06 ([[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]). ## Verification evidence journalctl --user -u hermes-gateway --since '2026-11-07 04:00' --until '2026-11-07 05:10' /home/hermes/.hermes/cron/executions.db (status/error per row; cron_incidents table) /home/hermes/.hermes/conduit-self-update.json /home/hermes/.hermes/cron/jobs.json (failure_streak, last_status) ## Open items - The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or fix the alerting path that let a `detected` row sit silent. ⚠ **2026-11-09 re-check: still open, and now three.** `1a8e90_628d914f1f97` and `1a8e90_2f370ef764f5` (both first seen 2026-11-07) are unchanged, and a **third** arrived — `1a8e90_a9cad99cb150`, `detected` 2026-11-08T04:35:38, `alerted_at: null`, from the 03:01:11 → 04:36:08 failure on `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending…`. **Unclosed incident rows 2 → 3.** The alert path itself is **not** broken: the *Newsletter Digest* row `85edf6_0877ca1e0eac` alerted at **07:23:07** and closed as `resolved` at **08:06:54** the same window. So the gap is specific to the **stocktake's** rows. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]] - Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and whether the 1800 s after-turn wait should refuse to drain long-running cron executions. ## See also - [[systems/scheduled-tasks]] - [[systems/hermes-agent]] - [[systems/conduit]] - [[systems/mnemosyne]] - [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]] - [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]