Files
agent-estate-wiki/incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md

8.7 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07) incident open 2026-11-08 2026-11-09 2026-11-09 2026-11-15 high
cron
fire-claim
scheduler
gateway-restart
drain
conduit
health-monitor
wiki-stocktake
live-executions-db
live-cron-incidents
live-agent-log
live-systemd-journal
conduit-self-update-json
live-jobs-json

Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)

Summary. A gateway restart was requested at ~04:29:52 AWST on 2026-11-07. Hermes held it in restart-after-turn mode for the full 1800 s window, then timed out and drained (#77184). The drain killed the two long-running cron executions that were still in flight, and both were closed failed with RuntimeError: Cron job '…' lost its durable fire claim ownership:

job id run window (ledger) closed
Mnemosyne Health Monitor ea79a2107e36 00:00:42.918 → 05:02:22.870 failed
wiki-stocktake-maintenance 1a8e90ea2271 03:00:31.515 → 05:02:22.174 failed

systemd restarted the unit at 05:01:00 (hermes-gateway.service: Scheduled restart job, restart counter is at 3.); the replacement process was the one that wrote both terminal failed rows at 05:02:22. The two runs were collateral, not the cause — nothing in the ledger shows them failing on their own.

Timeline (AWST; journalctl --user -u hermes-gateway unless stated)

time event
04:27:55 conduit_push self-update starts (0.122.2 → 0.134.4, systems/conduit) — ~/.hermes/conduit-self-update.json
~04:29:52 a gateway restart is requested — inferred, not logged: the after-turn wait that expires at 04:59:52 is 1800 s
04:29:54.774 / .796 ERROR cron.scheduler: Job 'ea79a2107e36'/'1a8e90ea2271': Restart-safe cron worker exited with status 1 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown. → incident rows raised
04:29:5x → 04:59 the gateway itself keeps running — pid 175269 keeps logging at 04:34, 04:39 … 04:59
04:32:32 conduit's ledger marks its update step: failed, failed_step: restarting_gateway, detail "The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host."
04:59:52.433 WARNING gateway.run: Restart after-turn wait timed out after 1800s with 3 still active; proceeding to stop()/drain which may interrupt remaining work (#77184)
05:00:43.469 ERROR cron.scheduler: Job 'Mnemosyne Health Monitor' failed: RuntimeError: … lost its durable fire claim ownership; admitted turn cc64fdf07d8c4dbd898d83dc2fa74894 fails
05:00:47.344 the same for wiki-stocktake-maintenance; admitted turn 56c8fe136a1241e7827316299b490ecb fails
05:00:50.863 Force-killed MCP process 175401 (cua-driver) after SIGTERM timeout
05:01:00 systemd: Scheduled restart job, restart counter is at 3
05:01:41 gateway back up as pid 2981474
05:02:22 both ledger rows finalised failed
05:02:41 → 05:02:59 job 96eb95ce12f2 (agent-log ERROR watchdog) exits 1 having caught the health monitor's traceback — it reported the event correctly, ~19 s later
05:43:03/04 hermes-dashboard stopped and started (the 0.22.5 → 0.23.0 update, systems/hermes-agent)
11:11:44 → 11:11:55 the gateway is restarted again to load the reinstalled hermes-hq-push 0.9.42 routes (systems/hermes-hq-push)
15:43:17 / 17:08:52 the health monitor's two incident rows are closed / resolved
2026-11-08 00:00 this audit: both stocktake rows are still detected, alerted_at: null

Incident-ledger state (live read 2026-11-08 00:0x AWST)

id job state first seen alerted_at closed
1a8e90_2f370ef764f5 wiki-stocktake-maintenance detected (open) 2026-11-07T05:02:21.573 null —
1a8e90_628d914f1f97 wiki-stocktake-maintenance detected (open) 2026-11-07T04:29:54.815 null —
ea79a2_111e67a3d83c Mnemosyne Health Monitor closed 2026-11-07T04:29:54.795 04:29:57.581 15:43:17
ea79a2_f317fcd843ae Mnemosyne Health Monitor resolved 2026-11-07T13:36:56.713 13:36:59 17:08:52 — "Cron execution unverified; reconcile …/admissions/df79590101fb3b3164e2261a1018f78473c5"

The day ends with two open rows on one job, neither alerted (alerted_at null) — the same red row, no alert shape the 2026-11-06 audit recorded for Job Radar, here on a different failure path.

Root cause and working hypotheses

Established. This is a restart-drain event, not a scheduler or model fault: the ledger error string, the 1800 s after-turn timeout, and the systemd restart on the same minute line up. The runtime behaviour is #77184 — a requested restart is deferred until active turns finish, and when the 1800 s window expires the gateway proceeds to stop()/drain "which may interrupt remaining work". The two cron runs were that remaining work.

Not established — the restart's origin. The request at ~04:29:52 is:

  • 2 s before both restart-safe cron workers exited status 1 (04:29:54), and
  • inside the conduit_push self-update window (0.122.2 → 0.134.4, 04:27:55 → 04:32:32), whose own final step is recorded as restarting_gateway = failed.

Working hypothesis: the conduit self-update requested the restart, could not complete it (the gateway runs under systemd), recorded failure — and the gateway went on to drain and restart 31 minutes later. Every timestamp fits; no log line names the requester, so this is not proven. The alternative (another component requested it) cannot be excluded. The 0.22.5 → 0.23.0 updater is excluded: its own receipt does not begin until 05:05:13, after the restart was already pending.

Second, independent defect — adoption lost the claim instead of resuming it. Both workers adopted their executions at 04:29:54 and then exited status 1 without a durable terminal state; the adopted runs were never re-driven, and the claims were only closed by the next process at 05:00:43/47. The ledger cannot say whether their side effects ran ("whether side effects ran is unknown"), so the stocktake's 2026-11-07 edits are known to exist only because an in-day session committed them (systems/scheduled-tasks).

Impact

  • Mnemosyne Health Monitor delivered nothing from its 00:00 slot. Its 11-06 slot had taken 7 h 47 m 56 s and its 11-05 slot 12 h 13 m 54 s, so this slot was already deep in its multi-hour tail (systems/mnemosyne).
  • wiki-stocktake-maintenance lost its 03:00 slot for a second consecutive day (failure_streak: 2, last_status: error) — no stocktake ran on 2026-11-07.
  • Job Radar was not affected on 11-07: no slot failed and the day's report landed normally, unlike 2026-11-06 (incidents/2026-11-06-job-radar-concurrent-fire-claim-discard).

Verification evidence

journalctl --user -u hermes-gateway --since '2026-11-07 04:00' --until '2026-11-07 05:10'
/home/hermes/.hermes/cron/executions.db    (status/error per row; cron_incidents table)
/home/hermes/.hermes/conduit-self-update.json
/home/hermes/.hermes/cron/jobs.json        (failure_streak, last_status)

Open items

  • The two open stocktake rows have never been reconciled (alerted_at null) — either resolve them or fix the alerting path that let a detected row sit silent. ⚠ 2026-11-09 re-check: still open, and now three. 1a8e90_628d914f1f97 and 1a8e90_2f370ef764f5 (both first seen 2026-11-07) are unchanged, and a third arrived — 1a8e90_a9cad99cb150, detected 2026-11-08T04:35:38, alerted_at: null, from the 03:01:11 → 04:36:08 failure on RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending…. Unclosed incident rows 2 → 3. The alert path itself is not broken: the Newsletter Digest row 85edf6_0877ca1e0eac alerted at 07:23:07 and closed as resolved at 08:06:54 the same window. So the gap is specific to the stocktake's rows. → incidents/2026-11-08-pre-tool-call-hook-timeout-cascade
  • Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and whether the 1800 s after-turn wait should refuse to drain long-running cron executions.

See also