8.7 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07) | incident | open | 2026-11-08 | 2026-11-09 | 2026-11-09 | 2026-11-15 | high |
|
|
Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
Summary. A gateway restart was requested at ~04:29:52 AWST on 2026-11-07. Hermes held it in
restart-after-turn mode for the full 1800 s window, then timed out and drained (#77184). The drain
killed the two long-running cron executions that were still in flight, and both were closed failed
with RuntimeError: Cron job '…' lost its durable fire claim ownership:
| job | id | run window (ledger) | closed |
|---|---|---|---|
| Mnemosyne Health Monitor | ea79a2107e36 |
00:00:42.918 → 05:02:22.870 | failed |
| wiki-stocktake-maintenance | 1a8e90ea2271 |
03:00:31.515 → 05:02:22.174 | failed |
systemd restarted the unit at 05:01:00 (hermes-gateway.service: Scheduled restart job, restart counter is at 3.); the replacement process was the one that wrote both terminal failed rows at 05:02:22.
The two runs were collateral, not the cause — nothing in the ledger shows them failing on their own.
Timeline (AWST; journalctl --user -u hermes-gateway unless stated)
| time | event |
|---|---|
| 04:27:55 | conduit_push self-update starts (0.122.2 → 0.134.4, systems/conduit) — ~/.hermes/conduit-self-update.json |
| ~04:29:52 | a gateway restart is requested — inferred, not logged: the after-turn wait that expires at 04:59:52 is 1800 s |
| 04:29:54.774 / .796 | ERROR cron.scheduler: Job 'ea79a2107e36'/'1a8e90ea2271': Restart-safe cron worker exited with status 1 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown. → incident rows raised |
| 04:29:5x → 04:59 | the gateway itself keeps running — pid 175269 keeps logging at 04:34, 04:39 … 04:59 |
| 04:32:32 | conduit's ledger marks its update step: failed, failed_step: restarting_gateway, detail "The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host." |
| 04:59:52.433 | WARNING gateway.run: Restart after-turn wait timed out after 1800s with 3 still active; proceeding to stop()/drain which may interrupt remaining work (#77184) |
| 05:00:43.469 | ERROR cron.scheduler: Job 'Mnemosyne Health Monitor' failed: RuntimeError: … lost its durable fire claim ownership; admitted turn cc64fdf07d8c4dbd898d83dc2fa74894 fails |
| 05:00:47.344 | the same for wiki-stocktake-maintenance; admitted turn 56c8fe136a1241e7827316299b490ecb fails |
| 05:00:50.863 | Force-killed MCP process 175401 (cua-driver) after SIGTERM timeout |
| 05:01:00 | systemd: Scheduled restart job, restart counter is at 3 |
| 05:01:41 | gateway back up as pid 2981474 |
| 05:02:22 | both ledger rows finalised failed |
| 05:02:41 → 05:02:59 | job 96eb95ce12f2 (agent-log ERROR watchdog) exits 1 having caught the health monitor's traceback — it reported the event correctly, ~19 s later |
| 05:43:03/04 | hermes-dashboard stopped and started (the 0.22.5 → 0.23.0 update, systems/hermes-agent) |
| 11:11:44 → 11:11:55 | the gateway is restarted again to load the reinstalled hermes-hq-push 0.9.42 routes (systems/hermes-hq-push) |
| 15:43:17 / 17:08:52 | the health monitor's two incident rows are closed / resolved |
| 2026-11-08 00:00 | this audit: both stocktake rows are still detected, alerted_at: null |
Incident-ledger state (live read 2026-11-08 00:0x AWST)
| id | job | state | first seen | alerted_at | closed |
|---|---|---|---|---|---|
1a8e90_2f370ef764f5 |
wiki-stocktake-maintenance | detected (open) |
2026-11-07T05:02:21.573 | null | — |
1a8e90_628d914f1f97 |
wiki-stocktake-maintenance | detected (open) |
2026-11-07T04:29:54.815 | null | — |
ea79a2_111e67a3d83c |
Mnemosyne Health Monitor | closed |
2026-11-07T04:29:54.795 | 04:29:57.581 | 15:43:17 |
ea79a2_f317fcd843ae |
Mnemosyne Health Monitor | resolved |
2026-11-07T13:36:56.713 | 13:36:59 | 17:08:52 — "Cron execution unverified; reconcile …/admissions/df79590101fb3b3164e2261a1018f78473c5" |
The day ends with two open rows on one job, neither alerted (alerted_at null) — the same red row, no
alert shape the 2026-11-06 audit recorded for Job Radar, here on a different failure path.
Root cause and working hypotheses
Established. This is a restart-drain event, not a scheduler or model fault: the ledger error string,
the 1800 s after-turn timeout, and the systemd restart on the same minute line up. The runtime behaviour is
#77184 — a requested restart is deferred until active turns finish, and when the 1800 s window expires
the gateway proceeds to stop()/drain "which may interrupt remaining work". The two cron runs were that
remaining work.
Not established — the restart's origin. The request at ~04:29:52 is:
- 2 s before both restart-safe cron workers exited status 1 (04:29:54), and
- inside the
conduit_pushself-update window (0.122.2 → 0.134.4, 04:27:55 → 04:32:32), whose own final step is recorded asrestarting_gateway= failed.
Working hypothesis: the conduit self-update requested the restart, could not complete it (the gateway runs under systemd), recorded failure — and the gateway went on to drain and restart 31 minutes later. Every timestamp fits; no log line names the requester, so this is not proven. The alternative (another component requested it) cannot be excluded. The 0.22.5 → 0.23.0 updater is excluded: its own receipt does not begin until 05:05:13, after the restart was already pending.
Second, independent defect — adoption lost the claim instead of resuming it. Both workers adopted their executions at 04:29:54 and then exited status 1 without a durable terminal state; the adopted runs were never re-driven, and the claims were only closed by the next process at 05:00:43/47. The ledger cannot say whether their side effects ran ("whether side effects ran is unknown"), so the stocktake's 2026-11-07 edits are known to exist only because an in-day session committed them (systems/scheduled-tasks).
Impact
- Mnemosyne Health Monitor delivered nothing from its 00:00 slot. Its 11-06 slot had taken 7 h 47 m 56 s and its 11-05 slot 12 h 13 m 54 s, so this slot was already deep in its multi-hour tail (systems/mnemosyne).
- wiki-stocktake-maintenance lost its 03:00 slot for a second consecutive day (
failure_streak: 2,last_status: error) — no stocktake ran on 2026-11-07. - Job Radar was not affected on 11-07: no slot failed and the day's report landed normally, unlike 2026-11-06 (incidents/2026-11-06-job-radar-concurrent-fire-claim-discard).
Verification evidence
journalctl --user -u hermes-gateway --since '2026-11-07 04:00' --until '2026-11-07 05:10'
/home/hermes/.hermes/cron/executions.db (status/error per row; cron_incidents table)
/home/hermes/.hermes/conduit-self-update.json
/home/hermes/.hermes/cron/jobs.json (failure_streak, last_status)
Open items
- The two open stocktake rows have never been reconciled (
alerted_atnull) — either resolve them or fix the alerting path that let adetectedrow sit silent. ⚠ 2026-11-09 re-check: still open, and now three.1a8e90_628d914f1f97and1a8e90_2f370ef764f5(both first seen 2026-11-07) are unchanged, and a third arrived —1a8e90_a9cad99cb150,detected2026-11-08T04:35:38,alerted_at: null, from the 03:01:11 → 04:36:08 failure onRuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending…. Unclosed incident rows 2 → 3. The alert path itself is not broken: the Newsletter Digest row85edf6_0877ca1e0eacalerted at 07:23:07 and closed asresolvedat 08:06:54 the same window. So the gap is specific to the stocktake's rows. → incidents/2026-11-08-pre-tool-call-hook-timeout-cascade - Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and whether the 1800 s after-turn wait should refuse to drain long-running cron executions.