Files
agent-estate-wiki/incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain.md
T

122 lines
8.0 KiB
Markdown

---
title: Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
type: incident
status: open
created: 2026-11-08
updated: 2026-11-08
verified_on: 2026-11-08
review_after: 2026-11-15
confidence: high
tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake]
sources: [live-executions-db, live-cron-incidents, live-agent-log, live-systemd-journal, conduit-self-update-json, live-jobs-json]
---
# Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
**Summary.** A gateway restart was **requested at ~04:29:52 AWST on 2026-11-07**. Hermes held it in
*restart-after-turn* mode for the full **1800 s** window, then timed out and drained (`#77184`). The drain
killed the two long-running cron executions that were still in flight, and both were closed **`failed`**
with `RuntimeError: Cron job '…' lost its durable fire claim ownership`:
| job | id | run window (ledger) | closed |
|---|---|---|---|
| *Mnemosyne Health Monitor* | `ea79a2107e36` | 00:00:42.918 → 05:02:22.870 | `failed` |
| *wiki-stocktake-maintenance* | `1a8e90ea2271` | 03:00:31.515 → 05:02:22.174 | `failed` |
systemd restarted the unit at **05:01:00** (`hermes-gateway.service: Scheduled restart job, restart counter
is at 3.`); the replacement process was the one that wrote both terminal `failed` rows at 05:02:22.
**The two runs were collateral, not the cause** — nothing in the ledger shows them failing on their own.
## Timeline (AWST; `journalctl --user -u hermes-gateway` unless stated)
| time | event |
|---|---|
| 04:27:55 | `conduit_push` self-update starts (`0.122.2 → 0.134.4`, [[systems/conduit]]) — `~/.hermes/conduit-self-update.json` |
| ~04:29:52 | a gateway restart is requested — **inferred**, not logged: the after-turn wait that expires at 04:59:52 is 1800 s |
| 04:29:54.774 / .796 | `ERROR cron.scheduler: Job 'ea79a2107e36'/'1a8e90ea2271': Restart-safe cron worker exited with status 1 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown.` → **incident rows raised** |
| 04:29:5x → 04:59 | the gateway **itself keeps running** — `pid 175269` keeps logging at 04:34, 04:39 … 04:59 |
| 04:32:32 | conduit's ledger marks its update `step: failed`, `failed_step: restarting_gateway`, detail *"The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host."* |
| 04:59:52.433 | `WARNING gateway.run: Restart after-turn wait timed out after 1800s with 3 still active; proceeding to stop()/drain which may interrupt remaining work (#77184)` |
| 05:00:43.469 | `ERROR cron.scheduler: Job 'Mnemosyne Health Monitor' failed: RuntimeError: … lost its durable fire claim ownership`; admitted turn `cc64fdf07d8c4dbd898d83dc2fa74894` fails |
| 05:00:47.344 | the same for `wiki-stocktake-maintenance`; admitted turn `56c8fe136a1241e7827316299b490ecb` fails |
| 05:00:50.863 | `Force-killed MCP process 175401 (cua-driver) after SIGTERM timeout` |
| 05:01:00 | systemd: *Scheduled restart job, restart counter is at 3* |
| 05:01:41 | gateway back up as `pid 2981474` |
| 05:02:22 | both ledger rows finalised `failed` |
| 05:02:41 → 05:02:59 | job `96eb95ce12f2` (agent-log ERROR watchdog) exits 1 having caught the health monitor's traceback — **it reported the event correctly, ~19 s later** |
| 05:43:03/04 | `hermes-dashboard` stopped and started (the 0.22.5 → 0.23.0 update, [[systems/hermes-agent]]) |
| 11:11:44 → 11:11:55 | the gateway is restarted again to load the reinstalled `hermes-hq-push` 0.9.42 routes ([[systems/hermes-hq-push]]) |
| 15:43:17 / 17:08:52 | the health monitor's two incident rows are closed / resolved |
| 2026-11-08 00:00 | **this audit:** both stocktake rows are still `detected`, `alerted_at: null` |
## Incident-ledger state (live read 2026-11-08 00:0x AWST)
| id | job | state | first seen | alerted_at | closed |
|---|---|---|---|---|---|
| `1a8e90_2f370ef764f5` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T05:02:21.573 | **null** | — |
| `1a8e90_628d914f1f97` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T04:29:54.815 | **null** | — |
| `ea79a2_111e67a3d83c` | Mnemosyne Health Monitor | `closed` | 2026-11-07T04:29:54.795 | 04:29:57.581 | 15:43:17 |
| `ea79a2_f317fcd843ae` | Mnemosyne Health Monitor | `resolved` | 2026-11-07T13:36:56.713 | 13:36:59 | 17:08:52 — *"Cron execution unverified; reconcile `…/admissions/df79590101fb3b3164e2261a1018f78473c5`"* |
The day ends with **two open rows on one job, neither alerted** (`alerted_at` null) — the same *red row, no
alert* shape the 2026-11-06 audit recorded for Job Radar, here on a different failure path.
## Root cause and working hypotheses
**Established.** This is a **restart-drain** event, not a scheduler or model fault: the ledger error string,
the 1800 s after-turn timeout, and the systemd restart on the same minute line up. The runtime behaviour is
`#77184` — a requested restart is deferred until active turns finish, and when the 1800 s window expires
the gateway proceeds to `stop()`/drain *"which may interrupt remaining work"*. The two cron runs were that
remaining work.
**Not established — the restart's origin.** The request at ~04:29:52 is:
- **2 s before** both restart-safe cron workers exited status 1 (04:29:54), and
- inside the `conduit_push` self-update window (`0.122.2 → 0.134.4`, 04:27:55 → 04:32:32), whose own final
step is recorded as **`restarting_gateway` = failed**.
*Working hypothesis:* **the conduit self-update requested the restart, could not complete it (the gateway
runs under systemd), recorded failure — and the gateway went on to drain and restart 31 minutes later.**
Every timestamp fits; no log line names the requester, so this is not proven. The alternative (another
component requested it) cannot be excluded. The 0.22.5 → 0.23.0 updater is **excluded**: its own receipt
does not begin until **05:05:13**, after the restart was already pending.
**Second, independent defect — adoption lost the claim instead of resuming it.** Both workers *adopted*
their executions at 04:29:54 and then exited status 1 without a durable terminal state; the adopted runs
were never re-driven, and the claims were only closed by the next process at 05:00:43/47. The ledger
cannot say whether their side effects ran (*"whether side effects ran is unknown"*), so the stocktake's
2026-11-07 edits are known to exist only because an in-day session committed them
([[systems/scheduled-tasks]]).
## Impact
- **Mnemosyne Health Monitor** delivered nothing from its 00:00 slot. Its 11-06 slot had taken
**7 h 47 m 56 s** and its 11-05 slot **12 h 13 m 54 s**, so this slot was already deep in its multi-hour
tail ([[systems/mnemosyne]]).
- **wiki-stocktake-maintenance** lost its 03:00 slot for a **second consecutive day** (`failure_streak: 2`,
`last_status: error`) — no stocktake ran on 2026-11-07.
- **Job Radar was not affected on 11-07:** no slot failed and the day's report landed normally, unlike
2026-11-06 ([[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]).
## Verification evidence
journalctl --user -u hermes-gateway --since '2026-11-07 04:00' --until '2026-11-07 05:10'
/home/hermes/.hermes/cron/executions.db (status/error per row; cron_incidents table)
/home/hermes/.hermes/conduit-self-update.json
/home/hermes/.hermes/cron/jobs.json (failure_streak, last_status)
## Open items
- The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or
fix the alerting path that let a `detected` row sit silent.
- Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and
whether the 1800 s after-turn wait should refuse to drain long-running cron executions.
## See also
- [[systems/scheduled-tasks]]
- [[systems/hermes-agent]]
- [[systems/conduit]]
- [[systems/mnemosyne]]
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]