7.1 KiB
title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
| title | type | status | created | updated | verified_on | review_after | confidence | tags | sources | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in) | incident | active | 2026-10-27 | 2026-10-27 | 2026-10-27 | 2026-11-03 | high |
|
|
Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)
Headline
On 2026-10-26 at 14:33:35 AWST the gateway received a planned SIGTERM (under_systemd=yes,
parent_name=systemd) — and the shutdown drain still interrupted a running cron execution, because
the drain reported timed_out=True after only 1.35 s. The orphaned run was
ea79a2107e36 Mnemosyne Health Monitor (12-hourly 0 */12 * * *), which had been claimed at
12:01:13 and was 2 h 34 m into its work.
This is the second instance of the orphaned-cron-run class documented for 2026-10-20 (incidents/2026-10-20-gateway-unclean-exit-orphaned-run), with three material differences:
| 2026-10-20 | 2026-10-26 | |
|---|---|---|
| Stop type | unclean exit (no exit path, suspected_oom=False) |
planned SIGTERM under systemd |
| Execution closed as | unknown ("owner exited before a durable terminal state") |
failed with a new error string |
| Incident row raised | no | yes — two |
| Failure delivered | no | yes, 14:35:59 |
So the loud-orphan recommendation from the 10-20 page is now partly implemented — the run no longer disappears silently. What it did not do is finish the work or avoid the interruption.
Timeline (all times AWST, 2026-10-26)
| Time | Event | Source |
|---|---|---|
| 12:01:13 | ea79a2107e36 Mnemosyne Health Monitor execution claimed, running |
executions.db |
| 00:01:46 → 00:10:45 | the same job's previous (00:01) slot — completed in 9 m | executions.db |
| 14:33:35 | Received SIGTERM as a planned gateway stop, under_systemd=yes, parent_name=systemd, loadavg_1m=2.15 |
gateway.log |
| 14:33:37 | Thread 34795 not found while sending the shutdown notice (same dead thread as the birthday/Uber jobs) |
gateway.log |
| 14:33:37 | incident row ea79a2_c3ab2d784ba3 opens — "Cron execution unverified; reconcile ~/.hermes/cron/admissions/724a69108b…json" (alerted) |
cron_incidents |
| 14:33:38 | Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True → interrupting remaining work |
gateway.log |
| 14:34:25 | 1 authority task(s) did not settle within 40s; left for recovery; Profile /home/hermes/.hermes still has live writers at shutdown; its admissions stay for recovery |
gateway.log |
| 14:34:27 | ERROR gateway.session_authority: Admitted turn 28ad127ffb2a44c0bcd17283a93aa496 failed; cron session local-253a96… paused: runtime_draining |
gateway.log |
| 14:34:27 | that session finalises with end_reason: cron_complete (66 msgs, 39 tool calls, 127 API calls) |
state.db |
| 14:34:29 | Gateway stopped (total teardown 54.06s); Skipping .clean_shutdown marker — drain timed out with interrupted agents |
gateway.log |
| 14:35:18 | next gateway life starts (Starting Hermes Gateway...) |
gateway.log |
| 14:35:59 | execution closes failed: RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership |
executions.db |
| 14:35:59 | incident row ea79a2_a4661554a3c2 opens on that RuntimeError (alerted); execution c5ac27e3… delivered the failure |
cron_incidents, deliveries.db |
| 14:46:12 / 15:01:15 | Mnemosyne Watchdog 96eb95ce12f2 fails twice (Script exited with code 1) reacting to the log ERROR — rows 96eb95_0c89bd373a0a, 96eb95_e32174134546 |
cron_incidents |
| 15:16:12 | both watchdog rows resolved (the error aged out of its 15-min log window); watchdog back to ok |
cron_incidents |
What is new or unexplained
- A planned stop is enough to orphan a run. No crash was required. The drain had a 30 s budget and
reported
timed_out=Trueafter 1.35 s withcron_now=0— i.e. the drain did not wait out its budget before taking the interrupt path. Logged verbatim; not diagnosed. - New terminal error string:
lost its durable fire claim ownership. The 10-20 orphan closed asunknownwith a scheduler-side note; this one closed asfailedfrom the worker's own perspective. Any alerting or triage that pattern-matches on the 10-20 wording will miss this one. - Two rows for one fault.
ea79a2_a4661554a3c2(fire claim) andea79a2_c3ab2d784ba3(unverified / reconcile admission) both remainalerted, unclosed as of 2026-10-27 00:1x, and the job carriesfailure_streak: 2from a single interrupted fire. - The admission was reconciled.
~/.hermes/cron/admissions/no longer holds724a69108b…json(only the 00:00 / 00:01 files from the following night), so the "reconcile" step in the row's text has been satisfied — the rows were simply never closed. - The watchdog works. This fault did alert, twice, and self-resolved — which is a genuine improvement over 10-20 (one wakes up via the incident ledger, not via a missing digest).
Impact
- One 12-hourly memory consolidation cycle did not run.
ea79a2107e36is the Mnemosyne health/consolidation job → systems/mnemosyne. The 00:01 slot completed normally; the 12:01 slot produced no completed work. - Day tally: 2026-10-26 settled 169
completed/ 3failed— and all three failures are this one fault (1 Health Monitor + 2 derivative Watchdog reds). 2026-10-25 was 172 / 2. - Session cost sunk: the interrupted run burned 66 messages / 39 tool calls / 127 API calls (~597 k input, 254 k output tokens) before the interrupt.
- Unclosed incident rows 2 → 3 (
0e753e_e7f8194a7d06carried over from 2026-10-01, plus the two newea79a2_*rows).
Recommended follow-ups (not yet done)
- Decide the drain policy for long-running cron jobs: this job legitimately occupies 9 m → 2 h+.
Either exempt a cron execution from the drain deadline, or interrupt it with a durable
"interrupted → re-fire this slot" marker instead of a terminal
RuntimeError. - Close or ack the two
ea79a2_*rows — the admitted turn failed and the admission is gone, so they are historical. - Verify the drain-timeout arithmetic (
timed_out=Trueat 1.35 s against a 30 s budget) — if the deadline is being bypassed whenever a writer is live, every future restart can interrupt work. - Confirm
failure_streakshould count the unverified row at all — a single interrupted fire reading streak 2 inflates the job's apparent health.
Related
- incidents/2026-10-20-gateway-unclean-exit-orphaned-run — first instance of this class
- systems/scheduled-tasks — fleet ledger and per-window tallies
- systems/mnemosyne — the interrupted job's purpose and settings
- systems/hermes-agent — gateway lifecycle and restart history