Files
agent-estate-wiki/incidents/2026-10-26-planned-restart-orphaned-cron-run.md
T
Tony0410 f880ba981d audit: 2026-10-28 nightly — radar budget overrun now a recorded failure (new incident page); radar split into two cron jobs; tailscale peer table re-verified; TTS provider corrected to the Mac mini speech server
Window 2026-10-27 settled 172 completed / 1 failed. The failure is dca8482e4f76
(Daily Marketing Job Radar - Discovery): 07:01:08 -> 11:03:54 (4h 2m 46s), closed
failed, delivered report opens "INCOMPLETE (tool-call budget exhausted before
Notion write stage)" — discovery/verify/dedup/classify finished, Notion writes and
cover letters never ran (Notion DB held 451 entries, 0 added). Incident row
dca848_e93a538ade6b opened 11:03:53, still open, failure_type: auth (misclassified).

Response the same day: split into stage 1 dca8482e4f76 (Discovery, 0 7 * * 1-5,
deliver local) + new stage 2 5fc292ff0f3b (Write and Letters, 30 11,14 * * 1-5,
topic 1030, created 2026-10-27T17:34:15). Fleet 28 -> 29 jobs (26/3).

Also: the two incident rows the 2026-10-26 planned restart raised auto-closed
00:38:32 (unclosed 3 -> 2) and the orphan class did not recur; non-clean
last_status 4 -> 5; skills-index.md regenerated by the 10-27 03:00 slot but never
committed (302 skills / 69 categories; freshness 0 stale / 104 fresh / 4 no-date of
108) — committed here and index.md corrected; tailscale.md peer table rebuilt from
tailscale status --json (14 rows, m93p peer flag disputed and ping-verified, sftpgo
CT299 running but its peer dropped); systems/hermes-agent.md TTS corrected to
provider openai -> Mac mini speech server (phonon/alloy), systems/conduit.md gained
the voice pipeline section.

New: incidents/2026-10-27-job-radar-budget-exhausted.md
Nothing removed or archived. No secrets.
2026-10-28 00:20:59 +08:00

7.8 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in) incident active 2026-10-27 2026-10-28 2026-10-27 2026-11-03 high
gateway
cron
scheduler
lifecycle
restart
memory
silent-failure
live-logs
live-gateway-ledger
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-sqlite-sessions
live-probe

Planned Gateway Restart on 2026-10-26 Orphaned a Live Cron Run — Mnemosyne Health Monitor (2 h 34 m in)

Headline

On 2026-10-26 at 14:33:35 AWST the gateway received a planned SIGTERM (under_systemd=yes, parent_name=systemd) — and the shutdown drain still interrupted a running cron execution, because the drain reported timed_out=True after only 1.35 s. The orphaned run was ea79a2107e36 Mnemosyne Health Monitor (12-hourly 0 */12 * * *), which had been claimed at 12:01:13 and was 2 h 34 m into its work.

This is the second instance of the orphaned-cron-run class documented for 2026-10-20 (incidents/2026-10-20-gateway-unclean-exit-orphaned-run), with three material differences:

2026-10-20 2026-10-26
Stop type unclean exit (no exit path, suspected_oom=False) planned SIGTERM under systemd
Execution closed as unknown ("owner exited before a durable terminal state") failed with a new error string
Incident row raised no yes — two
Failure delivered no yes, 14:35:59

So the loud-orphan recommendation from the 10-20 page is now partly implemented — the run no longer disappears silently. What it did not do is finish the work or avoid the interruption.

Timeline (all times AWST, 2026-10-26)

Time Event Source
12:01:13 ea79a2107e36 Mnemosyne Health Monitor execution claimed, running executions.db
00:01:46 → 00:10:45 the same job's previous (00:01) slot — completed in 9 m executions.db
14:33:35 Received SIGTERM as a planned gateway stop, under_systemd=yes, parent_name=systemd, loadavg_1m=2.15 gateway.log
14:33:37 Thread 34795 not found while sending the shutdown notice (same dead thread as the birthday/Uber jobs) gateway.log
14:33:37 incident row ea79a2_c3ab2d784ba3 opens — "Cron execution unverified; reconcile ~/.hermes/cron/admissions/724a69108b…json" (alerted) cron_incidents
14:33:38 Shutdown drain: 1 in-flight cron job(s) … drain took 1.35s, timed_out=True → interrupting remaining work gateway.log
14:34:25 1 authority task(s) did not settle within 40s; left for recovery; Profile /home/hermes/.hermes still has live writers at shutdown; its admissions stay for recovery gateway.log
14:34:27 ERROR gateway.session_authority: Admitted turn 28ad127ffb2a44c0bcd17283a93aa496 failed; cron session local-253a96… paused: runtime_draining gateway.log
14:34:27 that session finalises with end_reason: cron_complete (66 msgs, 39 tool calls, 127 API calls) state.db
14:34:29 Gateway stopped (total teardown 54.06s); Skipping .clean_shutdown marker — drain timed out with interrupted agents gateway.log
14:35:18 next gateway life starts (Starting Hermes Gateway...) gateway.log
14:35:59 execution closes failed: RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership executions.db
14:35:59 incident row ea79a2_a4661554a3c2 opens on that RuntimeError (alerted); execution c5ac27e3… delivered the failure cron_incidents, deliveries.db
14:46:12 / 15:01:15 Mnemosyne Watchdog 96eb95ce12f2 fails twice (Script exited with code 1) reacting to the log ERROR — rows 96eb95_0c89bd373a0a, 96eb95_e32174134546 cron_incidents
15:16:12 both watchdog rows resolved (the error aged out of its 15-min log window); watchdog back to ok cron_incidents

What is new or unexplained

  1. A planned stop is enough to orphan a run. No crash was required. The drain had a 30 s budget and reported timed_out=True after 1.35 s with cron_now=0 — i.e. the drain did not wait out its budget before taking the interrupt path. Logged verbatim; not diagnosed.
  2. New terminal error string: lost its durable fire claim ownership. The 10-20 orphan closed as unknown with a scheduler-side note; this one closed as failed from the worker's own perspective. Any alerting or triage that pattern-matches on the 10-20 wording will miss this one.
  3. Two rows for one fault. ea79a2_a4661554a3c2 (fire claim) and ea79a2_c3ab2d784ba3 (unverified / reconcile admission) both remain alerted, unclosed as of 2026-10-27 00:1x, and the job carries failure_streak: 2 from a single interrupted fire.
  4. The admission was reconciled. ~/.hermes/cron/admissions/ no longer holds 724a69108b…json (only the 00:00 / 00:01 files from the following night), so the "reconcile" step in the row's text has been satisfied — the rows were simply never closed.
  5. The watchdog works. This fault did alert, twice, and self-resolved — which is a genuine improvement over 10-20 (one wakes up via the incident ledger, not via a missing digest).

Impact

  • One 12-hourly memory consolidation cycle did not run. ea79a2107e36 is the Mnemosyne health/consolidation job → systems/mnemosyne. The 00:01 slot completed normally; the 12:01 slot produced no completed work.
  • Day tally: 2026-10-26 settled 169 completed / 3 failed — and all three failures are this one fault (1 Health Monitor + 2 derivative Watchdog reds). 2026-10-25 was 172 / 2.
  • Session cost sunk: the interrupted run burned 66 messages / 39 tool calls / 127 API calls (~597 k input, 254 k output tokens) before the interrupt.
  • Unclosed incident rows 2 → 3 (0e753e_e7f8194a7d06 carried over from 2026-10-01, plus the two new ea79a2_* rows).
  1. Decide the drain policy for long-running cron jobs: this job legitimately occupies 9 m → 2 h+. Either exempt a cron execution from the drain deadline, or interrupt it with a durable "interrupted → re-fire this slot" marker instead of a terminal RuntimeError.
  2. Close or ack the two ea79a2_* rows — the admitted turn failed and the admission is gone, so they are historical.
  3. Verify the drain-timeout arithmetic (timed_out=True at 1.35 s against a 30 s budget) — if the deadline is being bypassed whenever a writer is live, every future restart can interrupt work.
  4. Confirm failure_streak should count the unverified row at all — a single interrupted fire reading streak 2 inflates the job's apparent health.

Update — 2026-10-28 (nightly audit)

  • No recurrence. Every run in the 2026-10-27 window except the radar's budget overrun completed, and the watchdog logged no alert for the day.
  • Both incident rows this fault raised closed automatically — ea79a2_a4661554a3c2 and ea79a2_c3ab2d784ba3 auto-closed 2026-10-27T00:38:32+08:00 (unclosed rows fleet-wide fell 3 → 2).
  • The lost its durable fire claim ownership string did not appear in any 10-27 execution row.
  • Open question from the original write-up still stands: why a planned stop bypasses the drain deadline (timed_out=True at 1.35 s of a 30 s budget).