Files
agent-estate-wiki/incidents/2026-10-31-gateway-reload-force-drain-orphaned-run.md

6.6 KiB

title, type, status, created, updated, verified_on, review_after, confidence, tags, sources
title type status created updated verified_on review_after confidence tags sources
Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in) incident active 2026-11-01 2026-11-01 2026-11-01 2026-11-08 high
gateway
cron
scheduler
lifecycle
restart
memory
silent-failure
live-logs
live-gateway-ledger
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-sqlite-deliveries
live-probe

Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)

Headline

On 2026-10-31 at 18:06:41 AWST the gateway received a SIGUSR1 reload (systemctl reload / hermes gateway restart). It deferred its own stop for in-flight work — and this time it waited the whole deferral cap, hit it at 18:37:38 with "drain timed out with interrupted agents", and force-restarted. The casualty was ea79a2107e36 Mnemosyne Health Monitor (12-hourly 0 */12 * * *), whose 12:01:06 slot had been running for 6 h 38 m 09 s. It closed failed at 18:39:15 on RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership.

This is the third variant of the orphan class — after the 2026-10-20 unclean crash exit and the 2026-10-26 planned SIGTERM whose drain timed out after 1.35 s — and the first where the drain mechanism behaved as designed and still ended in a lost run.

Timeline (live gateway log, ~/.hermes/logs/agent.log)

Time (AWST) Event
18:06:41.772 SIGUSR1 received (systemctl reload / hermes gateway restart): performing a graceful gateway restart — drain active turns, exit, supervisor relaunches.
18:06:41.788 Restart requested with 2 active work unit(s); deferring stop() until they finish (cap=1800s) so in-flight turns are not amputated (#77184)
18:06:41.899 Deferred on 1 unit: the cron unit ea79a2107e36 (elapsed_s: 21950.7, external: True, wedged: False, restart_safe: True — excluded from the wait) plus {'kind': 'managed', 'pid': 397}
18:07:11 → 18:11:12 Countdown logged every 30 s: 1770 s → 1529 s remaining, same single managed pid 397 unit
18:11:23 Second SIGUSR1 arrives while the first restart is still deferred
18:20:13 Restart deferred: … (988s remaining before force drain): [{'kind': 'managed', 'pid': 397}]
≈18:36:41 Cap reached (18:06:41 + 1 800 s)
18:37:38.341 Skipping .clean_shutdown marker — drain timed out with interrupted agents; next startup will recover their interrupted turns.
18:38:43.284 New gateway starts; 18:39:14 reconnection watcher for 1 failed platform (relay)
18:39:15.208 Failure delivered (delivery row 9e7da0fa…, for_failure=1, status delivered)
18:39:15.563 Execution 9e7da0fafd2847b69643c6f235cb8651 closes failed — 6 h 38 m 09 s
18:39:23 / 18:45:23 / 19:00:22 Three derivative 96eb95ce12f2 (log-error watchdog) reds quote the cron error; the watchdog clears on its 19:15:21 slot
19:23:39 → 19:24:40 Second restart, clean: drain took 0.00s, timed_out=False; Previous gateway exited cleanly — skipping session suspension

Current gateway: hermes-gateway.service active since 19:23:56, PID 175269 (4 h 43 m at the 2026-11-01 00:07 probe). Nothing failed for the remaining ~5 h of 10-31 (every 96eb95ce12f2 and 0c771463c3cd slot from 19:15:21 onward completed).

Incident-ledger effect

Row Job State First seen Last seen
ea79a2_9197df32320b ea79a2107e36 alerted (open) 2026-10-31T18:06:44 2026-10-31T18:06:44
ea79a2_a4661554a3c2 ea79a2107e36 alerted (open, reused row) 2026-10-26T14:35:59 2026-10-31T18:39:15
0e753e_e7f8194a7d06 0e753e9720fd detected (open, unchanged) 2026-10-01T09:00:55 2026-10-01T09:00:55

The error_sig match re-opened the 2026-10-26 row rather than filing a fresh one, so two rows now describe one event: the earlier "Cron execution unverified; reconcile admissions/292056e5742c1ba4…" (failure_type: unknown) and the definitive fire-claim row. Unclosed incident rows 1 → 3.

What is genuinely new here

  1. The deferral cap is reachable in normal operation, not just under a stuck process. The 10-26 incident drained in 1.35 s; this one waited the full 1 800 s because a managed unit (pid 397) stayed registered. Force drain is therefore a routine outcome of any reload that overlaps a long-running cron job.
  2. restart_safe: True did not protect the execution. The scheduler explicitly classified the cron unit as restart-safe and excluded it from the wait — and the run still died with a lost claim. The flag protects the scheduler's bookkeeping, not the running agent's lease.
  3. A SIGUSR1 reload is not a lighter-weight restart. The pre-restart log shows plugin/update machinery active minutes earlier (.update-<id>/plugin capability checks at 18:05:41 and 18:19:53, plugin scan passed at 18:20:22) — the reload behaved exactly like a full stop/start for the in-flight job.

Open questions

  • Why is a managed (non-cron) work unit allowed to hold the gateway for the entire 30-minute cap? What is pid 397, and should it be exempt from the "wait" set (or bounded separately)?
  • The 12:01 slot had already run 6 h 38 m against 10 h 01 m for its 10-30 catch-up. Should a reload deferral weight long cron runs differently — or should the cron lease survive a supervisor relaunch, as the restart_safe flag implies?
  • Two SIGUSR1s 4 m 42 s apart produced a single interruption and then a second clean restart at 19:23 — is the double signal expected?

Impact

  • One job failure + three derivative watchdog reds = all four of 2026-10-31's failed executions (day closed 160 completed / 4 failed).
  • ea79a2107e36 now reads last_status: error, failure_streak: 2 — the only non-zero streak in the fleet; non-clean last_status 3 → 4, failing (run failed outright) 0 → 1.
  • Its next slot (2026-11-01 00:00:41) was already running at audit time.