--- title: Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in) type: incident status: active created: 2026-11-01 updated: 2026-11-01 verified_on: 2026-11-01 review_after: 2026-11-08 confidence: high tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure] sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-probe] --- # Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in) ## Headline On **2026-10-31 at 18:06:41 AWST** the gateway received a **`SIGUSR1` reload** (`systemctl reload` / `hermes gateway restart`). It deferred its own stop for in-flight work — and this time it **waited the whole deferral cap**, hit it at **18:37:38** with *"drain timed out with interrupted agents"*, and force-restarted. The casualty was `ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`), whose 12:01:06 slot had been running for **6 h 38 m 09 s**. It closed **`failed`** at 18:39:15 on **`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership`**. This is the **third variant of the orphan class** — after the 2026-10-20 *unclean crash* exit and the 2026-10-26 *planned `SIGTERM` whose drain timed out after 1.35 s* — and the first where the drain mechanism behaved as designed and still ended in a lost run. ## Timeline (live gateway log, `~/.hermes/logs/agent.log`) | Time (AWST) | Event | |---|---| | 18:06:41.772 | `SIGUSR1 received (systemctl reload / hermes gateway restart): performing a graceful gateway restart — drain active turns, exit, supervisor relaunches.` | | 18:06:41.788 | `Restart requested with 2 active work unit(s); deferring stop() until they finish (cap=1800s) so in-flight turns are not amputated (#77184)` | | 18:06:41.899 | Deferred on **1** unit: the cron unit `ea79a2107e36` (`elapsed_s: 21950.7`, `external: True`, `wedged: False`, **`restart_safe: True`** — *excluded* from the wait) plus `{'kind': 'managed', 'pid': 397}` | | 18:07:11 → 18:11:12 | Countdown logged every 30 s: 1770 s → 1529 s remaining, same single `managed pid 397` unit | | 18:11:23 | **Second `SIGUSR1`** arrives while the first restart is still deferred | | 18:20:13 | `Restart deferred: … (988s remaining before force drain): [{'kind': 'managed', 'pid': 397}]` | | **≈18:36:41** | **Cap reached** (18:06:41 + 1 800 s) | | 18:37:38.341 | `Skipping .clean_shutdown marker — drain timed out with interrupted agents; next startup will recover their interrupted turns.` | | 18:38:43.284 | New gateway starts; 18:39:14 reconnection watcher for 1 failed platform (`relay`) | | 18:39:15.208 | Failure **delivered** (delivery row `9e7da0fa…`, `for_failure=1`, status `delivered`) | | 18:39:15.563 | Execution `9e7da0fafd2847b69643c6f235cb8651` closes **`failed`** — 6 h 38 m 09 s | | 18:39:23 / 18:45:23 / 19:00:22 | Three derivative `96eb95ce12f2` (log-error watchdog) reds quote the cron error; the watchdog clears on its 19:15:21 slot | | 19:23:39 → 19:24:40 | **Second restart, clean**: `drain took 0.00s, timed_out=False`; `Previous gateway exited cleanly — skipping session suspension` | Current gateway: `hermes-gateway.service` active since **19:23:56**, PID `175269` (4 h 43 m at the 2026-11-01 00:07 probe). Nothing failed for the remaining ~5 h of 10-31 (every `96eb95ce12f2` and `0c771463c3cd` slot from 19:15:21 onward `completed`). ## Incident-ledger effect | Row | Job | State | First seen | Last seen | |---|---|---|---|---| | `ea79a2_9197df32320b` | `ea79a2107e36` | **`alerted` (open)** | 2026-10-31T18:06:44 | 2026-10-31T18:06:44 | | `ea79a2_a4661554a3c2` | `ea79a2107e36` | **`alerted` (open, reused row)** | **2026-10-26**T14:35:59 | 2026-10-31T18:39:15 | | `0e753e_e7f8194a7d06` | `0e753e9720fd` | `detected` (open, unchanged) | 2026-10-01T09:00:55 | 2026-10-01T09:00:55 | The `error_sig` match re-opened the **2026-10-26 row** rather than filing a fresh one, so two rows now describe one event: the earlier *"Cron execution unverified; reconcile `admissions/292056e5742c1ba4…`"* (`failure_type: unknown`) and the definitive *fire-claim* row. **Unclosed incident rows 1 → 3.** ## What is genuinely new here 1. **The deferral cap is reachable in normal operation, not just under a stuck process.** The 10-26 incident drained in **1.35 s**; this one waited the full **1 800 s** because a `managed` unit (`pid 397`) stayed registered. Force drain is therefore a *routine* outcome of any reload that overlaps a long-running cron job. 2. **`restart_safe: True` did not protect the execution.** The scheduler explicitly classified the cron unit as restart-safe and *excluded* it from the wait — and the run still died with a lost claim. The flag protects the *scheduler's* bookkeeping, not the running agent's lease. 3. **A `SIGUSR1` reload is not a lighter-weight restart.** The pre-restart log shows plugin/update machinery active minutes earlier (`.update-/plugin` capability checks at 18:05:41 and 18:19:53, `plugin scan passed` at 18:20:22) — the reload behaved exactly like a full stop/start for the in-flight job. ## Open questions - Why is a `managed` (non-cron) work unit allowed to hold the gateway for the entire 30-minute cap? What is `pid 397`, and should it be exempt from the "wait" set (or bounded separately)? - The 12:01 slot had already run **6 h 38 m** against **10 h 01 m** for its 10-30 catch-up. Should a reload deferral weight long cron runs differently — or should the cron lease survive a supervisor relaunch, as the `restart_safe` flag implies? - Two `SIGUSR1`s 4 m 42 s apart produced a **single** interruption and then a second clean restart at 19:23 — is the double signal expected? ## Impact - One job failure + three derivative watchdog reds = **all four** of 2026-10-31's failed executions (day closed **160 `completed` / 4 `failed`**). - `ea79a2107e36` now reads `last_status: error`, **`failure_streak: 2`** — the only non-zero streak in the fleet; non-clean `last_status` **3 → 4**, failing (run failed outright) **0 → 1**. - Its next slot (2026-11-01 **00:00:41**) was already `running` at audit time. ## Related - [[systems/scheduled-tasks]] — fleet ledger and the 10-31 window numbers - [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — variant 2, the 1.35 s drain timeout - [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — variant 1, the unclean crash exit - [[systems/hermes-agent]] — gateway lifecycle record - [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same job, the day before