Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)
incident
active
2026-11-01
2026-11-01
2026-11-01
2026-11-08
high
gateway
cron
scheduler
lifecycle
restart
memory
silent-failure
live-logs
live-gateway-ledger
live-sqlite-cron-executions
live-sqlite-cron-incidents
live-sqlite-deliveries
live-probe
Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)
Headline
On 2026-10-31 at 18:06:41 AWST the gateway received a SIGUSR1 reload (systemctl reload /
hermes gateway restart). It deferred its own stop for in-flight work — and this time it waited the
whole deferral cap, hit it at 18:37:38 with "drain timed out with interrupted agents", and
force-restarted. The casualty was ea79a2107e36Mnemosyne Health Monitor (12-hourly 0 */12 * * *),
whose 12:01:06 slot had been running for 6 h 38 m 09 s. It closed failed at 18:39:15 on
RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership.
This is the third variant of the orphan class — after the 2026-10-20 unclean crash exit and the
2026-10-26 planned SIGTERM whose drain timed out after 1.35 s — and the first where the drain
mechanism behaved as designed and still ended in a lost run.
SIGUSR1 received (systemctl reload / hermes gateway restart): performing a graceful gateway restart — drain active turns, exit, supervisor relaunches.
18:06:41.788
Restart requested with 2 active work unit(s); deferring stop() until they finish (cap=1800s) so in-flight turns are not amputated (#77184)
18:06:41.899
Deferred on 1 unit: the cron unit ea79a2107e36 (elapsed_s: 21950.7, external: True, wedged: False, restart_safe: True — excluded from the wait) plus {'kind': 'managed', 'pid': 397}
18:07:11 → 18:11:12
Countdown logged every 30 s: 1770 s → 1529 s remaining, same single managed pid 397 unit
18:11:23
Second SIGUSR1 arrives while the first restart is still deferred
18:20:13
Restart deferred: … (988s remaining before force drain): [{'kind': 'managed', 'pid': 397}]
≈18:36:41
Cap reached (18:06:41 + 1 800 s)
18:37:38.341
Skipping .clean_shutdown marker — drain timed out with interrupted agents; next startup will recover their interrupted turns.
18:38:43.284
New gateway starts; 18:39:14 reconnection watcher for 1 failed platform (relay)
18:39:15.208
Failure delivered (delivery row 9e7da0fa…, for_failure=1, status delivered)
18:39:15.563
Execution 9e7da0fafd2847b69643c6f235cb8651 closes failed — 6 h 38 m 09 s
18:39:23 / 18:45:23 / 19:00:22
Three derivative 96eb95ce12f2 (log-error watchdog) reds quote the cron error; the watchdog clears on its 19:15:21 slot
19:23:39 → 19:24:40
Second restart, clean: drain took 0.00s, timed_out=False; Previous gateway exited cleanly — skipping session suspension
Current gateway: hermes-gateway.service active since 19:23:56, PID 175269 (4 h 43 m at the
2026-11-01 00:07 probe). Nothing failed for the remaining ~5 h of 10-31 (every 96eb95ce12f2 and
0c771463c3cd slot from 19:15:21 onward completed).
Incident-ledger effect
Row
Job
State
First seen
Last seen
ea79a2_9197df32320b
ea79a2107e36
alerted (open)
2026-10-31T18:06:44
2026-10-31T18:06:44
ea79a2_a4661554a3c2
ea79a2107e36
alerted (open, reused row)
2026-10-26T14:35:59
2026-10-31T18:39:15
0e753e_e7f8194a7d06
0e753e9720fd
detected (open, unchanged)
2026-10-01T09:00:55
2026-10-01T09:00:55
The error_sig match re-opened the 2026-10-26 row rather than filing a fresh one, so two rows now
describe one event: the earlier "Cron execution unverified; reconcile admissions/292056e5742c1ba4…"
(failure_type: unknown) and the definitive fire-claim row. Unclosed incident rows 1 → 3.
What is genuinely new here
The deferral cap is reachable in normal operation, not just under a stuck process. The 10-26
incident drained in 1.35 s; this one waited the full 1 800 s because a managed unit
(pid 397) stayed registered. Force drain is therefore a routine outcome of any reload that
overlaps a long-running cron job.
restart_safe: True did not protect the execution. The scheduler explicitly classified the cron
unit as restart-safe and excluded it from the wait — and the run still died with a lost claim. The
flag protects the scheduler's bookkeeping, not the running agent's lease.
A SIGUSR1 reload is not a lighter-weight restart. The pre-restart log shows plugin/update
machinery active minutes earlier (.update-<id>/plugin capability checks at 18:05:41 and 18:19:53,
plugin scan passed at 18:20:22) — the reload behaved exactly like a full stop/start for the
in-flight job.
Open questions
Why is a managed (non-cron) work unit allowed to hold the gateway for the entire 30-minute cap?
What is pid 397, and should it be exempt from the "wait" set (or bounded separately)?
The 12:01 slot had already run 6 h 38 m against 10 h 01 m for its 10-30 catch-up. Should a
reload deferral weight long cron runs differently — or should the cron lease survive a supervisor
relaunch, as the restart_safe flag implies?
Two SIGUSR1s 4 m 42 s apart produced a single interruption and then a second clean restart at
19:23 — is the double signal expected?
Impact
One job failure + three derivative watchdog reds = all four of 2026-10-31's failed executions
(day closed 160 completed / 4 failed).
ea79a2107e36 now reads last_status: error, failure_streak: 2 — the only non-zero streak in
the fleet; non-clean last_status3 → 4, failing (run failed outright) 0 → 1.
Its next slot (2026-11-01 00:00:41) was already running at audit time.