audit(2026-11-01): Conduit updated to 0.122.2 (memory-timeout patch now upstream, call store re-shaped); document the local-voice-mode fault, the force-drained gateway reload that orphaned a cron run, and both dead Asana paths; 2026-10-31 settled 160/4
This commit is contained in:
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-11-01
|
||||
updated: 2026-11-01
|
||||
verified_on: 2026-11-01
|
||||
review_after: 2026-11-08
|
||||
confidence: high
|
||||
tags: [gateway, cron, scheduler, lifecycle, restart, memory, silent-failure]
|
||||
sources: [live-logs, live-gateway-ledger, live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sqlite-deliveries, live-probe]
|
||||
---
|
||||
|
||||
# Gateway Reload on 2026-10-31 Force-Drained After Its Full 1 800 s Cap and Orphaned a Live Cron Run — Mnemosyne Health Monitor (6 h 38 m in)
|
||||
|
||||
## Headline
|
||||
|
||||
On **2026-10-31 at 18:06:41 AWST** the gateway received a **`SIGUSR1` reload** (`systemctl reload` /
|
||||
`hermes gateway restart`). It deferred its own stop for in-flight work — and this time it **waited the
|
||||
whole deferral cap**, hit it at **18:37:38** with *"drain timed out with interrupted agents"*, and
|
||||
force-restarted. The casualty was `ea79a2107e36` **Mnemosyne Health Monitor** (12-hourly `0 */12 * * *`),
|
||||
whose 12:01:06 slot had been running for **6 h 38 m 09 s**. It closed **`failed`** at 18:39:15 on
|
||||
**`RuntimeError: Cron job 'Mnemosyne Health Monitor' lost its durable fire claim ownership`**.
|
||||
|
||||
This is the **third variant of the orphan class** — after the 2026-10-20 *unclean crash* exit and the
|
||||
2026-10-26 *planned `SIGTERM` whose drain timed out after 1.35 s* — and the first where the drain
|
||||
mechanism behaved as designed and still ended in a lost run.
|
||||
|
||||
## Timeline (live gateway log, `~/.hermes/logs/agent.log`)
|
||||
|
||||
| Time (AWST) | Event |
|
||||
|---|---|
|
||||
| 18:06:41.772 | `SIGUSR1 received (systemctl reload / hermes gateway restart): performing a graceful gateway restart — drain active turns, exit, supervisor relaunches.` |
|
||||
| 18:06:41.788 | `Restart requested with 2 active work unit(s); deferring stop() until they finish (cap=1800s) so in-flight turns are not amputated (#77184)` |
|
||||
| 18:06:41.899 | Deferred on **1** unit: the cron unit `ea79a2107e36` (`elapsed_s: 21950.7`, `external: True`, `wedged: False`, **`restart_safe: True`** — *excluded* from the wait) plus `{'kind': 'managed', 'pid': 397}` |
|
||||
| 18:07:11 → 18:11:12 | Countdown logged every 30 s: 1770 s → 1529 s remaining, same single `managed pid 397` unit |
|
||||
| 18:11:23 | **Second `SIGUSR1`** arrives while the first restart is still deferred |
|
||||
| 18:20:13 | `Restart deferred: … (988s remaining before force drain): [{'kind': 'managed', 'pid': 397}]` |
|
||||
| **≈18:36:41** | **Cap reached** (18:06:41 + 1 800 s) |
|
||||
| 18:37:38.341 | `Skipping .clean_shutdown marker — drain timed out with interrupted agents; next startup will recover their interrupted turns.` |
|
||||
| 18:38:43.284 | New gateway starts; 18:39:14 reconnection watcher for 1 failed platform (`relay`) |
|
||||
| 18:39:15.208 | Failure **delivered** (delivery row `9e7da0fa…`, `for_failure=1`, status `delivered`) |
|
||||
| 18:39:15.563 | Execution `9e7da0fafd2847b69643c6f235cb8651` closes **`failed`** — 6 h 38 m 09 s |
|
||||
| 18:39:23 / 18:45:23 / 19:00:22 | Three derivative `96eb95ce12f2` (log-error watchdog) reds quote the cron error; the watchdog clears on its 19:15:21 slot |
|
||||
| 19:23:39 → 19:24:40 | **Second restart, clean**: `drain took 0.00s, timed_out=False`; `Previous gateway exited cleanly — skipping session suspension` |
|
||||
|
||||
Current gateway: `hermes-gateway.service` active since **19:23:56**, PID `175269` (4 h 43 m at the
|
||||
2026-11-01 00:07 probe). Nothing failed for the remaining ~5 h of 10-31 (every `96eb95ce12f2` and
|
||||
`0c771463c3cd` slot from 19:15:21 onward `completed`).
|
||||
|
||||
## Incident-ledger effect
|
||||
|
||||
| Row | Job | State | First seen | Last seen |
|
||||
|---|---|---|---|---|
|
||||
| `ea79a2_9197df32320b` | `ea79a2107e36` | **`alerted` (open)** | 2026-10-31T18:06:44 | 2026-10-31T18:06:44 |
|
||||
| `ea79a2_a4661554a3c2` | `ea79a2107e36` | **`alerted` (open, reused row)** | **2026-10-26**T14:35:59 | 2026-10-31T18:39:15 |
|
||||
| `0e753e_e7f8194a7d06` | `0e753e9720fd` | `detected` (open, unchanged) | 2026-10-01T09:00:55 | 2026-10-01T09:00:55 |
|
||||
|
||||
The `error_sig` match re-opened the **2026-10-26 row** rather than filing a fresh one, so two rows now
|
||||
describe one event: the earlier *"Cron execution unverified; reconcile `admissions/292056e5742c1ba4…`"*
|
||||
(`failure_type: unknown`) and the definitive *fire-claim* row. **Unclosed incident rows 1 → 3.**
|
||||
|
||||
## What is genuinely new here
|
||||
|
||||
1. **The deferral cap is reachable in normal operation, not just under a stuck process.** The 10-26
|
||||
incident drained in **1.35 s**; this one waited the full **1 800 s** because a `managed` unit
|
||||
(`pid 397`) stayed registered. Force drain is therefore a *routine* outcome of any reload that
|
||||
overlaps a long-running cron job.
|
||||
2. **`restart_safe: True` did not protect the execution.** The scheduler explicitly classified the cron
|
||||
unit as restart-safe and *excluded* it from the wait — and the run still died with a lost claim. The
|
||||
flag protects the *scheduler's* bookkeeping, not the running agent's lease.
|
||||
3. **A `SIGUSR1` reload is not a lighter-weight restart.** The pre-restart log shows plugin/update
|
||||
machinery active minutes earlier (`.update-<id>/plugin` capability checks at 18:05:41 and 18:19:53,
|
||||
`plugin scan passed` at 18:20:22) — the reload behaved exactly like a full stop/start for the
|
||||
in-flight job.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Why is a `managed` (non-cron) work unit allowed to hold the gateway for the entire 30-minute cap?
|
||||
What is `pid 397`, and should it be exempt from the "wait" set (or bounded separately)?
|
||||
- The 12:01 slot had already run **6 h 38 m** against **10 h 01 m** for its 10-30 catch-up. Should a
|
||||
reload deferral weight long cron runs differently — or should the cron lease survive a supervisor
|
||||
relaunch, as the `restart_safe` flag implies?
|
||||
- Two `SIGUSR1`s 4 m 42 s apart produced a **single** interruption and then a second clean restart at
|
||||
19:23 — is the double signal expected?
|
||||
|
||||
## Impact
|
||||
|
||||
- One job failure + three derivative watchdog reds = **all four** of 2026-10-31's failed executions
|
||||
(day closed **160 `completed` / 4 `failed`**).
|
||||
- `ea79a2107e36` now reads `last_status: error`, **`failure_streak: 2`** — the only non-zero streak in
|
||||
the fleet; non-clean `last_status` **3 → 4**, failing (run failed outright) **0 → 1**.
|
||||
- Its next slot (2026-11-01 **00:00:41**) was already `running` at audit time.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/scheduled-tasks]] — fleet ledger and the 10-31 window numbers
|
||||
- [[incidents/2026-10-26-planned-restart-orphaned-cron-run]] — variant 2, the 1.35 s drain timeout
|
||||
- [[incidents/2026-10-20-gateway-unclean-exit-orphaned-run]] — variant 1, the unclean crash exit
|
||||
- [[systems/hermes-agent]] — gateway lifecycle record
|
||||
- [[incidents/2026-10-30-mnemosyne-health-monitor-paused]] — the same job, the day before
|
||||
Reference in New Issue
Block a user