audit 2026-11-08 (window 11-07): cron claim loss in gateway restart drain (2 runs orphaned, 2 incident rows unalerted); Hermes 0.22.5->0.23.0 (snapshot skipped), camofox plugin 1.1.0, conduit 0.134.4, hq-push 0.9.42, opencode-go key expired; skills 302->299 (4 archived); freshness 0/126/3 of 129
This commit is contained in:
@@ -105,6 +105,19 @@ exist, and the affected rows carry no marker distinguishing them from ordinary c
|
||||
- **Not done:** no value was printed, logged or written to any wiki page; nothing was deleted, rotated or
|
||||
scrubbed.
|
||||
|
||||
## 2026-11-08 00:5x AWST — nightly re-check (still OPEN; one narrow improvement)
|
||||
|
||||
- **Cron job prompts: clean.** A regex sweep of every job in `/home/hermes/.hermes/cron/jobs.json` for
|
||||
secret-shaped strings (`sk-…`, `ghp_…`, and any `<key|token|password|secret> = <16+ chars>`) returned
|
||||
**0 hits** on 2026-11-08 00:5x. Only the *shape* was tested — a secret that does not look like a secret
|
||||
would still pass.
|
||||
- **Redaction setting: not confirmable from the file.** `~/.hermes/config.yaml` still carries a
|
||||
`redact_secrets:` key, but its on-disk value reads as a masked token when printed, so the effective
|
||||
on/off state could not be established by reading it. The 2026-11-05 finding — *"no capture-time or
|
||||
delivery-time redaction filter is known to exist"* (line 80 above) — therefore **stands unrefuted**.
|
||||
- **Nothing else moved:** no key rotation, no affected row redacted or forgotten, no remediation option
|
||||
executed. Both **Remediation options 2 and 4 above remain open**.
|
||||
|
||||
## See also
|
||||
|
||||
- [[infrastructure/credentials-map]] - where secrets live, by design
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-11-08
|
||||
updated: 2026-11-08
|
||||
verified_on: 2026-11-08
|
||||
review_after: 2026-11-15
|
||||
confidence: high
|
||||
tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake]
|
||||
sources: [live-executions-db, live-cron-incidents, live-agent-log, live-systemd-journal, conduit-self-update-json, live-jobs-json]
|
||||
---
|
||||
|
||||
# Two cron runs lost their durable fire claims in a gateway restart drain (2026-11-07)
|
||||
|
||||
**Summary.** A gateway restart was **requested at ~04:29:52 AWST on 2026-11-07**. Hermes held it in
|
||||
*restart-after-turn* mode for the full **1800 s** window, then timed out and drained (`#77184`). The drain
|
||||
killed the two long-running cron executions that were still in flight, and both were closed **`failed`**
|
||||
with `RuntimeError: Cron job '…' lost its durable fire claim ownership`:
|
||||
|
||||
| job | id | run window (ledger) | closed |
|
||||
|---|---|---|---|
|
||||
| *Mnemosyne Health Monitor* | `ea79a2107e36` | 00:00:42.918 → 05:02:22.870 | `failed` |
|
||||
| *wiki-stocktake-maintenance* | `1a8e90ea2271` | 03:00:31.515 → 05:02:22.174 | `failed` |
|
||||
|
||||
systemd restarted the unit at **05:01:00** (`hermes-gateway.service: Scheduled restart job, restart counter
|
||||
is at 3.`); the replacement process was the one that wrote both terminal `failed` rows at 05:02:22.
|
||||
**The two runs were collateral, not the cause** — nothing in the ledger shows them failing on their own.
|
||||
|
||||
## Timeline (AWST; `journalctl --user -u hermes-gateway` unless stated)
|
||||
|
||||
| time | event |
|
||||
|---|---|
|
||||
| 04:27:55 | `conduit_push` self-update starts (`0.122.2 → 0.134.4`, [[systems/conduit]]) — `~/.hermes/conduit-self-update.json` |
|
||||
| ~04:29:52 | a gateway restart is requested — **inferred**, not logged: the after-turn wait that expires at 04:59:52 is 1800 s |
|
||||
| 04:29:54.774 / .796 | `ERROR cron.scheduler: Job 'ea79a2107e36'/'1a8e90ea2271': Restart-safe cron worker exited with status 1 after adopting this execution, without writing a durable terminal state; whether side effects ran is unknown.` → **incident rows raised** |
|
||||
| 04:29:5x → 04:59 | the gateway **itself keeps running** — `pid 175269` keeps logging at 04:34, 04:39 … 04:59 |
|
||||
| 04:32:32 | conduit's ledger marks its update `step: failed`, `failed_step: restarting_gateway`, detail *"The plugin is updated, but the gateway didn't restart. Run hermes gateway status on the host."* |
|
||||
| 04:59:52.433 | `WARNING gateway.run: Restart after-turn wait timed out after 1800s with 3 still active; proceeding to stop()/drain which may interrupt remaining work (#77184)` |
|
||||
| 05:00:43.469 | `ERROR cron.scheduler: Job 'Mnemosyne Health Monitor' failed: RuntimeError: … lost its durable fire claim ownership`; admitted turn `cc64fdf07d8c4dbd898d83dc2fa74894` fails |
|
||||
| 05:00:47.344 | the same for `wiki-stocktake-maintenance`; admitted turn `56c8fe136a1241e7827316299b490ecb` fails |
|
||||
| 05:00:50.863 | `Force-killed MCP process 175401 (cua-driver) after SIGTERM timeout` |
|
||||
| 05:01:00 | systemd: *Scheduled restart job, restart counter is at 3* |
|
||||
| 05:01:41 | gateway back up as `pid 2981474` |
|
||||
| 05:02:22 | both ledger rows finalised `failed` |
|
||||
| 05:02:41 → 05:02:59 | job `96eb95ce12f2` (agent-log ERROR watchdog) exits 1 having caught the health monitor's traceback — **it reported the event correctly, ~19 s later** |
|
||||
| 05:43:03/04 | `hermes-dashboard` stopped and started (the 0.22.5 → 0.23.0 update, [[systems/hermes-agent]]) |
|
||||
| 11:11:44 → 11:11:55 | the gateway is restarted again to load the reinstalled `hermes-hq-push` 0.9.42 routes ([[systems/hermes-hq-push]]) |
|
||||
| 15:43:17 / 17:08:52 | the health monitor's two incident rows are closed / resolved |
|
||||
| 2026-11-08 00:00 | **this audit:** both stocktake rows are still `detected`, `alerted_at: null` |
|
||||
|
||||
## Incident-ledger state (live read 2026-11-08 00:0x AWST)
|
||||
|
||||
| id | job | state | first seen | alerted_at | closed |
|
||||
|---|---|---|---|---|---|
|
||||
| `1a8e90_2f370ef764f5` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T05:02:21.573 | **null** | — |
|
||||
| `1a8e90_628d914f1f97` | wiki-stocktake-maintenance | **`detected` (open)** | 2026-11-07T04:29:54.815 | **null** | — |
|
||||
| `ea79a2_111e67a3d83c` | Mnemosyne Health Monitor | `closed` | 2026-11-07T04:29:54.795 | 04:29:57.581 | 15:43:17 |
|
||||
| `ea79a2_f317fcd843ae` | Mnemosyne Health Monitor | `resolved` | 2026-11-07T13:36:56.713 | 13:36:59 | 17:08:52 — *"Cron execution unverified; reconcile `…/admissions/df79590101fb3b3164e2261a1018f78473c5`"* |
|
||||
|
||||
The day ends with **two open rows on one job, neither alerted** (`alerted_at` null) — the same *red row, no
|
||||
alert* shape the 2026-11-06 audit recorded for Job Radar, here on a different failure path.
|
||||
|
||||
## Root cause and working hypotheses
|
||||
|
||||
**Established.** This is a **restart-drain** event, not a scheduler or model fault: the ledger error string,
|
||||
the 1800 s after-turn timeout, and the systemd restart on the same minute line up. The runtime behaviour is
|
||||
`#77184` — a requested restart is deferred until active turns finish, and when the 1800 s window expires
|
||||
the gateway proceeds to `stop()`/drain *"which may interrupt remaining work"*. The two cron runs were that
|
||||
remaining work.
|
||||
|
||||
**Not established — the restart's origin.** The request at ~04:29:52 is:
|
||||
- **2 s before** both restart-safe cron workers exited status 1 (04:29:54), and
|
||||
- inside the `conduit_push` self-update window (`0.122.2 → 0.134.4`, 04:27:55 → 04:32:32), whose own final
|
||||
step is recorded as **`restarting_gateway` = failed**.
|
||||
|
||||
*Working hypothesis:* **the conduit self-update requested the restart, could not complete it (the gateway
|
||||
runs under systemd), recorded failure — and the gateway went on to drain and restart 31 minutes later.**
|
||||
Every timestamp fits; no log line names the requester, so this is not proven. The alternative (another
|
||||
component requested it) cannot be excluded. The 0.22.5 → 0.23.0 updater is **excluded**: its own receipt
|
||||
does not begin until **05:05:13**, after the restart was already pending.
|
||||
|
||||
**Second, independent defect — adoption lost the claim instead of resuming it.** Both workers *adopted*
|
||||
their executions at 04:29:54 and then exited status 1 without a durable terminal state; the adopted runs
|
||||
were never re-driven, and the claims were only closed by the next process at 05:00:43/47. The ledger
|
||||
cannot say whether their side effects ran (*"whether side effects ran is unknown"*), so the stocktake's
|
||||
2026-11-07 edits are known to exist only because an in-day session committed them
|
||||
([[systems/scheduled-tasks]]).
|
||||
|
||||
## Impact
|
||||
|
||||
- **Mnemosyne Health Monitor** delivered nothing from its 00:00 slot. Its 11-06 slot had taken
|
||||
**7 h 47 m 56 s** and its 11-05 slot **12 h 13 m 54 s**, so this slot was already deep in its multi-hour
|
||||
tail ([[systems/mnemosyne]]).
|
||||
- **wiki-stocktake-maintenance** lost its 03:00 slot for a **second consecutive day** (`failure_streak: 2`,
|
||||
`last_status: error`) — no stocktake ran on 2026-11-07.
|
||||
- **Job Radar was not affected on 11-07:** no slot failed and the day's report landed normally, unlike
|
||||
2026-11-06 ([[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]).
|
||||
|
||||
## Verification evidence
|
||||
|
||||
journalctl --user -u hermes-gateway --since '2026-11-07 04:00' --until '2026-11-07 05:10'
|
||||
/home/hermes/.hermes/cron/executions.db (status/error per row; cron_incidents table)
|
||||
/home/hermes/.hermes/conduit-self-update.json
|
||||
/home/hermes/.hermes/cron/jobs.json (failure_streak, last_status)
|
||||
|
||||
## Open items
|
||||
|
||||
- The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or
|
||||
fix the alerting path that let a `detected` row sit silent.
|
||||
- Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and
|
||||
whether the 1800 s after-turn wait should refuse to drain long-running cron executions.
|
||||
|
||||
## See also
|
||||
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[systems/hermes-agent]]
|
||||
- [[systems/conduit]]
|
||||
- [[systems/mnemosyne]]
|
||||
- [[incidents/2026-11-06-job-radar-concurrent-fire-claim-discard]]
|
||||
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]
|
||||
Reference in New Issue
Block a user