audit 2026-11-08 (window 11-07): cron claim loss in gateway restart drain (2 runs orphaned, 2 incident rows unalerted); Hermes 0.22.5->0.23.0 (snapshot skipped), camofox plugin 1.1.0, conduit 0.134.4, hq-push 0.9.42, opencode-go key expired; skills 302->299 (4 archived); freshness 0/126/3 of 129
This commit is contained in:
@@ -2330,3 +2330,57 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
|
||||
`RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership`
|
||||
(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are
|
||||
included in this commit so the tree is not left dirty.
|
||||
|
||||
## [2026-11-08] audit - window 2026-11-07
|
||||
|
||||
- **Day ledger:** 2026-11-07 closed **163 executions — 159 `completed` / 3 `failed` / 1 `running`** (still
|
||||
open at probe time). Fleet unchanged: **29 jobs / 26 enabled**. Deliveries **7 / 7 `delivered`**,
|
||||
0 `for_failure`.
|
||||
- **New incident page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].** Two in-flight
|
||||
cron executions lost their durable fire claims when a gateway restart drained at **05:00:43/47**:
|
||||
`ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 → 05:02:22.870) and `1a8e90ea2271`
|
||||
(*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174). A restart requested at **~04:29:52** sat in
|
||||
*after-turn* mode for the full **1800 s** (`#77184`), timed out, and the drain proceeded; systemd
|
||||
restarted the unit at **05:01:00** (*restart counter is at 3*). The third `failed` row of the night
|
||||
(`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog correctly reporting the event
|
||||
~19 s later. Request origin: **inside the `conduit_push` self-update window** (04:27:55 → 04:32:32, whose
|
||||
own ledger records `restarting_gateway` = *failed*) — stated on the page as a **working hypothesis, not
|
||||
proven**; the 0.22.5 → 0.23.0 updater is excluded (its receipt begins 05:05:13).
|
||||
- **Incident ledger: unclosed rows 0 → 2, and neither alerted.** Both are the stocktake job's:
|
||||
`1a8e90_628d914f1f97` (first seen 04:29:54.815) and `1a8e90_2f370ef764f5` (05:02:21.573), both
|
||||
`detected` with **`alerted_at: null`**. The health monitor's two rows closed 15:43:17 / resolved 17:08:52.
|
||||
- **Hermes 0.22.5 → 0.23.0 on CT460** (`hermes update`, 05:05:13 → 05:43:10, `bf0203a794a0` → `2bd378ed1b8d`);
|
||||
the **`snapshot` stage was skipped again** (`updates.pre_update_backup` mode `off`) → **still no rollback
|
||||
point**. Gateway restarted 05:37:34, dashboard 05:43:04, gateway again 11:11:44 → 11:11:55 (live
|
||||
`pid 3109529`). CT450's install stamp now also reads **0.23.0**.
|
||||
- **Plugin/credential state changes:** `camofox` **v1.1.0** is now an **enabled plugin** (browser backend is
|
||||
no longer core); `conduit_push` self-updated **0.122.2 → 0.134.4** (tree HEAD `ef3d2f2`); `hermes-hq-push`
|
||||
reinstalled **0.9.27 → 0.9.42** (upstream `d26850e7`, 60/60 files, tree mtime 2026-11-07 06:42) with its
|
||||
routes only loaded by the 11:11 restart; the **OpenCode Go key is expired** — HTTP 401
|
||||
`api_key_expired` on `…/zen/go/v1/models`, probed twice in-window, with no 2026-11-07 execution failing on
|
||||
an auth error.
|
||||
- **Secrets incident re-checked, still OPEN:** a regex sweep of every `cron/jobs.json` prompt for
|
||||
secret-shaped strings returned **0 hits**, but `config.yaml`'s `redact_secrets:` value could not be read
|
||||
as an on/off state, so the 2026-11-05 finding stands unrefuted. Nothing rotated, redacted or scrubbed.
|
||||
- ⚠ **Skill inventory moved: 302 → 299 skills, 68 → 67 categories.** Four skills moved into
|
||||
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** (`cron-model-repair`,
|
||||
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`); parked archive **46 → 50**, and the
|
||||
`Cron Model Repair` category disappeared with its only member. A grep of every active `SKILL.md` finds no
|
||||
absorbing skill, so the *"absorbed into X"* pointer is missing. `microsoft-office` appeared in-window
|
||||
(Productivity 25 → 26). Disk shows **300 `SKILL.md` vs 299 listed** (generator de-dups one nested copy).
|
||||
- **Freshness:** 0 stale / **126 fresh** / 3 no-date of **129 pages** — `raw/stocktake/freshness-20261108.txt`.
|
||||
- **Pages touched:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new),
|
||||
[[systems/hermes-agent]], [[systems/scheduled-tasks]], [[systems/browser-backend]], [[systems/conduit]],
|
||||
[[systems/hermes-hq-push]], [[systems/opencode-go]], [[systems/skills-index]] (regenerated),
|
||||
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]], [[agents/hermes-production]],
|
||||
[[agents/hermes-chaos]], [[current-state]], [[index.md]], this log, and the vault mirror
|
||||
(`/home/hermes/wiki`).
|
||||
- **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script, via the `~/.hermes/scripts/` wrapper) run
|
||||
**after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz
|
||||
rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at
|
||||
**2026-11-08 00:38:29 AWST**.
|
||||
- **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.)
|
||||
- **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and
|
||||
the upstream error code only.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user