audit 2026-11-08 (window 11-07): cron claim loss in gateway restart drain (2 runs orphaned, 2 incident rows unalerted); Hermes 0.22.5->0.23.0 (snapshot skipped), camofox plugin 1.1.0, conduit 0.134.4, hq-push 0.9.42, opencode-go key expired; skills 302->299 (4 archived); freshness 0/126/3 of 129

This commit is contained in:
Tony0410
2026-11-08 00:41:58 +08:00
parent 0b4ff25fd2
commit 4fd138bb18
15 changed files with 563 additions and 41 deletions
+54
View File
@@ -2330,3 +2330,57 @@ ledger activity remains the 2026-11-05T12:36:17 closures), so **nothing alerted*
`RuntimeError: Cron job 'wiki-stocktake-maintenance' lost its durable fire claim ownership`
(failure_streak 2), leaving its stocktake edits uncommitted in the working tree. Those edits are
included in this commit so the tree is not left dirty.
## [2026-11-08] audit - window 2026-11-07
- **Day ledger:** 2026-11-07 closed **163 executions — 159 `completed` / 3 `failed` / 1 `running`** (still
open at probe time). Fleet unchanged: **29 jobs / 26 enabled**. Deliveries **7 / 7 `delivered`**,
0 `for_failure`.
- **New incident page: [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]].** Two in-flight
cron executions lost their durable fire claims when a gateway restart drained at **05:00:43/47**:
`ea79a2107e36` (*Mnemosyne Health Monitor*, 00:00:42.918 → 05:02:22.870) and `1a8e90ea2271`
(*wiki-stocktake-maintenance*, 03:00:31.515 → 05:02:22.174). A restart requested at **~04:29:52** sat in
*after-turn* mode for the full **1800 s** (`#77184`), timed out, and the drain proceeded; systemd
restarted the unit at **05:01:00** (*restart counter is at 3*). The third `failed` row of the night
(`96eb95ce12f2`, 05:02:41 → 05:02:59) is the agent-log ERROR watchdog correctly reporting the event
~19 s later. Request origin: **inside the `conduit_push` self-update window** (04:27:55 → 04:32:32, whose
own ledger records `restarting_gateway` = *failed*) — stated on the page as a **working hypothesis, not
proven**; the 0.22.5 → 0.23.0 updater is excluded (its receipt begins 05:05:13).
- **Incident ledger: unclosed rows 0 → 2, and neither alerted.** Both are the stocktake job's:
`1a8e90_628d914f1f97` (first seen 04:29:54.815) and `1a8e90_2f370ef764f5` (05:02:21.573), both
`detected` with **`alerted_at: null`**. The health monitor's two rows closed 15:43:17 / resolved 17:08:52.
- **Hermes 0.22.5 → 0.23.0 on CT460** (`hermes update`, 05:05:13 → 05:43:10, `bf0203a794a0` → `2bd378ed1b8d`);
the **`snapshot` stage was skipped again** (`updates.pre_update_backup` mode `off`) → **still no rollback
point**. Gateway restarted 05:37:34, dashboard 05:43:04, gateway again 11:11:44 → 11:11:55 (live
`pid 3109529`). CT450's install stamp now also reads **0.23.0**.
- **Plugin/credential state changes:** `camofox` **v1.1.0** is now an **enabled plugin** (browser backend is
no longer core); `conduit_push` self-updated **0.122.2 → 0.134.4** (tree HEAD `ef3d2f2`); `hermes-hq-push`
reinstalled **0.9.27 → 0.9.42** (upstream `d26850e7`, 60/60 files, tree mtime 2026-11-07 06:42) with its
routes only loaded by the 11:11 restart; the **OpenCode Go key is expired** — HTTP 401
`api_key_expired` on `…/zen/go/v1/models`, probed twice in-window, with no 2026-11-07 execution failing on
an auth error.
- **Secrets incident re-checked, still OPEN:** a regex sweep of every `cron/jobs.json` prompt for
secret-shaped strings returned **0 hits**, but `config.yaml`'s `redact_secrets:` value could not be read
as an on/off state, so the 2026-11-05 finding stands unrefuted. Nothing rotated, redacted or scrubbed.
- ⚠ **Skill inventory moved: 302 → 299 skills, 68 → 67 categories.** Four skills moved into
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** (`cron-model-repair`,
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup`); parked archive **46 → 50**, and the
`Cron Model Repair` category disappeared with its only member. A grep of every active `SKILL.md` finds no
absorbing skill, so the *"absorbed into X"* pointer is missing. `microsoft-office` appeared in-window
(Productivity 25 → 26). Disk shows **300 `SKILL.md` vs 299 listed** (generator de-dups one nested copy).
- **Freshness:** 0 stale / **126 fresh** / 3 no-date of **129 pages** — `raw/stocktake/freshness-20261108.txt`.
- **Pages touched:** [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]] (new),
[[systems/hermes-agent]], [[systems/scheduled-tasks]], [[systems/browser-backend]], [[systems/conduit]],
[[systems/hermes-hq-push]], [[systems/opencode-go]], [[systems/skills-index]] (regenerated),
[[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]], [[agents/hermes-production]],
[[agents/hermes-chaos]], [[current-state]], [[index.md]], this log, and the vault mirror
(`/home/hermes/wiki`).
- **Wiki maintenance:** `wiki-maintain.sh` (canonical repo script, via the `~/.hermes/scripts/` wrapper) run
**after** this audit's writes — skills-index regenerated (**299 / 67**), freshness report written, Quartz
rebuilt (**129 markdown → 779 files** deployed to `public/`) and `hermes-wiki-serve.service` restarted at
**2026-11-08 00:38:29 AWST**.
- **No wiki page removed or archived.** No job added, retired, disabled or repinned; no system obsoleted; no
decision reversed. (The four skill archives above happened on the filesystem, not in this wiki.)
- **No secrets, keys, tokens or passwords written** — the opencode-go finding records the HTTP status and
the upstream error code only.