audit 2026-11-08 (window 11-07): cron claim loss in gateway restart drain (2 runs orphaned, 2 incident rows unalerted); Hermes 0.22.5->0.23.0 (snapshot skipped), camofox plugin 1.1.0, conduit 0.134.4, hq-push 0.9.42, opencode-go key expired; skills 302->299 (4 archived); freshness 0/126/3 of 129

This commit is contained in:
Tony0410
2026-11-08 00:41:58 +08:00
parent 0b4ff25fd2
commit 4fd138bb18
15 changed files with 563 additions and 41 deletions
+43 -1
View File
@@ -12,7 +12,49 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
---
# Current State
Updated: 2026-11-07
Updated: 2026-11-08
## Recent Changes (2026-11-08, nightly audit; live probes 2026-11-07 23:5x – 2026-11-08 00:5x AWST)
- **Hermes on CT460 is now `0.23.0`** (was `0.22.5`): `hermes update`, 2026-11-07 05:05:13 → 05:43:10,
`bf0203a794a07aac3ff764972ab9087baa4e4f4e` → `2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`. ⚠ **The `snapshot`
stage was skipped again** (`updates.pre_update_backup` mode `off`) → **there is still no rollback point**.
Gateway restarted 05:37:34, dashboard 05:43:04, and the gateway once more at 11:11:44 → 11:11:55
(live `pid 3109529`). → [[systems/hermes-agent]], [[agents/hermes-production]]
- ⚠ **The restart drain cost two cron runs their fire claims.** A restart requested at ~04:29:52 sat in
*after-turn* mode for the full 1800 s, timed out, and drained at 05:00:43/47 — killing *Mnemosyne Health
Monitor* (`ea79a2107e36`, 00:00:42.918 → 05:02:22.870) and *wiki-stocktake-maintenance*
(`1a8e90ea2271`, 03:00:31.515 → 05:02:22.174), both closed `failed` on
`lost its durable fire claim ownership`. systemd restarted the unit at **05:01:00**
(`restart counter is at 3`). → [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- ⚠ **Two incident rows are open and neither alerted.** `1a8e90_628d914f1f97` (first seen 04:29:54.815) and
`1a8e90_2f370ef764f5` (05:02:21.573) both sit `detected` for *wiki-stocktake-maintenance* with
**`alerted_at: null`** — the *red row, no alert* shape first recorded for Job Radar on 11-06.
- ⚠ **The 03:00 stocktake slot has now failed two days running** (`failure_streak: 2`, `last_status: error`,
2026-11-06 + 2026-11-07). The 11-07 in-day entry committed the stocktake's own edits so the tree was not
left dirty.
- **`camofox` is now an enabled plugin (`v1.1.0`)**, not a core feature — installed by the 0.23.0 update.
No `browser.*` config key changed. → [[systems/browser-backend]]
- **`conduit_push` self-updated `0.122.2 → 0.134.4`** (`git` source, tree HEAD `ef3d2f2`) and recorded its own
last step **`restarting_gateway` = failed** — the gateway it wanted to restart is systemd-managed. →
[[systems/conduit]]
- **`hermes-hq-push` reinstalled `0.9.27 → 0.9.42`** (upstream `d26850e7`, 60/60 files) at 06:42; its web
routes only load at service start, so the 11:11:44 gateway restart is what put them on the wire. →
[[systems/hermes-hq-push]]
- ⚠ **The OpenCode Go credential is expired** — HTTP 401, `code: api_key_expired`, on
`…/zen/go/v1/models` (probed twice in-window). Configuration alive, credential dead; no 2026-11-07
execution failed on an auth error, so nothing was consuming it yet. → [[systems/opencode-go]]
- ⚠ **Skill inventory moved: 302 → 299 skills (68 → 67 categories).** Four skills went into
`~/.hermes/skills/.archive/` in one pass at **2026-11-07 07:43** — `cron-model-repair`,
`fleet-wide-failure-triage`, `hermes-instance-update`, `mail-lookup` — and the parked archive grew
**46 → 50**. No active `SKILL.md` references any of them, so if this was a curator consolidation the
*"absorbed into X"* pointer is missing. `microsoft-office` appeared in the same window (Productivity
25 → 26). → [[systems/skills-index]], [[systems/scheduled-tasks]]
- **Cron day (2026-11-07): 163 rows — 159 `completed` / 3 `failed` / 1 `running`**; deliveries 7/7
`delivered`, 0 `for_failure`. Fleet unchanged at **29 jobs / 26 enabled**. →
[[systems/scheduled-tasks]]
- [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]] **remains OPEN** — the re-check
found cron job prompts clean of secret-shaped strings, but the redaction setting could not be confirmed.
## Recent Changes (2026-11-07, stocktake; live probes 2026-11-07 03:1x-03:2x AWST)