nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130

This commit is contained in:
Tony0410
2026-11-09 00:21:01 +08:00
parent 4fd138bb18
commit e2f1ae7a0b
12 changed files with 422 additions and 29 deletions
+5 -3
View File
@@ -3,11 +3,11 @@ title: Hermes Agent
type: system
status: active
created: 2026-07-22
updated: 2026-11-08
verified_on: 2026-11-08
updated: 2026-11-09
verified_on: 2026-11-09
confidence: high
tags: [system, hermes, agent, llm]
sources: [live-hermes-version, live-config, live-skill-inventory]
sources: [live-hermes-version, live-config, live-skill-inventory, live-errors-log, live-state-db]
---
# Hermes Agent
@@ -17,6 +17,8 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
## Current Status
- **Version:** ⚠ **`0.23.0`** — updated **2026-11-07 05:05:13 → 05:43:10 AWST** by `hermes update` (method `git`, outcome **`success`**, 37 m 57 s): `bf0203a794a07aac3ff764972ab9087baa4e4f4e` → **`2bd378ed1b8d6bf2192cb26e0a55ff3091e6bd19`** (upstream commit dated 2026-11-06 03:57:38 AWST). ⚠ **The `snapshot` stage was skipped again** — `disabled by updates.pre_update_backup (mode: off)`, so there is **still no rollback point** ([[infrastructure/backups]]). Restarted: `hermes-gateway` (pre-update `pid 2981474`) and `hermes-dashboard` (`Stopping` 05:43:03 → `Started` 05:43:04, `:9119`). The gateway was then restarted **again 11:11:44 → 11:11:55** (live process `pid 3109529`, seen logging 11:12:30 and 12:01:32). **Side effect of the same night:** two cron runs lost their fire claims in the restart drain — [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]. Previous: `0.22.3` — install stamp `5b147ab036d6d19131bff8d20701b5001eab844a`, built 2026-10-21T14:57:38Z; git-describe token `v0.21.5+15956.g5b147ab` (detached HEAD). **Last update: 2026-10-21 22:47:25 → 22:57:38 AWST** via `hermes update` (action `9eed3a99e799347c691ea52a72e122a1`, receipt `update_20261021_144738_866674_82212ebd…`): stages `plan` → `snapshot` (**skipped**) → `apply` (mode git) → `deps` → `build`, all success except the skip. The pre-update tree read `0.22.2` / `d923a5eb487a17c0a68801bba9ef0f56c81040a8` — an intermediate rebuild never recorded on this page, so the previously-noted `v0.21.5+6169.gab86a20` (2026-10-04) is at least two builds behind. Runtimes in the plan: `gateway` (systemd) and `dashboard` (`manual-serve`, restarted by argv). Post-restart state: gateway pid **868087**, `phase: running`, `api_server: connected` (heartbeat 2026-10-22 00:13 AWST); dashboard pid **869852** on `:9119`.
- **Hook budget raised: `plugins.hook_callback_timeout: 120`** (built-in default **30 s**), set **2026-11-08 13:3x AWST** in `~/.hermes/config.yaml`. Reason: the `conduit_push` plugin's `pre_tool_call` callback — the **only** `pre_tool_call` registrant on the host — overran the 30 s budget **58 times** between **2026-11-07 14:14:03** and **2026-11-08 08:21:03** (`~/.hermes/logs/errors.log`: 12 lines on 11-07, 46 on 11-08; plus **89** *"skipped after previous timeout or while still running"* suppression lines), which degraded every tool call to slow-then-suppressed. Verified live in config, and verified effective — **no timeout line after 08:21:03**. → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- ⚠ **Session store: `~/.hermes/state.db` is 1.89 GB (1,890,656,256 B) with 2,901 sessions / 197,682 messages and a live 40,157,672 B WAL** (measured 2026-11-09 00:0x AWST). It grew **1.7 GB → 1.89 GB** since 2026-11-06; the 2-core CT460 was at load average 5.89 while it checkpointed. The `state.db.corrupted.20260811_151052` artifact (**624.1 MB**, plus a 32 KB `-shm` and a 0-byte `-wal` sibling) was **deleted** in the 2026-11-08 window; `state.db.recovered.recovery.json` (14,566 B, 2026-08-11) remains. **No retention or checkpoint policy is set** — see [[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]].
- **Install method:** git
- **Python:** managed runtime 3.14 (legacy `venv/` on 3.11.15)
- **Upstream status:** advanced 2026-10-21 to commit `5b147ab036` (detached HEAD, package version 0.22.2 → 0.22.3). ⚠ **That update ran with its own safety net switched off**: the `snapshot` stage is recorded as `skipped`, reason `disabled by updates.pre_update_backup (mode: off)` — the same `pre_update_backup: false` recorded in [[infrastructure/backups]], while the off-box config backup push has been dead since 2026-08-08. Two advisories were printed after the run: rebuild the search index (`hermes sessions optimize-storage` — "on the compact layout from an earlier optimization, but a later release changed it"), and `→ Preparing pinned cua-driver (Computer Use)...`, whose completion line is **absent from the log** — status unverified. The `.restart_pending.json` request (`via_service: true`, requested 2026-10-21 22:48:42 AWST) was delivered to telegram, discord and ntfy. Neither advisory has been acted on.