nightly audit 2026-11-09: pre_tool_call hook-timeout cascade (new incident page), hook budget 30->120s, state.db 1.89GB + corruption remnant, no-agent health monitor verified 73s, third silent stocktake incident, skills 299/67, freshness 0 stale/127 fresh of 130

This commit is contained in:
Tony0410
2026-11-09 00:21:01 +08:00
parent 4fd138bb18
commit e2f1ae7a0b
12 changed files with 422 additions and 29 deletions
@@ -3,8 +3,8 @@ title: Live Secrets Stored Verbatim in Mnemosyne Rows, Session Transcripts and T
type: incident
status: open
created: 2026-11-06
updated: 2026-11-06
verified_on: 2026-11-06
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-11-13
confidence: high
tags: [security, credentials, mnemosyne, memory, sessions, cron, incident]
@@ -57,7 +57,9 @@ carries any tested value; `cron/executions.db`, `cron/notepad.db` and `data/shar
### 3. Session transcripts hold all four values
`~/.hermes/state.db` (1.7 GB, the session store) contains all four values. Measured as 4 MB chunks
`~/.hermes/state.db` (1.7 GB *at discovery* — **1.89 GB / 1,890,656,256 B as at 2026-11-09 00:0x AWST**, with
**2,901 sessions / 197,682 messages** and a live **40,157,672 B WAL**; the store grew while nothing was
retentioned), the session store, contains all four values. Measured as 4 MB chunks
containing the value: **`CAMOFOX_API_KEY` 178, `MATTERMOST_TOKEN` 64,
`HERMES_CUSTOM_OMNIROUTE_20128_API_KEY` 61, `TELEGRAM_BOT_TOKEN` 22**. This is the **most likely origin of
exposure 1** - the memory-capture path writes from session text, so a session in which a key was printed
@@ -72,7 +74,11 @@ becomes a memory row - and it is itself a plaintext at-rest store that deleting
**rotation is the only remedy that fully closes it**.
- `jobs.json` is the fleet's source of truth and is exported and examined routinely; a prompt-embedded key
is exposed to anything that reads job definitions.
- `state.db` is the fleet's largest artifact (1.7 GB) and is copied into snapshots and repair backups.
- `state.db` is the fleet's largest artifact (**1.89 GB as at 2026-11-09**, up from 1.7 GB) and is copied into
snapshots and repair backups. ⚠ **It is also the load source behind the 2026-11-07/08 hook-timeout
cascade** — a 40 MB WAL checkpointing on the same spindle as a 2-core guest → [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]].
The **2026-08-11 corruption remnant `state.db.corrupted.20260811_151052` (624.1 MB)** was deleted in the
2026-11-08 window, so one stale copy did leave the box — but **`mnemosyne/data/mnemosyne.db.corrupted.20260809_120404` (23.6 MB) is still there**, so the rotation-snapshot exposure above is untouched.
## Origin (inferred - not established)
@@ -91,7 +97,7 @@ exist, and the affected rows carry no marker distinguishing them from ordinary c
4. **Add capture-time and delivery-time redaction** so `.env` values cannot enter memory rows, job prompts or
delivery records again.
5. **Decide a retention/serialisation policy for `state.db`** - it currently holds every transcript
verbatim at 1.7 GB.
verbatim at 1.89 GB (2,901 sessions / 197,682 messages) and it is growing.
## Verification evidence
@@ -3,8 +3,8 @@ title: Two cron runs lost their durable fire claims in a gateway restart drain (
type: incident
status: open
created: 2026-11-08
updated: 2026-11-08
verified_on: 2026-11-08
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-11-15
confidence: high
tags: [cron, fire-claim, scheduler, gateway-restart, drain, conduit, health-monitor, wiki-stocktake]
@@ -108,6 +108,14 @@ cannot say whether their side effects ran (*"whether side effects ran is unknown
- The two open stocktake rows have **never been reconciled** (`alerted_at` null) — either resolve them or
fix the alerting path that let a `detected` row sit silent.
⚠ **2026-11-09 re-check: still open, and now three.** `1a8e90_628d914f1f97` and `1a8e90_2f370ef764f5`
(both first seen 2026-11-07) are unchanged, and a **third** arrived —
`1a8e90_a9cad99cb150`, `detected` 2026-11-08T04:35:38, `alerted_at: null`, from the 03:01:11 → 04:36:08
failure on `RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending…`.
**Unclosed incident rows 2 → 3.** The alert path itself is **not** broken: the *Newsletter Digest* row
`85edf6_0877ca1e0eac` alerted at **07:23:07** and closed as `resolved` at **08:06:54** the same window.
So the gap is specific to the **stocktake's** rows.
→ [[incidents/2026-11-08-pre-tool-call-hook-timeout-cascade]]
- Decide whether the conduit self-update should stop requesting a gateway restart it cannot perform, and
whether the 1800 s after-turn wait should refuse to drain long-running cron executions.
@@ -0,0 +1,101 @@
---
title: The conduit pre_tool_call hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
type: incident
status: resolved
created: 2026-11-09
updated: 2026-11-09
verified_on: 2026-11-09
review_after: 2026-11-16
confidence: high
tags: [cron, plugin, conduit, hook, timeout, state-db, load, watchdog, health-monitor]
sources: [live-hermes-errors-log, live-hermes-agent-log, live-config-yaml, plugin-tree, live-state-db, live-executions-db, live-cron-incidents, live-jobs-json]
---
# The `conduit_push` `pre_tool_call` hook jammed every tool call for ~18 hours (2026-11-07 → 2026-11-08)
**Summary.** From **2026-11-07 14:14:03** to **2026-11-08 08:21:03** the `conduit_push` plugin's
**sole `pre_tool_call` callback** overran Hermes's **30 s hook budget** and was skipped. Each skip
suppresses the callback for a window, so the *pre-tool* side of tool dispatch degraded from *slow* to
*blocked* — this is the fault that ate the 2026-11-08 morning and stretched that day's cron runs to
**6 h** (*Mnemosyne Health Monitor* 12:00:51 → 18:00:42) and **1 h 13 m** (*FreshRSS Midday* 12:00:50 →
14:10:43). Fixed **2026-11-08 13:35 AWST** by raising `plugins.hook_callback_timeout` **30 → 120**.
## Evidence (live reads, 2026-11-08 23:5x – 2026-11-09 00:1x AWST)
grep -c "callback _pre_tool_call timed out" ~/.hermes/logs/errors.log -> 58
# 2026-11-07: 12 lines 2026-11-08: 46 lines
# first 2026-11-07 14:14:03,075 last 2026-11-08 08:21:03,863
grep -c "skipped after previous timeout" ~/.hermes/logs/*.log -> 89
The exact pair of lines (`~/.hermes/logs/errors.log`):
2026-11-08 08:19:40,942 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call skipped after previous timeout or while still running
2026-11-08 08:21:03,863 WARNING hermes_cli.plugins: Hook 'pre_tool_call' callback _pre_tool_call timed out after 30s — skipping
**The register (correction to a plausible-but-wrong diagnosis).** `conduit_push/__init__.py:119` is the
**only** `pre_tool_call` registration on the host, and the plugin's caches are already in place —
`session_titles.py:21 CACHE_SECONDS = 30.0`. Timed in-process by the 2026-11-08 session:
**3.1 s cold, 0.00 s warm**. The *"the hook re-opens a 1.87 GB state.db per call, ~9 s each"* theory was
therefore **wrong** — warm calls were free. The timeouts track **load spikes** on the 2-core CT460
(load average **5.89** at 13:27:50; `state.db` carrying a **40 MB WAL** checkpointing on the same spindle),
not per-call database opens.
## Fix applied (2026-11-08 13:35 AWST)
~/.hermes/config.yaml
plugins:
hook_callback_timeout: 120 # was the built-in 30 s default
- **Verified live:** `plugins.hook_callback_timeout = 120`.
- **Verified effective:** no `_pre_tool_call` timeout has been logged since **08:21:03** — re-checked at
the end of the fix window and again at audit time (2026-11-09 00:0x). The **58th** line is still the last.
- **Deliberately NOT done:** rewriting the plugin's caching layer. The caches work; the budget was the
binding constraint. **Next lever is `state.db` checkpoint/retention discipline**, not more caching.
## Consequence — the same signature landed in the cron ledger
The 2026-11-08 03:00 *wiki-stocktake-maintenance* slot (`1a8e90ea2271`, 03:01:11 → 04:36:08) closed
**`failed`** on a **new terminal signature for that job**:
RuntimeError: ⚠️ No reply: the turn stopped while a tool result was still pending and the model
produced no follow-up text. Send `continue` to let it summarize.
*Working hypothesis, not proven:* a `pre_tool_call` block holding a tool result open is exactly what this
error describes, and it sits inside the same 14 h timeout window. It is **consistent**, not established —
the error string does not name the hook.
## Impact
| item | effect |
|---|---|
| *Mnemosyne Health Monitor* `ea79a2107e36` | 00:01:20 → **08:29:05** (8 h 27 m) and 12:00:51 → **18:00:42** (5 h 59 m 51 s) — its two slowest slots on record, both delivered |
| *wiki-stocktake-maintenance* `1a8e90ea2271` | 03:01:11 → 04:36:08 `failed`; **`failure_streak` 3**; a **third** unalerted `detected` incident row |
| *FreshRSS Midday Briefing* `ecc1fc13655a` | 12:00:50 → **14:10:43** (2 h 09 m 53 s) |
| *Tech & AI Newsletter Digest* `85edf6772bca` | an admission-read `database is locked` incident (`85edf6_0877ca1e0eac`) at **07:23:02**, **resolved** at 08:06:54 when the run completed |
| every interactive tool call | slow, then suppressed — the user-visible symptom that opened the investigation |
## Verification evidence
~/.hermes/logs/errors.log (58 timeout lines; last 2026-11-08 08:21:03)
~/.hermes/config.yaml (plugins.hook_callback_timeout: 120)
~/.hermes/plugins/conduit_push/__init__.py:119,257 (the only pre_tool_call registration)
~/.hermes/plugins/conduit_push/session_titles.py:21 (CACHE_SECONDS = 30.0)
~/.hermes/cron/executions.db, cron_incidents, jobs.json (run windows, streak, incident rows)
## Open items
- **`state.db` is 1.89 GB with 2,901 sessions / 197,682 messages and a live 40 MB WAL** — it is the load
source behind the spikes. Checkpoint and retention policy are still undecided
([[incidents/2026-11-05-secrets-at-rest-in-mnemosyne-and-cron-prompts]]).
- Confirm whether the stocktake's `No reply: … tool result was still pending` failure is really the same
fault; if it recurs while the hook budget is 120 s, this hypothesis is dead.
- Decide whether a skipped `pre_tool_call` should **fail loudly** rather than silently degrade tool dispatch.
## See also
- [[systems/conduit]]
- [[systems/hermes-agent]]
- [[systems/scheduled-tasks]]
- [[systems/mnemosyne]]
- [[incidents/2026-11-07-cron-claim-loss-in-gateway-restart-drain]]
- [[current-state]]